lda-wf/docs/superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md

Durable Workflow Runs and Resume Design

Status: v1 implemented; future protocol-native progress and broader recovery remain

Durable workflow runs turn the current process-local run_deployment / resume_run behavior into platform state. The runtime already exposes the essential execution snapshot as RunState; the missing boundary is a durable run repository and a strict persisted codec for that state.

This design intentionally implements safe stopped-run persistence first. It does not promise replay of an in-flight external side effect or arbitrary automatic retries for failed MCP tool calls.

Goals

• Persist workflow runs that stop as interrupted, completed, or failed.
• Let an explicitly interrupted deployment resume after MCP server restart or from another process using a durable run_id.
• Pin the artifact, deployment binding environment, and prepared dependency environment used by a run when it starts.
• Revalidate pinned dependencies before a persisted interrupted run continues.
• Provide compact run inspection and bounded trace retrieval suitable for MCP clients and a future dashboard.
• Keep run persistence outside wf_core and separate from auth/catalog storage.
• Leave room for finer-grained checkpoints without requiring them in v1.

Non-Goals

• Do not pause automatically when an MCP connection or tool call fails.
• Do not implement automatic retry or timeout behavior in this slice.
• Do not checkpoint after every node or scheduler tick in v1.
• Do not add time travel, replay from arbitrary historical state, or state edits.
• Do not add LangGraph-style cross-run memory available to arbitrary nodes.
• Do not move artifact, deployment, auth, or catalog storage into one store.

Current State

wf_core.RunState already owns the data necessary to continue a stopped run:

• workflow input, committed state, terminal outcome, and public output
• trace entries
• frames and ready-frame queue
• runtime scopes and lineage writes
• current execution cursor
• error information
• interrupt request and nested subgraph interrupt route

wf_mcp.workflow_surface.handlers.ActiveWorkflowRun currently retains the remaining platform context only in memory:

• WorkflowDeployment
• WorkflowArtifact
• RawWorkflowPlan
• RunState

That class is the temporary seam to replace. It is not evidence that run state belongs inside MCP: MCP is merely one front door for a workflow platform concern.

Relationship to LangGraph Persistence

LangGraph distinguishes execution checkpoints from a general memory store:

• A checkpointer stores execution snapshots so interrupted graphs can be inspected and resumed.
• A store provides arbitrary memory shared across threads/runs.

This design needs the first concept only. A workflow run/checkpoint repository is appropriate now; cross-run node-accessible memory is separate future work.

Platform Ownership

The conceptual ownership remains:

wf_core
  workflow execution models and runtime

wf_artifacts
  immutable workflow definitions and deployment contracts

wf_platform (future extraction target)
  runs, checkpoints, schedules, run history, UI/admin policy

wf_mcp
  MCP exposure of platform operations

The current package graph already has wf_artifacts depending on wf_platform for capability/source refs. Placing typed run records in wf_platform while they snapshot WorkflowArtifact and WorkflowDeployment would introduce a cycle or weaken those fields into untyped dictionaries. V1 should therefore place run models/store code under a focused wf_artifacts.runs subpackage. Names and module boundaries must make later extraction straightforward if the artifact/platform dependency direction is reorganized. Do not expand wf_mcp.storage.Store, which owns MCP auth/catalog snapshots with a different lifecycle.

Domain Model

Workflow Run

A WorkflowRun identifies one execution attempt of one pinned deployment environment.

Required information:

WorkflowRun
  id: stable run id
  deployment_id
  artifact_id
  artifact_version
  status: interrupted | completed | failed
  resume_readiness: ready | blocked | not_applicable
  created_at
  updated_at
  latest_checkpoint_id
  pinned_environment
  diagnostics

unrunnable is not a run status. It is a pre-start response when deployment dependency validation fails before an execution begins.

Run Checkpoint

A RunCheckpoint is a stored stopped-state snapshot of a run.

RunCheckpoint
  id
  run_id
  sequence
  reason: interrupted | completed | failed
  created_at
  persisted_run_state

V1 writes a checkpoint only when a public run operation stops:

• run_deployment returns an interrupted, completed, or failed execution.
• resume_run returns an interrupted, completed, or failed execution.

V1 does not guarantee recovery after a process crash while a node or external tool call is in progress.

Pinned Execution Environment

Runs must not reinterpret bindings when resumed. The persisted run pins:

• root artifact snapshot, in addition to its exact id/version
• deployment snapshot, including the binding selection used at start
• resolved saved child artifact snapshots, in addition to exact child versions
• dependency contract information needed to revalidate those bindings

A configuration edit after the run starts does not silently redirect a paused run from one MCP account/source to another. If a future repair/migration flow allows rebinding a stopped run, that must be explicit and auditable.

Snapshotting artifact contents is required by the current implementation: FileWorkflowArtifactStore stores versioned paths but does not yet reject an overwrite of an existing version. A durable run must not depend on a later filesystem overwrite preserving the graph it originally executed. A future store that enforces immutable artifact writes may still retain snapshots as an audit record.

Persisted Runtime Codec

RunState.to_dict() is useful for inspection but is not by itself a durable storage contract. Nested dataclasses, enums, path types, reducer refs, workflow refs, scopes, lineages, and interrupt routes must round-trip through a validated persisted model or codec.

The storage codec must:

• serialize the full stopped RunState needed for resume
• restore equivalent enum/path/ref/dataclass values rather than loose dicts
• reject corrupted or unsupported stored state with a clear diagnostic
• be versioned or structured so later checkpoint migrations are possible
• contain JSON-compatible values only at the persistence boundary

Core should own the execution-state codec shape because it understands RunState. The platform store should persist the codec output, not reconstruct runtime internals itself.

Lifecycle Semantics

Start

validate deployment and resolved dependencies
if blocking diagnostics:
  return unrunnable response; do not create a run

pin resolved execution environment
execute workflow
persist WorkflowRun and stopped-state checkpoint
return run summary

Completed and failed runs are persisted as well as interrupted runs. This makes inspection and debugging useful immediately without materially increasing storage implementation complexity.

Intentional Pause

Only a declared InterruptNode creates a resumable pause.

InterruptNode reached
  -> RunStatus.INTERRUPTED
  -> persist interrupted checkpoint
  -> resume_readiness = ready when pinned dependencies validate

Interrupts raised within a native saved child subgraph retain the typed interrupt route already carried in RunState, so durable resume re-enters the original child scope.

Resume

load run and latest interrupted checkpoint
verify run is interrupted and resumable
revalidate exact pinned dependency environment
if dependency validation fails:
  keep run status interrupted
  set resume_readiness = blocked
  return dependency diagnostics without mutating execution state
else:
  resume from restored RunState
  persist next stopped-state checkpoint

When a missing or disabled pinned dependency returns, a later explicit validate_run, inspect_run, or resume_run may report it ready again. There is no background auto-resume.

Failure

An ordinary runtime exception, MCP transport failure, or tool connection loss while execution is running fails the run.

This is deliberately distinct from interruption. An external tool may have completed a side effect before losing its response; automatically presenting that condition as safely resumable could duplicate effects.

tool/source failure during execution
  -> status = failed
  -> persist failed checkpoint and error details
  -> no automatic resume

Recovery from such failures must later be declared in workflow semantics, such as explicit outcome mapping or a carefully specified retry policy.

Dependency Validation Semantics

Dependency status has three positions:

Point	Result
Before start	unrunnable; no execution run is created
At a declared interrupt	interrupted; persisted and potentially resumable
Before resume when pinned dependency is broken	run remains interrupted, resume_readiness = blocked, diagnostics returned
During live execution	failed, unless future workflow logic explicitly models recovery

Dependency diagnostics are platform/control-plane information, not graph outcomes. They must not be forced through a user's declared ok / error workflow outcome routes.

Public Surface

The stable MCP workflow surface should eventually expose:

wf.workflow.run_deployment
  starts a run; always returns a stable run_id once execution starts

wf.workflow.resume_run
  resumes one durable interrupted run after pinned dependency validation

wf.workflow.inspect_run
  returns compact run status, readiness, terminal output/outcome,
  interrupt summary, diagnostics, and trace_count

wf.workflow.read_run_trace
  returns a bounded trace slice selected by range

The response style stays progressive:

• start/resume returns compact stopped-state status
• inspect returns detail needed for decision making
• trace is fetched only in bounded slices

Store Interface Direction

Use a dedicated protocol rather than expanding MCP auth/catalog persistence:

RunStore
  save_run(run)
  get_run(run_id)
  save_checkpoint(checkpoint)
  get_latest_checkpoint(run_id)
  list_runs(query, cursor, limit)
  read_checkpoints(run_id, cursor, limit)

The initial backend may be file-based under the configured application store root:

.wf_mcp_store/
  auth/
  catalog/
  workflows/
  deployments/
  runs/
    <run_id>/
      run.json
      checkpoints/
        000001.json
        000002.json

The protocol must allow later SQLite/Postgres storage without changing the MCP surface or core runtime model.

Retry and Timeout Fields

The codebase already carries retry and timeout_seconds fields in node/draft models, plus a runtime retry_count context field. They are not currently an implemented runtime policy.

For durable-run v1:

• treat these fields as declared-but-unsupported behavior
• do not interpret failed MCP calls as retryable
• do not add recovery checkpoints around external calls
• document clearly when a future slice implements retry/timeout semantics

A future retry design must distinguish known-not-executed failures from unknown-side-effect failures, or require idempotency/author-provided policy.

Testing Strategy

Implementation tests must cover:

• persisted RunState round-trip for root interrupts
• persisted RunState round-trip for native child-subgraph interrupts
• a completed run saved with output and terminal outcome
• a failed run saved with error details
• server/service recreation followed by successful resume_run
• pinned dependency disabled after pause: resume is blocked and state remains interrupted and unmodified
• pinned dependency restored: later resume succeeds
• changed live configuration cannot silently redirect pinned source bindings
• transport failure during execution produces failed, not interrupted
• trace retrieval is ranged and reports total trace count

Future Expansion

Once stopped-run persistence is stable:

1. Add checkpoint writes at explicit scheduler-safe boundaries if crash recovery during long execution is needed.
2. Investigate MCP tasks/progress reporting for live long-running operations.
3. Design retry/timeout semantics explicitly rather than activating dormant fields casually.
4. Consider replay/time-travel only after external side-effect semantics are documented.
5. Add general cross-run memory separately if nodes need it; it is not a replacement for checkpoints.

Implemented V1

Durable stopped-run snapshots now provide:

1. A validated persisted RunState codec.
2. WorkflowRunRecord, RunCheckpoint, and RunStore.
3. Checkpoints for interrupted, completed, and failed public executions.
4. Durable resume retrieval instead of process-local _active_runs.
5. Compact inspect_run and bounded read_run_trace tools.
6. Pinned dependency revalidation before interrupted runs resume.

This delivers durable human-in-the-loop execution and run inspection while preserving the correctness boundary that live external-call failures are not safe implicit pauses.