durable stopped workflow runs and resume

docs/current_roadmap.md

@@ -62,9 +62,8 @@ implementation state.
   `WorkflowBuilder.prepare_subgraph()` and `WorkflowBuilder.resume()` make the
   local runnable/resumable path available without core-runtime plumbing.
   Saved interrupting artifacts can now pause and resume through
-  `run_deployment`/`resume_run` for the duration of the MCP server process
-  (in-memory only). Persisted resume across process restarts remains future
-  work.
+  `run_deployment`/`resume_run` across handler/server recreation by restoring
+  stopped checkpoints and pinned root/child artifact definitions.
 - **Concurrent foreach**: implemented in core with explicit scheduling,
   reducer/merge semantics, item error policy, async handler batching, and
   quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -77,14 +76,14 @@ implementation state.
   metadata and compatibility patches. Scope-root commits now apply to both the
   root workflow and prepared native child scopes through the explicit
   scope/lineage commit helper.
-- **Durable run history and resume**: design is recorded in
+- **Durable run history and resume**: the design is recorded in
   [2026-05-26 durable workflow runs](./superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md).
-  Add a validated `RunState` storage codec and dedicated run/checkpoint store.
-  Persist stopped snapshots for interrupted, completed, and failed executions;
-  replace process-local resume handles with stable `run_id` values; add compact
-  `inspect_run` and bounded `read_run_trace` APIs. Resuming an interrupted run
-  must revalidate its pinned dependency environment; ordinary live tool/source
-  failures remain failed runs, not implicit pauses.
+  A validated `RunState` codec and dedicated run/checkpoint store now persist
+  interrupted, completed, and failed stopped snapshots. Stable `run_id` values
+  support compact `inspect_run` and bounded `read_run_trace` reads. Resume
+  revalidates its pinned dependency environment and reports `blocked` without
+  consuming input when a required source is unavailable. Ordinary live
+  tool/source failures remain failed runs, not implicit pauses.
 - **Protocol-native long-running runs**: investigate MCP tasks/progress
   notifications for long-running workflow execution. Avoid inventing a custom
   "start" convention unless protocol-native behavior is insufficient.
@@ -111,8 +110,9 @@ Frame stress points remaining for native subgraphs and future fork/gather:
 ## Why This Order
 
 The MCP workflow authoring path is now usable enough for real testing. The next
-bottleneck is runtime/platform correctness: durable resume/run history,
-optional per-use-site child deployment overrides, and protocol-native progress
-reporting. Concurrent foreach, native saved child execution, and process-local
-interrupt resume now supply scheduler/lineage precedent. Those remaining pieces
-should come before adding more high-level authoring sugar.
+bottleneck is runtime/platform correctness: optional per-use-site child
+deployment overrides, protocol-native progress reporting, and stronger durable
+run operations beyond stopped checkpoints. Concurrent foreach, native saved
+child execution, and durable interrupt resume now supply scheduler/lineage
+precedent. Those remaining pieces should come before adding more high-level
+authoring sugar.

docs/superpowers/specs/2026-05-24-native-subgraphs-design.md

@@ -1,7 +1,7 @@
 # Native Subgraphs Design
 
-Status: prepared-child execution, saved-artifact resolution, and process-local
-interrupt resume implemented; durable resume specified separately
+Status: prepared-child execution, saved-artifact resolution, and durable
+stopped-run interrupt resume implemented
 
 Native subgraphs should make a workflow usable as a workflow step without
 collapsing the child run into one opaque Python node call. The current
@@ -409,10 +409,11 @@ child artifact twice against different accounts or capability bindings.
 ### Saved Child Interrupt Status
 
 Native prepared children can interrupt and resume in core. The workflow surface
-now exposes process-local `run_deployment` / `resume_run` support for saved
+now exposes durable `run_deployment` / `resume_run` support for saved
 interrupting artifacts, including interrupts raised by saved descendants.
-That support does not survive process restart yet. Durable run persistence is
-specified separately in
+Stopped checkpoints pin the deployment and exact root/child artifact
+definitions; dependency drift can block resume without mutating the stopped
+execution. Durable run persistence is specified in
 [`2026-05-26-durable-workflow-runs-and-resume-design.md`](2026-05-26-durable-workflow-runs-and-resume-design.md).
 
 ## Implementation Slices
@@ -462,8 +463,8 @@ specified separately in
 - Dependencies, missing artifacts, and saved-child cycles validate before
   execution.
 - Tests cover saved child execution through deployment bindings, nested saved
-  child dependencies, missing/cyclic diagnostics, and process-local
-  interrupt/resume for saved children.
+  child dependencies, missing/cyclic diagnostics, and durable interrupt/resume
+  for saved children.
 
 ### Slice 4: Optional Policy Expansion
 
@@ -491,12 +492,8 @@ specified separately in
 
 ## Recommendation
 
-The typed boundary scaffold, prepared-child runtime, and routed interrupt
-resume are complete. Implement Slice 3 in the platform layer: prepare saved
-non-interrupting child artifacts recursively under one deployment environment
-and pass those prepared dependencies into core execution.
-
-The process-local saved-interrupt execution path is complete. Next add durable
-run/checkpoint persistence so interrupted saved children survive process
-restart. Do not delete wrapper-node helpers yet; they remain compatibility APIs
-while saved native execution matures.
+The typed boundary scaffold, prepared-child runtime, saved-child resolution,
+and durable routed interrupt resume are complete. Do not delete wrapper-node
+helpers yet; they remain compatibility APIs while saved native execution
+matures. Next policy work is optional per-use-site child deployment overrides
+and richer protocol-native long-running progress/reporting.

docs/superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md

@@ -1,6 +1,6 @@
 # Durable Workflow Runs and Resume Design
 
-Status: design approved for implementation planning
+Status: v1 implemented; future protocol-native progress and broader recovery remain
 
 Durable workflow runs turn the current process-local `run_deployment` /
 `resume_run` behavior into platform state. The runtime already exposes the
@@ -376,16 +376,16 @@ Once stopped-run persistence is stable:
 5. Add general cross-run memory separately if nodes need it; it is not a
    replacement for checkpoints.
 
-## Recommendation
+## Implemented V1
 
-Implement durable stopped-run snapshots first:
+Durable stopped-run snapshots now provide:
 
-1. Add a validated persisted `RunState` codec.
-2. Add `WorkflowRun`, `RunCheckpoint`, and `RunStore`.
-3. Save interrupted, completed, and failed runs after start/resume returns.
-4. Replace process-local `_active_runs` lookup with durable run retrieval.
-5. Add compact `inspect_run` and bounded `read_run_trace`.
-6. Revalidate pinned dependencies before resuming interrupted runs.
+1. A validated persisted `RunState` codec.
+2. `WorkflowRunRecord`, `RunCheckpoint`, and `RunStore`.
+3. Checkpoints for interrupted, completed, and failed public executions.
+4. Durable resume retrieval instead of process-local `_active_runs`.
+5. Compact `inspect_run` and bounded `read_run_trace` tools.
+6. Pinned dependency revalidation before interrupted runs resume.
 
 This delivers durable human-in-the-loop execution and run inspection while
 preserving the correctness boundary that live external-call failures are not

docs/wf_core_architecture.md

@@ -143,8 +143,8 @@ limits and intended adapter seam.
   the parent routes by the child's terminal workflow outcome. Saved/deployed
   workflow resolution remains outside core; the workflow platform can now
   supply saved child artifacts as prepared dependencies using one inherited
-  deployment binding environment, including process-local pause/resume for
-  child interrupts. For local authoring,
+  deployment binding environment, including durable stopped-run pause/resume
+  for child interrupts. For local authoring,
   `WorkflowBuilder.prepare_subgraph()`
   registers a child builder and `WorkflowBuilder.resume()` continues a paused
   prepared-child interrupt without requiring direct core-runtime calls.

docs/wf_mcp_operator_manual.md

@@ -182,6 +182,11 @@ Primary:
 - `wf.workflow.run_deployment`: execute a saved deployment with input. The
   default response is compact and returns `trace_count`; pass `trace_range`
   only when debugging a failed or surprising run.
+- `wf.workflow.inspect_run`: inspect a durable stopped run without trace detail.
+- `wf.workflow.read_run_trace`: retrieve only an explicit bounded debug trace
+  slice for a durable run.
+- `wf.workflow.resume_run`: resume an interrupted durable run when its pinned
+  dependencies remain available.
 
 Advanced:
 
@@ -357,6 +362,13 @@ response. Trace entries may include resolved node inputs, node outputs, and
 state changes, so treat them as debug payloads rather than ordinary list/summary
 data.
 
+Every started deployment receives a durable `run_id`, including completed and
+failed runs. Use `inspect_run` for the compact stored result and
+`read_run_trace` only for an explicit bounded debug range. Interrupted runs can
+be resumed after server/handler recreation; if a pinned source is missing or
+disabled, `resume_run` returns `resume_readiness="blocked"` without advancing
+the execution checkpoint.
+
 ## Which Tool Do I Use?
 
 | I want to... | Use |
@@ -383,6 +395,9 @@ data.
 | Bind a saved workflow to concrete sources | `wf.workflow.save_deployment` |
 | Check whether a deployment can run | `wf.workflow.validate_deployment` |
 | Execute a saved workflow | `wf.workflow.run_deployment` |
+| Inspect a stopped workflow run | `wf.workflow.inspect_run` |
+| Read bounded debug trace entries | `wf.workflow.read_run_trace` |
+| Resume an interrupted workflow run | `wf.workflow.resume_run` |
 
 ## Common Confusions
 

docs/wf_mcp_troubleshooting.md

@@ -305,7 +305,8 @@ The deployment paused at an interrupt node. The response should include:
 ```text
 status: interrupted
 outcome: null
-run_id: <process-local id>
+run_id: <durable id>
+resume_readiness: ready
 interrupt: <request payload and metadata>
 ```
 
@@ -315,11 +316,17 @@ Send the requested resume payload to:
 wf.workflow.resume_run
 ```
 
-The `run_id` is intentionally process-local. It is valid only while the current
-MCP server process keeps the paused run in memory. If the server restarts,
-there is no durable run store yet; rerun the deployment from the beginning.
-After resume completes, `status` is `completed` and `outcome` reports the
-workflow terminal outcome such as `ok` or `error`.
+The `run_id` identifies a stored stopped-state checkpoint and survives handler
+or server recreation. Before applying the resume payload, the platform
+revalidates the pinned deployment/source environment. If it returns
+`resume_readiness: blocked`, inspect the diagnostics, restore the missing or
+disabled source, and call `resume_run` again; the blocked attempt has not
+advanced workflow state. After resume completes, `status` is `completed` and
+`outcome` reports the workflow terminal outcome such as `ok` or `error`.
+
+For debugging a completed, failed, or interrupted run, call
+`wf.workflow.inspect_run` first. Only call `wf.workflow.read_run_trace` with a
+small explicit range when node-level detail is necessary.
 
 ## A Raw MCP Tool Works But The Workflow Version Is Awkward
 

docs/workflow_artifacts.md

@@ -70,12 +70,10 @@ Dynamic projection of saved workflows as individual MCP tools can exist later,
 but it should be optional. The stable run tool is the reliable base layer.
 
 Current `run_deployment` calls are synchronous request/response executions until
-the workflow pauses or completes. They return compact execution status,
-terminal workflow outcome, output, diagnostics, and `trace_count`; optional
-ranged trace detail is for debugging
-only. If a run pauses at an interrupt, the response includes a process-local
-`run_id` and interrupt payload. Use `wf.workflow.resume_run` with that `run_id`
-to continue while the same MCP server process is alive.
+the workflow pauses or completes. They return a durable `run_id`, compact
+execution status, terminal workflow outcome when available, output, diagnostics,
+and `trace_count`; optional ranged trace detail is for debugging only. If a run
+pauses at an interrupt, use `wf.workflow.resume_run` with that `run_id`.
 
 Non-interrupting saved workflow children can now execute natively through this
 deployment surface. A parent deployment resolves its saved descendants by exact
@@ -86,21 +84,21 @@ the subgraph use site rather than only the child artifact id.
 
 Interrupting saved artifacts can pause and resume through the current
 deployment surface, including interrupts raised inside saved child workflows.
-This support is in-memory only: server restart, reload that replaces process
-state, or another frontend process invalidates the `run_id`.
+Stopped snapshots pin the deployment, root artifact, and saved child artifact
+definitions so handler/server recreation does not invalidate the `run_id`.
 
 Durable run history is specified in
 [`2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md).
-It should replace process-local `run_id` values with durable run records. The
-planned surface is:
+The implemented surface is:
 
 - `run_deployment` starts or completes a run and returns `run_id`
 - `inspect_run(run_id)` returns status, output, diagnostics, and trace metadata
 - `read_run_trace(run_id, range)` returns bounded trace slices
 
-Until that exists, clients should treat the current response as the complete
-ephemeral run result for this request, or as a process-local resume handle when
-`status` is `interrupted`.
+Before applying a resume payload, the pinned dependency environment is
+revalidated. If it is unavailable or incompatible, the run stays
+`interrupted`, returns `resume_readiness="blocked"` and diagnostics, and does
+not append a new execution checkpoint.
 
 For long-running workflow execution, prefer MCP-native execution mechanisms
 where available:
@@ -686,12 +684,10 @@ native child pause and builder-driven resume. In the wrapper example the
 parent trace sees one node call; in the native examples child trace entries
 remain in the parent run state.
 
-Persisted resume is still not implemented. In-memory resume works because the
-server keeps the paused `RunState`; the durable run design persists stopped
-snapshots for interrupted, completed, and failed runs and pins root workflow,
-prepared child dependencies, deployment bindings, and trace metadata before a
-paused run can survive restart or move across processes. Only declared
-interrupts become resumable pauses; live tool/source failures remain failures.
+Durable resume persists stopped snapshots for interrupted, completed, and
+failed runs and pins root workflow, prepared child dependencies, deployment
+bindings, and trace metadata. Only declared interrupts become resumable
+pauses; live tool/source failures remain failures.
 
 Blocking dependency failures happen before workflow execution and are not normal
 workflow outcomes. A missing source, disabled source, unresolved binding, or

docs/workflow_drafts.md

@@ -446,11 +446,11 @@ provided on resume back into workflow state. Older map-shaped `request` and
 `resume` values are accepted only as parse compatibility and dump back to the
 canonical list shape.
 
-Saved interrupting artifacts can pause and resume through deployment runs while
-the MCP server process stays alive. The run response includes a process-local
-`run_id`; pass that to `wf.workflow.resume_run` with the resume payload.
-Persisted run storage is still future work, so a server restart invalidates
-that in-memory run id.
+Saved interrupting artifacts can pause and resume through deployment runs. The
+run response includes a durable `run_id`; pass that to
+`wf.workflow.resume_run` with the resume payload. Before advancing a resumed
+run, the platform revalidates its pinned dependency environment and can return
+`resume_readiness="blocked"` without consuming input.
 
 ### `end`