plan for durable run state storage idfk

big guy rambles.md

@@ -1,102 +0,0 @@
-# lda rambles
-
-that was surprisingly tuff response from our first user. it made me think about how we handle @node atm.
-
-atm only one way is allowed: path. If you want to inbuilt a value; please use the the graph input.
-
-our first user doesnt understand this. AT ALL.
-
-this calls to several problems.
-
-## dict use, instead of list[InputMap]
-
-why dict, when we can:
-
-```python
-class InputMap(BaseModel, Generic[TypeT]):
-  key: SupportToValue[TypeT]
-  input: Path
-```
-
-same thing
-
-```python
-class OutputMap(BaseModel, Generic[TypeT]):
-  output: Pathexpr
-  key (give me a better key name): SupportToValue[TypeT]
-```
-
-look at this support to value, or use another fitting name.
-what it does is:
-
-Path("state.foo") -> TypeT
-Literal(TypeT) -> TypeT
-
-two modes that we currently need to PATCH IN. we do not PATCH wf_core if there isnt a greater problem.
-
-### default arguments, and literal()
-
-expand on literal, in wf_authoring we can have some Field bs:
-
-```python
-@node(
-  in_map = [
-    InputMap(dont support this either, use g.use()),
-  ]
-)
-def baz(spam: InputT = "literal input", egg: Input2T = Path(do not support this.)):
-  ...
-```
-
-so that means wf_authoring.node shouldnt do any field augmentation.
-
-```python
-g.use(
-  baz,
-  in_map = [
-    now were talking
-  ]
-)
-```
-
-this SHOULD support both path and literal.
-
-## Context, of some sort
-
-langgraph has 3 ways of storage, if you read tests/rewrite
-
-lets copy that over:
-
-```
- According to https://docs.langchain.com/oss/python/concepts/context, there are three types:
-
- | type | mut | lifetime |
- | --- | --- | --- |
- |static runtime (context) | static | single run |
- |dynamic runtime (state) | mut | single run |
- |dynamic cross-convo (store) | mut | cross-conversation |
-
- now what the hell is store
-## store
-
- store is used in langgraph-demo for debugging. but it can be used for more things.
- it saves every turn. every graph nodes. I use InMemoryStore, you can use psql store!
- This allows for picking the work up again after a while for example.
- a lot more versatility there.
-```
-
-now we 100% has state. we kinda have store if you skim it (trace), but trace is only 50% store. and Context is patched in through input/state.
-
-this can be done with support for async, because lowk store is going to give us superpowers.
-
-## how this would aid adding async + foreach parallel
-
-(probably async gather)
-
-currently frame feels like magic + its core i dont understand. + the current serial mode impl is lowk smart.
-its reasonable; it spawns how many child frames as how many loop in foreach container item. Serially, like how `for in` in python works.
-complaints are only API improvements; not so much aid into adding these
-
-## complaints come when user uses
-
-not a lot from me rn.

docs/current_roadmap.md

@@ -77,9 +77,14 @@ implementation state.
   metadata and compatibility patches. Scope-root commits now apply to both the
   root workflow and prepared native child scopes through the explicit
   scope/lineage commit helper.
-- **Persistent run history**: add a run store before adding stable `run_id`,
+- **Durable run history and resume**: design is recorded in
-  `inspect_run`, or `read_run_trace(run_id, range)` APIs. Current traces are
+  [2026-05-26 durable workflow runs](./superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md).
-  returned directly from immediate run responses.
+  Add a validated `RunState` storage codec and dedicated run/checkpoint store.
+  Persist stopped snapshots for interrupted, completed, and failed executions;
+  replace process-local resume handles with stable `run_id` values; add compact
+  `inspect_run` and bounded `read_run_trace` APIs. Resuming an interrupted run
+  must revalidate its pinned dependency environment; ordinary live tool/source
+  failures remain failed runs, not implicit pauses.
 - **Protocol-native long-running runs**: investigate MCP tasks/progress
   notifications for long-running workflow execution. Avoid inventing a custom
   "start" convention unless protocol-native behavior is insufficient.

docs/superpowers/specs/2026-05-24-native-subgraphs-design.md

@@ -1,7 +1,7 @@
 # Native Subgraphs Design
 
-Status: prepared-child execution and interrupt resume implemented; saved-artifact
+Status: prepared-child execution, saved-artifact resolution, and process-local
-resolution scoped for platform implementation
+interrupt resume implemented; durable resume specified separately
 
 Native subgraphs should make a workflow usable as a workflow step without
 collapsing the child run into one opaque Python node call. The current
@@ -10,10 +10,9 @@ compatibility wrappers, but they hide the child trace, child frames, and child
 interrupt lifecycle from `wf_core`.
 
 This design defines the core runtime shape. The boundary model, prepared-child
-execution, and routed child interrupt resume are implemented. The remaining
+execution, routed child interrupt resume, and platform loading of immutable
-saved-workflow work is platform preparation: load immutable child artifact
+saved-child artifact versions are implemented. Durable persisted resume is the
-versions, resolve their runtime dependencies, and supply prepared children to
+remaining run-lifecycle concern and is specified separately.
-the already-implemented core runtime.
 
 ## Goals
 
@@ -407,15 +406,14 @@ must be keyed by the parent subgraph dependency/use site, not only by child
 artifact id: one parent graph may intentionally invoke the same immutable
 child artifact twice against different accounts or capability bindings.
 
-### Saved Child Interrupt Limitation
+### Saved Child Interrupt Status
 
-Native prepared children can interrupt and resume in core. The current
+Native prepared children can interrupt and resume in core. The workflow surface
-workflow-surface `run_deployment` entrypoint is still a one-shot execution
+now exposes process-local `run_deployment` / `resume_run` support for saved
-call, however, and does not expose persisted platform resume for a saved run.
+interrupting artifacts, including interrupts raised by saved descendants.
-Therefore this slice preserves the existing unrunnable behavior for saved
+That support does not survive process restart yet. Durable run persistence is
-artifacts containing interrupts, including interrupting descendant artifacts.
+specified separately in
-The platform must report that diagnostic before execution rather than starting
+[`2026-05-26-durable-workflow-runs-and-resume-design.md`](2026-05-26-durable-workflow-runs-and-resume-design.md).
-a run it cannot resume through its public surface.
 
 ## Implementation Slices
 
@@ -454,18 +452,18 @@ a run it cannot resume through its public surface.
 - Tests: child interrupt pauses parent, resume continues child, parent completes,
   wrong resume target fails clearly.
 
-### Slice 3: Saved Workflow References
+### Completed Slice 3: Saved Workflow References
 
-- Structural saved-workflow references and conversion helpers already exist;
+- Structural saved-workflow references and conversion helpers identify exact
-  this slice is execution resolution, not a new identity shape.
+  saved child versions without runtime string parsing.
-- Add platform-level resolution for saved workflow artifacts.
+- Platform-level resolution prepares saved workflow artifacts before runtime.
-- Apply the parent deployment binding environment transitively to each exact
+- The parent deployment binding environment applies transitively to each exact
   saved child artifact version.
-- Validate dependencies, missing artifacts, saved-child cycles, and existing
+- Dependencies, missing artifacts, and saved-child cycles validate before
-  unsupported interrupt diagnostics before execution.
+  execution.
-- Tests: saved child workflow runs through a deployment binding, nested saved
+- Tests cover saved child execution through deployment bindings, nested saved
-  child dependencies use the same binding environment, and missing,
+  child dependencies, missing/cyclic diagnostics, and process-local
-  cyclic, or interrupting child artifacts report an unrunnable dependency.
+  interrupt/resume for saved children.
 
 ### Slice 4: Optional Policy Expansion
 
@@ -498,7 +496,7 @@ resume are complete. Implement Slice 3 in the platform layer: prepare saved
 non-interrupting child artifacts recursively under one deployment environment
 and pass those prepared dependencies into core execution.
 
-Do not broaden saved interrupt support through `run_deployment` until the
+The process-local saved-interrupt execution path is complete. Next add durable
-platform has a public persisted resume path. Do not delete wrapper-node
+run/checkpoint persistence so interrupted saved children survive process
-helpers yet; they remain compatibility APIs while saved native execution
+restart. Do not delete wrapper-node helpers yet; they remain compatibility APIs
-matures.
+while saved native execution matures.

docs/superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md

@@ -0,0 +1,392 @@
+# Durable Workflow Runs and Resume Design
+
+Status: design approved for implementation planning
+
+Durable workflow runs turn the current process-local `run_deployment` /
+`resume_run` behavior into platform state. The runtime already exposes the
+essential execution snapshot as `RunState`; the missing boundary is a durable
+run repository and a strict persisted codec for that state.
+
+This design intentionally implements safe stopped-run persistence first. It
+does not promise replay of an in-flight external side effect or arbitrary
+automatic retries for failed MCP tool calls.
+
+## Goals
+
+- Persist workflow runs that stop as `interrupted`, `completed`, or `failed`.
+- Let an explicitly interrupted deployment resume after MCP server restart or
+  from another process using a durable `run_id`.
+- Pin the artifact, deployment binding environment, and prepared dependency
+  environment used by a run when it starts.
+- Revalidate pinned dependencies before a persisted interrupted run continues.
+- Provide compact run inspection and bounded trace retrieval suitable for MCP
+  clients and a future dashboard.
+- Keep run persistence outside `wf_core` and separate from auth/catalog storage.
+- Leave room for finer-grained checkpoints without requiring them in v1.
+
+## Non-Goals
+
+- Do not pause automatically when an MCP connection or tool call fails.
+- Do not implement automatic retry or timeout behavior in this slice.
+- Do not checkpoint after every node or scheduler tick in v1.
+- Do not add time travel, replay from arbitrary historical state, or state edits.
+- Do not add LangGraph-style cross-run memory available to arbitrary nodes.
+- Do not move artifact, deployment, auth, or catalog storage into one store.
+
+## Current State
+
+`wf_core.RunState` already owns the data necessary to continue a stopped run:
+
+- workflow input, committed state, terminal outcome, and public output
+- trace entries
+- frames and ready-frame queue
+- runtime scopes and lineage writes
+- current execution cursor
+- error information
+- interrupt request and nested subgraph interrupt route
+
+`wf_mcp.workflow_surface.handlers.ActiveWorkflowRun` currently retains the
+remaining platform context only in memory:
+
+- `WorkflowDeployment`
+- `WorkflowArtifact`
+- `RawWorkflowPlan`
+- `RunState`
+
+That class is the temporary seam to replace. It is not evidence that run state
+belongs inside MCP: MCP is merely one front door for a workflow platform
+concern.
+
+## Relationship to LangGraph Persistence
+
+LangGraph distinguishes execution checkpoints from a general memory store:
+
+- A **checkpointer** stores execution snapshots so interrupted graphs can be
+  inspected and resumed.
+- A **store** provides arbitrary memory shared across threads/runs.
+
+This design needs the first concept only. A workflow run/checkpoint repository
+is appropriate now; cross-run node-accessible memory is separate future work.
+
+## Platform Ownership
+
+The conceptual ownership remains:
+
+```text
+wf_core
+  workflow execution models and runtime
+
+wf_artifacts
+  immutable workflow definitions and deployment contracts
+
+wf_platform (future extraction target)
+  runs, checkpoints, schedules, run history, UI/admin policy
+
+wf_mcp
+  MCP exposure of platform operations
+```
+
+The current package graph already has `wf_artifacts` depending on `wf_platform`
+for capability/source refs. Placing typed run records in `wf_platform` while
+they snapshot `WorkflowArtifact` and `WorkflowDeployment` would introduce a
+cycle or weaken those fields into untyped dictionaries. V1 should therefore
+place run models/store code under a focused `wf_artifacts.runs` subpackage.
+Names and module boundaries must make later extraction straightforward if the
+artifact/platform dependency direction is reorganized. Do not expand
+`wf_mcp.storage.Store`, which owns MCP auth/catalog snapshots with a different
+lifecycle.
+
+## Domain Model
+
+### Workflow Run
+
+A `WorkflowRun` identifies one execution attempt of one pinned deployment
+environment.
+
+Required information:
+
+```text
+WorkflowRun
+  id: stable run id
+  deployment_id
+  artifact_id
+  artifact_version
+  status: interrupted | completed | failed
+  resume_readiness: ready | blocked | not_applicable
+  created_at
+  updated_at
+  latest_checkpoint_id
+  pinned_environment
+  diagnostics
+```
+
+`unrunnable` is not a run status. It is a pre-start response when deployment
+dependency validation fails before an execution begins.
+
+### Run Checkpoint
+
+A `RunCheckpoint` is a stored stopped-state snapshot of a run.
+
+```text
+RunCheckpoint
+  id
+  run_id
+  sequence
+  reason: interrupted | completed | failed
+  created_at
+  persisted_run_state
+```
+
+V1 writes a checkpoint only when a public run operation stops:
+
+- `run_deployment` returns an interrupted, completed, or failed execution.
+- `resume_run` returns an interrupted, completed, or failed execution.
+
+V1 does not guarantee recovery after a process crash while a node or external
+tool call is in progress.
+
+### Pinned Execution Environment
+
+Runs must not reinterpret bindings when resumed. The persisted run pins:
+
+- root artifact snapshot, in addition to its exact id/version
+- deployment snapshot, including the binding selection used at start
+- resolved saved child artifact snapshots, in addition to exact child versions
+- dependency contract information needed to revalidate those bindings
+
+A configuration edit after the run starts does not silently redirect a paused
+run from one MCP account/source to another. If a future repair/migration flow
+allows rebinding a stopped run, that must be explicit and auditable.
+
+Snapshotting artifact contents is required by the current implementation:
+`FileWorkflowArtifactStore` stores versioned paths but does not yet reject an
+overwrite of an existing version. A durable run must not depend on a later
+filesystem overwrite preserving the graph it originally executed. A future
+store that enforces immutable artifact writes may still retain snapshots as an
+audit record.
+
+## Persisted Runtime Codec
+
+`RunState.to_dict()` is useful for inspection but is not by itself a durable
+storage contract. Nested dataclasses, enums, path types, reducer refs, workflow
+refs, scopes, lineages, and interrupt routes must round-trip through a
+validated persisted model or codec.
+
+The storage codec must:
+
+- serialize the full stopped `RunState` needed for resume
+- restore equivalent enum/path/ref/dataclass values rather than loose dicts
+- reject corrupted or unsupported stored state with a clear diagnostic
+- be versioned or structured so later checkpoint migrations are possible
+- contain JSON-compatible values only at the persistence boundary
+
+Core should own the execution-state codec shape because it understands
+`RunState`. The platform store should persist the codec output, not reconstruct
+runtime internals itself.
+
+## Lifecycle Semantics
+
+### Start
+
+```text
+validate deployment and resolved dependencies
+if blocking diagnostics:
+  return unrunnable response; do not create a run
+
+pin resolved execution environment
+execute workflow
+persist WorkflowRun and stopped-state checkpoint
+return run summary
+```
+
+Completed and failed runs are persisted as well as interrupted runs. This makes
+inspection and debugging useful immediately without materially increasing
+storage implementation complexity.
+
+### Intentional Pause
+
+Only a declared `InterruptNode` creates a resumable pause.
+
+```text
+InterruptNode reached
+  -> RunStatus.INTERRUPTED
+  -> persist interrupted checkpoint
+  -> resume_readiness = ready when pinned dependencies validate
+```
+
+Interrupts raised within a native saved child subgraph retain the typed
+interrupt route already carried in `RunState`, so durable resume re-enters the
+original child scope.
+
+### Resume
+
+```text
+load run and latest interrupted checkpoint
+verify run is interrupted and resumable
+revalidate exact pinned dependency environment
+if dependency validation fails:
+  keep run status interrupted
+  set resume_readiness = blocked
+  return dependency diagnostics without mutating execution state
+else:
+  resume from restored RunState
+  persist next stopped-state checkpoint
+```
+
+When a missing or disabled pinned dependency returns, a later explicit
+`validate_run`, `inspect_run`, or `resume_run` may report it ready again.
+There is no background auto-resume.
+
+### Failure
+
+An ordinary runtime exception, MCP transport failure, or tool connection loss
+while execution is running fails the run.
+
+This is deliberately distinct from interruption. An external tool may have
+completed a side effect before losing its response; automatically presenting
+that condition as safely resumable could duplicate effects.
+
+```text
+tool/source failure during execution
+  -> status = failed
+  -> persist failed checkpoint and error details
+  -> no automatic resume
+```
+
+Recovery from such failures must later be declared in workflow semantics, such
+as explicit outcome mapping or a carefully specified retry policy.
+
+## Dependency Validation Semantics
+
+Dependency status has three positions:
+
+| Point | Result |
+| --- | --- |
+| Before start | `unrunnable`; no execution run is created |
+| At a declared interrupt | `interrupted`; persisted and potentially resumable |
+| Before resume when pinned dependency is broken | run remains `interrupted`, `resume_readiness = blocked`, diagnostics returned |
+| During live execution | `failed`, unless future workflow logic explicitly models recovery |
+
+Dependency diagnostics are platform/control-plane information, not graph
+outcomes. They must not be forced through a user's declared `ok` / `error`
+workflow outcome routes.
+
+## Public Surface
+
+The stable MCP workflow surface should eventually expose:
+
+```text
+wf.workflow.run_deployment
+  starts a run; always returns a stable run_id once execution starts
+
+wf.workflow.resume_run
+  resumes one durable interrupted run after pinned dependency validation
+
+wf.workflow.inspect_run
+  returns compact run status, readiness, terminal output/outcome,
+  interrupt summary, diagnostics, and trace_count
+
+wf.workflow.read_run_trace
+  returns a bounded trace slice selected by range
+```
+
+The response style stays progressive:
+
+- start/resume returns compact stopped-state status
+- inspect returns detail needed for decision making
+- trace is fetched only in bounded slices
+
+## Store Interface Direction
+
+Use a dedicated protocol rather than expanding MCP auth/catalog persistence:
+
+```text
+RunStore
+  save_run(run)
+  get_run(run_id)
+  save_checkpoint(checkpoint)
+  get_latest_checkpoint(run_id)
+  list_runs(query, cursor, limit)
+  read_checkpoints(run_id, cursor, limit)
+```
+
+The initial backend may be file-based under the configured application store
+root:
+
+```text
+.wf_mcp_store/
+  auth/
+  catalog/
+  workflows/
+  deployments/
+  runs/
+    <run_id>/
+      run.json
+      checkpoints/
+        000001.json
+        000002.json
+```
+
+The protocol must allow later SQLite/Postgres storage without changing the MCP
+surface or core runtime model.
+
+## Retry and Timeout Fields
+
+The codebase already carries `retry` and `timeout_seconds` fields in node/draft
+models, plus a runtime `retry_count` context field. They are not currently an
+implemented runtime policy.
+
+For durable-run v1:
+
+- treat these fields as declared-but-unsupported behavior
+- do not interpret failed MCP calls as retryable
+- do not add recovery checkpoints around external calls
+- document clearly when a future slice implements retry/timeout semantics
+
+A future retry design must distinguish known-not-executed failures from
+unknown-side-effect failures, or require idempotency/author-provided policy.
+
+## Testing Strategy
+
+Implementation tests must cover:
+
+- persisted `RunState` round-trip for root interrupts
+- persisted `RunState` round-trip for native child-subgraph interrupts
+- a completed run saved with output and terminal outcome
+- a failed run saved with error details
+- server/service recreation followed by successful `resume_run`
+- pinned dependency disabled after pause: resume is blocked and state remains
+  interrupted and unmodified
+- pinned dependency restored: later resume succeeds
+- changed live configuration cannot silently redirect pinned source bindings
+- transport failure during execution produces failed, not interrupted
+- trace retrieval is ranged and reports total trace count
+
+## Future Expansion
+
+Once stopped-run persistence is stable:
+
+1. Add checkpoint writes at explicit scheduler-safe boundaries if crash
+   recovery during long execution is needed.
+2. Investigate MCP tasks/progress reporting for live long-running operations.
+3. Design retry/timeout semantics explicitly rather than activating dormant
+   fields casually.
+4. Consider replay/time-travel only after external side-effect semantics are
+   documented.
+5. Add general cross-run memory separately if nodes need it; it is not a
+   replacement for checkpoints.
+
+## Recommendation
+
+Implement durable stopped-run snapshots first:
+
+1. Add a validated persisted `RunState` codec.
+2. Add `WorkflowRun`, `RunCheckpoint`, and `RunStore`.
+3. Save interrupted, completed, and failed runs after start/resume returns.
+4. Replace process-local `_active_runs` lookup with durable run retrieval.
+5. Add compact `inspect_run` and bounded `read_run_trace`.
+6. Revalidate pinned dependencies before resuming interrupted runs.
+
+This delivers durable human-in-the-loop execution and run inspection while
+preserving the correctness boundary that live external-call failures are not
+safe implicit pauses.

docs/workflow_artifacts.md

@@ -89,8 +89,10 @@ deployment surface, including interrupts raised inside saved child workflows.
 This support is in-memory only: server restart, reload that replaces process
 state, or another frontend process invalidates the `run_id`.
 
-Future run history should replace process-local `run_id` values with durable
+Durable run history is specified in
-run records. The likely shape is:
+[`2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md).
+It should replace process-local `run_id` values with durable run records. The
+planned surface is:
 
 - `run_deployment` starts or completes a run and returns `run_id`
 - `inspect_run(run_id)` returns status, output, diagnostics, and trace metadata
@@ -685,9 +687,11 @@ parent trace sees one node call; in the native examples child trace entries
 remain in the parent run state.
 
 Persisted resume is still not implemented. In-memory resume works because the
-server keeps the paused `RunState`; a durable run store will need to snapshot
+server keeps the paused `RunState`; the durable run design persists stopped
-the root workflow, prepared child dependencies, deployment bindings, and trace
+snapshots for interrupted, completed, and failed runs and pins root workflow,
-metadata before this can survive restart or move across processes.
+prepared child dependencies, deployment bindings, and trace metadata before a
+paused run can survive restart or move across processes. Only declared
+interrupts become resumable pauses; live tool/source failures remain failures.
 
 Blocking dependency failures happen before workflow execution and are not normal
 workflow outcomes. A missing source, disabled source, unresolved binding, or