durable stopped workflow runs and resume

This commit is contained in:
lda
2026-05-26 12:20:40 +07:00 Verified
parent d37077397a
commit ba8fd2b614
30 changed files with 1216 additions and 146 deletions
@@ -1,7 +1,7 @@
# Native Subgraphs Design
Status: prepared-child execution, saved-artifact resolution, and process-local
interrupt resume implemented; durable resume specified separately
Status: prepared-child execution, saved-artifact resolution, and durable
stopped-run interrupt resume implemented
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
@@ -409,10 +409,11 @@ child artifact twice against different accounts or capability bindings.
### Saved Child Interrupt Status
Native prepared children can interrupt and resume in core. The workflow surface
now exposes process-local `run_deployment` / `resume_run` support for saved
now exposes durable `run_deployment` / `resume_run` support for saved
interrupting artifacts, including interrupts raised by saved descendants.
That support does not survive process restart yet. Durable run persistence is
specified separately in
Stopped checkpoints pin the deployment and exact root/child artifact
definitions; dependency drift can block resume without mutating the stopped
execution. Durable run persistence is specified in
[`2026-05-26-durable-workflow-runs-and-resume-design.md`](2026-05-26-durable-workflow-runs-and-resume-design.md).
## Implementation Slices
@@ -462,8 +463,8 @@ specified separately in
- Dependencies, missing artifacts, and saved-child cycles validate before
execution.
- Tests cover saved child execution through deployment bindings, nested saved
child dependencies, missing/cyclic diagnostics, and process-local
interrupt/resume for saved children.
child dependencies, missing/cyclic diagnostics, and durable interrupt/resume
for saved children.
### Slice 4: Optional Policy Expansion
@@ -491,12 +492,8 @@ specified separately in
## Recommendation
The typed boundary scaffold, prepared-child runtime, and routed interrupt
resume are complete. Implement Slice 3 in the platform layer: prepare saved
non-interrupting child artifacts recursively under one deployment environment
and pass those prepared dependencies into core execution.
The process-local saved-interrupt execution path is complete. Next add durable
run/checkpoint persistence so interrupted saved children survive process
restart. Do not delete wrapper-node helpers yet; they remain compatibility APIs
while saved native execution matures.
The typed boundary scaffold, prepared-child runtime, saved-child resolution,
and durable routed interrupt resume are complete. Do not delete wrapper-node
helpers yet; they remain compatibility APIs while saved native execution
matures. Next policy work is optional per-use-site child deployment overrides
and richer protocol-native long-running progress/reporting.
@@ -1,6 +1,6 @@
# Durable Workflow Runs and Resume Design
Status: design approved for implementation planning
Status: v1 implemented; future protocol-native progress and broader recovery remain
Durable workflow runs turn the current process-local `run_deployment` /
`resume_run` behavior into platform state. The runtime already exposes the
@@ -376,16 +376,16 @@ Once stopped-run persistence is stable:
5. Add general cross-run memory separately if nodes need it; it is not a
replacement for checkpoints.
## Recommendation
## Implemented V1
Implement durable stopped-run snapshots first:
Durable stopped-run snapshots now provide:
1. Add a validated persisted `RunState` codec.
2. Add `WorkflowRun`, `RunCheckpoint`, and `RunStore`.
3. Save interrupted, completed, and failed runs after start/resume returns.
4. Replace process-local `_active_runs` lookup with durable run retrieval.
5. Add compact `inspect_run` and bounded `read_run_trace`.
6. Revalidate pinned dependencies before resuming interrupted runs.
1. A validated persisted `RunState` codec.
2. `WorkflowRunRecord`, `RunCheckpoint`, and `RunStore`.
3. Checkpoints for interrupted, completed, and failed public executions.
4. Durable resume retrieval instead of process-local `_active_runs`.
5. Compact `inspect_run` and bounded `read_run_trace` tools.
6. Pinned dependency revalidation before interrupted runs resume.
This delivers durable human-in-the-loop execution and run inspection while
preserving the correctness boundary that live external-call failures are not