# Persisted Run/Resume Contract

Date: 2026-06-03

Status: contract clarification; current V1 mostly implemented

Related:

- [Durable workflow runs and resume design](./2026-05-26-durable-workflow-runs-and-resume-design.md)
- [Store transaction and locking boundary](./2026-06-09-store-transaction-boundary.md)
- [Durable run operations](../../durable_run_operations.md)
- [WorkflowOperationContext audit](../research/2026-06-03-workflow-operation-context-audit.md)

## Purpose

This spec sharpens the public and internal contract for persisted deployment
runs. The older durable-run design describes the broad model and implemented V1.
This document defines the invariants that future MCP, CLI, and HTTP frontends
must preserve when starting, inspecting, tracing, and resuming stored runs.

The main rule: persisted resume is a human-in-the-loop interrupt mechanism, not
a generic retry/recovery mechanism for dead external sources.

## Contract Summary

- `run_deployment` creates a durable run only after deployment validation passes.
- Every started run receives a stable `run_id`.
- Stopped runs are persisted when execution returns `completed`, `failed`, or
  `interrupted`.
- `resume_run` only resumes runs whose latest stored status is `interrupted`.
- Resume validates the pinned execution environment before mutating execution
  state.
- If pinned dependency validation fails, resume returns `blocked` readiness and
  does not consume the resume payload or write a new execution checkpoint.
- If a live tool/source fails during execution, the run fails. It is not paused.
- Trace entries are never returned wholesale by default; callers must request a
  bounded range.

## Data Model Contract

### WorkflowRunRecord

`WorkflowRunRecord` is the durable summary for one started execution attempt.

Required invariants:

- `id` is a safe run id matching `RUN_ID_PATTERN`.
- `status` is one of `interrupted`, `completed`, or `failed`.
- `resume_readiness` is:
  - `ready` for interrupted runs that are currently resumable.
  - `blocked` for interrupted runs whose pinned dependency environment is
    currently invalid.
  - `not_applicable` for completed and failed runs.
- `environment` pins the exact deployment, root artifact, and child artifacts
  captured at run start.
- `latest_checkpoint_id` points to the latest stored execution checkpoint.
- `diagnostics` stores control-plane diagnostics, especially blocked-resume
  dependency diagnostics.
- `created_at` is stable for the run id.
- `updated_at` changes when the summary changes.

### RunCheckpoint

`RunCheckpoint` is the stored execution state at a public stopped boundary.

Required invariants:

- `run_id` matches the owning `WorkflowRunRecord.id`.
- `sequence` starts at `1` and increases monotonically per run.
- `reason` matches the stopped runtime status that caused checkpoint creation.
- `state` is a validated `PersistedRunState`, not an untyped dict.
- V1 writes checkpoints only when public run operations return stopped states.

### PinnedRunEnvironment

The pinned environment must be sufficient to resume without re-reading mutable
deployment or artifact definitions:

- `deployment`: exact deployment binding snapshot used at start.
- `root_artifact`: exact root artifact snapshot used at start.
- `child_artifacts`: exact saved child artifact snapshots used at start.

Changing or deleting a deployment after a run starts must not silently redirect
or erase that run's resume environment.

## Operation Contract

### `run_deployment`

Input:

- deployment id
- workflow input
- optional trace range

Behavior:

1. Load deployment and artifact from the configured artifact store.
2. Resolve saved child artifact tree.
3. Validate root and child dependency bindings against current available
   sources.
4. If validation has blocking diagnostics:
   - return `status="unrunnable"`
   - return `run_id=None`
   - do not create a run record
   - do not write a checkpoint
5. If validation passes:
   - create pinned environment
   - execute workflow through `WorkflowRuntimeRunner`
   - persist stopped run and checkpoint
   - return compact run payload

Current implementation:

- `WorkflowRunApi.run_deployment()` follows this contract.
- `persist_stopped_run()` rejects active runtime statuses.

### `inspect_run`

Input:

- run id

Behavior:

1. Load `WorkflowRunRecord`.
2. Load latest checkpoint.
3. Decode checkpoint state into `RunState`.
4. Return compact summary:
   - status
   - run id
   - resume readiness
   - interrupt payload when present
   - outcome/error/output when present
   - diagnostics
   - trace count
5. Do not return trace entries.

Current implementation:

- `WorkflowRunApi.inspect_run()` follows this contract.

### `read_run_trace`

Input:

- run id
- trace range with `start >= 0` and `limit > 0`

Behavior:

1. Validate trace range before store lookup.
2. Load run and latest checkpoint.
3. Return compact run summary plus the bounded trace slice.
4. Return trace metadata:
   - `trace_start`
   - `trace_limit`
   - `trace_truncated`
   - `trace_count`

Current implementation:

- `WorkflowRunApi.read_run_trace()` follows this contract.

### `list_runs`

Returns paged compact summaries for stopped durable runs. The list payload
contains run id, deployment id, artifact id/version, status, resume readiness,
diagnostic count, and timestamps. It never returns trace entries, checkpoint
state, runtime output, or pinned environment bodies.

### `resume_run`

Input:

- run id
- resume payload
- resume outcome, default `submitted`
- optional trace range

Behavior:

1. Load run and latest checkpoint.
2. Reject if stored run status is not `interrupted`.
3. Decode checkpoint state into `RunState`.
4. Validate pinned environment against current available sources.
5. If validation has blocking diagnostics:
   - keep run status `interrupted`
   - set `resume_readiness="blocked"`
   - save updated run summary diagnostics
   - do not write a new execution checkpoint
   - do not apply resume payload
6. If validation passes:
   - restore saved child artifact tree from pinned environment
   - resume workflow through `WorkflowRuntimeRunner`
   - persist next stopped run/checkpoint with same `run_id`
   - return compact run payload

Current implementation:

- `WorkflowRunApi.resume_run()` follows this contract for stored interrupted
  runs and blocked dependency validation.
- Same-process callers are serialized per `run_id` around the restore,
  dependency validation, runtime resume, and checkpoint write sequence.
- `restore_interrupted_run()` rejects non-interrupted statuses.
- `mark_resume_blocked()` updates run summary without writing a checkpoint.

## Status Semantics

| Condition | Public result |
| --- | --- |
| Deployment dependencies invalid before start | `status="unrunnable"`, no run id |
| Workflow reaches explicit interrupt | `status="interrupted"`, `resume_readiness="ready"` |
| Interrupted run has broken pinned dependency before resume | `status="interrupted"`, `resume_readiness="blocked"` |
| Workflow completes | `status="completed"`, `resume_readiness="not_applicable"` |
| Workflow/runtime/tool fails during execution | `status="failed"`, `resume_readiness="not_applicable"` |

`unrunnable` is not a stored run status. It is a pre-start response.

## External Source Failure Rule

External source failure during execution is not a resumable pause.

Rationale:

- A tool may have performed a side effect before disconnecting or failing to
  return a response.
- Resuming from that point without explicit workflow semantics could duplicate
  external side effects.
- Retry/timeout fields exist in models but are not yet an implemented runtime
  policy.

Future retry support must explicitly define idempotency, unknown-side-effect
behavior, and checkpoint boundaries.

## Store Contract

`RunStore` must provide:

- save/get/list run records
- save/get/list checkpoints
- latest-checkpoint lookup
- safe run id validation

`FileRunStore` currently stores:

```text
<store-root>/runs/<run-id>/run.json
<store-root>/runs/<run-id>/checkpoints/000001.json
<store-root>/runs/<run-id>/checkpoints/000002.json
```

Current limits:

- `WorkflowRunApi.resume_run()` provides a process-local per-run critical
  section for one API/server process.
- `FileRunStore` locks individual file writes only; it does not provide
  compare-and-swap or cross-process transactions.
- The file store is appropriate for local/dev/single-process use.
- A multi-worker or cloud deployment still needs SQLite/Postgres or another
  transactional store before claiming strong concurrent resume safety.

## Frontend Contract

MCP, CLI, and future HTTP surfaces should preserve the same operation semantics:

- Start: `run_deployment`
- List compact summaries: `list_runs`
- Inspect: `inspect_run`
- Debug trace: `read_run_trace`
- Continue explicit interrupt: `resume_run`

Frontend-specific names may differ, but they must not change:

- status meanings
- trace range requirement
- blocked resume behavior
- run id stability
- pinned environment semantics
- no-implicit-pause rule for dead tools/sources

## Current Gaps / Next Implementation Work

The core V1 behavior exists. Remaining implementation work should focus on
hardening and frontend durability:

1. **Required stores for durable API**
   - Implemented for process-local frontends through
     `wf_api.durable_context.require_workflow_stores()` and
     `wf_api.durable_context.durable_workflow_api()`.
   - `WorkflowOperationContext` still allows optional stores for MCP test and
     compatibility paths.

2. **Run listing and checkpoint listing**
   - `RunStore` can list runs, and the public workflow API exposes compact paged
     run listing. Checkpoint listing remains intentionally private for now.

3. **Transactional backend**
   - `FileRunStore` is fine for local process use.
   - Multi-process/cloud use needs SQLite/Postgres or another transactional
     store to avoid lost writes and weak concurrent resume behavior.

4. **Resume concurrency guard**
   - Implemented for same-process API callers through a per-`run_id` async
     critical section in `WorkflowRunApi.resume_run()`.
   - Cross-process protection remains part of the transactional backend gap.

5. **Protocol-native long-running progress**
   - MCP tasks/progress or an HTTP streaming/event surface should report active
     long-running runs without bloating stopped-run responses.

6. **Retry/timeout policy**
   - Do not activate existing retry/timeout fields casually.
   - Specify idempotency and unknown-side-effect semantics first.

## Implementation Order

Recommended order after this contract:

1. Contract regression tests now cover:
   - non-interrupted `resume_run` rejection
   - deleted deployment does not erase existing run inspection
   - trace range validates before store lookup
   - blocked resume writes no checkpoint (`tests/wf_mcp/test_saved_subgraphs.py`)
2. Add a required-store context/factory for durable API surfaces.
3. Specify and implement a transactional run store backend when a cloud/API
   deployment is real.
4. Add paged run listing/checkpoint listing only after the storage boundary is
   stable.

## Non-Goals

- Do not redesign `RunState`.
- Do not checkpoint every node.
- Do not add automatic retry.
- Do not treat source death as an interrupt.
- Do not require MCP clients to reload dynamic tools to run workflows.
- Do not make HTTP/API design depend on `WfMcpService`.