docs n + 1

2026-06-03 01:18:17 +07:00
parent 8101cbad5c
commit 170bb1110d
4 changed files with 552 additions and 3 deletions
@@ -47,6 +47,8 @@ implementation state.
     hints, next actions, raw workflow plans, runtime dependencies, saved
     subgraph preparation, and durable run lifecycle helpers. Old
     `wf_mcp.workflow_surface` helper paths remain compatibility shims.
   - The boundary is documented in
     [wf_api architecture](./wf_api_architecture.md).
 ## Active Next Roadmap
@@ -59,9 +61,11 @@ implementation state.
     [2026-06-03 WorkflowOperationContext shape audit](./superpowers/research/2026-06-03-workflow-operation-context-audit.md).
 2. **Persisted run/resume spec**
-   - Define the process-restart resume contract around run records, pinned
+   - Completed: the process-restart resume contract is defined in
-     deployment/artifact/subgraph environment, source/capability validation,
+     [2026-06-03 persisted run/resume contract](./superpowers/specs/2026-06-03-persisted-run-resume-contract.md).
-     trace paging, and interrupt-only pause semantics.
+   - It covers run records, pinned deployment/artifact/subgraph environment,
     source/capability validation, trace paging, and interrupt-only pause
     semantics.
   - Keep ordinary dead tools/sources as diagnostics or failed runs, not implicit
     pauses.
@@ -0,0 +1,325 @@
 # Persisted Run/Resume Contract
 Date: 2026-06-03
 Status: contract clarification; current V1 mostly implemented
 Related:
 - [Durable workflow runs and resume design](./2026-05-26-durable-workflow-runs-and-resume-design.md)
 - [Durable run operations](../../durable_run_operations.md)
 - [WorkflowOperationContext audit](../research/2026-06-03-workflow-operation-context-audit.md)
 ## Purpose
 This spec sharpens the public and internal contract for persisted deployment
 runs. The older durable-run design describes the broad model and implemented V1.
 This document defines the invariants that future MCP, CLI, and HTTP frontends
 must preserve when starting, inspecting, tracing, and resuming stored runs.
 The main rule: persisted resume is a human-in-the-loop interrupt mechanism, not
 a generic retry/recovery mechanism for dead external sources.
 ## Contract Summary
 - `run_deployment` creates a durable run only after deployment validation passes.
 - Every started run receives a stable `run_id`.
 - Stopped runs are persisted when execution returns `completed`, `failed`, or
  `interrupted`.
 - `resume_run` only resumes runs whose latest stored status is `interrupted`.
 - Resume validates the pinned execution environment before mutating execution
  state.
 - If pinned dependency validation fails, resume returns `blocked` readiness and
  does not consume the resume payload or write a new execution checkpoint.
 - If a live tool/source fails during execution, the run fails. It is not paused.
 - Trace entries are never returned wholesale by default; callers must request a
  bounded range.
 ## Data Model Contract
 ### WorkflowRunRecord
 `WorkflowRunRecord` is the durable summary for one started execution attempt.
 Required invariants:
 - `id` is a safe run id matching `RUN_ID_PATTERN`.
 - `status` is one of `interrupted`, `completed`, or `failed`.
 - `resume_readiness` is:
  - `ready` for interrupted runs that are currently resumable.
  - `blocked` for interrupted runs whose pinned dependency environment is
    currently invalid.
  - `not_applicable` for completed and failed runs.
 - `environment` pins the exact deployment, root artifact, and child artifacts
  captured at run start.
 - `latest_checkpoint_id` points to the latest stored execution checkpoint.
 - `diagnostics` stores control-plane diagnostics, especially blocked-resume
  dependency diagnostics.
 - `created_at` is stable for the run id.
 - `updated_at` changes when the summary changes.
 ### RunCheckpoint
 `RunCheckpoint` is the stored execution state at a public stopped boundary.
 Required invariants:
 - `run_id` matches the owning `WorkflowRunRecord.id`.
 - `sequence` starts at `1` and increases monotonically per run.
 - `reason` matches the stopped runtime status that caused checkpoint creation.
 - `state` is a validated `PersistedRunState`, not an untyped dict.
 - V1 writes checkpoints only when public run operations return stopped states.
 ### PinnedRunEnvironment
 The pinned environment must be sufficient to resume without re-reading mutable
 deployment or artifact definitions:
 - `deployment`: exact deployment binding snapshot used at start.
 - `root_artifact`: exact root artifact snapshot used at start.
 - `child_artifacts`: exact saved child artifact snapshots used at start.
 Changing or deleting a deployment after a run starts must not silently redirect
 or erase that run's resume environment.
 ## Operation Contract
 ### `run_deployment`
 Input:
 - deployment id
 - workflow input
 - optional trace range
 Behavior:
 1. Load deployment and artifact from the configured artifact store.
 2. Resolve saved child artifact tree.
 3. Validate root and child dependency bindings against current available
   sources.
 4. If validation has blocking diagnostics:
   - return `status="unrunnable"`
   - return `run_id=None`
   - do not create a run record
   - do not write a checkpoint
 5. If validation passes:
   - create pinned environment
   - execute workflow through `WorkflowRuntimeRunner`
   - persist stopped run and checkpoint
   - return compact run payload
 Current implementation:
 - `WorkflowRunApi.run_deployment()` follows this contract.
 - `persist_stopped_run()` rejects active runtime statuses.
 ### `inspect_run`
 Input:
 - run id
 Behavior:
 1. Load `WorkflowRunRecord`.
 2. Load latest checkpoint.
 3. Decode checkpoint state into `RunState`.
 4. Return compact summary:
   - status
   - run id
   - resume readiness
   - interrupt payload when present
   - outcome/error/output when present
   - diagnostics
   - trace count
 5. Do not return trace entries.
 Current implementation:
 - `WorkflowRunApi.inspect_run()` follows this contract.
 ### `read_run_trace`
 Input:
 - run id
 - trace range with `start >= 0` and `limit > 0`
 Behavior:
 1. Validate trace range before store lookup.
 2. Load run and latest checkpoint.
 3. Return compact run summary plus the bounded trace slice.
 4. Return trace metadata:
   - `trace_start`
   - `trace_limit`
   - `trace_truncated`
   - `trace_count`
 Current implementation:
 - `WorkflowRunApi.read_run_trace()` follows this contract.
 ### `resume_run`
 Input:
 - run id
 - resume payload
 - resume outcome, default `submitted`
 - optional trace range
 Behavior:
 1. Load run and latest checkpoint.
 2. Reject if stored run status is not `interrupted`.
 3. Decode checkpoint state into `RunState`.
 4. Validate pinned environment against current available sources.
 5. If validation has blocking diagnostics:
   - keep run status `interrupted`
   - set `resume_readiness="blocked"`
   - save updated run summary diagnostics
   - do not write a new execution checkpoint
   - do not apply resume payload
 6. If validation passes:
   - restore saved child artifact tree from pinned environment
   - resume workflow through `WorkflowRuntimeRunner`
   - persist next stopped run/checkpoint with same `run_id`
   - return compact run payload
 Current implementation:
 - `WorkflowRunApi.resume_run()` follows this contract for stored interrupted
  runs and blocked dependency validation.
 - `restore_interrupted_run()` rejects non-interrupted statuses.
 - `mark_resume_blocked()` updates run summary without writing a checkpoint.
 ## Status Semantics
 | Condition | Public result |
 | --- | --- |
 | Deployment dependencies invalid before start | `status="unrunnable"`, no run id |
 | Workflow reaches explicit interrupt | `status="interrupted"`, `resume_readiness="ready"` |
 | Interrupted run has broken pinned dependency before resume | `status="interrupted"`, `resume_readiness="blocked"` |
 | Workflow completes | `status="completed"`, `resume_readiness="not_applicable"` |
 | Workflow/runtime/tool fails during execution | `status="failed"`, `resume_readiness="not_applicable"` |
 `unrunnable` is not a stored run status. It is a pre-start response.
 ## External Source Failure Rule
 External source failure during execution is not a resumable pause.
 Rationale:
 - A tool may have performed a side effect before disconnecting or failing to
  return a response.
 - Resuming from that point without explicit workflow semantics could duplicate
  external side effects.
 - Retry/timeout fields exist in models but are not yet an implemented runtime
  policy.
 Future retry support must explicitly define idempotency, unknown-side-effect
 behavior, and checkpoint boundaries.
 ## Store Contract
 `RunStore` must provide:
 - save/get/list run records
 - save/get/list checkpoints
 - latest-checkpoint lookup
 - safe run id validation
 `FileRunStore` currently stores:
 ```text
 <store-root>/runs/<run-id>/run.json
 <store-root>/runs/<run-id>/checkpoints/000001.json
 <store-root>/runs/<run-id>/checkpoints/000002.json
 ```
 Current limits:
 - The file store uses per-process locking only.
 - It is appropriate for local/dev/single-process use.
 - A long-lived API or multi-worker deployment should use a transactional store
  later.
 ## Frontend Contract
 MCP, CLI, and future HTTP surfaces should preserve the same operation semantics:
 - Start: `run_deployment`
 - Inspect: `inspect_run`
 - Debug trace: `read_run_trace`
 - Continue explicit interrupt: `resume_run`
 Frontend-specific names may differ, but they must not change:
 - status meanings
 - trace range requirement
 - blocked resume behavior
 - run id stability
 - pinned environment semantics
 - no-implicit-pause rule for dead tools/sources
 ## Current Gaps / Next Implementation Work
 The core V1 behavior exists. Remaining implementation work should focus on
 hardening and frontend durability:
 1. **Required stores for durable API**
   - Current `WorkflowOperationContext` allows optional stores for MCP test and
     compatibility paths.
   - A durable HTTP/API backend should construct a stricter context where
     artifact, draft, and run stores are required.
 2. **Run listing and checkpoint listing**
   - `RunStore` can list runs/checkpoints, but the public workflow API does not
     yet expose a mature paged run catalog.
   - Add only after inspect/trace semantics remain compact and stable.
 3. **Transactional backend**
   - `FileRunStore` is fine for local process use.
   - Multi-process/cloud use needs SQLite/Postgres or another transactional
     store to avoid lost writes and weak concurrent resume behavior.
 4. **Resume concurrency guard**
   - Concurrent `resume_run` calls for the same interrupted run should not both
     advance from the same checkpoint.
   - V1 file store does not provide compare-and-swap semantics.
 5. **Protocol-native long-running progress**
   - MCP tasks/progress or an HTTP streaming/event surface should report active
     long-running runs without bloating stopped-run responses.
 6. **Retry/timeout policy**
   - Do not activate existing retry/timeout fields casually.
   - Specify idempotency and unknown-side-effect semantics first.
 ## Implementation Order
 Recommended order after this contract:
 1. Add tests that explicitly lock down the current contract where coverage is
   weak:
   - non-interrupted `resume_run` rejection
   - blocked resume writes no checkpoint
   - deleted deployment does not erase existing run inspection
   - trace range validates before store lookup
 2. Add a required-store context/factory for durable API surfaces.
 3. Specify and implement a transactional run store backend when a cloud/API
   deployment is real.
 4. Add paged run listing/checkpoint listing only after the storage boundary is
   stable.
 ## Non-Goals
 - Do not redesign `RunState`.
 - Do not checkpoint every node.
 - Do not add automatic retry.
 - Do not treat source death as an interrupt.
 - Do not require MCP clients to reload dynamic tools to run workflows.
 - Do not make HTTP/API design depend on `WfMcpService`.
@@ -0,0 +1,217 @@
 # wf_api Architecture Boundary
 `wf_api` is the process-local workflow application service layer. It is not just
 a DTO wrapper around `wf_core`, and it is not an MCP transport package.
 The package owns workflow-facing application operations that combine execution,
 saved artifacts, deployment validation, authoring guidance, progressive payloads,
 and run lifecycle policy into a stable API that CLI, MCP, and future HTTP
 frontends can share.
 ## Responsibility Map
 | Package | Responsibility |
 | --- | --- |
 | `wf_core` | Workflow execution semantics: graph models, runtime state, scheduler, path/state mapping, interrupts, subgraphs, foreach, and run-state codec. |
 | `wf_artifacts` | Saved definitions and persistence contracts: workflow artifacts, deployments, draft workspaces, run records, checkpoints, artifact/deployment validation models. |
 | `wf_platform` | Shared capability/source concepts and platform-facing contracts such as capability refs, source inventory models, documentation source models, and JSON schema helpers. |
 | `wf_api` | Application workflows over core/artifacts/platform: capability discovery, wrapper hints, draft editing, artifact/deployment operations, run/resume operations, next actions, and progressive response shaping. |
 | `wf_mcp` | MCP-specific transport, tool schemas, upstream MCP adapters, broker services, proxy/admin tools, config reload, and `WorkflowApi` context construction for MCP. |
 | future `wf_http` | HTTP transport over `wf_api`, not a reimplementation of workflow business logic. |
 | `wf_cli` | CLI frontend over `wf_api`; it may run locally against process-local stores or later target an HTTP backend. |
 ## What Belongs In wf_api
 `wf_api` should contain code that is:
 - protocol-neutral between MCP, CLI, and future HTTP
 - workflow-application policy rather than low-level graph execution
 - response shaping for human/LLM clients, such as compact list payloads and
  bounded trace slices
 - authoring guidance, such as wrapper hints and next actions
 - orchestration across `wf_core`, `wf_artifacts`, and `wf_platform`
 - durable run lifecycle policy over `RunStore` and `WorkflowRuntimeRunner`
 Examples that belong in `wf_api`:
 - `WorkflowApi`
 - `WorkflowCapabilityApi`
 - `WorkflowDraftApi`
 - `WorkflowArtifactApi`
 - `WorkflowDeploymentApi`
 - `WorkflowRunApi`
 - wrapper hints
 - next actions
 - runtime dependency resolution
 - saved subgraph preparation helpers
 - run lifecycle helpers such as `persist_stopped_run()` and
  `validate_pinned_resume_environment()`
 ## What Does Not Belong In wf_api
 `wf_api` must not contain:
 - MCP SDK calls
 - FastMCP tool/resource/prompt registration
 - MCP content block models or MCP schema workarounds
 - broker config file mutation
 - upstream MCP session/runtime management
 - proxy mounting or reload logic
 - local process service construction that assumes `WfMcpService`
 - scheduler/execution semantics that belong in `wf_core`
 - persistence model definitions that belong in `wf_artifacts`
 The hard import rule remains:
 ```text
 wf_mcp -> wf_api is allowed
 wf_api -> wf_mcp is forbidden
 ```
 ## WorkflowOperationContext
 `WorkflowOperationContext` is the adapter seam that lets `wf_api` stay
 transport-neutral.
 Current shape:
 ```python
 WorkflowOperationContext(
    artifact_store=...,
    draft_workspace_store=...,
    run_store=...,
    events=...,
    specs=...,
    runtime=...,
    live_sources=...,
 )
 ```
 Important rules:
 - Source inventory goes through `context.specs.capability_sources`.
 - Qualified node lookup goes through `context.specs.get_qualified_spec()`.
 - Runtime execution goes through `context.runtime`.
 - Workflow events go through `context.events.record_workflow_event()`.
 - Optional live source checks go through `context.live_sources`.
 - Stores are still optional for MCP/test compatibility, but durable API
  frontends should construct stricter contexts with required stores.
 Do not add a catch-all `service` field to the context. If a domain API needs a
 new dependency, add a narrow protocol or explicit field.
 ## Domain Services
 `WorkflowApi` composes focused domain APIs:
 ```text
 WorkflowApi
  capabilities: WorkflowCapabilityApi
  drafts: WorkflowDraftApi
  artifacts: WorkflowArtifactApi
  deployments: WorkflowDeploymentApi
  runs: WorkflowRunApi
 ```
 These domain services are allowed to return `dict[str, Any]` payloads because
 they define application-facing response contracts consumed by multiple
 frontends. Internally, they should prefer typed models from `wf_core`,
 `wf_artifacts`, and `wf_platform`, then serialize at the boundary.
 ## Relationship To wf_core
 `wf_core` is lower-level than `wf_api`.
 `wf_api` may:
 - compile or validate workflow-facing requests into core/artifact models
 - call runtime runners that execute core workflows
 - shape run traces and diagnostics for clients
 `wf_api` must not:
 - decide scheduler semantics
 - mutate `RunState` internals directly
 - add graph node semantics
 - encode MCP/client-specific behavior into core execution
 If a behavior changes how workflows execute, it probably belongs in `wf_core`
 or in an explicit runtime dependency injected into `wf_core`, not in `wf_api`.
 ## Relationship To wf_artifacts
 `wf_artifacts` owns durable definitions and storage contracts. `wf_api` owns
 operations over them.
 Examples:
 - `WorkflowArtifact`, `WorkflowDeployment`, `WorkflowRunRecord`, and
  `RunCheckpoint` belong in `wf_artifacts`.
 - `create_artifact_from_plan`, `save_deployment`, `run_deployment`, and
  `resume_run` application flows belong in `wf_api`.
 `wf_api` should not define parallel persistence models for the same durable
 concepts. If a response needs a different shape, create response payload helpers
 or next actions rather than duplicating the storage model.
 ## Relationship To wf_mcp
 `wf_mcp` adapts MCP into `wf_api`.
 Current MCP path:
 ```text
 wf_mcp.workflow_surface.tools
  -> WorkflowApi(context_from_service(service))
  -> wf_api domain service
  -> WorkflowOperationContext protocol
  -> focused broker service / store / runtime implementation
 ```
 MCP-specific concerns stay outside `wf_api`:
 - tool schema names and safe tool names
 - MCP resources/prompts-as-tools registration
 - upstream MCP source liveness checks
 - FastMCP notifications
 - MCP content block normalization
 - broker connection config/reload
 ## Future HTTP/API Boundary
 A future HTTP API should reuse `WorkflowApi`, not copy MCP handlers.
 Expected shape:
 ```text
 wf_http route/controller
  -> WorkflowApi(required_store_context)
  -> wf_api domain services
 ```
 The HTTP layer should own:
 - request/response framework models
 - auth/session policy
 - API routing
 - streaming/progress transport if needed
 - construction of a durable `WorkflowOperationContext`
 The HTTP layer should not own:
 - workflow validation semantics
 - run resume semantics
 - wrapper hint policy
 - deployment dependency validation
 ## Current Roadmap Implication
 The next implementation work should follow this order:
 1. Harden persisted run/resume contract tests in `wf_api`.
 2. Introduce a stricter required-store context/factory for durable API surfaces.
 3. Design the durable HTTP/API frontend around `WorkflowApi`.
 4. Align CLI so it can target either local process stores or the future HTTP API.
 Workflow primitives such as fork/gather and additional authoring sugar should
 resume after the durability/platform boundary is stable.
@@ -60,6 +60,9 @@ New code should treat `wf_api.WorkflowApi` as the application-facing API. Do not
 add new callers that import `WorkflowSurfaceHandlers` directly unless they are
 compatibility tests.
 The broader application-service boundary is documented in
 [`wf_api_architecture.md`](wf_api_architecture.md).
 This is a dependency-direction cleanup, not a full domain split. Most API
 methods still mirror the old workflow-surface payloads and return
 `dict[str, Any]`.