520 lines
14 KiB
Markdown
520 lines
14 KiB
Markdown
# Persisted Run/Resume Hardening Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Lock down the persisted run/resume contract with missing regression tests and add a fail-fast required-store factory for durable API frontends.
|
|
|
|
**Architecture:** The current V1 run/resume behavior mostly exists in `wf_api.runs` and `wf_api.run_lifecycle`. This plan avoids redesigning `RunState`, retry, checkpoint cadence, or transactional storage. It adds tests for weakly covered contract rules, then introduces a protocol-neutral durable-context helper that future HTTP/API frontends can use before constructing `WorkflowApi`.
|
|
|
|
**Tech Stack:** Python 3.14, pytest, Pydantic v2, `wf_api`, `wf_artifacts`, `wf_core`, existing MCP-backed test fixtures.
|
|
|
|
---
|
|
|
|
## File Map
|
|
|
|
- Modify `tests/wf_api/test_run_api.py`
|
|
- Add contract tests for rejected resume of non-interrupted stored runs.
|
|
- Add contract test proving deleted deployments do not break stored run inspection.
|
|
- Strengthen trace-range validation test to prove store lookup is not touched.
|
|
- Create `src/wf_api/durable_context.py`
|
|
- Add required-store validation for durable frontends.
|
|
- Add `durable_workflow_api(context)` helper returning `WorkflowApi`.
|
|
- Modify `src/wf_api/__init__.py`
|
|
- Export the durable-context helpers.
|
|
- Create `tests/wf_api/test_durable_context.py`
|
|
- Cover success and missing-store failures.
|
|
- Modify `docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md`
|
|
- Mark these V1 hardening items as implemented after tests/helper land.
|
|
- Modify `docs/current_roadmap.md`
|
|
- Add a short note that durable required-store factory exists; transactional backend remains future work.
|
|
|
|
Out of scope:
|
|
|
|
- No transactional run store.
|
|
- No concurrent resume compare-and-swap.
|
|
- No paged run listing/checkpoint listing.
|
|
- No automatic retry/timeout policy.
|
|
- No HTTP server routes.
|
|
|
|
---
|
|
|
|
## Task 1: Add Missing Run Contract Tests
|
|
|
|
**Files:**
|
|
|
|
- Modify: `tests/wf_api/test_run_api.py`
|
|
|
|
- [ ] **Step 1: Add imports for stored-run helpers**
|
|
|
|
At the top of `tests/wf_api/test_run_api.py`, extend imports:
|
|
|
|
```python
|
|
from wf_artifacts import (
|
|
FileWorkflowArtifactStore,
|
|
FileRunStore,
|
|
ResumeReadiness,
|
|
WorkflowDeployment,
|
|
)
|
|
```
|
|
|
|
If `ResumeReadiness` is already imported later by another task, keep one import only.
|
|
|
|
- [ ] **Step 2: Add test for non-interrupted resume rejection**
|
|
|
|
Append this test after `test_run_api_completed_run_persists`:
|
|
|
|
```python
|
|
def test_run_api_rejects_resume_for_completed_run() -> None:
|
|
root = local_temp_root() / "run_api_resume_completed_rejected"
|
|
service, _ = _service_with_echo(root)
|
|
context = context_from_service(service)
|
|
api = WorkflowRunApi(context)
|
|
|
|
result = asyncio.run(
|
|
api.run_deployment(
|
|
deployment_id="echo.personal",
|
|
workflow_input={"text": "hello"},
|
|
)
|
|
)
|
|
|
|
with pytest.raises(ValueError, match="is not interrupted"):
|
|
asyncio.run(
|
|
api.resume_run(
|
|
run_id=result["run_id"],
|
|
resume_payload={"answer": "ignored"},
|
|
)
|
|
)
|
|
```
|
|
|
|
This locks down: `resume_run` only resumes stored runs whose latest status is `interrupted`.
|
|
|
|
- [ ] **Step 3: Add test that deleting deployment does not erase stored inspection**
|
|
|
|
Append this test after the non-interrupted resume test:
|
|
|
|
```python
|
|
def test_run_api_inspect_uses_pinned_environment_after_deployment_deleted() -> None:
|
|
root = local_temp_root() / "run_api_inspect_after_deployment_deleted"
|
|
service, artifact_store = _service_with_echo(root)
|
|
context = context_from_service(service)
|
|
api = WorkflowRunApi(context)
|
|
|
|
result = asyncio.run(
|
|
api.run_deployment(
|
|
deployment_id="echo.personal",
|
|
workflow_input={"text": "hello"},
|
|
)
|
|
)
|
|
artifact_store.delete_deployment("echo.personal")
|
|
|
|
summary = asyncio.run(api.inspect_run(run_id=result["run_id"]))
|
|
|
|
assert summary["status"] == "completed"
|
|
assert summary["run_id"] == result["run_id"]
|
|
assert summary["deployment_id"] == "echo.personal"
|
|
assert summary["artifact_id"] == "echo"
|
|
assert summary["output"]["echoed"] == "hello"
|
|
```
|
|
|
|
This locks down: stored run inspection reads pinned environment from `WorkflowRunRecord`, not mutable deployment storage.
|
|
|
|
- [ ] **Step 4: Strengthen trace-range-before-store validation**
|
|
|
|
In `test_run_api_rejects_invalid_trace_range_before_store_lookup`, replace the setup with a service whose run store raises if accessed:
|
|
|
|
```python
|
|
class ExplodingRunStore(FileRunStore):
|
|
def get_run(self, run_id: str):
|
|
raise AssertionError("run store must not be read before trace_range validation")
|
|
```
|
|
|
|
Then build the service explicitly:
|
|
|
|
```python
|
|
root = local_temp_root() / "run_api_invalid_trace_range"
|
|
service, _ = _service_with_echo(root)
|
|
service.run_store = ExplodingRunStore(root / "exploding_runs")
|
|
context = context_from_service(service)
|
|
api = WorkflowRunApi(context)
|
|
```
|
|
|
|
Keep the two existing `pytest.raises(ValueError, ...)` assertions unchanged. The test must still pass, proving invalid trace ranges fail before store lookup.
|
|
|
|
- [ ] **Step 5: Run focused run API tests**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api/test_run_api.py -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
all tests in tests/wf_api/test_run_api.py pass
|
|
```
|
|
|
|
- [ ] **Step 6: Commit Task 1**
|
|
|
|
```bash
|
|
git add tests/wf_api/test_run_api.py
|
|
git commit -m "test: lock down durable run resume contract"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 2: Add Required-Store Durable Context Helper
|
|
|
|
**Files:**
|
|
|
|
- Create: `src/wf_api/durable_context.py`
|
|
- Modify: `src/wf_api/__init__.py`
|
|
- Test: `tests/wf_api/test_durable_context.py`
|
|
|
|
- [ ] **Step 1: Write failing durable-context tests**
|
|
|
|
Create `tests/wf_api/test_durable_context.py`:
|
|
|
|
```python
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from wf_api import WorkflowApi
|
|
from wf_api.durable_context import durable_workflow_api, require_workflow_stores
|
|
from wf_api.operation_context import WorkflowOperationContext
|
|
from wf_api.stores import file_workflow_stores
|
|
from wf_mcp.broker import WfMcpService
|
|
from wf_mcp.broker.service.workflow_operation_context import context_from_service
|
|
from wf_mcp.storage import FileStore
|
|
|
|
|
|
def test_require_workflow_stores_returns_existing_store_bundle(tmp_path) -> None:
|
|
stores = file_workflow_stores(tmp_path / "workflow_stores")
|
|
service = WfMcpService(
|
|
store=FileStore(tmp_path / "mcp"),
|
|
artifact_store=stores.artifact_store,
|
|
draft_workspace_store=stores.draft_workspace_store,
|
|
run_store=stores.run_store,
|
|
)
|
|
context = context_from_service(service)
|
|
|
|
required = require_workflow_stores(context)
|
|
|
|
assert required.artifact_store is stores.artifact_store
|
|
assert required.draft_workspace_store is stores.draft_workspace_store
|
|
assert required.run_store is stores.run_store
|
|
|
|
|
|
def test_require_workflow_stores_rejects_missing_store(tmp_path) -> None:
|
|
service = WfMcpService(
|
|
store=FileStore(tmp_path / "mcp"),
|
|
artifact_store=None,
|
|
draft_workspace_store=None,
|
|
run_store=None,
|
|
)
|
|
context = context_from_service(service)
|
|
|
|
with pytest.raises(ValueError, match="durable workflow API requires stores"):
|
|
require_workflow_stores(context)
|
|
|
|
|
|
def test_durable_workflow_api_returns_workflow_api_with_same_context(tmp_path) -> None:
|
|
stores = file_workflow_stores(tmp_path / "workflow_stores")
|
|
service = WfMcpService(
|
|
store=FileStore(tmp_path / "mcp"),
|
|
artifact_store=stores.artifact_store,
|
|
draft_workspace_store=stores.draft_workspace_store,
|
|
run_store=stores.run_store,
|
|
)
|
|
context = context_from_service(service)
|
|
|
|
api = durable_workflow_api(context)
|
|
|
|
assert isinstance(api, WorkflowApi)
|
|
assert api.context is context
|
|
```
|
|
|
|
- [ ] **Step 2: Run tests and verify they fail**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api/test_durable_context.py -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
FAIL with ModuleNotFoundError: No module named 'wf_api.durable_context'
|
|
```
|
|
|
|
- [ ] **Step 3: Implement durable context helper**
|
|
|
|
Create `src/wf_api/durable_context.py`:
|
|
|
|
```python
|
|
from __future__ import annotations
|
|
|
|
from .operation_context import WorkflowOperationContext
|
|
from .service import WorkflowApi
|
|
from .stores import WorkflowStores
|
|
|
|
|
|
def require_workflow_stores(context: WorkflowOperationContext) -> WorkflowStores:
|
|
"""Return required stores or fail before constructing durable frontends.
|
|
|
|
`WorkflowOperationContext` keeps stores optional for compatibility tests and
|
|
lightweight MCP surfaces. Durable API surfaces need all stores up front so a
|
|
run cannot start without somewhere to persist artifacts, drafts, and stopped
|
|
execution state.
|
|
"""
|
|
missing = []
|
|
if context.artifact_store is None:
|
|
missing.append("artifact_store")
|
|
if context.draft_workspace_store is None:
|
|
missing.append("draft_workspace_store")
|
|
if context.run_store is None:
|
|
missing.append("run_store")
|
|
if missing:
|
|
raise ValueError(
|
|
"durable workflow API requires stores: " + ", ".join(missing)
|
|
)
|
|
return WorkflowStores(
|
|
artifact_store=context.artifact_store,
|
|
draft_workspace_store=context.draft_workspace_store,
|
|
run_store=context.run_store,
|
|
)
|
|
|
|
|
|
def durable_workflow_api(context: WorkflowOperationContext) -> WorkflowApi:
|
|
"""Construct a WorkflowApi only after durable store dependencies exist."""
|
|
require_workflow_stores(context)
|
|
return WorkflowApi(context)
|
|
|
|
|
|
__all__ = ["durable_workflow_api", "require_workflow_stores"]
|
|
```
|
|
|
|
- [ ] **Step 4: Export helpers from wf_api**
|
|
|
|
Modify `src/wf_api/__init__.py` to import and export:
|
|
|
|
```python
|
|
from .durable_context import durable_workflow_api, require_workflow_stores
|
|
```
|
|
|
|
Add these names to `__all__`:
|
|
|
|
```python
|
|
"durable_workflow_api",
|
|
"require_workflow_stores",
|
|
```
|
|
|
|
- [ ] **Step 5: Run durable-context tests**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api/test_durable_context.py -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
3 passed
|
|
```
|
|
|
|
- [ ] **Step 6: Run wf_api import-direction test**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api/test_import_direction.py -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
1 passed
|
|
```
|
|
|
|
- [ ] **Step 7: Commit Task 2**
|
|
|
|
```bash
|
|
git add src/wf_api/durable_context.py src/wf_api/__init__.py tests/wf_api/test_durable_context.py
|
|
git commit -m "feat: add durable workflow API store guard"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 3: Document Implemented Hardening
|
|
|
|
**Files:**
|
|
|
|
- Modify: `docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md`
|
|
- Modify: `docs/current_roadmap.md`
|
|
|
|
- [ ] **Step 1: Update spec current gaps**
|
|
|
|
In `docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md`, under `## Current Gaps / Next Implementation Work`, replace item 1 with:
|
|
|
|
```markdown
|
|
1. **Required stores for durable API**
|
|
- Implemented for process-local frontends through
|
|
`wf_api.durable_context.require_workflow_stores()` and
|
|
`wf_api.durable_context.durable_workflow_api()`.
|
|
- `WorkflowOperationContext` still allows optional stores for MCP test and
|
|
compatibility paths.
|
|
```
|
|
|
|
Under `## Implementation Order`, replace item 1 with:
|
|
|
|
```markdown
|
|
1. Contract regression tests now cover:
|
|
- non-interrupted `resume_run` rejection
|
|
- deleted deployment does not erase existing run inspection
|
|
- trace range validates before store lookup
|
|
- blocked resume writes no checkpoint (`tests/wf_mcp/test_saved_subgraphs.py`)
|
|
```
|
|
|
|
- [ ] **Step 2: Update current roadmap**
|
|
|
|
In `docs/current_roadmap.md`, find the durable/persisted run section and add:
|
|
|
|
```markdown
|
|
- `wf_api.durable_context` now provides a required-store guard for future
|
|
durable HTTP/API frontends. It preserves the current process-local behavior
|
|
while failing fast if artifact, draft, or run stores are missing.
|
|
```
|
|
|
|
If no durable/persisted run section exists, add it under the current `wf_api` or runtime roadmap notes.
|
|
|
|
- [ ] **Step 3: Run docs grep sanity check**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
rg -n "durable_workflow_api|require_workflow_stores|non-interrupted|blocked resume writes no checkpoint" docs
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
matches in persisted-run spec and/or current roadmap
|
|
```
|
|
|
|
- [ ] **Step 4: Commit Task 3**
|
|
|
|
```bash
|
|
git add docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md docs/current_roadmap.md
|
|
git commit -m "docs: record durable run hardening status"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 4: Final Verification
|
|
|
|
**Files:**
|
|
|
|
- Verify all touched code/tests/docs.
|
|
|
|
- [ ] **Step 1: Run focused tests**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api/test_run_api.py tests/wf_api/test_durable_context.py tests/wf_api/test_import_direction.py tests/wf_mcp/test_saved_subgraphs.py -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
all selected tests pass
|
|
```
|
|
|
|
- [ ] **Step 2: Run broader wf_api suite**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run pytest tests/wf_api -q
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
all wf_api tests pass
|
|
```
|
|
|
|
- [ ] **Step 3: Run ruff**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run ruff check src/wf_api tests/wf_api tests/wf_mcp/test_saved_subgraphs.py
|
|
uv run ruff format --check src/wf_api tests/wf_api tests/wf_mcp/test_saved_subgraphs.py
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
All checks passed
|
|
```
|
|
|
|
- [ ] **Step 4: Run basedpyright**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
uv run basedpyright --level error
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
0 errors, 0 warnings, 0 notes
|
|
```
|
|
|
|
Known caveat: this repo may still exit nonzero with the workspace enumeration warning even when it reports `0 errors`.
|
|
|
|
- [ ] **Step 5: Final report**
|
|
|
|
Report:
|
|
|
|
```text
|
|
Implemented persisted run/resume hardening:
|
|
- added missing contract regression tests
|
|
- added wf_api durable required-store guard
|
|
- updated persisted-run spec and roadmap
|
|
|
|
Verification:
|
|
- focused tests: ...
|
|
- wf_api tests: ...
|
|
- ruff: ...
|
|
- basedpyright: ...
|
|
```
|
|
|
|
---
|
|
|
|
## Self-Review
|
|
|
|
Spec coverage:
|
|
|
|
- `resume_run` only resumes interrupted runs: Task 1.
|
|
- Pinned environment survives deployment deletion: Task 1.
|
|
- Trace range validates before store lookup: Task 1.
|
|
- Blocked resume writes no checkpoint: already covered by `tests/wf_mcp/test_saved_subgraphs.py`; referenced in Task 3 and final focused verification.
|
|
- Required-store context/factory for durable API surfaces: Task 2.
|
|
- Transactional backend, CAS resume guard, run listing, progress, retry/timeout: explicitly out of scope.
|
|
|
|
Placeholder scan:
|
|
|
|
- No placeholder implementation steps.
|
|
- Code snippets include exact functions and expected assertions.
|
|
|
|
Type consistency:
|
|
|
|
- `WorkflowStores`, `WorkflowOperationContext`, and `WorkflowApi` names match current `src/wf_api`.
|
|
- `TraceRangeLike` remains unchanged; this plan does not alter MCP/CLI schemas.
|