lda-wf/docs/historical/superpowers/plans/2026-06-03-persisted-run-resume-hardening.md

Persisted Run/Resume Hardening Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Lock down the persisted run/resume contract with missing regression tests and add a fail-fast required-store factory for durable API frontends.

Architecture: The current V1 run/resume behavior mostly exists in wf_api.runs and wf_api.run_lifecycle. This plan avoids redesigning RunState, retry, checkpoint cadence, or transactional storage. It adds tests for weakly covered contract rules, then introduces a protocol-neutral durable-context helper that future HTTP/API frontends can use before constructing WorkflowApi.

Tech Stack: Python 3.14, pytest, Pydantic v2, wf_api, wf_artifacts, wf_core, existing MCP-backed test fixtures.

---

File Map

• Modify tests/wf_api/test_run_api.py
  • Add contract tests for rejected resume of non-interrupted stored runs.
  • Add contract test proving deleted deployments do not break stored run inspection.
  • Strengthen trace-range validation test to prove store lookup is not touched.
• Create src/wf_api/durable_context.py
  • Add required-store validation for durable frontends.
  • Add durable_workflow_api(context) helper returning WorkflowApi.
• Modify src/wf_api/__init__.py
  • Export the durable-context helpers.
• Create tests/wf_api/test_durable_context.py
  • Cover success and missing-store failures.
• Modify docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md
  • Mark these V1 hardening items as implemented after tests/helper land.
• Modify docs/current_roadmap.md
  • Add a short note that durable required-store factory exists; transactional backend remains future work.

Out of scope:

• No transactional run store.
• No concurrent resume compare-and-swap.
• No paged run listing/checkpoint listing.
• No automatic retry/timeout policy.
• No HTTP server routes.

---

Task 1: Add Missing Run Contract Tests

Files:

• Modify: tests/wf_api/test_run_api.py

• Step 1: Add imports for stored-run helpers

At the top of tests/wf_api/test_run_api.py, extend imports:

from wf_artifacts import (
    FileWorkflowArtifactStore,
    FileRunStore,
    ResumeReadiness,
    WorkflowDeployment,
)

If ResumeReadiness is already imported later by another task, keep one import only.

• Step 2: Add test for non-interrupted resume rejection

Append this test after test_run_api_completed_run_persists:

def test_run_api_rejects_resume_for_completed_run() -> None:
    root = local_temp_root() / "run_api_resume_completed_rejected"
    service, _ = _service_with_echo(root)
    context = context_from_service(service)
    api = WorkflowRunApi(context)

    result = asyncio.run(
        api.run_deployment(
            deployment_id="echo.personal",
            workflow_input={"text": "hello"},
        )
    )

    with pytest.raises(ValueError, match="is not interrupted"):
        asyncio.run(
            api.resume_run(
                run_id=result["run_id"],
                resume_payload={"answer": "ignored"},
            )
        )

This locks down: resume_run only resumes stored runs whose latest status is interrupted.

• Step 3: Add test that deleting deployment does not erase stored inspection

Append this test after the non-interrupted resume test:

def test_run_api_inspect_uses_pinned_environment_after_deployment_deleted() -> None:
    root = local_temp_root() / "run_api_inspect_after_deployment_deleted"
    service, artifact_store = _service_with_echo(root)
    context = context_from_service(service)
    api = WorkflowRunApi(context)

    result = asyncio.run(
        api.run_deployment(
            deployment_id="echo.personal",
            workflow_input={"text": "hello"},
        )
    )
    artifact_store.delete_deployment("echo.personal")

    summary = asyncio.run(api.inspect_run(run_id=result["run_id"]))

    assert summary["status"] == "completed"
    assert summary["run_id"] == result["run_id"]
    assert summary["deployment_id"] == "echo.personal"
    assert summary["artifact_id"] == "echo"
    assert summary["output"]["echoed"] == "hello"

This locks down: stored run inspection reads pinned environment from WorkflowRunRecord, not mutable deployment storage.

• Step 4: Strengthen trace-range-before-store validation

In test_run_api_rejects_invalid_trace_range_before_store_lookup, replace the setup with a service whose run store raises if accessed:

class ExplodingRunStore(FileRunStore):
    def get_run(self, run_id: str):
        raise AssertionError("run store must not be read before trace_range validation")

Then build the service explicitly:

root = local_temp_root() / "run_api_invalid_trace_range"
service, _ = _service_with_echo(root)
service.run_store = ExplodingRunStore(root / "exploding_runs")
context = context_from_service(service)
api = WorkflowRunApi(context)

Keep the two existing pytest.raises(ValueError, ...) assertions unchanged. The test must still pass, proving invalid trace ranges fail before store lookup.

• Step 5: Run focused run API tests

Run:

uv run pytest tests/wf_api/test_run_api.py -q

Expected:

all tests in tests/wf_api/test_run_api.py pass

• Step 6: Commit Task 1

git add tests/wf_api/test_run_api.py
git commit -m "test: lock down durable run resume contract"

---

Task 2: Add Required-Store Durable Context Helper

Files:

• Create: src/wf_api/durable_context.py

• Modify: src/wf_api/__init__.py

• Test: tests/wf_api/test_durable_context.py

• Step 1: Write failing durable-context tests

Create tests/wf_api/test_durable_context.py:

from __future__ import annotations

import pytest

from wf_api import WorkflowApi
from wf_api.durable_context import durable_workflow_api, require_workflow_stores
from wf_api.operation_context import WorkflowOperationContext
from wf_api.stores import file_workflow_stores
from wf_mcp.broker import WfMcpService
from wf_mcp.broker.service.workflow_operation_context import context_from_service
from wf_mcp.storage import FileStore


def test_require_workflow_stores_returns_existing_store_bundle(tmp_path) -> None:
    stores = file_workflow_stores(tmp_path / "workflow_stores")
    service = WfMcpService(
        store=FileStore(tmp_path / "mcp"),
        artifact_store=stores.artifact_store,
        draft_workspace_store=stores.draft_workspace_store,
        run_store=stores.run_store,
    )
    context = context_from_service(service)

    required = require_workflow_stores(context)

    assert required.artifact_store is stores.artifact_store
    assert required.draft_workspace_store is stores.draft_workspace_store
    assert required.run_store is stores.run_store


def test_require_workflow_stores_rejects_missing_store(tmp_path) -> None:
    service = WfMcpService(
        store=FileStore(tmp_path / "mcp"),
        artifact_store=None,
        draft_workspace_store=None,
        run_store=None,
    )
    context = context_from_service(service)

    with pytest.raises(ValueError, match="durable workflow API requires stores"):
        require_workflow_stores(context)


def test_durable_workflow_api_returns_workflow_api_with_same_context(tmp_path) -> None:
    stores = file_workflow_stores(tmp_path / "workflow_stores")
    service = WfMcpService(
        store=FileStore(tmp_path / "mcp"),
        artifact_store=stores.artifact_store,
        draft_workspace_store=stores.draft_workspace_store,
        run_store=stores.run_store,
    )
    context = context_from_service(service)

    api = durable_workflow_api(context)

    assert isinstance(api, WorkflowApi)
    assert api.context is context

• Step 2: Run tests and verify they fail

Run:

uv run pytest tests/wf_api/test_durable_context.py -q

Expected:

FAIL with ModuleNotFoundError: No module named 'wf_api.durable_context'

• Step 3: Implement durable context helper

Create src/wf_api/durable_context.py:

from __future__ import annotations

from .operation_context import WorkflowOperationContext
from .service import WorkflowApi
from .stores import WorkflowStores


def require_workflow_stores(context: WorkflowOperationContext) -> WorkflowStores:
    """Return required stores or fail before constructing durable frontends.

    `WorkflowOperationContext` keeps stores optional for compatibility tests and
    lightweight MCP surfaces. Durable API surfaces need all stores up front so a
    run cannot start without somewhere to persist artifacts, drafts, and stopped
    execution state.
    """
    missing = []
    if context.artifact_store is None:
        missing.append("artifact_store")
    if context.draft_workspace_store is None:
        missing.append("draft_workspace_store")
    if context.run_store is None:
        missing.append("run_store")
    if missing:
        raise ValueError(
            "durable workflow API requires stores: " + ", ".join(missing)
        )
    return WorkflowStores(
        artifact_store=context.artifact_store,
        draft_workspace_store=context.draft_workspace_store,
        run_store=context.run_store,
    )


def durable_workflow_api(context: WorkflowOperationContext) -> WorkflowApi:
    """Construct a WorkflowApi only after durable store dependencies exist."""
    require_workflow_stores(context)
    return WorkflowApi(context)


__all__ = ["durable_workflow_api", "require_workflow_stores"]

• Step 4: Export helpers from wf_api

Modify src/wf_api/__init__.py to import and export:

from .durable_context import durable_workflow_api, require_workflow_stores

Add these names to __all__:

"durable_workflow_api",
"require_workflow_stores",

• Step 5: Run durable-context tests

Run:

uv run pytest tests/wf_api/test_durable_context.py -q

Expected:

3 passed

• Step 6: Run wf_api import-direction test

Run:

uv run pytest tests/wf_api/test_import_direction.py -q

Expected:

1 passed

• Step 7: Commit Task 2

git add src/wf_api/durable_context.py src/wf_api/__init__.py tests/wf_api/test_durable_context.py
git commit -m "feat: add durable workflow API store guard"

---

Task 3: Document Implemented Hardening

Files:

• Modify: docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md

• Modify: docs/current_roadmap.md

• Step 1: Update spec current gaps

In docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md, under ## Current Gaps / Next Implementation Work, replace item 1 with:

1. **Required stores for durable API**
   - Implemented for process-local frontends through
     `wf_api.durable_context.require_workflow_stores()` and
     `wf_api.durable_context.durable_workflow_api()`.
   - `WorkflowOperationContext` still allows optional stores for MCP test and
     compatibility paths.

Under ## Implementation Order, replace item 1 with:

1. Contract regression tests now cover:
   - non-interrupted `resume_run` rejection
   - deleted deployment does not erase existing run inspection
   - trace range validates before store lookup
   - blocked resume writes no checkpoint (`tests/wf_mcp/test_saved_subgraphs.py`)

• Step 2: Update current roadmap

In docs/current_roadmap.md, find the durable/persisted run section and add:

- `wf_api.durable_context` now provides a required-store guard for future
  durable HTTP/API frontends. It preserves the current process-local behavior
  while failing fast if artifact, draft, or run stores are missing.

If no durable/persisted run section exists, add it under the current wf_api or runtime roadmap notes.

• Step 3: Run docs grep sanity check

Run:

rg -n "durable_workflow_api|require_workflow_stores|non-interrupted|blocked resume writes no checkpoint" docs

Expected:

matches in persisted-run spec and/or current roadmap

• Step 4: Commit Task 3

git add docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md docs/current_roadmap.md
git commit -m "docs: record durable run hardening status"

---

Task 4: Final Verification

Files:

• Verify all touched code/tests/docs.

• Step 1: Run focused tests

Run:

uv run pytest tests/wf_api/test_run_api.py tests/wf_api/test_durable_context.py tests/wf_api/test_import_direction.py tests/wf_mcp/test_saved_subgraphs.py -q

Expected:

all selected tests pass

• Step 2: Run broader wf_api suite

Run:

uv run pytest tests/wf_api -q

Expected:

all wf_api tests pass

• Step 3: Run ruff

Run:

uv run ruff check src/wf_api tests/wf_api tests/wf_mcp/test_saved_subgraphs.py
uv run ruff format --check src/wf_api tests/wf_api tests/wf_mcp/test_saved_subgraphs.py

Expected:

All checks passed

• Step 4: Run basedpyright

Run:

uv run basedpyright --level error

Expected:

0 errors, 0 warnings, 0 notes

Known caveat: this repo may still exit nonzero with the workspace enumeration warning even when it reports 0 errors.

• Step 5: Final report

Report:

Implemented persisted run/resume hardening:
- added missing contract regression tests
- added wf_api durable required-store guard
- updated persisted-run spec and roadmap

Verification:
- focused tests: ...
- wf_api tests: ...
- ruff: ...
- basedpyright: ...

---

Self-Review

Spec coverage:

• resume_run only resumes interrupted runs: Task 1.
• Pinned environment survives deployment deletion: Task 1.
• Trace range validates before store lookup: Task 1.
• Blocked resume writes no checkpoint: already covered by tests/wf_mcp/test_saved_subgraphs.py; referenced in Task 3 and final focused verification.
• Required-store context/factory for durable API surfaces: Task 2.
• Transactional backend, CAS resume guard, run listing, progress, retry/timeout: explicitly out of scope.

Placeholder scan:

• No placeholder implementation steps.
• Code snippets include exact functions and expected assertions.

Type consistency:

• WorkflowStores, WorkflowOperationContext, and WorkflowApi names match current src/wf_api.
• TraceRangeLike remains unchanged; this plan does not alter MCP/CLI schemas.