lda-wf/docs/historical/superpowers/plans/2026-06-09-resume-run-concurrency-guard.md

Resume Run Concurrency Guard Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Prevent two concurrent resume_run calls in the same server process from both advancing the same interrupted run from the same checkpoint.

Architecture: Add a small process-local keyed async lock helper in wf_api, then wrap WorkflowRunApi.resume_run() with a critical section keyed by run_id. This intentionally protects one long-lived API process; it does not pretend the JSON filesystem store is safe for multi-process/cloud concurrency, which remains a transactional-store follow-up.

Tech Stack: Python 3.14, asyncio.Lock, pytest, pytest-asyncio, wf_api, wf_artifacts.FileRunStore.

---

File Structure

• Create src/wf_api/run_locks.py
  • Owns one focused helper, AsyncKeyedLock, for process-local async critical sections keyed by string ids.
  • No dependency on run models or stores.
• Modify src/wf_api/runs.py
  • Instantiate a resume lock registry on WorkflowRunApi.
  • Wrap the existing resume implementation in async with self._resume_locks.lock(run_id).
  • Add a short comment explaining why the lock lives above RunStore.
• Create tests/wf_api/test_run_locks.py
  • Tests lock serialization for the same key and concurrency for different keys.
• Create tests/wf_api/test_resume_concurrency.py
  • Seeds an interrupted run directly in FileRunStore.
  • Uses a fake runtime that blocks inside resume_workflow_from_plan.
  • Starts two concurrent resumes for the same run_id.
  • Asserts only the first reaches runtime and the second rejects after the first persists completion.
• Modify docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md
  • Mark the process-local guard as the next concrete slice.
  • Keep the transactional-store limitation explicit.
• Modify docs/current_roadmap.md
  • Link this active implementation plan under Priority 2.

---

Task 1: Add Keyed Async Lock Tests

Files:

• Create: tests/wf_api/test_run_locks.py

• Step 1: Write failing tests for the lock helper

Create tests/wf_api/test_run_locks.py:

from __future__ import annotations

import asyncio

from wf_api.run_locks import AsyncKeyedLock


async def test_async_keyed_lock_serializes_same_key() -> None:
    locks = AsyncKeyedLock()
    entered: list[str] = []
    release_first = asyncio.Event()

    async def first() -> None:
        async with locks.lock("run_123"):
            entered.append("first")
            await release_first.wait()

    async def second() -> None:
        async with locks.lock("run_123"):
            entered.append("second")

    first_task = asyncio.create_task(first())
    await asyncio.sleep(0)
    second_task = asyncio.create_task(second())
    await asyncio.sleep(0)

    assert entered == ["first"]

    release_first.set()
    await asyncio.gather(first_task, second_task)

    assert entered == ["first", "second"]


async def test_async_keyed_lock_allows_different_keys_concurrently() -> None:
    locks = AsyncKeyedLock()
    entered: list[str] = []
    release = asyncio.Event()

    async def hold(key: str) -> None:
        async with locks.lock(key):
            entered.append(key)
            await release.wait()

    first_task = asyncio.create_task(hold("run_a"))
    second_task = asyncio.create_task(hold("run_b"))
    await asyncio.sleep(0)

    assert entered == ["run_a", "run_b"]

    release.set()
    await asyncio.gather(first_task, second_task)

• Step 2: Run tests to verify they fail

Run:

uv run pytest tests/wf_api/test_run_locks.py -q

Expected: FAIL with ModuleNotFoundError: No module named 'wf_api.run_locks'.

---

Task 2: Implement AsyncKeyedLock

Files:

• Create: src/wf_api/run_locks.py

• Step 1: Add the keyed lock helper

Create src/wf_api/run_locks.py:

from __future__ import annotations

import asyncio
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from dataclasses import dataclass


@dataclass(slots=True)
class _LockEntry:
    lock: asyncio.Lock
    users: int = 0


class AsyncKeyedLock:
    """Process-local async critical sections keyed by a stable string id.

    This helper intentionally does not provide cross-process locking. Use it for
    one API/server process; use a transactional store for multi-worker safety.
    """

    def __init__(self) -> None:
        self._guard = asyncio.Lock()
        self._entries: dict[str, _LockEntry] = {}

    @asynccontextmanager
    async def lock(self, key: str) -> AsyncIterator[None]:
        """Hold the critical section for `key` until the context exits."""
        entry = await self._retain(key)
        await entry.lock.acquire()
        try:
            yield
        finally:
            entry.lock.release()
            await self._release(key, entry)

    async def _retain(self, key: str) -> _LockEntry:
        async with self._guard:
            entry = self._entries.get(key)
            if entry is None:
                entry = _LockEntry(lock=asyncio.Lock())
                self._entries[key] = entry
            entry.users += 1
            return entry

    async def _release(self, key: str, entry: _LockEntry) -> None:
        async with self._guard:
            entry.users -= 1
            if entry.users == 0 and not entry.lock.locked():
                self._entries.pop(key, None)

• Step 2: Run keyed lock tests

Run:

uv run pytest tests/wf_api/test_run_locks.py -q

Expected: PASS, 2 passed.

• Step 3: Run lint/typecheck on the new helper

Run:

uv run ruff check src/wf_api/run_locks.py tests/wf_api/test_run_locks.py
uv run basedpyright --level error src/wf_api/run_locks.py tests/wf_api/test_run_locks.py

Expected:

All checks passed!
0 errors, 0 warnings, 0 notes

• Step 4: Commit the helper

Run:

git add src/wf_api/run_locks.py tests/wf_api/test_run_locks.py
git commit -m "feat: add async keyed lock helper"

---

Task 3: Add Same-Run Resume Race Regression Test

Files:

• Create: tests/wf_api/test_resume_concurrency.py

• Step 1: Write the failing resume race test

Create tests/wf_api/test_resume_concurrency.py:

from __future__ import annotations

import asyncio
from pathlib import Path
from typing import Any

import pytest

from wf_api.models import RawWorkflowPlan
from wf_api.operation_context import WorkflowOperationContext
from wf_api.run_lifecycle import create_pinned_environment, persist_stopped_run
from wf_api.runs import WorkflowRunApi
from wf_api.saved_subgraphs import SavedSubgraphTree
from wf_artifacts import (
    FileRunStore,
    WorkflowArtifact,
    WorkflowDeployment,
)
from wf_authoring import NodeSpec
from wf_core import InterruptRequest, RunState, RunStatus
from wf_platform import CapabilityBuckets, CapabilitySource


class DummyEvents:
    def record_event(self, event: object) -> None:
        pass

    def record_workflow_event(
        self,
        event_type: str,
        *,
        capability_id: str,
        payload: dict[str, Any],
    ) -> None:
        pass


class EmptySpecProvider:
    @property
    def capability_sources(self) -> dict[str, CapabilitySource]:
        return {}

    def get_qualified_spec(self, qualified_name: str) -> NodeSpec[Any, Any]:
        raise KeyError(f"unknown capability {qualified_name!r}")


class BlockingResumeRuntime:
    def __init__(self) -> None:
        self.entered = 0
        self.first_entered = asyncio.Event()
        self.release_first = asyncio.Event()

    async def run_workflow_from_plan(
        self,
        plan: RawWorkflowPlan,
        workflow_input: dict[str, Any],
        deployment: WorkflowDeployment | None = None,
        artifact: WorkflowArtifact | None = None,
        saved_subgraph_tree: SavedSubgraphTree | None = None,
    ) -> RunState:
        raise AssertionError("test should not start new workflow runs")

    async def resume_workflow_from_plan(
        self,
        plan: RawWorkflowPlan,
        run: RunState,
        *,
        resume_payload: dict[str, Any],
        resume_outcome: str,
        deployment: WorkflowDeployment | None = None,
        artifact: WorkflowArtifact | None = None,
        saved_subgraph_tree: SavedSubgraphTree | None = None,
    ) -> RunState:
        self.entered += 1
        self.first_entered.set()
        await self.release_first.wait()
        return RunState(
            workflow_name=plan.name,
            status=RunStatus.COMPLETED,
            workflow_input=run.workflow_input,
            state={"answer": resume_payload["answer"]},
            outcome=resume_outcome,
            output={"answer": resume_payload["answer"]},
        )


def test_resume_run_serializes_same_run_attempts(tmp_path: Path) -> None:
    async def scenario() -> None:
        store = FileRunStore(tmp_path / "runs")
        runtime = BlockingResumeRuntime()
        run_id = _seed_interrupted_run(store)
        api = WorkflowRunApi(
            WorkflowOperationContext(
                artifact_store=None,
                draft_workspace_store=None,
                run_store=store,
                events=DummyEvents(),
                specs=EmptySpecProvider(),
                runtime=runtime,
                live_sources=None,
            )
        )

        first = asyncio.create_task(
            api.resume_run(run_id=run_id, resume_payload={"answer": "first"})
        )
        await runtime.first_entered.wait()

        second = asyncio.create_task(
            api.resume_run(run_id=run_id, resume_payload={"answer": "second"})
        )
        await asyncio.sleep(0)

        assert runtime.entered == 1

        runtime.release_first.set()
        first_payload = await first

        assert first_payload["status"] == "completed"
        assert first_payload["output"] == {"answer": "first"}

        with pytest.raises(ValueError, match="is not interrupted"):
            await second

        assert runtime.entered == 1
        assert store.get_latest_checkpoint(run_id).sequence == 2
        assert len(store.list_checkpoints(run_id)) == 2

    asyncio.run(scenario())


def _seed_interrupted_run(store: FileRunStore) -> str:
    artifact = _artifact()
    deployment = WorkflowDeployment(
        id="pause.default",
        artifact_id=artifact.id,
        artifact_version=artifact.version,
        bindings=[],
    )
    interrupted = RunState(
        workflow_name="pause",
        status=RunStatus.INTERRUPTED,
        workflow_input={"question": "continue?"},
        state={},
        interrupt=InterruptRequest(
            id="interrupt:approval",
            frame_id="root",
            node_id="approval",
            kind="approval",
            payload={"question": "continue?"},
        ),
    )
    record = persist_stopped_run(
        store=store,
        environment=create_pinned_environment(
            deployment=deployment,
            artifact=artifact,
            tree=SavedSubgraphTree(artifacts_by_ref={}),
        ),
        run=interrupted,
    )
    return record.id


def _artifact() -> WorkflowArtifact:
    return WorkflowArtifact(
        id="pause",
        version=1,
        title="Pause",
        input_schema={"type": "object", "properties": {}},
        output_schema={"type": "object", "properties": {}},
        outcomes=("ok", "submitted"),
        plan={
            "name": "pause",
            "input_schema": {"type": "object", "properties": {}},
            "state_schema": {"type": "object", "properties": {}},
            "output_schema": {"type": "object", "properties": {}},
            "outcomes": ["ok", "submitted"],
            "start": "end_submitted",
            "nodes": [
                {"id": "end_submitted", "type": "end", "outcome": "submitted"}
            ],
            "edges": [],
        },
    )

• Step 2: Run the test to verify the race exists

Run:

uv run pytest tests/wf_api/test_resume_concurrency.py -q

Expected before wiring the lock: FAIL. The likely failure is:

assert runtime.entered == 1
E assert 2 == 1

If the failure appears later as duplicate checkpoint sequence or final output from "second", that is also acceptable. The test must fail because the second concurrent resume reaches runtime or writes from the same starting checkpoint.

---

Task 4: Wrap WorkflowRunApi.resume_run With the Keyed Lock

Files:

• Modify: src/wf_api/runs.py

• Step 1: Import and instantiate the lock helper

In src/wf_api/runs.py, add the import:

from .run_locks import AsyncKeyedLock

Change the constructor from:

    def __init__(self, context: WorkflowOperationContext) -> None:
        self.context = context
        self.deployments = WorkflowDeploymentApi(context)

to:

    def __init__(
        self,
        context: WorkflowOperationContext,
        *,
        resume_locks: AsyncKeyedLock | None = None,
    ) -> None:
        self.context = context
        self.deployments = WorkflowDeploymentApi(context)
        self._resume_locks = resume_locks or AsyncKeyedLock()

• Step 2: Split the unlocked resume body into a private helper

Replace the current resume_run method with this wrapper plus private method:

    async def resume_run(
        self,
        *,
        run_id: str,
        resume_payload: dict[str, Any],
        resume_outcome: str = "submitted",
        trace_range: TraceRangeLike | None = None,
    ) -> dict[str, Any]:
        """Resume one durable interrupted deployment run."""
        # FileRunStore locks individual file writes only. The API layer owns the
        # process-local read/execute/write critical section for one run id.
        async with self._resume_locks.lock(run_id):
            return await self._resume_run_unlocked(
                run_id=run_id,
                resume_payload=resume_payload,
                resume_outcome=resume_outcome,
                trace_range=trace_range,
            )

    async def _resume_run_unlocked(
        self,
        *,
        run_id: str,
        resume_payload: dict[str, Any],
        resume_outcome: str,
        trace_range: TraceRangeLike | None,
    ) -> dict[str, Any]:
        trace_values = _trace_range_values(trace_range)
        record, stopped_run = restore_interrupted_run(self._run_store(), run_id)
        environment = record.environment
        diagnostics = validate_pinned_resume_environment(
            record=record,
            sources=_available_sources(self.context.specs.capability_sources),
        )
        if has_blocking_diagnostics(diagnostics):
            blocked = mark_resume_blocked(
                store=self._run_store(),
                record=record,
                diagnostics=diagnostics,
            )
            return _run_payload(
                deployment=environment.deployment,
                artifact=environment.root_artifact,
                status=stopped_run.status.value,
                run_id=blocked.id,
                resume_readiness=blocked.resume_readiness.value,
                interrupt=_interrupt_payload(stopped_run),
                outcome=stopped_run.outcome,
                error=stopped_run.error,
                output=stopped_run.output,
                diagnostics=diagnostics,
                trace_count=len(stopped_run.trace),
            )
        plan = raw_plan_from_artifact(environment.root_artifact)
        tree = saved_subgraph_tree_from_snapshots(environment.child_artifacts)
        run = await self.context.runtime.resume_workflow_from_plan(
            plan,
            stopped_run,
            resume_payload=resume_payload,
            resume_outcome=resume_outcome,
            deployment=environment.deployment,
            artifact=environment.root_artifact,
            saved_subgraph_tree=tree,
        )
        next_record = persist_stopped_run(
            store=self._run_store(),
            environment=environment,
            run=run,
            run_id=run_id,
        )
        return _run_payload(
            deployment=environment.deployment,
            artifact=environment.root_artifact,
            status=run.status.value,
            run_id=next_record.id,
            resume_readiness=next_record.resume_readiness.value,
            interrupt=_interrupt_payload(run),
            outcome=run.outcome,
            error=run.error,
            output=run.output,
            trace_count=len(run.trace),
            **_trace_slice_fields(run, trace_values),
        )

The body is the existing implementation moved under _resume_run_unlocked; do not change blocked-resume or persistence semantics.

• Step 3: Run the resume race test

Run:

uv run pytest tests/wf_api/test_resume_concurrency.py -q

Expected: PASS, 1 passed.

• Step 4: Run existing run API tests

Run:

uv run pytest tests/wf_api/test_run_api.py tests/wf_api/test_resume_concurrency.py tests/wf_api/test_run_locks.py -q

Expected: all tests pass.

• Step 5: Commit the API wiring

Run:

git add src/wf_api/runs.py tests/wf_api/test_resume_concurrency.py
git commit -m "fix: serialize same-run resume attempts"

---

Task 5: Update Durable Resume Docs

Files:

• Modify: docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md

• Modify: docs/current_roadmap.md

• Step 1: Update the resume contract current implementation section

In docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md, under ### resume_run, change the current implementation bullets to:

Current implementation:

- `WorkflowRunApi.resume_run()` follows this contract for stored interrupted
  runs and blocked dependency validation.
- Same-process callers are serialized per `run_id` around the restore,
  dependency validation, runtime resume, and checkpoint write sequence.
- `restore_interrupted_run()` rejects non-interrupted statuses.
- `mark_resume_blocked()` updates run summary without writing a checkpoint.

• Step 2: Update the store contract limitation

In the same spec, replace the Current limits: list under ## Store Contract with:

Current limits:

- `WorkflowRunApi.resume_run()` provides a process-local per-run critical
  section for one API/server process.
- `FileRunStore` locks individual file writes only; it does not provide
  compare-and-swap or cross-process transactions.
- The file store is appropriate for local/dev/single-process use.
- A multi-worker or cloud deployment still needs SQLite/Postgres or another
  transactional store before claiming strong concurrent resume safety.

• Step 3: Update the gap list

In the same spec, replace gap item 4. **Resume concurrency guard** with:

4. **Resume concurrency guard**
   - Implemented for same-process API callers through a per-`run_id` async
     critical section in `WorkflowRunApi.resume_run()`.
   - Cross-process protection remains part of the transactional backend gap.

• Step 4: Update the roadmap

In docs/current_roadmap.md, under ## Priority 2: Durable Run/Resume Hardening, replace:

- Add same-run concurrency protection around `resume_run`.
- Clarify store-level locking/transaction expectations for filesystem stores.

with:

- Completed: same-process `resume_run` calls are serialized per run id.
  Implementation:
  [`resume run concurrency guard`](historical/superpowers/plans/2026-06-09-resume-run-concurrency-guard.md).
- Clarify store-level locking/transaction expectations for future filesystem
  and transactional stores.

• Step 5: Move this plan to historical after implementation

After all code and docs pass verification, move this plan:

git mv docs/superpowers/plans/2026-06-09-resume-run-concurrency-guard.md docs/historical/superpowers/plans/2026-06-09-resume-run-concurrency-guard.md

Then commit docs:

git add docs/current_roadmap.md docs/superpowers/specs/2026-06-03-persisted-run-resume-contract.md docs/historical/superpowers/plans/2026-06-09-resume-run-concurrency-guard.md
git commit -m "docs: record resume concurrency guard"

---

Task 6: Final Verification

Files:

• No new files.

• Step 1: Run focused tests

Run:

uv run pytest tests/wf_api/test_run_locks.py tests/wf_api/test_resume_concurrency.py tests/wf_api/test_run_api.py tests/wf_mcp/test_saved_subgraphs.py tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py -q

Expected: all selected tests pass.

• Step 2: Run broader API tests

Run:

uv run pytest tests/wf_api -q

Expected: all wf_api tests pass.

• Step 3: Run lint and typecheck

Run:

uv run ruff check src/wf_api tests/wf_api
uv run ruff format --check src/wf_api tests/wf_api
uv run basedpyright --level error src/wf_api tests/wf_api/test_run_locks.py tests/wf_api/test_resume_concurrency.py tests/wf_api/test_run_api.py

Expected:

All checks passed!
0 errors, 0 warnings, 0 notes

• Step 4: Inspect git status

Run:

git status --short

Expected: clean working tree after the commits above.

---

Self-Review

• Spec coverage: The plan implements the roadmap gap "same-run concurrency protection around resume_run" and explicitly preserves the store-level transactional limitation.
• Placeholder scan: No TBD, vague "handle errors", or unspecified test work remains.
• Type consistency: AsyncKeyedLock, WorkflowRunApi.resume_run, RawWorkflowPlan, WorkflowOperationContext, FileRunStore, and SavedSubgraphTree names match current code.
• Scope control: This plan does not add run listing, checkpoint listing, retries, cloud transactions, or public API shape changes.