lda-wf/docs/historical/superpowers/plans/2026-09-09-caller-cancellation-capacity.md

Scheduled Caller Cancellation Capacity Implementation Plan

For agentic workers: Execute this plan inline in the supplied isolated worktree. Preserve the existing shutdown fence and use a run-scoped reconciliation path; never perform whole-store recovery for caller cancellation.

Goal: Reconcile a cancelled scheduled resume while the scheduler remains running so its durable ambiguity is terminal and its execution slot is reclaimed without permitting replay.

Architecture: Keep shutdown cancellation on the existing fence() path, because shutdown must retain the durable executing marker and ACTIVE attempt until restart recovery decides the ambiguous outcome. Add a distinct normal- operation cancellation path that invokes the existing recover(..., only_run_id=...) reconciliation under the scheduler's ownership/lock, then releases only that run's in-memory and durable capacity state. Healthy sibling runs and the rest of the store remain untouched.

Tech Stack: Python 3.14, asyncio, pytest/pytest-asyncio, file-backed run and schedule stores, Ruff, basedpyright.

Spec: docs/superpowers/specs/2026-09-08-deployment-scheduling-design.md

Global Constraints

• Scheduled execution is bounded by the configured shared capacity.
• A cancelled or abandoned execution must disclose unknown outcome and must never replay the occurrence.
• Restart recovery is run-scoped when repairing a live execution; do not use whole-store recovery for caller cancellation.
• Shutdown cancellation must preserve the existing durable fence and cancel-and-join-before-ownership-release ordering.
• Existing B1–B4, drain, restart, sibling, and capacity behavior must remain green.

---

Task 1: Add the caller-cancellation regression test

Files:

• Modify: tests/wf_server/test_scheduler_integration.py

Interfaces:

• Reuse _interrupt_then_gate_plan, _gate_open, _scheduler, _wait_for, _entries, and _kinds from the integration test module.

• The test must observe the public server.api.resume_run behavior plus the durable FileRunStore state, not only private bookkeeping.

• Step 1: Write the failing test

  Add an async integration test with capacity=1 and auto_tick=False that:

  1. schedules _interrupt_then_gate_plan, polls until it is interrupted;
  2. starts server.api.resume_run while _gate_open is closed;
  3. waits until the run is in service._live_resumes, then cancels and awaits the task with pytest.raises(asyncio.CancelledError);
  4. asserts the service is still running, the live-resume set is empty, the run is durably failed, its executing marker is cleared, and its resume attempt remains ACTIVE as the no-replay ambiguity record;
  5. schedules an unrelated ordinary constant workflow and polls until it completes, proving the cancelled run no longer occupies capacity;
  6. asserts the cancelled run has exactly one failed occurrence entry and that the sibling has only its normal admitted/completed history;
  7. attempts to resume the cancelled run and asserts the API rejects the ambiguous attempt without dispatching or adding completion history.

• Step 2: Run the focused test to verify the red failure

  Run:

  uv run pytest -q tests/wf_server/test_scheduler_integration.py -k caller_cancellation
  

  Expected: the new test fails because the cancelled run remains interrupted with executing=True, the attempt remains active, and the ordinary sibling cannot dispatch.

Task 2: Separate normal-operation cancellation reconciliation from shutdown fencing

Files:

• Modify: src/wf_api/runs.py
• Modify: src/wf_scheduling/resume_gate.py
• Modify: tests/wf_server/test_scheduler_integration.py if a focused assertion needs a small adjustment after the first red run

Interfaces:

• Preserve SchedulerResumeGate.fence(run_id) for shutdown cancellation.

• Add a narrowly named gate operation for a cancelled live scheduled resume; it must reconcile only run_id through wf_scheduling.recovery.recover with only_run_id=run_id, while holding the service lock and requiring the live service's ownership.

• Keep WorkflowRunApi._resume_run_unlocked re-raising CancelledError.

• Step 1: Implement the minimal normal-operation branch

  In the CancelledError handler, choose the normal-operation reconciliation only while the scheduler is still started and not stopping. If shutdown has begun, retain the existing fence() call unchanged. The normal branch must run the existing scoped recovery for this run_id, preserve the ACTIVE attempt and ambiguous failure reason, clear the durable executing marker, remove the run from _live_resumes/_resume_tasks, and re-raise the cancellation. If scoped reconciliation fails, retain the durable marker and report the failure through the scheduler's existing error path rather than claiming capacity was reclaimed.

• Step 2: Run the regression and shutdown tests

  Run:

  uv run pytest -q tests/wf_server/test_scheduler_integration.py -k "caller_cancellation or shutdown or sibling"
  

  Expected: the new caller-cancellation test passes, and the existing shutdown timeout/restart, drain rejection, healthy sibling, and live resume tests remain green.

• Step 3: Run the focused scheduling/API suites

  Run:

  uv run pytest -q tests/scheduling tests/wf_api/test_resume_attempt.py tests/wf_api/test_resume_concurrency.py tests/wf_server/test_scheduler_integration.py
  uv run ruff check src/wf_api/runs.py src/wf_scheduling/resume_gate.py tests/wf_server/test_scheduler_integration.py
  uv run basedpyright --level error src/wf_api/runs.py src/wf_scheduling/resume_gate.py
  

  Expected: all selected tests pass, Ruff reports no violations, and basedpyright reports no errors.

• Step 4: Review the full feature range independently

  Review git diff d5fa1809...HEAD plus the new fix commit against the deployment scheduling spec and historical plan. Inspect admission, dispatch, resume, cancellation, shutdown, ownership, persistence, integration, pagination, and test boundaries. Reproduce any suspected defect before changing it; fix only confirmed in-scope blockers and then re-review the fix and its interactions.

• Step 5: Run final verification and commit locally

  Run the appropriately scoped suite, uv run ruff check, uv run basedpyright --level error, and a diff/status check. Commit the focused implementation and regression coverage locally on opencode/sched-verify-plan; do not merge or push.