diff --git a/docs/historical/superpowers/plans/2026-09-09-caller-cancellation-capacity.md b/docs/historical/superpowers/plans/2026-09-09-caller-cancellation-capacity.md new file mode 100644 index 00000000..8d35b2ca --- /dev/null +++ b/docs/historical/superpowers/plans/2026-09-09-caller-cancellation-capacity.md @@ -0,0 +1,147 @@ +# Scheduled Caller Cancellation Capacity Implementation Plan + +> **For agentic workers:** Execute this plan inline in the supplied isolated +> worktree. Preserve the existing shutdown fence and use a run-scoped +> reconciliation path; never perform whole-store recovery for caller +> cancellation. + +**Goal:** Reconcile a cancelled scheduled resume while the scheduler remains +running so its durable ambiguity is terminal and its execution slot is +reclaimed without permitting replay. + +**Architecture:** Keep shutdown cancellation on the existing `fence()` path, +because shutdown must retain the durable executing marker and ACTIVE attempt +until restart recovery decides the ambiguous outcome. Add a distinct normal- +operation cancellation path that invokes the existing `recover(..., +only_run_id=...)` reconciliation under the scheduler's ownership/lock, then +releases only that run's in-memory and durable capacity state. Healthy sibling +runs and the rest of the store remain untouched. + +**Tech Stack:** Python 3.14, asyncio, pytest/pytest-asyncio, file-backed run +and schedule stores, Ruff, basedpyright. + +**Spec:** `docs/superpowers/specs/2026-09-08-deployment-scheduling-design.md` + +## Global Constraints + +- Scheduled execution is bounded by the configured shared capacity. +- A cancelled or abandoned execution must disclose unknown outcome and must + never replay the occurrence. +- Restart recovery is run-scoped when repairing a live execution; do not use + whole-store recovery for caller cancellation. +- Shutdown cancellation must preserve the existing durable fence and + cancel-and-join-before-ownership-release ordering. +- Existing B1–B4, drain, restart, sibling, and capacity behavior must remain + green. + +--- + +### Task 1: Add the caller-cancellation regression test + +**Files:** +- Modify: `tests/wf_server/test_scheduler_integration.py` + +**Interfaces:** +- Reuse `_interrupt_then_gate_plan`, `_gate_open`, `_scheduler`, `_wait_for`, + `_entries`, and `_kinds` from the integration test module. +- The test must observe the public `server.api.resume_run` behavior plus the + durable `FileRunStore` state, not only private bookkeeping. + +- [x] **Step 1: Write the failing test** + + Add an async integration test with `capacity=1` and `auto_tick=False` that: + + 1. schedules `_interrupt_then_gate_plan`, polls until it is interrupted; + 2. starts `server.api.resume_run` while `_gate_open` is closed; + 3. waits until the run is in `service._live_resumes`, then cancels and + awaits the task with `pytest.raises(asyncio.CancelledError)`; + 4. asserts the service is still running, the live-resume set is empty, the + run is durably `failed`, its `executing` marker is cleared, and its + resume attempt remains `ACTIVE` as the no-replay ambiguity record; + 5. schedules an unrelated ordinary constant workflow and polls until it + completes, proving the cancelled run no longer occupies capacity; + 6. asserts the cancelled run has exactly one failed occurrence entry and + that the sibling has only its normal admitted/completed history; + 7. attempts to resume the cancelled run and asserts the API rejects the + ambiguous attempt without dispatching or adding completion history. + +- [x] **Step 2: Run the focused test to verify the red failure** + + Run: + + ```powershell + uv run pytest -q tests/wf_server/test_scheduler_integration.py -k caller_cancellation + ``` + + Expected: the new test fails because the cancelled run remains + `interrupted` with `executing=True`, the attempt remains active, and the + ordinary sibling cannot dispatch. + +### Task 2: Separate normal-operation cancellation reconciliation from shutdown fencing + +**Files:** +- Modify: `src/wf_api/runs.py` +- Modify: `src/wf_scheduling/resume_gate.py` +- Modify: `tests/wf_server/test_scheduler_integration.py` if a focused + assertion needs a small adjustment after the first red run + +**Interfaces:** +- Preserve `SchedulerResumeGate.fence(run_id)` for shutdown cancellation. +- Add a narrowly named gate operation for a cancelled live scheduled resume; + it must reconcile only `run_id` through `wf_scheduling.recovery.recover` with + `only_run_id=run_id`, while holding the service lock and requiring the live + service's ownership. +- Keep `WorkflowRunApi._resume_run_unlocked` re-raising `CancelledError`. + +- [x] **Step 1: Implement the minimal normal-operation branch** + + In the `CancelledError` handler, choose the normal-operation reconciliation + only while the scheduler is still started and not stopping. If shutdown has + begun, retain the existing `fence()` call unchanged. The normal branch must + run the existing scoped recovery for this `run_id`, preserve the `ACTIVE` + attempt and ambiguous failure reason, clear the durable executing marker, + remove the run from `_live_resumes`/`_resume_tasks`, and re-raise the + cancellation. If scoped reconciliation fails, retain the durable marker and + report the failure through the scheduler's existing error path rather than + claiming capacity was reclaimed. + +- [x] **Step 2: Run the regression and shutdown tests** + + Run: + + ```powershell + uv run pytest -q tests/wf_server/test_scheduler_integration.py -k "caller_cancellation or shutdown or sibling" + ``` + + Expected: the new caller-cancellation test passes, and the existing shutdown + timeout/restart, drain rejection, healthy sibling, and live resume tests + remain green. + +- [x] **Step 3: Run the focused scheduling/API suites** + + Run: + + ```powershell + uv run pytest -q tests/scheduling tests/wf_api/test_resume_attempt.py tests/wf_api/test_resume_concurrency.py tests/wf_server/test_scheduler_integration.py + uv run ruff check src/wf_api/runs.py src/wf_scheduling/resume_gate.py tests/wf_server/test_scheduler_integration.py + uv run basedpyright --level error src/wf_api/runs.py src/wf_scheduling/resume_gate.py + ``` + + Expected: all selected tests pass, Ruff reports no violations, and + basedpyright reports no errors. + +- [x] **Step 4: Review the full feature range independently** + + Review `git diff d5fa1809...HEAD` plus the new fix commit against the + deployment scheduling spec and historical plan. Inspect admission, + dispatch, resume, cancellation, shutdown, ownership, persistence, + integration, pagination, and test boundaries. Reproduce any suspected + defect before changing it; fix only confirmed in-scope blockers and then + re-review the fix and its interactions. + +- [x] **Step 5: Run final verification and commit locally** + + Run the appropriately scoped suite, `uv run ruff check`, + `uv run basedpyright --level error`, and a diff/status check. Commit the + focused implementation and regression coverage locally on + `opencode/sched-verify-plan`; do not merge or push.