sched: docs, probe retirement, MCP guard, coverage pins, executable example (T14)

This commit is contained in:
lda
2026-09-09 12:21:13 +07:00 Verified
parent e09c08899a
commit 9c4f7de086
17 changed files with 601 additions and 2398 deletions
@@ -1,6 +1,9 @@
# Deployment Scheduling
Status: draft for review; not implemented.
Status: implemented (slices T01T14, reviews R0R4 passed); this document
remains the current contract. The implementation plan that built it is
archived at
`docs/historical/superpowers/plans/2026-09-09-deployment-scheduling-implementation-plan.md`.
## Purpose and scope
@@ -247,6 +250,16 @@ anything else under an active attempt is stale and fails closed without
retry. Recovery must distinguish both cases from an interruption that was
merely waiting across server restart.
Implemented checkpoint authority: a checkpoint decides a summary only when
its content is coherent — same run identity, checkpoint id of the form
`{run_id}.{sequence:06d}`, runtime state decodable, and decoded stopped
status equal to the outer reason. A durable failed decision is superseded
only by a present, coherent, strictly newer checkpoint with matching
attempt provenance; a missing or older referenced checkpoint keeps the
decision (noted, history preserved) and never rolls a failed run back to
interrupted. A genuinely newer coherent result still repairs a torn
summary, and repeated recovery is silent and stable.
The process must own the store exclusively before recovery. Enforce scheduler
ownership with a held cross-process lock, not a stale PID file or a lease that
can expire while the old owner still runs. Unsupported locking must reject
@@ -254,6 +267,17 @@ scheduler startup. Other processes mutating/resuming the same store remain
unsupported. This does not upgrade the rest of the file stores to multi-writer
safety or claim exactly-once external effects.
Implemented lock identity: one store composition has exactly one lock file,
at the deepest composition root containing every store root as itself or a
direct child (`canonical_lock_root`). Identical, sibling, and nested roots
covering the same store files contend on that one file; ancestor locks prove
nothing and are rejected, and cross pairs reusing one protected store with a
different partner have no lock identity at all and are rejected before any
write. The acquired identity is frozen at acquisition, so mutating the
handle's root afterwards cannot redirect authority. The server layout points
both stores at the composition root itself (run data at `<root>/runs`,
schedules at `<root>/schedules`, one lock at `<root>/scheduler.lock`).
## Lifecycle, administration, and resource bounds
Expose create/get/list/update/pause/resume/delete and paginated occurrence
@@ -290,6 +314,43 @@ grace period. Record cancellation/failure when possible; abrupt termination
uses startup recovery. Paused/deleted schedule definitions must not prevent
run completion or resume from updating retained occurrence history.
Implemented service and administration surface. The opt-in same-server
scheduler (`SchedulerService`, enabled by `server.scheduler.enabled` or
`wf-rpc-server --enable-scheduler`; local/static servers only) acquires
canonical ownership, recovers without executing, then ticks calendar
polling without blocking on long workflows: each dispatch spawns exactly
one bounded execution task behind the async-completion seam, and the
scheduler's own capacity gate is the execution-slot bound (an executing
run keeps its slot until it stops). Shutdown stops admission, drains
within `drain_grace_s`, leaves unfinished work under its executing mark
for startup recovery to abandon truthfully, and releases ownership last.
A failed startup releases the lock and raises.
Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
RPC, Python client): `create/get/list/update/pause/resume/delete_schedule`
plus paginated `list_schedule_occurrences`. Creation validates the
trigger, the deployment, the binding shapes, and a sample-occurrence
resolution against the pinned root schema, and starts the consumed
watermark at creation (no pre-creation backfill). Updates are
revision-checked; pause/resume/delete mirror the poll-loop transitions;
all mutating admin ops clear the old revision's unadmitted work and
advance the watermark BEFORE the revision bump or flag flip lands, so a
crash can only leave the op unapplied (retryable), never a new revision
that backfills. Occurrence pages carry the stored history plus a live
held-candidate `pending` synthesis on the first page. Manual runs and
resumes bypass scheduler capacity by design (unchanged API behavior);
scheduler capacity governs scheduled dispatch only, and a scheduled
interrupted run resumed manually reconciles its terminal history through
recovery.
Known limitations: pointing one composition's stores inside another live
composition's store subtree (without sharing its identical roots) is
unsupported operator error; `max_steps: None` means "unpatched" on
update (a set budget cannot be cleared back to unset); a first
occurrence page may carry one row more than `limit` while a candidate is
held; calendar iteration within a tick may use the tick-start source
(trigger edits take effect on the next tick).
## Verification gates
Use injected clocks and controlled executors, not real-time sleeps: