sched: docs, probe retirement, MCP guard, coverage pins, executable example (T14)
This commit is contained in:
@@ -1,6 +1,9 @@
|
||||
# Deployment Scheduling
|
||||
|
||||
Status: draft for review; not implemented.
|
||||
Status: implemented (slices T01–T14, reviews R0–R4 passed); this document
|
||||
remains the current contract. The implementation plan that built it is
|
||||
archived at
|
||||
`docs/historical/superpowers/plans/2026-09-09-deployment-scheduling-implementation-plan.md`.
|
||||
|
||||
## Purpose and scope
|
||||
|
||||
@@ -247,6 +250,16 @@ anything else under an active attempt is stale and fails closed without
|
||||
retry. Recovery must distinguish both cases from an interruption that was
|
||||
merely waiting across server restart.
|
||||
|
||||
Implemented checkpoint authority: a checkpoint decides a summary only when
|
||||
its content is coherent — same run identity, checkpoint id of the form
|
||||
`{run_id}.{sequence:06d}`, runtime state decodable, and decoded stopped
|
||||
status equal to the outer reason. A durable failed decision is superseded
|
||||
only by a present, coherent, strictly newer checkpoint with matching
|
||||
attempt provenance; a missing or older referenced checkpoint keeps the
|
||||
decision (noted, history preserved) and never rolls a failed run back to
|
||||
interrupted. A genuinely newer coherent result still repairs a torn
|
||||
summary, and repeated recovery is silent and stable.
|
||||
|
||||
The process must own the store exclusively before recovery. Enforce scheduler
|
||||
ownership with a held cross-process lock, not a stale PID file or a lease that
|
||||
can expire while the old owner still runs. Unsupported locking must reject
|
||||
@@ -254,6 +267,17 @@ scheduler startup. Other processes mutating/resuming the same store remain
|
||||
unsupported. This does not upgrade the rest of the file stores to multi-writer
|
||||
safety or claim exactly-once external effects.
|
||||
|
||||
Implemented lock identity: one store composition has exactly one lock file,
|
||||
at the deepest composition root containing every store root as itself or a
|
||||
direct child (`canonical_lock_root`). Identical, sibling, and nested roots
|
||||
covering the same store files contend on that one file; ancestor locks prove
|
||||
nothing and are rejected, and cross pairs reusing one protected store with a
|
||||
different partner have no lock identity at all and are rejected before any
|
||||
write. The acquired identity is frozen at acquisition, so mutating the
|
||||
handle's root afterwards cannot redirect authority. The server layout points
|
||||
both stores at the composition root itself (run data at `<root>/runs`,
|
||||
schedules at `<root>/schedules`, one lock at `<root>/scheduler.lock`).
|
||||
|
||||
## Lifecycle, administration, and resource bounds
|
||||
|
||||
Expose create/get/list/update/pause/resume/delete and paginated occurrence
|
||||
@@ -290,6 +314,43 @@ grace period. Record cancellation/failure when possible; abrupt termination
|
||||
uses startup recovery. Paused/deleted schedule definitions must not prevent
|
||||
run completion or resume from updating retained occurrence history.
|
||||
|
||||
Implemented service and administration surface. The opt-in same-server
|
||||
scheduler (`SchedulerService`, enabled by `server.scheduler.enabled` or
|
||||
`wf-rpc-server --enable-scheduler`; local/static servers only) acquires
|
||||
canonical ownership, recovers without executing, then ticks calendar
|
||||
polling without blocking on long workflows: each dispatch spawns exactly
|
||||
one bounded execution task behind the async-completion seam, and the
|
||||
scheduler's own capacity gate is the execution-slot bound (an executing
|
||||
run keeps its slot until it stops). Shutdown stops admission, drains
|
||||
within `drain_grace_s`, leaves unfinished work under its executing mark
|
||||
for startup recovery to abandon truthfully, and releases ownership last.
|
||||
A failed startup releases the lock and raises.
|
||||
|
||||
Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
|
||||
RPC, Python client): `create/get/list/update/pause/resume/delete_schedule`
|
||||
plus paginated `list_schedule_occurrences`. Creation validates the
|
||||
trigger, the deployment, the binding shapes, and a sample-occurrence
|
||||
resolution against the pinned root schema, and starts the consumed
|
||||
watermark at creation (no pre-creation backfill). Updates are
|
||||
revision-checked; pause/resume/delete mirror the poll-loop transitions;
|
||||
all mutating admin ops clear the old revision's unadmitted work and
|
||||
advance the watermark BEFORE the revision bump or flag flip lands, so a
|
||||
crash can only leave the op unapplied (retryable), never a new revision
|
||||
that backfills. Occurrence pages carry the stored history plus a live
|
||||
held-candidate `pending` synthesis on the first page. Manual runs and
|
||||
resumes bypass scheduler capacity by design (unchanged API behavior);
|
||||
scheduler capacity governs scheduled dispatch only, and a scheduled
|
||||
interrupted run resumed manually reconciles its terminal history through
|
||||
recovery.
|
||||
|
||||
Known limitations: pointing one composition's stores inside another live
|
||||
composition's store subtree (without sharing its identical roots) is
|
||||
unsupported operator error; `max_steps: None` means "unpatched" on
|
||||
update (a set budget cannot be cleared back to unset); a first
|
||||
occurrence page may carry one row more than `limit` while a candidate is
|
||||
held; calendar iteration within a tick may use the tick-start source
|
||||
(trigger edits take effect on the next tick).
|
||||
|
||||
## Verification gates
|
||||
|
||||
Use injected clocks and controlled executors, not real-time sleeps:
|
||||
|
||||
Reference in New Issue
Block a user