sched: docs, probe retirement, MCP guard, coverage pins, executable example (T14)

2026-09-09 12:21:13 +07:00
parent e09c08899a
commit 9c4f7de086
17 changed files with 601 additions and 2398 deletions
@@ -39,6 +39,8 @@ implementation plans are kept for context, not as active instructions.
  deployments, dependency compatibility, and interrupt limitations.
 - [`durable_run_operations.md`](durable_run_operations.md): `run_deployment`,
  `inspect_run`, bounded trace reads, and `resume_run` behavior.
+- [`deployment_scheduling.md`](deployment_scheduling.md): opt-in schedule
+  administration, occurrence inspection, and server lifecycle behavior.
 - [`workflow_drafts.md`](workflow_drafts.md): LLM/human draft authoring format
  above raw workflow plans.

@@ -149,6 +149,16 @@ The active sequence can assume these foundations:
 - Python client reconstruction of capabilities, artifacts, deployments, and
  runs through the API

+The active sequence can also assume deployment scheduling: an opt-in
+same-server scheduler starts ordinary deployment runs without a connected
+client (one-shot and recurring cron, durable admission, coalesced
+missed-start recovery, bounded parallel runs, occurrence inspection, and
+API/Python-client administration). Scheduling is disabled by default and
+enabled per server. The current contract is
+[`deployment scheduling`](superpowers/specs/2026-09-08-deployment-scheduling-design.md);
+operator usage is under
+[`deployment scheduling operations`](deployment_scheduling.md).
+
 The current foreach return contract is
 [`foreach back-edge design`](superpowers/specs/2026-09-04-foreach-back-edge-design.md).
 The current context contract is
@@ -0,0 +1,198 @@
+# Deployment Scheduling Operations
+
+Schedules start ordinary deployment runs without a connected client. The
+scheduler is opt-in, lives in the workflow server, and introduces no
+workflow node type. It is not the core runtime's frame scheduler.
+
+Current contract:
+[`deployment scheduling spec`](superpowers/specs/2026-09-08-deployment-scheduling-design.md).
+
+## Enablement
+
+Scheduling is disabled by default. Enable it for a local/static server
+with the config section or the CLI flag (MCP-backed servers reject it):
+
+```json
+{"server": {"scheduler": {"enabled": true}}}
+```
+
+```bash
+wf-rpc-server --store-root .wf_store --enable-scheduler
+```
+
+Tuning (`server.scheduler`): `poll_interval_s` (default 1.0),
+`max_concurrent_runs` (default 4, the execution-slot bound),
+`drain_grace_s` (default 30.0). Schedule data lives at
+`<store_root>/schedules` next to run data; one lock file at
+`<store_root>/scheduler.lock` proves exclusive ownership. A second
+scheduler over the same stores is rejected; shut the first down before
+starting another.
+
+## Mental model: schedule vs occurrence vs run
+
+- A **schedule** is a durable definition: deployment, trigger
+  (one-shot or cron), input bindings, overlap/misfire policies, and a
+  revision. Edits bump the revision and affect only future admissions.
+- An **occurrence** is one resolved calendar instant, identified by
+  `(schedule_id, resolved UTC instant)`. An occurrence is immutable once
+  admitted and is never replayed.
+- A **run** is the execution of one admitted occurrence, with a
+  store-backed `run-000001`-style id, a pinned input/artifact snapshot,
+  and a stopped checkpoint when it stops.
+
+## Triggers and time zones
+
+One-shot timestamps must include an offset. Cron uses five-field Unix
+expressions (`0` and `7` both mean Sunday; day-of-month/day-of-week
+match with OR) plus an explicit IANA time zone, default UTC.
+Occurrence instants persist as UTC; the definition keeps the zone name.
+`croniter` owns calendar resolution including DST gaps and folds; there
+is no custom calendar filtering. Reject invalid zones and naive times.
+
+## Overlap
+
+Overlap is per schedule, not per deployment. With default
+`overlap="skip"`, an unfinished scheduled run — including an
+interrupted run awaiting input — blocks another occurrence from that
+schedule. Manual runs and other schedules never participate.
+`overlap="parallel"` admits independent runs up to a required positive
+`max_active_runs`; admitted, running, and interrupted runs all count,
+and lowering the limit blocks new admission without cancelling existing
+runs. A known completed or failed run releases overlap.
+
+## Misfire
+
+Default `misfire="skip"` drops missed starts (past the 60-second
+configurable lateness allowance). `misfire="latest"` retains at most
+one latest unadmitted candidate per schedule and runs it as soon as
+overlap and capacity allow; it never replays a burst. Creation never
+backfills time before the revision: the consumed watermark starts at
+creation.
+
+## Pause
+
+Pausing stops future admission, not an active run. Resume selects the
+next future occurrence; paused times are not replayed. Pause is not
+downtime: the paused span is consumed, so downtime catch-up never
+resurrects it.
+
+## Deletion
+
+Deleting a schedule stops future admission. Existing runs and
+occurrence history survive, including their schedule identity; active
+runs continue and may still complete or resume against the retained
+history. Schedule ids are never reused.
+
+## Failure and restart
+
+An abandoned in-flight execution (the process died mid-run) becomes
+failed without retrying its occurrence; the failure discloses that
+external effects may already have occurred. A durably interrupted run
+is not abandoned: it stays resumable and keeps occupying its
+schedule's overlap slot across restarts. Every resume marks a
+store-backed attempt first, so recovery can tell a fresh result from a
+stale checkpoint and fail the ambiguous case closed instead of
+retrying it. Corrupt or contradictory records fail closed with
+diagnostics and block the schedule rather than clearing overlap.
+
+On shutdown the server stops admission first and drains active tasks
+within the grace period; anything still running keeps its executing
+mark, and the next startup recovery abandons it truthfully.
+
+## Occurrence inspection
+
+`list_schedule_occurrences` pages stored history (`pending`,
+`coalesced`/`superseded`, `skipped-overlap`, `skipped-misfire`,
+`preflight-rejected`, `admitted`/`running`, `interrupted`,
+`completed`, `failed`, `exhausted`) oldest-first with a `next_cursor`.
+A currently held (unadmitted) candidate is synthesized as a `pending`
+row at the top of the first page; once admitted, the durable
+`admitted` entry replaces it. While a candidate is held, the first
+page may carry one row more than `limit`.
+
+## Administration surface
+
+Python API (`server.api.schedules`), JSON-RPC (`workflow.schedules.*`),
+and the Python client (`App` schedule methods) share these operations:
+
+- `create_schedule` (`workflow.schedules.create`): caller-chosen id;
+  trigger, deployment, binding, and sample-schema checks.
+- `get_schedule` (`workflow.schedules.get`): full definition payload.
+- `list_schedules` (`workflow.schedules.list`): deleted excluded
+  unless asked.
+- `update_schedule` (`workflow.schedules.update`): `expected_revision`
+  required; provided fields only; `None` means unpatched.
+- `pause_schedule` (`workflow.schedules.pause`): idempotent; clears
+  candidates, consumes the span.
+- `resume_schedule` (`workflow.schedules.resume`): idempotent; resumes
+  from the next future instant.
+- `delete_schedule` (`workflow.schedules.delete`): soft delete; runs
+  and history survive.
+- `list_schedule_occurrences`
+  (`workflow.schedules.occurrences.list`): cursor pages, `limit` 1–100,
+  live pending synthesis.
+
+`inspect_run` also reads admitted (checkpoint-less) runs: status
+`admitted` with no fabricated trace, output, or checkpoint.
+
+## Hypothetically used as follows
+
+EXECUTABLE example (runs in CI as
+`tests/examples/test_scheduled_deployment_example.py`; run it with
+`uv run pytest -q tests/examples/test_scheduled_deployment_example.py`):
+
+```python
+server = build_local_static_workflow_server(root, schedules=True)
+await server.api.create_artifact_from_plan(
+    artifact_id="scheduled_hello", version=1, ...,
+)
+await server.api.save_deployment({...})
+due = datetime.now(UTC) + timedelta(seconds=0.5)
+await server.api.schedules.create_schedule(
+    schedule_id="hello-once",
+    deployment_id="scheduled_hello.default",
+    trigger={"kind": "oneshot", "at": due.isoformat()},
+)
+service = build_scheduler_service(server, SchedulerServiceConfig(...))
+await service.start()
+# ... the service admits the occurrence and completes the run ...
+inspected = await server.api.inspect_run(run_id=run_id)
+assert inspected["output"]["result"] == "hello on a schedule"
+page = await server.api.schedules.list_schedule_occurrences(
+    schedule_id="hello-once"
+)
+await service.stop()
+```
+
+ILLUSTRATIVE example (not executed; shows a cron week with an operator
+pause — same calls, longer horizons):
+
+```python
+# Monday: an hourly report, latest-wins catch-up, at most two at once.
+await schedules.create_schedule(
+    schedule_id="hourly-report",
+    deployment_id="report.default",
+    trigger={"kind": "cron", "expression": "0 * * * *", "timezone": "UTC"},
+    misfire="latest",
+    overlap="parallel",
+    max_active_runs=2,
+)
+# Friday: pause for maintenance; Monday: resume from the next hour.
+await schedules.pause_schedule(schedule_id="hourly-report")
+await schedules.resume_schedule(schedule_id="hourly-report")
+# A bad edit is rejected without touching the running definition:
+await schedules.update_schedule(
+    schedule_id="hourly-report", expected_revision=1, ...
+)
+```
+
+## Known limitations
+
+- One composition's stores nested inside another live composition's
+  store subtree (without sharing its identical roots) is unsupported
+  operator error; shared-store cross layouts are rejected outright.
+- Manual runs and resumes bypass scheduler capacity by design;
+  capacity governs scheduled dispatch only.
+- A set `max_steps` budget cannot be cleared back to unset through
+  update (recreate the schedule for an unbounded budget).
+- MCP-backed servers reject scheduler enablement for now.
@@ -16,6 +16,48 @@ Scheduling policies remain settled. Calendar behavior follows the subsequent
 approved simplification: croniter owns DST resolution. Findings below record
 the missing implementation seams and the completed verification work.

+## Completion status (2026-09-09; plan retired to historical/)
+
+All gates and tasks completed on branch `opencode/sched-verify-plan`:
+
+- R0, R1, R2, R3: passed during phased implementation.
+- R4 (fault-injection review): passed in three waves — wave 2 bound
+  ownership to store composition and stabilized recovery failure; wave 3
+  added one canonical lock identity plus checkpoint coherence and
+  ordering authority; wave 4 closed the shared-store overlap hole
+  (sibling-distinct identity, overlapping layouts rejected).
+- T01–T11: implemented per phase (calendar, expressions, admission,
+  scheduler core, resume safety, recovery, ownership).
+- T12: opt-in server lifecycle with bounded real execution, drain, and
+  subprocess-tested death paths; independent review passed.
+- T13: administration API + JSON-RPC surface + Python client; an
+  independent review gated it on torn-admin ordering and admin/poll
+  staleness, both fixed (crash-safe admin ordering, creation watermark,
+  per-schedule poll freshness) and re-reviewed to pass.
+- T14: this completion record; spec updated in place; roadmap and
+  user docs updated; disposable probes deleted after production-test
+  equivalence was verified item by item (calendar Part A mirrored in
+  `test_calendar_adapter.py`; APScheduler rejected-candidate evidence
+  preserved in the design spec; expression pins mirrored or superseded
+  by T03/T04 plus core tests; all 31 state-model behaviors mapped to
+  production tests, adding overlap-independence and store-backed
+  run-identity pins where no equivalent existed).
+
+Probe retirement map: `probes/deployment_scheduling_verify/` deleted.
+`test_calendar_probe.py` Part A is subsumed by
+`tests/scheduling/test_calendar_adapter.py`; Part B (APScheduler gap
+phantom + fold replay strict xfails) is preserved as narrative evidence
+in the design spec, not as runnable tests (it needs an isolated env
+with `apscheduler` installed). `test_expression_contract_probe.py` is
+subsumed by `tests/scheduling/test_schedule_expressions.py`,
+`tests/core/test_input_sources.py`, and core strict-JSON tests, except
+the pre-T03 no-seam observation (deliberately superseded) and one
+unowned `InputBinding` micro-pin of untouched core code (noted, not
+ported). `test_schedule_state_model.py` is subsumed by
+`tests/scheduling/` (matrix, coalescing, slots, pause/edit/delete,
+restart, downtime, fairness, fault injection, ownership, corrupt
+handling) plus the two pins added at retirement.
+
 ## Gate 1 — Spec audit against actual code

 Three parallel audit sweeps (expression bindings, deployment invocation,
@@ -1,6 +1,9 @@
 # Deployment Scheduling

-Status: draft for review; not implemented.
+Status: implemented (slices T01–T14, reviews R0–R4 passed); this document
+remains the current contract. The implementation plan that built it is
+archived at
+`docs/historical/superpowers/plans/2026-09-09-deployment-scheduling-implementation-plan.md`.

 ## Purpose and scope

@@ -247,6 +250,16 @@ anything else under an active attempt is stale and fails closed without
 retry. Recovery must distinguish both cases from an interruption that was
 merely waiting across server restart.

+Implemented checkpoint authority: a checkpoint decides a summary only when
+its content is coherent — same run identity, checkpoint id of the form
+`{run_id}.{sequence:06d}`, runtime state decodable, and decoded stopped
+status equal to the outer reason. A durable failed decision is superseded
+only by a present, coherent, strictly newer checkpoint with matching
+attempt provenance; a missing or older referenced checkpoint keeps the
+decision (noted, history preserved) and never rolls a failed run back to
+interrupted. A genuinely newer coherent result still repairs a torn
+summary, and repeated recovery is silent and stable.
+
 The process must own the store exclusively before recovery. Enforce scheduler
 ownership with a held cross-process lock, not a stale PID file or a lease that
 can expire while the old owner still runs. Unsupported locking must reject
@@ -254,6 +267,17 @@ scheduler startup. Other processes mutating/resuming the same store remain
 unsupported. This does not upgrade the rest of the file stores to multi-writer
 safety or claim exactly-once external effects.

+Implemented lock identity: one store composition has exactly one lock file,
+at the deepest composition root containing every store root as itself or a
+direct child (`canonical_lock_root`). Identical, sibling, and nested roots
+covering the same store files contend on that one file; ancestor locks prove
+nothing and are rejected, and cross pairs reusing one protected store with a
+different partner have no lock identity at all and are rejected before any
+write. The acquired identity is frozen at acquisition, so mutating the
+handle's root afterwards cannot redirect authority. The server layout points
+both stores at the composition root itself (run data at `<root>/runs`,
+schedules at `<root>/schedules`, one lock at `<root>/scheduler.lock`).
+
 ## Lifecycle, administration, and resource bounds

 Expose create/get/list/update/pause/resume/delete and paginated occurrence
@@ -290,6 +314,43 @@ grace period. Record cancellation/failure when possible; abrupt termination
 uses startup recovery. Paused/deleted schedule definitions must not prevent
 run completion or resume from updating retained occurrence history.

+Implemented service and administration surface. The opt-in same-server
+scheduler (`SchedulerService`, enabled by `server.scheduler.enabled` or
+`wf-rpc-server --enable-scheduler`; local/static servers only) acquires
+canonical ownership, recovers without executing, then ticks calendar
+polling without blocking on long workflows: each dispatch spawns exactly
+one bounded execution task behind the async-completion seam, and the
+scheduler's own capacity gate is the execution-slot bound (an executing
+run keeps its slot until it stops). Shutdown stops admission, drains
+within `drain_grace_s`, leaves unfinished work under its executing mark
+for startup recovery to abandon truthfully, and releases ownership last.
+A failed startup releases the lock and raises.
+
+Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
+RPC, Python client): `create/get/list/update/pause/resume/delete_schedule`
+plus paginated `list_schedule_occurrences`. Creation validates the
+trigger, the deployment, the binding shapes, and a sample-occurrence
+resolution against the pinned root schema, and starts the consumed
+watermark at creation (no pre-creation backfill). Updates are
+revision-checked; pause/resume/delete mirror the poll-loop transitions;
+all mutating admin ops clear the old revision's unadmitted work and
+advance the watermark BEFORE the revision bump or flag flip lands, so a
+crash can only leave the op unapplied (retryable), never a new revision
+that backfills. Occurrence pages carry the stored history plus a live
+held-candidate `pending` synthesis on the first page. Manual runs and
+resumes bypass scheduler capacity by design (unchanged API behavior);
+scheduler capacity governs scheduled dispatch only, and a scheduled
+interrupted run resumed manually reconciles its terminal history through
+recovery.
+
+Known limitations: pointing one composition's stores inside another live
+composition's store subtree (without sharing its identical roots) is
+unsupported operator error; `max_steps: None` means "unpatched" on
+update (a set budget cannot be cleared back to unset); a first
+occurrence page may carry one row more than `limit` while a candidate is
+held; calendar iteration within a tick may use the tick-start source
+(trigger edits take effect on the next tick).
+
 ## Verification gates

 Use injected clocks and controlled executors, not real-time sleeps:
@@ -233,6 +233,7 @@ WorkflowApiSurface
  WorkflowArtifactSurface
  WorkflowDeploymentSurface
  WorkflowRunSurface
+  WorkflowScheduleSurface
 ```

 The domain services below are the process-local implementation pieces, not the
@@ -109,6 +109,17 @@ small and avoids dumping arbitrary MCP resource payloads.
 registry. `--store-root` is for the local/static server path and cannot be
 combined with `--mcp-config`.

+Opt in to deployment scheduling on a local/static server (MCP-backed
+servers reject it for now):
+
+```bash
+wf-rpc-server --store-root .wf_store --enable-scheduler
+```
+
+or set `server.scheduler.enabled` (plus optional `poll_interval_s`,
+`max_concurrent_runs`, `drain_grace_s`) in the neutral config. See
+[`deployment scheduling operations`](deployment_scheduling.md).
+
 `admin registry` shows desired persisted source entries. It is separate from
 workflow artifacts and deployments, so it can be empty even when the server has
 runtime sources and saved workflows.