sched: docs, probe retirement, MCP guard, coverage pins, executable example (T14)
This commit is contained in:
@@ -0,0 +1,198 @@
|
||||
# Deployment Scheduling Operations
|
||||
|
||||
Schedules start ordinary deployment runs without a connected client. The
|
||||
scheduler is opt-in, lives in the workflow server, and introduces no
|
||||
workflow node type. It is not the core runtime's frame scheduler.
|
||||
|
||||
Current contract:
|
||||
[`deployment scheduling spec`](superpowers/specs/2026-09-08-deployment-scheduling-design.md).
|
||||
|
||||
## Enablement
|
||||
|
||||
Scheduling is disabled by default. Enable it for a local/static server
|
||||
with the config section or the CLI flag (MCP-backed servers reject it):
|
||||
|
||||
```json
|
||||
{"server": {"scheduler": {"enabled": true}}}
|
||||
```
|
||||
|
||||
```bash
|
||||
wf-rpc-server --store-root .wf_store --enable-scheduler
|
||||
```
|
||||
|
||||
Tuning (`server.scheduler`): `poll_interval_s` (default 1.0),
|
||||
`max_concurrent_runs` (default 4, the execution-slot bound),
|
||||
`drain_grace_s` (default 30.0). Schedule data lives at
|
||||
`<store_root>/schedules` next to run data; one lock file at
|
||||
`<store_root>/scheduler.lock` proves exclusive ownership. A second
|
||||
scheduler over the same stores is rejected; shut the first down before
|
||||
starting another.
|
||||
|
||||
## Mental model: schedule vs occurrence vs run
|
||||
|
||||
- A **schedule** is a durable definition: deployment, trigger
|
||||
(one-shot or cron), input bindings, overlap/misfire policies, and a
|
||||
revision. Edits bump the revision and affect only future admissions.
|
||||
- An **occurrence** is one resolved calendar instant, identified by
|
||||
`(schedule_id, resolved UTC instant)`. An occurrence is immutable once
|
||||
admitted and is never replayed.
|
||||
- A **run** is the execution of one admitted occurrence, with a
|
||||
store-backed `run-000001`-style id, a pinned input/artifact snapshot,
|
||||
and a stopped checkpoint when it stops.
|
||||
|
||||
## Triggers and time zones
|
||||
|
||||
One-shot timestamps must include an offset. Cron uses five-field Unix
|
||||
expressions (`0` and `7` both mean Sunday; day-of-month/day-of-week
|
||||
match with OR) plus an explicit IANA time zone, default UTC.
|
||||
Occurrence instants persist as UTC; the definition keeps the zone name.
|
||||
`croniter` owns calendar resolution including DST gaps and folds; there
|
||||
is no custom calendar filtering. Reject invalid zones and naive times.
|
||||
|
||||
## Overlap
|
||||
|
||||
Overlap is per schedule, not per deployment. With default
|
||||
`overlap="skip"`, an unfinished scheduled run — including an
|
||||
interrupted run awaiting input — blocks another occurrence from that
|
||||
schedule. Manual runs and other schedules never participate.
|
||||
`overlap="parallel"` admits independent runs up to a required positive
|
||||
`max_active_runs`; admitted, running, and interrupted runs all count,
|
||||
and lowering the limit blocks new admission without cancelling existing
|
||||
runs. A known completed or failed run releases overlap.
|
||||
|
||||
## Misfire
|
||||
|
||||
Default `misfire="skip"` drops missed starts (past the 60-second
|
||||
configurable lateness allowance). `misfire="latest"` retains at most
|
||||
one latest unadmitted candidate per schedule and runs it as soon as
|
||||
overlap and capacity allow; it never replays a burst. Creation never
|
||||
backfills time before the revision: the consumed watermark starts at
|
||||
creation.
|
||||
|
||||
## Pause
|
||||
|
||||
Pausing stops future admission, not an active run. Resume selects the
|
||||
next future occurrence; paused times are not replayed. Pause is not
|
||||
downtime: the paused span is consumed, so downtime catch-up never
|
||||
resurrects it.
|
||||
|
||||
## Deletion
|
||||
|
||||
Deleting a schedule stops future admission. Existing runs and
|
||||
occurrence history survive, including their schedule identity; active
|
||||
runs continue and may still complete or resume against the retained
|
||||
history. Schedule ids are never reused.
|
||||
|
||||
## Failure and restart
|
||||
|
||||
An abandoned in-flight execution (the process died mid-run) becomes
|
||||
failed without retrying its occurrence; the failure discloses that
|
||||
external effects may already have occurred. A durably interrupted run
|
||||
is not abandoned: it stays resumable and keeps occupying its
|
||||
schedule's overlap slot across restarts. Every resume marks a
|
||||
store-backed attempt first, so recovery can tell a fresh result from a
|
||||
stale checkpoint and fail the ambiguous case closed instead of
|
||||
retrying it. Corrupt or contradictory records fail closed with
|
||||
diagnostics and block the schedule rather than clearing overlap.
|
||||
|
||||
On shutdown the server stops admission first and drains active tasks
|
||||
within the grace period; anything still running keeps its executing
|
||||
mark, and the next startup recovery abandons it truthfully.
|
||||
|
||||
## Occurrence inspection
|
||||
|
||||
`list_schedule_occurrences` pages stored history (`pending`,
|
||||
`coalesced`/`superseded`, `skipped-overlap`, `skipped-misfire`,
|
||||
`preflight-rejected`, `admitted`/`running`, `interrupted`,
|
||||
`completed`, `failed`, `exhausted`) oldest-first with a `next_cursor`.
|
||||
A currently held (unadmitted) candidate is synthesized as a `pending`
|
||||
row at the top of the first page; once admitted, the durable
|
||||
`admitted` entry replaces it. While a candidate is held, the first
|
||||
page may carry one row more than `limit`.
|
||||
|
||||
## Administration surface
|
||||
|
||||
Python API (`server.api.schedules`), JSON-RPC (`workflow.schedules.*`),
|
||||
and the Python client (`App` schedule methods) share these operations:
|
||||
|
||||
- `create_schedule` (`workflow.schedules.create`): caller-chosen id;
|
||||
trigger, deployment, binding, and sample-schema checks.
|
||||
- `get_schedule` (`workflow.schedules.get`): full definition payload.
|
||||
- `list_schedules` (`workflow.schedules.list`): deleted excluded
|
||||
unless asked.
|
||||
- `update_schedule` (`workflow.schedules.update`): `expected_revision`
|
||||
required; provided fields only; `None` means unpatched.
|
||||
- `pause_schedule` (`workflow.schedules.pause`): idempotent; clears
|
||||
candidates, consumes the span.
|
||||
- `resume_schedule` (`workflow.schedules.resume`): idempotent; resumes
|
||||
from the next future instant.
|
||||
- `delete_schedule` (`workflow.schedules.delete`): soft delete; runs
|
||||
and history survive.
|
||||
- `list_schedule_occurrences`
|
||||
(`workflow.schedules.occurrences.list`): cursor pages, `limit` 1–100,
|
||||
live pending synthesis.
|
||||
|
||||
`inspect_run` also reads admitted (checkpoint-less) runs: status
|
||||
`admitted` with no fabricated trace, output, or checkpoint.
|
||||
|
||||
## Hypothetically used as follows
|
||||
|
||||
EXECUTABLE example (runs in CI as
|
||||
`tests/examples/test_scheduled_deployment_example.py`; run it with
|
||||
`uv run pytest -q tests/examples/test_scheduled_deployment_example.py`):
|
||||
|
||||
```python
|
||||
server = build_local_static_workflow_server(root, schedules=True)
|
||||
await server.api.create_artifact_from_plan(
|
||||
artifact_id="scheduled_hello", version=1, ...,
|
||||
)
|
||||
await server.api.save_deployment({...})
|
||||
due = datetime.now(UTC) + timedelta(seconds=0.5)
|
||||
await server.api.schedules.create_schedule(
|
||||
schedule_id="hello-once",
|
||||
deployment_id="scheduled_hello.default",
|
||||
trigger={"kind": "oneshot", "at": due.isoformat()},
|
||||
)
|
||||
service = build_scheduler_service(server, SchedulerServiceConfig(...))
|
||||
await service.start()
|
||||
# ... the service admits the occurrence and completes the run ...
|
||||
inspected = await server.api.inspect_run(run_id=run_id)
|
||||
assert inspected["output"]["result"] == "hello on a schedule"
|
||||
page = await server.api.schedules.list_schedule_occurrences(
|
||||
schedule_id="hello-once"
|
||||
)
|
||||
await service.stop()
|
||||
```
|
||||
|
||||
ILLUSTRATIVE example (not executed; shows a cron week with an operator
|
||||
pause — same calls, longer horizons):
|
||||
|
||||
```python
|
||||
# Monday: an hourly report, latest-wins catch-up, at most two at once.
|
||||
await schedules.create_schedule(
|
||||
schedule_id="hourly-report",
|
||||
deployment_id="report.default",
|
||||
trigger={"kind": "cron", "expression": "0 * * * *", "timezone": "UTC"},
|
||||
misfire="latest",
|
||||
overlap="parallel",
|
||||
max_active_runs=2,
|
||||
)
|
||||
# Friday: pause for maintenance; Monday: resume from the next hour.
|
||||
await schedules.pause_schedule(schedule_id="hourly-report")
|
||||
await schedules.resume_schedule(schedule_id="hourly-report")
|
||||
# A bad edit is rejected without touching the running definition:
|
||||
await schedules.update_schedule(
|
||||
schedule_id="hourly-report", expected_revision=1, ...
|
||||
)
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- One composition's stores nested inside another live composition's
|
||||
store subtree (without sharing its identical roots) is unsupported
|
||||
operator error; shared-store cross layouts are rejected outright.
|
||||
- Manual runs and resumes bypass scheduler capacity by design;
|
||||
capacity governs scheduled dispatch only.
|
||||
- A set `max_steps` budget cannot be cleared back to unset through
|
||||
update (recreate the schedule for an unbounded budget).
|
||||
- MCP-backed servers reject scheduler enablement for now.
|
||||
Reference in New Issue
Block a user