lda-wf/docs/superpowers/specs/2026-09-08-deployment-scheduling-design.md

# Deployment Scheduling

Status: implemented (slices T01–T14, reviews R0–R4 passed); this document
remains the current contract. The implementation plan that built it is
archived at
`docs/historical/superpowers/plans/2026-09-09-deployment-scheduling-implementation-plan.md`.

## Purpose and scope

A schedule starts ordinary runs of a deployment without a connected client.
The explicitly enabled scheduling service lives in the workflow server. It is
not the core runtime's frame scheduler and introduces no workflow node type.

First slice: one-shot and recurring cron schedules, durable admission,
coalesced missed-start recovery, bounded parallel runs, occurrence inspection,
and API/Python-client administration. Wait nodes,
distributed workers, automatic execution retries, and arbitrary in-flight
checkpoint recovery are outside this slice.

The existing file-store contract remains single-process. See
[store transactions](2026-06-09-store-transaction-boundary.md).

## Decided behavior

- A schedule follows its deployment. At admission, capture its configuration
  revision, resolved input, limits, deployment, and pinned artifact tree.
  Later edits cannot change that occurrence or its run.
- Overlap is per schedule, not per deployment. With default `overlap="skip"`,
  an unfinished scheduled run, including an interrupted run awaiting input,
  blocks another occurrence from that schedule. Manual runs and other
  schedules do not participate in that overlap check.
- `overlap="parallel"` admits independent runs up to a required positive
  `max_active_runs` limit per schedule, subject to server capacity. Admitted,
  running, and interrupted runs all count toward this limit. Do not store
  only one active run ID per schedule. Queue and replace-running policies
  are not included.
- Default `misfire="skip"` drops missed starts. Optional `misfire="latest"`
  means "run as soon as possible after a missed start": retain at most one
  latest unadmitted candidate per schedule. It is not replay-all/burst mode.
- Pausing stops future admission, not an active run. Resuming selects the next
  future occurrence; paused times are not replayed.
- Deleting a schedule stops future admission. Existing runs and occurrence
  history survive, including their schedule identity. Active runs continue.
  Schedule identifiers must not be reused to annex old history.
- A known completed or failed run releases overlap. After exclusive startup,
  an abandoned in-flight execution becomes failed without retrying its
  occurrence. Future occurrences may proceed. The failure must disclose that
  external effects may already have occurred; it does not assert rollback or
  remote cancellation.
- A durably interrupted execution is not abandoned. It remains resumable and
  continues to occupy its schedule's overlap/concurrency slot after restart.

## Time and triggers

Cron uses an explicit IANA time zone, default UTC. One-shot timestamps must
include an offset. Persist occurrence instants as UTC timestamps; retain the
cron time-zone name in the definition. Reject invalid zones and naive times.

Proposed default lateness allowance: 60 seconds, configurable per schedule as
a non-negative finite duration. During uninterrupted operation, a due instant
within the allowance can be admitted. Older instants and enabled times missed
while the server was unavailable follow the configured misfire policy.
With `skip`, startup selects the next future instant regardless of lateness
allowance, and an expired one-shot becomes exhausted. With `latest`, startup
retains the latest missed instant, including an expired one-shot, for prompt
admission subject to overlap and capacity. Never enumerate years of missed
occurrences on startup; find the latest eligible instant with bounded library
queries and record a skipped/coalesced interval summary instead.

### Missed starts versus overlap

`latest` has no age expiry while the schedule remains enabled. Its pending
candidate survives restart without becoming a run until admission. A newer
due instant replaces an older unadmitted candidate, with an inspectable
superseded/coalesced reason; admission always uses that candidate's
resolved occurrence instant, not the current clock time. Once admitted,
an occurrence is immutable
and can never be superseded or replayed by this policy.

For example, an hourly schedule returning at 12:20 after missing 10:00,
11:00, and 12:00 offers one 12:00 occurrence, not three runs. If global
capacity remains unavailable until 13:00, the pending candidate becomes
13:00. A tick at exactly 13:00 must not independently admit both candidates.

Overlap decisions take precedence over waiting for server capacity. With
`overlap="skip"`, a candidate examined while that schedule has an unfinished
run is terminally skipped-overlap, not held until the run finishes. With
`parallel`, reaching `max_active_runs` similarly produces skipped-overlap.
Neither policy resurrects that skipped instant as a later catch-up candidate.
Global execution-capacity shortage instead leaves a `latest` candidate pending;
with `skip`, it expires when its lateness allowance is exceeded.

Explicit pause is not downtime. Pause clears unadmitted candidates and excludes
the paused interval from catch-up under both policies. Resume starts from the
next future instant. Definition edits discard unadmitted candidates from the
old revision and begin the new revision at the edit time; creation and edits
do not backfill time before that revision. Deletion clears pending candidates
without touching admitted runs or retained history. An enabled one-shot missed
during downtime can catch up with `latest`; one missed while paused cannot.

Persist consumed/superseded interval progress with candidate selection so a
restart cannot reconstruct a terminally skipped or already-admitted candidate.
Pending selection and immutable admission are different states: only admission
freezes input and the deployment/artifact snapshot.

Task Scheduler is inspiration, not a compatibility target. Microsoft's
[StartWhenAvailable documentation](https://learn.microsoft.com/en-us/windows/win32/taskschd/tasksettings-startwhenavailable)
describes delayed starts, while its
[instance policy documentation](https://learn.microsoft.com/en-us/windows/win32/api/taskschd/ne-taskschd-task_instances_policy)
separately defines parallel, queue, ignore-new, and stop-existing. Those pages
do not specify our latest-candidate supersession rule. We choose that rule
explicitly and do not copy Windows' documented default ten-minute delay.

Occurrence identity is the schedule identity plus the resolved UTC instant.
Clock rollback cannot admit an already-consumed instant again. Use a monotonic
clock for sleeping, and a wall clock for calendar eligibility. A forward jump
applies lateness policy, not unconditional replay.

Use a library for calendar calculation. Decided: croniter behind a thin
next/previous-occurrence adapter (`next_after` / `prev_before` over
schedule-zone instants, UTC at the boundary), without adopting any job
store or executor. Probes (see the implementation plan) disqualified
APScheduler 3.x triggers: no bounded latest-missed seam, phantom wall
times across DST gaps, and replayed UTC minutes after fall folds. The
adapter converts the query instant into the schedule's named zone, asks
croniter for the next/previous occurrence, and converts the result to
UTC. It applies no calendar correction of its own: croniter owns DST
resolution, including nonexistent and repeated local times.

Probed and pinned on Python 3.14 (`croniter==6.2.4`, `tzdata` floor;
`python-dateutil` only transitive): some nonexistent scheduled wall
times resolve forward — daily 02:30 on the spring-gap day resolves to
03:00-04:00 the same day — and repeated times surface as distinct UTC
occurrences (both 01:30s on the fall-fold day). Occurrence identity is
`(schedule_id, resolved UTC instant)`. The definition's cron
expression, time-zone name, and admission snapshots are retained, but
the platform claims no finer intended-wall-time provenance than the
library supplies. Iteration is exclusive in both directions (a query
from exactly a due instant returns the neighboring occurrence), so the
adapter queries forward from the last-consumed instant and compares
catch-up results against the same watermark: a due occurrence is
admitted exactly once. Impossible schedules surface promptly as a
documented library exhaustion error, never as silent no-progress;
search exhaustion or backward/non-progressing results fail visibly.
Five-field cron scope; the adapter itself rejects invalid zones and
naive timestamps (the library does not reject naive). Re-probe on any
calendar-dependency upgrade; the strict failure pins guard the
APScheduler behaviors we rejected.

The adapter uses Unix cron dialect explicitly: numeric `0` and `7`
both mean Sunday (not Monday-first), and day-of-month/day-of-week
matching uses croniter's standard `day_or=True` (Unix OR), exposed
explicitly rather than as an undocumented promise. No jitter or
extended trigger combinations in this slice.

Alternatives considered: APScheduler 3.x triggers were probed and rejected
(bounded latest-missed lookup impossible, DST-gap phantoms, fold replay);
a full scheduling framework owns useful job machinery but would create a
second persistence/execution lifecycle alongside our run API. Prefer
croniter iteration plus our existing run lifecycle.

Sources checked on 2026-09-08:

- [APScheduler 3.x cron trigger](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/cron.html)
- [APScheduler date trigger](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/date.html)
- [croniter project documentation](https://pypi.org/project/croniter/)

## Input authoring and serialization

A schedule stores instructions for building a future workflow input object.
The deployment-run API still receives resolved data, not expression objects.
Do not add parallel raw-input and occurrence-binding mechanisms.

Reuse literal, object, array, target-path, strict-JSON, and expression-budget
semantics from the existing input-binding system. Add a typed occurrence
reference for the schedule environment; graph references to input/state/context
are invalid here. Do not extend GraphSourcePath with schedule-only roots.
Existing expressions compose data; they do not implement date formatting,
arithmetic, template evaluation, Python execution, or arbitrary transforms.

Proposed persisted binding example, pending concrete model names:

```json
{
  "input_bindings": [
    {"target": "team", "value": "engineering"},
    {
      "target": "report_time",
      "expression": {"kind": "occurrence", "field": "scheduled_at"}
    }
  ]
}
```

Occurrence references initially expose schedule_id, occurrence_id, and
scheduled_at. Date-time values serialize as UTC RFC 3339 strings. The admitted
run's resolved input is persisted once and never re-evaluated on restart.
Validate target conflicts, expression bounds, source fields, and the resulting
workflow input schema. Recheck the current deployment contract at admission;
an edit may have changed the expected input since schedule creation.

Extract the shared composition traversal and limits behind a typed source
resolver seam. Keep graph and schedule source models distinct. Do not copy
the recursive evaluator into a second package or use fake graph context to
smuggle occurrence values into graph-path evaluation.

Generic host Runtime[ContextT] remains a separate future feature. Schedule
provenance is persisted platform metadata, not an arbitrary host object and
not a new graph-visible context namespace. Child workflows receive business
values through their declared input bindings as before.

## Durable admission and recovery

Current seams needing change:

- wf_api/runs.py executes before persisting a stopped run.
- wf_artifacts/runs/models.py permits only stopped summaries, with a required
  checkpoint identifier.
- wf_api/run_lifecycle.py assumes an existing checkpoint when updating a run.

Introduce a durable admission representation with a preassigned run identity.
Run inspection must distinguish an admitted/in-flight run from a stopped run
with a checkpoint. Never fabricate a completed checkpoint, trace, output, or
successful step count for work whose outcome is unknown.

Required ordering under the single-owner admission lock:

1. Recheck schedule revision, enabled status, due time, capacity, and overlap.
2. Allocate occurrence/run identities and freeze the invocation data.
3. Atomically persist the authoritative admission record before dispatch.
4. Materialize the run admission view using that same identity.
5. Dispatch the captured invocation without resolving the deployment again.
6. Persist a stopped checkpoint and summary, then reconcile occurrence status.

The admission record is the recovery authority for partial multi-file writes;
atomic rename is not a transaction across files. A failed durable admission
must never dispatch. Reconciliation is idempotent: it completes missing views,
recognizes durable stopped results, or marks abandoned work failed without
redispatch. A persisted interruption must win over stale in-flight metadata.
Corrupt or contradictory records fail closed with diagnostics; do not silently
discard them to clear overlap.

Resume of a scheduled interrupted run must mark its active attempt durably
before executing again. Otherwise a crash during resume could leave an old
interrupted checkpoint looking safe to retry. The mark carries a
store-backed attempt identity, and every stopped result the attempt
produces echoes that identity back. Recovery matches result to active
attempt: a result belonging to the active attempt is fresh and resumable,
anything else under an active attempt is stale and fails closed without
retry. Recovery must distinguish both cases from an interruption that was
merely waiting across server restart.

Implemented checkpoint authority: a checkpoint decides a summary only when
its content is coherent — same run identity, checkpoint id of the form
`{run_id}.{sequence:06d}`, runtime state decodable, and decoded stopped
status equal to the outer reason. A durable failed decision is superseded
only by a present, coherent, strictly newer checkpoint with matching
attempt provenance; a missing or older referenced checkpoint keeps the
decision (noted, history preserved) and never rolls a failed run back to
interrupted. A genuinely newer coherent result still repairs a torn
summary, and repeated recovery is silent and stable.

The process must own the store exclusively before recovery. Enforce scheduler
ownership with a held cross-process lock, not a stale PID file or a lease that
can expire while the old owner still runs. Unsupported locking must reject
scheduler startup. Other processes mutating/resuming the same store remain
unsupported. This does not upgrade the rest of the file stores to multi-writer
safety or claim exactly-once external effects.

Implemented lock identity: one store composition has exactly one lock file,
at the deepest composition root containing every store root as itself or a
direct child (`canonical_lock_root`). Identical, sibling, and nested roots
covering the same store files contend on that one file; ancestor locks prove
nothing and are rejected, and cross pairs reusing one protected store with a
different partner have no lock identity at all and are rejected before any
write. The acquired identity is frozen at acquisition, so mutating the
handle's root afterwards cannot redirect authority. The server layout points
both stores at the composition root itself (run data at `<root>/runs`,
schedules at `<root>/schedules`, one lock at `<root>/scheduler.lock`).

## Lifecycle, administration, and resource bounds

Expose create/get/list/update/pause/resume/delete and paginated occurrence
inspection through the workflow API and Python client. Public client names
are finalized in the implementation plan, not treated as existing methods.
Reject stale schedule edits using revisions within the owning process.

Occurrence inspection distinguishes pending, coalesced/superseded,
skipped-overlap, skipped-misfire,
preflight rejection, admitted/running, interrupted, completed, and failed.
These are platform occurrence states, not business outcomes or core node
outcomes. Preflight rejection does not invent a run that never started.

Store resolved input and pinned environment with admitted runs. Inspection
includes resolved occurrence instant (UTC), admission time, actual start
when known, schedule
revision, run identity, and failure/skip reason. Preserve existing run limits;
allow a schedule-specific max_steps using the same validation as manual runs.

Bound active scheduled execution tasks and work per polling batch. Do not
block calendar polling on a long run or create unbounded pending tasks.
Capacity-delayed `skip` occurrences expire at their lateness deadline; `latest`
retains at most one candidate as specified above. Interrupted runs consume
per-schedule active-run slots but no executing-task slot while waiting.
Resumption must acquire a server execution slot before dispatch. Lowering a
schedule's active limit never cancels existing runs; block new admission until
the count drops below the new limit. Polling must be fair across schedules so
a frequently due schedule cannot monopolize available capacity. The concrete
server capacity default is deployment configuration, with deterministic tests
using a small injected limit.

On shutdown stop admission first and drain active tasks within a configured
grace period. Record cancellation/failure when possible; abrupt termination
uses startup recovery. Paused/deleted schedule definitions must not prevent
run completion or resume from updating retained occurrence history.

Implemented service and administration surface. The opt-in same-server
scheduler (`SchedulerService`, enabled by `server.scheduler.enabled` or
`wf-rpc-server --enable-scheduler`; local/static servers only) acquires
canonical ownership, recovers without executing, then ticks calendar
polling without blocking on long workflows: each dispatch spawns exactly
one bounded execution task behind the async-completion seam, and the
scheduler's own capacity gate is the execution-slot bound (an executing
run keeps its slot until it stops). Shutdown stops admission, drains
within `drain_grace_s`, leaves unfinished work under its executing mark
for startup recovery to abandon truthfully, and releases ownership last.
A failed startup releases the lock and raises.

Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
RPC, Python client): `create/get/list/update/pause/resume/delete_schedule`
plus paginated `list_schedule_occurrences`. Creation validates the
trigger, the deployment, the binding shapes, and a sample-occurrence
resolution against the pinned root schema, and starts the consumed
watermark at creation (no pre-creation backfill). Updates are
revision-checked; pause/resume/delete mirror the poll-loop transitions;
all mutating admin ops clear the old revision's unadmitted work and
advance the watermark BEFORE the revision bump or flag flip lands, so a
crash can only leave the op unapplied (retryable), never a new revision
that backfills. Occurrence pages carry the stored history plus a live
held-candidate `pending` synthesis on the first page. Manual runs bypass
scheduler capacity by design (unchanged API behavior); capacity governs
scheduled dispatch plus scheduled resumes. A scheduled interrupted run
resumed through the run API acquires a server execution slot through the
scheduler's own accounting before dispatch — rejection leaves no resume
attempt behind — holds the durable executing mark for the re-execution
(visible to capacity and drain like any live execution), and releases
the slot when its stopped result is persisted. A resumed scheduled run
reconciles its terminal history live through the same idempotent
recording as dispatch; restart recovery still repairs torn boundaries.

Known limitations: pointing one composition's stores inside another live
composition's store subtree (without sharing its identical roots) is
unsupported operator error; `max_steps: None` means "unpatched" on
update (a set budget cannot be cleared back to unset); a first
occurrence page may carry one row more than `limit` while a candidate is
held; calendar iteration within a tick may use the tick-start source
(trigger edits take effect on the next tick).

## Verification gates

Use injected clocks and controlled executors, not real-time sleeps:

- Cron parsing, time zones, invalid syntax, leap/calendar boundaries, DST gaps
  and folds, UTC uniqueness, impossible schedules, and bounded next-time search.
- One-shot success/exhaustion/catch-up; lateness boundary; startup behavior
  under both policies; explicit pause/resume exclusion; long downtime without
  unbounded enumeration; clock rollback/forward jumps.
- Latest-only coalescing across multiple missed times, candidate persistence,
  supersession at the next due instant, no double admission at that boundary,
  and no resurrection after overlap skips or admission.
- Literal and nested occurrence expressions round-trip through JSON; missing
  fields, graph-only paths, excessive trees, conflicting targets, and invalid
  resolved workflow input fail before dispatch.
- Same-schedule overlap across running and interrupted states; manual/other
  schedule independence; release on failure/completion and on resumed completion.
- Parallel admission limits, interrupted slot accounting, capacity-delayed
  candidates, fair polling, limit edits, and bounded task allocation. Exercise
  all four overlap/misfire combinations with controlled executors.
- Edit/repoint/delete races at admission; captured invocation stays unchanged.
- Fault injection before/after every persistence boundary, including resume:
  no dispatch before durable admission and no replay after ambiguous execution.
- Recovery preserves stopped interruptions, reconciles terminal results, marks
  abandoned attempts failed, and leaves corrupt records visibly blocked.
- Second-owner rejection, lock release on process death, bounded concurrency,
  shutdown drain, and full occurrence history after schedule deletion.
- Public API/client round trips, pagination, inspection without a checkpoint,
  and existing manual run/resume behavior remain valid.

## Review before implementation planning

User policy decisions above are settled. Review the proposed 60-second
allowance, expression seam, and admission/recovery representation together.
The calendar-library probe is an explicit gate, not a claimed passing test.
After approval, create a sequenced implementation plan with fault-injection
tests before enabling scheduling in the server. WaitNode is a later contract
that may reuse timed admission but must persist its own suspended execution.