lda-wf/docs/superpowers/specs/2026-09-08-deployment-scheduling-design.md

Deployment Scheduling

Status: implemented (slices T01–T14, reviews R0–R4 passed); this document remains the current contract. The implementation plan that built it is archived at docs/historical/superpowers/plans/2026-09-09-deployment-scheduling-implementation-plan.md.

Purpose and scope

A schedule starts ordinary runs of a deployment without a connected client. The explicitly enabled scheduling service lives in the workflow server. It is not the core runtime's frame scheduler and introduces no workflow node type.

First slice: one-shot and recurring cron schedules, durable admission, coalesced missed-start recovery, bounded parallel runs, occurrence inspection, and API/Python-client administration. Wait nodes, distributed workers, automatic execution retries, and arbitrary in-flight checkpoint recovery are outside this slice.

The existing file-store contract remains single-process. See store transactions.

Decided behavior

• A schedule follows its deployment. At admission, capture its configuration revision, resolved input, limits, deployment, and pinned artifact tree. Later edits cannot change that occurrence or its run.
• Overlap is per schedule, not per deployment. With default overlap="skip", an unfinished scheduled run, including an interrupted run awaiting input, blocks another occurrence from that schedule. Manual runs and other schedules do not participate in that overlap check.
• overlap="parallel" admits independent runs up to a required positive max_active_runs limit per schedule, subject to server capacity. Admitted, running, and interrupted runs all count toward this limit. Do not store only one active run ID per schedule. Queue and replace-running policies are not included.
• Default misfire="skip" drops missed starts. Optional misfire="latest" means "run as soon as possible after a missed start": retain at most one latest unadmitted candidate per schedule. It is not replay-all/burst mode.
• Pausing stops future admission, not an active run. Resuming selects the next future occurrence; paused times are not replayed.
• Deleting a schedule stops future admission. Existing runs and occurrence history survive, including their schedule identity. Active runs continue. Schedule identifiers must not be reused to annex old history.
• A known completed or failed run releases overlap. After exclusive startup, an abandoned in-flight execution becomes failed without retrying its occurrence. Future occurrences may proceed. The failure must disclose that external effects may already have occurred; it does not assert rollback or remote cancellation.
• A durably interrupted execution is not abandoned. It remains resumable and continues to occupy its schedule's overlap/concurrency slot after restart.

Time and triggers

Cron uses an explicit IANA time zone, default UTC. One-shot timestamps must include an offset. Persist occurrence instants as UTC timestamps; retain the cron time-zone name in the definition. Reject invalid zones and naive times.

Default lateness allowance: 60 seconds, configurable per schedule as a non-negative finite duration. During uninterrupted operation, a due instant within the allowance can be admitted. Older instants and enabled times missed while the server was unavailable follow the configured misfire policy. With skip, startup selects the next future instant regardless of lateness allowance, and an expired one-shot becomes exhausted. With latest, startup retains the latest missed instant, including an expired one-shot, for prompt admission subject to overlap and capacity. Never enumerate years of missed occurrences on startup; find the latest eligible instant with bounded library queries and record a skipped/coalesced interval summary instead.

Missed starts versus overlap

latest has no age expiry while the schedule remains enabled. Its pending candidate survives restart without becoming a run until admission. A newer due instant replaces an older unadmitted candidate, with an inspectable superseded/coalesced reason; admission always uses that candidate's resolved occurrence instant, not the current clock time. Once admitted, an occurrence is immutable and can never be superseded or replayed by this policy.

For example, an hourly schedule returning at 12:20 after missing 10:00, 11:00, and 12:00 offers one 12:00 occurrence, not three runs. If global capacity remains unavailable until 13:00, the pending candidate becomes 13:00. A tick at exactly 13:00 must not independently admit both candidates.

Overlap decisions take precedence over waiting for server capacity. With overlap="skip", a candidate examined while that schedule has an unfinished run is terminally skipped-overlap, not held until the run finishes. With parallel, reaching max_active_runs similarly produces skipped-overlap. Neither policy resurrects that skipped instant as a later catch-up candidate. Global execution-capacity shortage instead leaves a latest candidate pending; with skip, it expires when its lateness allowance is exceeded.

Explicit pause is not downtime. Pause clears unadmitted candidates and excludes the paused interval from catch-up under both policies. Resume starts from the next future instant. Definition edits discard unadmitted candidates from the old revision and begin the new revision at the edit time; creation and edits do not backfill time before that revision. Deletion clears pending candidates without touching admitted runs or retained history. An enabled one-shot missed during downtime can catch up with latest; one missed while paused cannot.

Persist consumed/superseded interval progress with candidate selection so a restart cannot reconstruct a terminally skipped or already-admitted candidate. Pending selection and immutable admission are different states: only admission freezes input and the deployment/artifact snapshot.

Task Scheduler is inspiration, not a compatibility target. Microsoft's StartWhenAvailable documentation describes delayed starts, while its instance policy documentation separately defines parallel, queue, ignore-new, and stop-existing. Those pages do not specify our latest-candidate supersession rule. We choose that rule explicitly and do not copy Windows' documented default ten-minute delay.

Occurrence identity is the schedule identity plus the resolved UTC instant. Clock rollback cannot admit an already-consumed instant again. Use a monotonic clock for sleeping, and a wall clock for calendar eligibility. A forward jump applies lateness policy, not unconditional replay.

Use a library for calendar calculation. Decided: croniter behind a thin next/previous-occurrence adapter (next_after / prev_before over schedule-zone instants, UTC at the boundary), without adopting any job store or executor. Probes (see the implementation plan) disqualified APScheduler 3.x triggers: no bounded latest-missed seam, phantom wall times across DST gaps, and replayed UTC minutes after fall folds. The adapter converts the query instant into the schedule's named zone, asks croniter for the next/previous occurrence, and converts the result to UTC. It applies no calendar correction of its own: croniter owns DST resolution, including nonexistent and repeated local times.

Probed and pinned on Python 3.14 (croniter==6.2.4, tzdata floor; python-dateutil only transitive): some nonexistent scheduled wall times resolve forward — daily 02:30 on the spring-gap day resolves to 03:00-04:00 the same day — and repeated times surface as distinct UTC occurrences (both 01:30s on the fall-fold day). Occurrence identity is (schedule_id, resolved UTC instant). The definition's cron expression, time-zone name, and admission snapshots are retained, but the platform claims no finer intended-wall-time provenance than the library supplies. Iteration is exclusive in both directions (a query from exactly a due instant returns the neighboring occurrence), so the adapter queries forward from the last-consumed instant and compares catch-up results against the same watermark: a due occurrence is admitted exactly once. Impossible schedules surface promptly as a documented library exhaustion error, never as silent no-progress; search exhaustion or backward/non-progressing results fail visibly. Five-field cron scope; the adapter itself rejects invalid zones and naive timestamps (the library does not reject naive). Re-probe on any calendar-dependency upgrade; the strict failure pins guard the APScheduler behaviors we rejected.

The adapter uses Unix cron dialect explicitly: numeric 0 and 7 both mean Sunday (not Monday-first), and day-of-month/day-of-week matching uses croniter's standard day_or=True (Unix OR), exposed explicitly rather than as an undocumented promise. No jitter or extended trigger combinations in this slice.

Alternatives considered: APScheduler 3.x triggers were probed and rejected (bounded latest-missed lookup impossible, DST-gap phantoms, fold replay); a full scheduling framework owns useful job machinery but would create a second persistence/execution lifecycle alongside our run API. Prefer croniter iteration plus our existing run lifecycle.

Sources checked on 2026-09-08:

• APScheduler 3.x cron trigger
• APScheduler date trigger
• croniter project documentation

Input authoring and serialization

A schedule stores instructions for building a future workflow input object. The deployment-run API still receives resolved data, not expression objects. Do not add parallel raw-input and occurrence-binding mechanisms.

Reuse literal, object, array, target-path, strict-JSON, and expression-budget semantics from the existing input-binding system. Add a typed occurrence reference for the schedule environment; graph references to input/state/context are invalid here. Do not extend GraphSourcePath with schedule-only roots. Existing expressions compose data; they do not implement date formatting, arithmetic, template evaluation, Python execution, or arbitrary transforms.

The persisted Schedule.input_bindings field is a list of ScheduleInputBinding objects. Each binding has a target: LocalPath and a discriminated ScheduleExpression, whose concrete models are LiteralExpression, OccurrenceExpression, ScheduleArrayExpression, and ScheduleObjectExpression. The API accepts and returns these bindings as JSON through Schedule.model_dump(mode="json"); for example:

{
  "input_bindings": [
    {
      "target": "team",
      "expression": {"kind": "literal", "value": "engineering"}
    },
    {
      "target": "report_time",
      "expression": {"kind": "occurrence", "field": "scheduled_at"}
    }
  ]
}

OccurrenceExpression exposes schedule_id, occurrence_id, and scheduled_at from the admitted occurrence. scheduled_at is serialized as a UTC RFC 3339 string; the other occurrence fields are strings as well. The admitted run's resolved input is persisted once and never re-evaluated on restart. Validate target conflicts, expression bounds, source fields, and the resulting workflow input schema. Recheck the current deployment contract at admission; an edit may have changed the expected input since schedule creation.

Extract the shared composition traversal and limits behind a typed source resolver seam. Keep graph and schedule source models distinct. Do not copy the recursive evaluator into a second package or use fake graph context to smuggle occurrence values into graph-path evaluation.

Generic host Runtime[ContextT] remains a separate future feature. Schedule provenance is persisted platform metadata, not an arbitrary host object and not a new graph-visible context namespace. Child workflows receive business values through their declared input bindings as before.

Durable admission and recovery

Historical pre-implementation seams

Before T01–T14 were implemented, these were the seams that required change:

• wf_api/runs.py executes before persisting a stopped run.
• wf_artifacts/runs/models.py permits only stopped summaries, with a required checkpoint identifier.
• wf_api/run_lifecycle.py assumes an existing checkpoint when updating a run.

Introduce a durable admission representation with a preassigned run identity. Run inspection must distinguish an admitted/in-flight run from a stopped run with a checkpoint. Never fabricate a completed checkpoint, trace, output, or successful step count for work whose outcome is unknown.

Required ordering under the single-owner admission lock:

1. Recheck schedule revision, enabled status, due time, capacity, and overlap.
2. Allocate occurrence/run identities and freeze the invocation data.
3. Atomically persist the authoritative admission record before dispatch.
4. Materialize the run admission view using that same identity.
5. Dispatch the captured invocation without resolving the deployment again.
6. Persist a stopped checkpoint and summary, then reconcile occurrence status.

The admission record is the recovery authority for partial multi-file writes; atomic rename is not a transaction across files. A failed durable admission must never dispatch. Reconciliation is idempotent: it completes missing views, recognizes durable stopped results, or marks abandoned work failed without redispatch. A persisted interruption must win over stale in-flight metadata. Corrupt or contradictory records fail closed with diagnostics; do not silently discard them to clear overlap.

Resume of a scheduled interrupted run must mark its active attempt durably before executing again. Otherwise a crash during resume could leave an old interrupted checkpoint looking safe to retry. The mark carries a store-backed attempt identity, and every stopped result the attempt produces echoes that identity back. Recovery matches result to active attempt: a result belonging to the active attempt is fresh and resumable, anything else under an active attempt is stale and fails closed without retry. Recovery must distinguish both cases from an interruption that was merely waiting across server restart.

Implemented checkpoint authority: a checkpoint decides a summary only when its content is coherent — same run identity, checkpoint id of the form {run_id}.{sequence:06d}, runtime state decodable, and decoded stopped status equal to the outer reason. A durable failed decision is superseded only by a present, coherent, strictly newer checkpoint with matching attempt provenance; a missing or older referenced checkpoint keeps the decision (noted, history preserved) and never rolls a failed run back to interrupted. A genuinely newer coherent result still repairs a torn summary, and repeated recovery is silent and stable.

The process must own the store exclusively before recovery. Enforce scheduler ownership with a held cross-process lock, not a stale PID file or a lease that can expire while the old owner still runs. Unsupported locking must reject scheduler startup. Other processes mutating/resuming the same store remain unsupported. This does not upgrade the rest of the file stores to multi-writer safety or claim exactly-once external effects.

Implemented lock identity: one store composition has exactly one lock file, at the deepest composition root containing every store root as itself or a direct child (canonical_lock_root). Identical, sibling, and nested roots covering the same store files contend on that one file; ancestor locks prove nothing and are rejected, and cross pairs reusing one protected store with a different partner have no lock identity at all and are rejected before any write. The acquired identity is frozen at acquisition, so mutating the handle's root afterwards cannot redirect authority. The server layout points both stores at the composition root itself (run data at <root>/runs, schedules at <root>/schedules, one lock at <root>/scheduler.lock).

Lifecycle, administration, and resource bounds

Expose create/get/list/update/pause/resume/delete and paginated occurrence inspection through the workflow API and Python client. The workflow API and transport names are create_schedule, get_schedule, list_schedules, update_schedule, pause_schedule, resume_schedule, delete_schedule, and list_schedule_occurrences; the Python client facade exposes the same operations as create_schedule, schedule, schedules, update_schedule, pause_schedule, resume_schedule, delete_schedule, and schedule_occurrences. Reject stale schedule edits using revisions within the owning process.

Occurrence inspection distinguishes pending, coalesced/superseded, skipped-overlap, skipped-misfire, preflight rejection, admitted/running, interrupted, completed, and failed. These are platform occurrence states, not business outcomes or core node outcomes. Preflight rejection does not invent a run that never started.

Store resolved input and pinned environment with admitted runs. Inspection includes resolved occurrence instant (UTC), admission time, actual start when known, schedule revision, run identity, and failure/skip reason. Preserve existing run limits; allow a schedule-specific max_steps using the same validation as manual runs.

Bound active scheduled execution tasks and work per polling batch. Do not block calendar polling on a long run or create unbounded pending tasks. Capacity-delayed skip occurrences expire at their lateness deadline; latest retains at most one candidate as specified above. Interrupted runs consume per-schedule active-run slots but no executing-task slot while waiting. Resumption must acquire a server execution slot before dispatch. Lowering a schedule's active limit never cancels existing runs; block new admission until the count drops below the new limit. Polling must be fair across schedules so a frequently due schedule cannot monopolize available capacity. The concrete server capacity default is deployment configuration, with deterministic tests using a small injected limit.

On shutdown stop admission first and drain active tasks within a configured grace period. New scheduled resumes are rejected once shutdown begins (no dispatch, no ungated fallback); in-flight scheduled resumes share the grace window, then are cancelled and joined before ownership is released, so no old execution persists after a new owner takes over. A cancelled resume keeps its executing mark and ACTIVE attempt, and startup recovery fails it closed exactly like a crash mid-resume. Record cancellation/failure when possible; abrupt termination uses startup recovery. Paused/deleted schedule definitions must not prevent run completion or resume from updating retained occurrence history.

Implemented service and administration surface. The opt-in same-server scheduler (SchedulerService, enabled by server.scheduler.enabled or wf-rpc-server --enable-scheduler; local/static servers only) acquires canonical ownership, recovers without executing, then ticks calendar polling without blocking on long workflows: each dispatch spawns exactly one bounded execution task behind the async-completion seam, and the scheduler's own capacity gate is the execution-slot bound (an executing run keeps its slot until it stops). Shutdown stops admission, rejects new scheduled resumes for the duration of the drain, cancels and joins unfinished scheduled resumes after drain_grace_s (cancelled work keeps its executing mark and ACTIVE attempt for startup recovery to abandon truthfully), and releases ownership last. A failed startup releases the lock and raises.

Administration (WorkflowApi schedules methods, workflow.schedules.* RPC, Python client): create/get/list/update/pause/resume/delete_schedule plus paginated list_schedule_occurrences. Creation validates the trigger, the deployment, the binding shapes, and a sample-occurrence resolution against the pinned root schema, and starts the consumed watermark at creation (no pre-creation backfill). Updates are revision-checked; pause/resume/delete mirror the poll-loop transitions; all mutating admin ops clear the old revision's unadmitted work and advance the watermark BEFORE the revision bump or flag flip lands, so a crash can only leave the op unapplied (retryable), never a new revision that backfills. Occurrence pages carry the stored history plus a live held-candidate pending synthesis on the first page. Manual runs bypass scheduler capacity by design (unchanged API behavior); capacity governs scheduled dispatch plus scheduled resumes. A scheduled interrupted run resumed through the run API acquires a server execution slot through the scheduler's own accounting before dispatch — rejection leaves no resume attempt behind — holds the durable executing mark for the re-execution (visible to capacity and drain like any live execution), and releases the slot when its stopped result is persisted. Shutdown drain rejects further scheduled resumes; a resume cancelled by the drain keeps its marks for recovery instead of releasing them. A resumed scheduled run reconciles its terminal history live through the same idempotent recording as dispatch; restart recovery still repairs torn boundaries.

Known limitations: pointing one composition's stores inside another live composition's store subtree (without sharing its identical roots) is unsupported operator error; max_steps: None means "unpatched" on update (a set budget cannot be cleared back to unset); a pending occurrence projection still honors the requested page limit, using an opaque continuation cursor when limit=1 so the first stored row remains reachable; calendar iteration within a tick may use the tick-start source (trigger edits take effect on the next tick).

Verification gates

Use injected clocks and controlled executors, not real-time sleeps:

• Cron parsing, time zones, invalid syntax, leap/calendar boundaries, DST gaps and folds, UTC uniqueness, impossible schedules, and bounded next-time search.
• One-shot success/exhaustion/catch-up; lateness boundary; startup behavior under both policies; explicit pause/resume exclusion; long downtime without unbounded enumeration; clock rollback/forward jumps.
• Latest-only coalescing across multiple missed times, candidate persistence, supersession at the next due instant, no double admission at that boundary, and no resurrection after overlap skips or admission.
• Literal and nested occurrence expressions round-trip through JSON; missing fields, graph-only paths, excessive trees, conflicting targets, and invalid resolved workflow input fail before dispatch.
• Same-schedule overlap across running and interrupted states; manual/other schedule independence; release on failure/completion and on resumed completion.
• Parallel admission limits, interrupted slot accounting, capacity-delayed candidates, fair polling, limit edits, and bounded task allocation. Exercise all four overlap/misfire combinations with controlled executors.
• Edit/repoint/delete races at admission; captured invocation stays unchanged.
• Fault injection before/after every persistence boundary, including resume: no dispatch before durable admission and no replay after ambiguous execution.
• Recovery preserves stopped interruptions, reconciles terminal results, marks abandoned attempts failed, and leaves corrupt records visibly blocked.
• Second-owner rejection, lock release on process death, bounded concurrency, shutdown drain, and full occurrence history after schedule deletion.
• Public API/client round trips, pagination, inspection without a checkpoint, and existing manual run/resume behavior remain valid.

Verification and operations guidance

The implementation plan is archived as historical context; no further design-approval or implementation-planning step remains for this slice. When changing the calendar dependency or adapter, rerun the calendar-library probe and retain the calendar boundary coverage above. Keep fault-injection tests at each persistence boundary, including resume, when changing admission or recovery. Use the deployment scheduling operations guide for local/static server configuration and runtime operations. WaitNode is a later contract that may reuse timed admission but must persist its own suspended execution.