lda-wf/docs/superpowers/specs/2026-09-08-deployment-scheduling-design.md

# Deployment Scheduling

Status: draft for review; not implemented.

## Purpose and scope

A schedule starts ordinary runs of a deployment without a connected client.
The explicitly enabled scheduling service lives in the workflow server. It is
not the core runtime's frame scheduler and introduces no workflow node type.

First slice: one-shot and recurring cron schedules, durable admission,
coalesced missed-start recovery, bounded parallel runs, occurrence inspection,
and API/Python-client administration. Wait nodes,
distributed workers, automatic execution retries, and arbitrary in-flight
checkpoint recovery are outside this slice.

The existing file-store contract remains single-process. See
[store transactions](2026-06-09-store-transaction-boundary.md).

## Decided behavior

- A schedule follows its deployment. At admission, capture its configuration
  revision, resolved input, limits, deployment, and pinned artifact tree.
  Later edits cannot change that occurrence or its run.
- Overlap is per schedule, not per deployment. With default `overlap="skip"`,
  an unfinished scheduled run, including an interrupted run awaiting input,
  blocks another occurrence from that schedule. Manual runs and other
  schedules do not participate in that overlap check.
- `overlap="parallel"` admits independent runs up to a required positive
  `max_active_runs` limit per schedule, subject to server capacity. Admitted,
  running, and interrupted runs all count toward this limit. Do not store
  only one active run ID per schedule. Queue and replace-running policies
  are not included.
- Default `misfire="skip"` drops missed starts. Optional `misfire="latest"`
  means "run as soon as possible after a missed start": retain at most one
  latest unadmitted candidate per schedule. It is not replay-all/burst mode.
- Pausing stops future admission, not an active run. Resuming selects the next
  future occurrence; paused times are not replayed.
- Deleting a schedule stops future admission. Existing runs and occurrence
  history survive, including their schedule identity. Active runs continue.
  Schedule identifiers must not be reused to annex old history.
- A known completed or failed run releases overlap. After exclusive startup,
  an abandoned in-flight execution becomes failed without retrying its
  occurrence. Future occurrences may proceed. The failure must disclose that
  external effects may already have occurred; it does not assert rollback or
  remote cancellation.
- A durably interrupted execution is not abandoned. It remains resumable and
  continues to occupy its schedule's overlap/concurrency slot after restart.

## Time and triggers

Cron uses an explicit IANA time zone, default UTC. One-shot timestamps must
include an offset. Persist occurrence instants as UTC timestamps; retain the
cron time-zone name in the definition. Reject invalid zones and naive times.

Proposed default lateness allowance: 60 seconds, configurable per schedule as
a non-negative finite duration. During uninterrupted operation, a due instant
within the allowance can be admitted. Older instants and enabled times missed
while the server was unavailable follow the configured misfire policy.
With `skip`, startup selects the next future instant regardless of lateness
allowance, and an expired one-shot becomes exhausted. With `latest`, startup
retains the latest missed instant, including an expired one-shot, for prompt
admission subject to overlap and capacity. Never enumerate years of missed
occurrences on startup; find the latest eligible instant with bounded library
queries and record a skipped/coalesced interval summary instead.

### Missed starts versus overlap

`latest` has no age expiry while the schedule remains enabled. Its pending
candidate survives restart without becoming a run until admission. A newer
due instant replaces an older unadmitted candidate, with an inspectable
superseded/coalesced reason; admission always uses that candidate's
resolved occurrence instant, not the current clock time. Once admitted,
an occurrence is immutable
and can never be superseded or replayed by this policy.

For example, an hourly schedule returning at 12:20 after missing 10:00,
11:00, and 12:00 offers one 12:00 occurrence, not three runs. If global
capacity remains unavailable until 13:00, the pending candidate becomes
13:00. A tick at exactly 13:00 must not independently admit both candidates.

Overlap decisions take precedence over waiting for server capacity. With
`overlap="skip"`, a candidate examined while that schedule has an unfinished
run is terminally skipped-overlap, not held until the run finishes. With
`parallel`, reaching `max_active_runs` similarly produces skipped-overlap.
Neither policy resurrects that skipped instant as a later catch-up candidate.
Global execution-capacity shortage instead leaves a `latest` candidate pending;
with `skip`, it expires when its lateness allowance is exceeded.

Explicit pause is not downtime. Pause clears unadmitted candidates and excludes
the paused interval from catch-up under both policies. Resume starts from the
next future instant. Definition edits discard unadmitted candidates from the
old revision and begin the new revision at the edit time; creation and edits
do not backfill time before that revision. Deletion clears pending candidates
without touching admitted runs or retained history. An enabled one-shot missed
during downtime can catch up with `latest`; one missed while paused cannot.

Persist consumed/superseded interval progress with candidate selection so a
restart cannot reconstruct a terminally skipped or already-admitted candidate.
Pending selection and immutable admission are different states: only admission
freezes input and the deployment/artifact snapshot.

Task Scheduler is inspiration, not a compatibility target. Microsoft's
[StartWhenAvailable documentation](https://learn.microsoft.com/en-us/windows/win32/taskschd/tasksettings-startwhenavailable)
describes delayed starts, while its
[instance policy documentation](https://learn.microsoft.com/en-us/windows/win32/api/taskschd/ne-taskschd-task_instances_policy)
separately defines parallel, queue, ignore-new, and stop-existing. Those pages
do not specify our latest-candidate supersession rule. We choose that rule
explicitly and do not copy Windows' documented default ten-minute delay.

Occurrence identity is the schedule identity plus the resolved UTC instant.
Clock rollback cannot admit an already-consumed instant again. Use a monotonic
clock for sleeping, and a wall clock for calendar eligibility. A forward jump
applies lateness policy, not unconditional replay.

Use a library for calendar calculation. Decided: croniter behind a thin
next/previous-occurrence adapter (`next_after` / `prev_before` over
schedule-zone instants, UTC at the boundary), without adopting any job
store or executor. Probes (see the implementation plan) disqualified
APScheduler 3.x triggers: no bounded latest-missed seam, phantom wall
times across DST gaps, and replayed UTC minutes after fall folds. The
adapter converts the query instant into the schedule's named zone, asks
croniter for the next/previous occurrence, and converts the result to
UTC. It applies no calendar correction of its own: croniter owns DST
resolution, including nonexistent and repeated local times.

Probed and pinned on Python 3.14 (`croniter==6.2.4`, `tzdata` floor;
`python-dateutil` only transitive): some nonexistent scheduled wall
times resolve forward — daily 02:30 on the spring-gap day resolves to
03:00-04:00 the same day — and repeated times surface as distinct UTC
occurrences (both 01:30s on the fall-fold day). Occurrence identity is
`(schedule_id, resolved UTC instant)`. The definition's cron
expression, time-zone name, and admission snapshots are retained, but
the platform claims no finer intended-wall-time provenance than the
library supplies. Iteration is exclusive in both directions (a query
from exactly a due instant returns the neighboring occurrence), so the
adapter queries forward from the last-consumed instant and compares
catch-up results against the same watermark: a due occurrence is
admitted exactly once. Impossible schedules surface promptly as a
documented library exhaustion error, never as silent no-progress;
search exhaustion or backward/non-progressing results fail visibly.
Five-field cron scope; the adapter itself rejects invalid zones and
naive timestamps (the library does not reject naive). Re-probe on any
calendar-dependency upgrade; the strict failure pins guard the
APScheduler behaviors we rejected.

The adapter uses Unix cron dialect explicitly: numeric `0` and `7`
both mean Sunday (not Monday-first), and day-of-month/day-of-week
matching uses croniter's standard `day_or=True` (Unix OR), exposed
explicitly rather than as an undocumented promise. No jitter or
extended trigger combinations in this slice.

Alternatives considered: APScheduler 3.x triggers were probed and rejected
(bounded latest-missed lookup impossible, DST-gap phantoms, fold replay);
a full scheduling framework owns useful job machinery but would create a
second persistence/execution lifecycle alongside our run API. Prefer
croniter iteration plus our existing run lifecycle.

Sources checked on 2026-09-08:

- [APScheduler 3.x cron trigger](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/cron.html)
- [APScheduler date trigger](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/date.html)
- [croniter project documentation](https://pypi.org/project/croniter/)

## Input authoring and serialization

A schedule stores instructions for building a future workflow input object.
The deployment-run API still receives resolved data, not expression objects.
Do not add parallel raw-input and occurrence-binding mechanisms.

Reuse literal, object, array, target-path, strict-JSON, and expression-budget
semantics from the existing input-binding system. Add a typed occurrence
reference for the schedule environment; graph references to input/state/context
are invalid here. Do not extend GraphSourcePath with schedule-only roots.
Existing expressions compose data; they do not implement date formatting,
arithmetic, template evaluation, Python execution, or arbitrary transforms.

Proposed persisted binding example, pending concrete model names:

```json
{
  "input_bindings": [
    {"target": "team", "value": "engineering"},
    {
      "target": "report_time",
      "expression": {"kind": "occurrence", "field": "scheduled_at"}
    }
  ]
}
```

Occurrence references initially expose schedule_id, occurrence_id, and
scheduled_at. Date-time values serialize as UTC RFC 3339 strings. The admitted
run's resolved input is persisted once and never re-evaluated on restart.
Validate target conflicts, expression bounds, source fields, and the resulting
workflow input schema. Recheck the current deployment contract at admission;
an edit may have changed the expected input since schedule creation.

Extract the shared composition traversal and limits behind a typed source
resolver seam. Keep graph and schedule source models distinct. Do not copy
the recursive evaluator into a second package or use fake graph context to
smuggle occurrence values into graph-path evaluation.

Generic host Runtime[ContextT] remains a separate future feature. Schedule
provenance is persisted platform metadata, not an arbitrary host object and
not a new graph-visible context namespace. Child workflows receive business
values through their declared input bindings as before.

## Durable admission and recovery

Current seams needing change:

- wf_api/runs.py executes before persisting a stopped run.
- wf_artifacts/runs/models.py permits only stopped summaries, with a required
  checkpoint identifier.
- wf_api/run_lifecycle.py assumes an existing checkpoint when updating a run.

Introduce a durable admission representation with a preassigned run identity.
Run inspection must distinguish an admitted/in-flight run from a stopped run
with a checkpoint. Never fabricate a completed checkpoint, trace, output, or
successful step count for work whose outcome is unknown.

Required ordering under the single-owner admission lock:

1. Recheck schedule revision, enabled status, due time, capacity, and overlap.
2. Allocate occurrence/run identities and freeze the invocation data.
3. Atomically persist the authoritative admission record before dispatch.
4. Materialize the run admission view using that same identity.
5. Dispatch the captured invocation without resolving the deployment again.
6. Persist a stopped checkpoint and summary, then reconcile occurrence status.

The admission record is the recovery authority for partial multi-file writes;
atomic rename is not a transaction across files. A failed durable admission
must never dispatch. Reconciliation is idempotent: it completes missing views,
recognizes durable stopped results, or marks abandoned work failed without
redispatch. A persisted interruption must win over stale in-flight metadata.
Corrupt or contradictory records fail closed with diagnostics; do not silently
discard them to clear overlap.

Resume of a scheduled interrupted run must mark its active attempt durably
before executing again. Otherwise a crash during resume could leave an old
interrupted checkpoint looking safe to retry. The mark carries a
store-backed attempt identity, and every stopped result the attempt
produces echoes that identity back. Recovery matches result to active
attempt: a result belonging to the active attempt is fresh and resumable,
anything else under an active attempt is stale and fails closed without
retry. Recovery must distinguish both cases from an interruption that was
merely waiting across server restart.

The process must own the store exclusively before recovery. Enforce scheduler
ownership with a held cross-process lock, not a stale PID file or a lease that
can expire while the old owner still runs. Unsupported locking must reject
scheduler startup. Other processes mutating/resuming the same store remain
unsupported. This does not upgrade the rest of the file stores to multi-writer
safety or claim exactly-once external effects.

## Lifecycle, administration, and resource bounds

Expose create/get/list/update/pause/resume/delete and paginated occurrence
inspection through the workflow API and Python client. Public client names
are finalized in the implementation plan, not treated as existing methods.
Reject stale schedule edits using revisions within the owning process.

Occurrence inspection distinguishes pending, coalesced/superseded,
skipped-overlap, skipped-misfire,
preflight rejection, admitted/running, interrupted, completed, and failed.
These are platform occurrence states, not business outcomes or core node
outcomes. Preflight rejection does not invent a run that never started.

Store resolved input and pinned environment with admitted runs. Inspection
includes resolved occurrence instant (UTC), admission time, actual start
when known, schedule
revision, run identity, and failure/skip reason. Preserve existing run limits;
allow a schedule-specific max_steps using the same validation as manual runs.

Bound active scheduled execution tasks and work per polling batch. Do not
block calendar polling on a long run or create unbounded pending tasks.
Capacity-delayed `skip` occurrences expire at their lateness deadline; `latest`
retains at most one candidate as specified above. Interrupted runs consume
per-schedule active-run slots but no executing-task slot while waiting.
Resumption must acquire a server execution slot before dispatch. Lowering a
schedule's active limit never cancels existing runs; block new admission until
the count drops below the new limit. Polling must be fair across schedules so
a frequently due schedule cannot monopolize available capacity. The concrete
server capacity default is deployment configuration, with deterministic tests
using a small injected limit.

On shutdown stop admission first and drain active tasks within a configured
grace period. Record cancellation/failure when possible; abrupt termination
uses startup recovery. Paused/deleted schedule definitions must not prevent
run completion or resume from updating retained occurrence history.

## Verification gates

Use injected clocks and controlled executors, not real-time sleeps:

- Cron parsing, time zones, invalid syntax, leap/calendar boundaries, DST gaps
  and folds, UTC uniqueness, impossible schedules, and bounded next-time search.
- One-shot success/exhaustion/catch-up; lateness boundary; startup behavior
  under both policies; explicit pause/resume exclusion; long downtime without
  unbounded enumeration; clock rollback/forward jumps.
- Latest-only coalescing across multiple missed times, candidate persistence,
  supersession at the next due instant, no double admission at that boundary,
  and no resurrection after overlap skips or admission.
- Literal and nested occurrence expressions round-trip through JSON; missing
  fields, graph-only paths, excessive trees, conflicting targets, and invalid
  resolved workflow input fail before dispatch.
- Same-schedule overlap across running and interrupted states; manual/other
  schedule independence; release on failure/completion and on resumed completion.
- Parallel admission limits, interrupted slot accounting, capacity-delayed
  candidates, fair polling, limit edits, and bounded task allocation. Exercise
  all four overlap/misfire combinations with controlled executors.
- Edit/repoint/delete races at admission; captured invocation stays unchanged.
- Fault injection before/after every persistence boundary, including resume:
  no dispatch before durable admission and no replay after ambiguous execution.
- Recovery preserves stopped interruptions, reconciles terminal results, marks
  abandoned attempts failed, and leaves corrupt records visibly blocked.
- Second-owner rejection, lock release on process death, bounded concurrency,
  shutdown drain, and full occurrence history after schedule deletion.
- Public API/client round trips, pagination, inspection without a checkpoint,
  and existing manual run/resume behavior remain valid.

## Review before implementation planning

User policy decisions above are settled. Review the proposed 60-second
allowance, expression seam, and admission/recovery representation together.
The calendar-library probe is an explicit gate, not a claimed passing test.
After approval, create a sequenced implementation plan with fault-injection
tests before enabling scheduling in the server. WaitNode is a later contract
that may reuse timed admission but must persist its own suspended execution.