sched: fence scheduled resumes at shutdown so none outlive ownership

This commit is contained in:
lda
2026-09-09 18:08:20 +07:00 Verified
parent 99c038a04f
commit 82dc5a80ae
6 changed files with 440 additions and 32 deletions
+8 -4
View File
@@ -96,8 +96,11 @@ retrying it. Corrupt or contradictory records fail closed with
diagnostics and block the schedule rather than clearing overlap.
On shutdown the server stops admission first and drains active tasks
within the grace period; anything still running keeps its executing
mark, and the next startup recovery abandons it truthfully.
within the grace period; new scheduled resumes are rejected for the
duration of the drain, and a resume still running past the deadline is
cancelled and joined before ownership is released, keeping its marks
for the next startup recovery. Anything else still running keeps its
executing mark, and the next startup recovery abandons it truthfully.
## Occurrence inspection
@@ -194,8 +197,9 @@ await schedules.update_schedule(
- Manual runs bypass scheduler capacity by design; capacity governs
scheduled dispatch plus scheduled resumes. Resuming a scheduled
interrupted run acquires a server execution slot first (rejected
while saturated, without dispatching) and releases it when the
resumed result is persisted.
while saturated or draining, without dispatching) and releases it when the
resumed result is persisted; a drain-cancelled resume keeps its marks
for recovery instead.
- A set `max_steps` budget cannot be cleared back to unset through
update (recreate the schedule for an unbounded budget).
- MCP-backed servers reject scheduler enablement for now.
@@ -310,7 +310,13 @@ server capacity default is deployment configuration, with deterministic tests
using a small injected limit.
On shutdown stop admission first and drain active tasks within a configured
grace period. Record cancellation/failure when possible; abrupt termination
grace period. New scheduled resumes are rejected once shutdown begins
(no dispatch, no ungated fallback); in-flight scheduled resumes share
the grace window, then are cancelled and joined before ownership is
released, so no old execution persists after a new owner takes over.
A cancelled resume keeps its executing mark and ACTIVE attempt, and
startup recovery fails it closed exactly like a crash mid-resume.
Record cancellation/failure when possible; abrupt termination
uses startup recovery. Paused/deleted schedule definitions must not prevent
run completion or resume from updating retained occurrence history.
@@ -321,9 +327,11 @@ canonical ownership, recovers without executing, then ticks calendar
polling without blocking on long workflows: each dispatch spawns exactly
one bounded execution task behind the async-completion seam, and the
scheduler's own capacity gate is the execution-slot bound (an executing
run keeps its slot until it stops). Shutdown stops admission, drains
within `drain_grace_s`, leaves unfinished work under its executing mark
for startup recovery to abandon truthfully, and releases ownership last.
run keeps its slot until it stops). Shutdown stops admission, rejects
new scheduled resumes for the duration of the drain, cancels and joins
unfinished scheduled resumes after `drain_grace_s` (cancelled work keeps
its executing mark and ACTIVE attempt for startup recovery to abandon
truthfully), and releases ownership last.
A failed startup releases the lock and raises.
Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
@@ -344,7 +352,9 @@ resumed through the run API acquires a server execution slot through the
scheduler's own accounting before dispatch — rejection leaves no resume
attempt behind — holds the durable executing mark for the re-execution
(visible to capacity and drain like any live execution), and releases
the slot when its stopped result is persisted. A resumed scheduled run
the slot when its stopped result is persisted. Shutdown drain rejects
further scheduled resumes; a resume cancelled by the drain keeps its
marks for recovery instead of releasing them. A resumed scheduled run
reconciles its terminal history live through the same idempotent
recording as dispatch; restart recovery still repairs torn boundaries.