sched: fence scheduled resumes at shutdown so none outlive ownership

This commit is contained in:
lda
2026-09-09 18:08:20 +07:00 Verified
parent 99c038a04f
commit 82dc5a80ae
6 changed files with 440 additions and 32 deletions
@@ -310,7 +310,13 @@ server capacity default is deployment configuration, with deterministic tests
using a small injected limit.
On shutdown stop admission first and drain active tasks within a configured
grace period. Record cancellation/failure when possible; abrupt termination
grace period. New scheduled resumes are rejected once shutdown begins
(no dispatch, no ungated fallback); in-flight scheduled resumes share
the grace window, then are cancelled and joined before ownership is
released, so no old execution persists after a new owner takes over.
A cancelled resume keeps its executing mark and ACTIVE attempt, and
startup recovery fails it closed exactly like a crash mid-resume.
Record cancellation/failure when possible; abrupt termination
uses startup recovery. Paused/deleted schedule definitions must not prevent
run completion or resume from updating retained occurrence history.
@@ -321,9 +327,11 @@ canonical ownership, recovers without executing, then ticks calendar
polling without blocking on long workflows: each dispatch spawns exactly
one bounded execution task behind the async-completion seam, and the
scheduler's own capacity gate is the execution-slot bound (an executing
run keeps its slot until it stops). Shutdown stops admission, drains
within `drain_grace_s`, leaves unfinished work under its executing mark
for startup recovery to abandon truthfully, and releases ownership last.
run keeps its slot until it stops). Shutdown stops admission, rejects
new scheduled resumes for the duration of the drain, cancels and joins
unfinished scheduled resumes after `drain_grace_s` (cancelled work keeps
its executing mark and ACTIVE attempt for startup recovery to abandon
truthfully), and releases ownership last.
A failed startup releases the lock and raises.
Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
@@ -344,7 +352,9 @@ resumed through the run API acquires a server execution slot through the
scheduler's own accounting before dispatch — rejection leaves no resume
attempt behind — holds the durable executing mark for the re-execution
(visible to capacity and drain like any live execution), and releases
the slot when its stopped result is persisted. A resumed scheduled run
the slot when its stopped result is persisted. Shutdown drain rejects
further scheduled resumes; a resume cancelled by the drain keeps its
marks for recovery instead of releasing them. A resumed scheduled run
reconciles its terminal history live through the same idempotent
recording as dispatch; restart recovery still repairs torn boundaries.