sched: fence scheduled resumes at shutdown so none outlive ownership
This commit is contained in:
@@ -96,8 +96,11 @@ retrying it. Corrupt or contradictory records fail closed with
|
||||
diagnostics and block the schedule rather than clearing overlap.
|
||||
|
||||
On shutdown the server stops admission first and drains active tasks
|
||||
within the grace period; anything still running keeps its executing
|
||||
mark, and the next startup recovery abandons it truthfully.
|
||||
within the grace period; new scheduled resumes are rejected for the
|
||||
duration of the drain, and a resume still running past the deadline is
|
||||
cancelled and joined before ownership is released, keeping its marks
|
||||
for the next startup recovery. Anything else still running keeps its
|
||||
executing mark, and the next startup recovery abandons it truthfully.
|
||||
|
||||
## Occurrence inspection
|
||||
|
||||
@@ -194,8 +197,9 @@ await schedules.update_schedule(
|
||||
- Manual runs bypass scheduler capacity by design; capacity governs
|
||||
scheduled dispatch plus scheduled resumes. Resuming a scheduled
|
||||
interrupted run acquires a server execution slot first (rejected
|
||||
while saturated, without dispatching) and releases it when the
|
||||
resumed result is persisted.
|
||||
while saturated or draining, without dispatching) and releases it when the
|
||||
resumed result is persisted; a drain-cancelled resume keeps its marks
|
||||
for recovery instead.
|
||||
- A set `max_steps` budget cannot be cleared back to unset through
|
||||
update (recreate the schedule for an unbounded budget).
|
||||
- MCP-backed servers reject scheduler enablement for now.
|
||||
|
||||
@@ -310,7 +310,13 @@ server capacity default is deployment configuration, with deterministic tests
|
||||
using a small injected limit.
|
||||
|
||||
On shutdown stop admission first and drain active tasks within a configured
|
||||
grace period. Record cancellation/failure when possible; abrupt termination
|
||||
grace period. New scheduled resumes are rejected once shutdown begins
|
||||
(no dispatch, no ungated fallback); in-flight scheduled resumes share
|
||||
the grace window, then are cancelled and joined before ownership is
|
||||
released, so no old execution persists after a new owner takes over.
|
||||
A cancelled resume keeps its executing mark and ACTIVE attempt, and
|
||||
startup recovery fails it closed exactly like a crash mid-resume.
|
||||
Record cancellation/failure when possible; abrupt termination
|
||||
uses startup recovery. Paused/deleted schedule definitions must not prevent
|
||||
run completion or resume from updating retained occurrence history.
|
||||
|
||||
@@ -321,9 +327,11 @@ canonical ownership, recovers without executing, then ticks calendar
|
||||
polling without blocking on long workflows: each dispatch spawns exactly
|
||||
one bounded execution task behind the async-completion seam, and the
|
||||
scheduler's own capacity gate is the execution-slot bound (an executing
|
||||
run keeps its slot until it stops). Shutdown stops admission, drains
|
||||
within `drain_grace_s`, leaves unfinished work under its executing mark
|
||||
for startup recovery to abandon truthfully, and releases ownership last.
|
||||
run keeps its slot until it stops). Shutdown stops admission, rejects
|
||||
new scheduled resumes for the duration of the drain, cancels and joins
|
||||
unfinished scheduled resumes after `drain_grace_s` (cancelled work keeps
|
||||
its executing mark and ACTIVE attempt for startup recovery to abandon
|
||||
truthfully), and releases ownership last.
|
||||
A failed startup releases the lock and raises.
|
||||
|
||||
Administration (`WorkflowApi` schedules methods, `workflow.schedules.*`
|
||||
@@ -344,7 +352,9 @@ resumed through the run API acquires a server execution slot through the
|
||||
scheduler's own accounting before dispatch — rejection leaves no resume
|
||||
attempt behind — holds the durable executing mark for the re-execution
|
||||
(visible to capacity and drain like any live execution), and releases
|
||||
the slot when its stopped result is persisted. A resumed scheduled run
|
||||
the slot when its stopped result is persisted. Shutdown drain rejects
|
||||
further scheduled resumes; a resume cancelled by the drain keeps its
|
||||
marks for recovery instead of releasing them. A resumed scheduled run
|
||||
reconciles its terminal history live through the same idempotent
|
||||
recording as dispatch; restart recovery still repairs torn boundaries.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user