153 lines
7.1 KiB
Markdown
153 lines
7.1 KiB
Markdown
# Scheduler Foundation Before Concurrent Foreach
|
|
|
|
Status: accepted
|
|
|
|
We will introduce an explicit scheduler foundation in `wf_core` before enabling
|
|
concurrent foreach. The first implementation preserves deterministic serial
|
|
behavior while adding ready/block/wake semantics, because concurrent foreach and
|
|
native subgraphs both require multiple runnable frames without treating
|
|
`current_frame_id` as the whole runtime state.
|
|
|
|
## Context
|
|
|
|
The current runtime behaves like a stack cursor. `RunState.current_frame_id` and
|
|
`current_node_id` identify the active frame, and foreach creates one child frame
|
|
at a time before control collapses back to the parent. This works for serial
|
|
execution, but it does not give the runtime a clear way to represent multiple
|
|
runnable frames, blocked parent frames, interrupt resume priority, or future
|
|
concurrent branch merge semantics.
|
|
|
|
Concurrent foreach is not just `asyncio.gather` over node calls. It changes state
|
|
visibility, error handling, interrupt behavior, trace ordering, reducer rules,
|
|
and parent completion semantics. Enabling it before those concepts exist would
|
|
make the runtime fragile.
|
|
|
|
## Decision
|
|
|
|
Add an internal scheduler foundation first. The first pass will keep serial
|
|
workflow behavior intact and will not enable `foreach(mode="concurrent")`.
|
|
|
|
The scheduler model is:
|
|
|
|
- `RunState.frames` is the frame set and source of truth for live runtime
|
|
cursors.
|
|
- `RunState.ready_frame_ids` is an explicit FIFO ready queue and is serialized
|
|
with run state.
|
|
- `current_frame_id` and `current_node_id` remain compatibility fields for the
|
|
selected cursor, not the source of all runnable work.
|
|
- A selected frame is removed from the ready queue, marked `RUNNING`, and stepped
|
|
once.
|
|
- A normal non-terminal step advance marks the same frame `PENDING` and
|
|
re-enqueues it.
|
|
- Terminal, blocked, interrupted, and failed frames are not re-enqueued.
|
|
- `FrameStatus.BLOCKED` represents a live frame waiting on child completion or a
|
|
future internal event.
|
|
- Blocked frames carry typed block-reason metadata so wakeups and deadlocks are
|
|
explainable.
|
|
- Scheduler helpers live in an internal runtime scheduler module, not public
|
|
`wf_core` exports.
|
|
- Sync and async runtime loops use the same scheduler rules; only node handler
|
|
execution differs.
|
|
|
|
Serial foreach keeps its current behavior through clearer runtime mechanics:
|
|
the foreach parent blocks on one iteration child, the child runs until
|
|
completion/interruption/failure, and child completion wakes the parent to create
|
|
the next item or finish.
|
|
|
|
## Rejected Alternatives
|
|
|
|
**Patch concurrent execution directly into foreach.**
|
|
This would keep the stack-cursor model and force foreach to own scheduling,
|
|
interrupt, merge, and failure policy. It would not help native subgraphs.
|
|
|
|
**Infer runnable frames by scanning `RunState.frames`.**
|
|
This hides scheduling order in dictionary iteration, cleanup behavior, and frame
|
|
creation side effects. A ready queue makes the policy testable and preserves
|
|
resume/checkpoint order.
|
|
|
|
**Treat `RUNNING` as async-in-flight immediately.**
|
|
The first pass is deterministic and serial. `RUNNING` means the scheduler has
|
|
selected the frame for the current step. Actual async task orchestration can
|
|
refine this later.
|
|
|
|
**Upgrade `JoinNode` into a real merge barrier now.**
|
|
The future barrier semantics are larger than the current join node. `JoinNode`
|
|
may be repurposed later, but not silently in this scheduler pass.
|
|
|
|
## Future Concurrent Semantics
|
|
|
|
Concurrent foreach and graph-level convergence require additional semantics before
|
|
they are enabled:
|
|
|
|
- A future foreach policy must define admission limits, item failure handling,
|
|
and quiescence behavior.
|
|
- Concurrent foreach should be bounded by default so workflows do not spawn
|
|
unbounded MCP, HTTP, browser, or external service calls.
|
|
- A future barrier must wait for multiple child or upstream frames and merge
|
|
their lineage patches.
|
|
- Missing reducers mean replace only within one serial lineage. At a barrier,
|
|
multiple writes to the same path without a reducer are conflicts.
|
|
- Future concurrent child frames should produce state patches that the scheduler
|
|
commits atomically.
|
|
- Concurrent branches require lineage isolation: a child frame sees parent-visible
|
|
state plus its own ancestor writes, not sibling branch writes.
|
|
- Public trace remains append-only chronological execution history. Scheduler
|
|
block/wake events are not added to public trace in this pass.
|
|
|
|
## Interrupt Semantics
|
|
|
|
A run has at most one outstanding interrupt. If any frame interrupts, the whole
|
|
run pauses and scheduling stops. Only the requesting frame becomes
|
|
`INTERRUPTED`; ancestors waiting on that frame remain `BLOCKED`.
|
|
|
|
Resume wakes the interrupted frame, clears the run interrupt, and places the
|
|
resumed frame at the front of the ready queue. Ready sibling frames are preserved
|
|
but do not run while the interrupt is outstanding.
|
|
|
|
Future concurrent execution should not assume in-flight node calls can be safely
|
|
cancelled. If one frame interrupts while sibling jobs are already started, the
|
|
runtime should stop scheduling new work, let started jobs drain to pending
|
|
results, and defer sibling state commits until resume/commit policy decides what
|
|
becomes visible. User-facing control should return only at a quiescent pause
|
|
point: no new work is scheduled, already-started jobs have drained, and their
|
|
results have been captured without unsafe commits. The first scheduler pass has
|
|
no simultaneous node jobs, so an interrupt still stops scheduling immediately.
|
|
|
|
## Failure Semantics
|
|
|
|
A runtime failure is distinct from a node returning an `error` outcome. Outcomes
|
|
are graph control flow. Runtime failures stop scheduling unless a future policy
|
|
explicitly handles them.
|
|
|
|
In the first scheduler pass, any runtime failure fails the whole run. Future
|
|
concurrent foreach policies may support `collect` or `skip`, but those are policy
|
|
features and should not be treated as implemented until runtime support exists.
|
|
|
|
## Compatibility Requirements
|
|
|
|
The first scheduler pass should be additive and regression-safe:
|
|
|
|
- Existing valid serial workflows keep the same final state, output, node
|
|
execution order, foreach item count, interrupt resume behavior, and meaningful
|
|
trace order.
|
|
- `RunState.to_dict()` may grow `ready_frame_ids`, but existing run fields and
|
|
trace entry shape should not churn.
|
|
- Completed frames remain in the frame set for the lifetime of an in-memory run.
|
|
Persistence can add explicit compaction later.
|
|
- Frame identifiers remain strings for now, but construction should be
|
|
centralized so structural frame IDs can replace the string format later.
|
|
|
|
## Test Boundary
|
|
|
|
The scheduler foundation is not complete without tests for:
|
|
|
|
- root frame initialization in the ready queue
|
|
- selecting a frame pops it and marks it `RUNNING`
|
|
- normal step advance re-enqueues the same frame
|
|
- duplicate frame creation rejection
|
|
- duplicate enqueue prevention and priority move-to-front
|
|
- blocked frames not being selectable
|
|
- empty ready queue resolution into completed, interrupted, failed, or deadlock
|
|
- serial foreach regression and parent block/child wake behavior
|
|
- interrupt resume priority and single outstanding interrupt behavior
|