# Scheduler Foundation Before Concurrent Foreach

Status: accepted

We will introduce an explicit scheduler foundation in `wf_core` before enabling
concurrent foreach. The first implementation preserves deterministic serial
behavior while adding ready/block/wake semantics, because concurrent foreach and
native subgraphs both require multiple runnable frames without treating
`current_frame_id` as the whole runtime state.

## Context

The current runtime behaves like a stack cursor. `RunState.current_frame_id` and
`current_node_id` identify the active frame, and foreach creates one child frame
at a time before control collapses back to the parent. This works for serial
execution, but it does not give the runtime a clear way to represent multiple
runnable frames, blocked parent frames, interrupt resume priority, or future
concurrent branch merge semantics.

Concurrent foreach is not just `asyncio.gather` over node calls. It changes state
visibility, error handling, interrupt behavior, trace ordering, reducer rules,
and parent completion semantics. Enabling it before those concepts exist would
make the runtime fragile.

## Decision

Add an internal scheduler foundation first. The first pass will keep serial
workflow behavior intact and will not enable `foreach(mode="concurrent")`.

The scheduler model is:

- `RunState.frames` is the frame set and source of truth for live runtime
  cursors.
- `RunState.ready_frame_ids` is an explicit FIFO ready queue and is serialized
  with run state.
- `current_frame_id` and `current_node_id` remain compatibility fields for the
  selected cursor, not the source of all runnable work.
- A selected frame is removed from the ready queue, marked `RUNNING`, and stepped
  once.
- A normal non-terminal step advance marks the same frame `PENDING` and
  re-enqueues it.
- Terminal, blocked, interrupted, and failed frames are not re-enqueued.
- `FrameStatus.BLOCKED` represents a live frame waiting on child completion or a
  future internal event.
- Blocked frames carry typed block-reason metadata so wakeups and deadlocks are
  explainable.
- Scheduler helpers live in an internal runtime scheduler module, not public
  `wf_core` exports.
- Sync and async runtime loops use the same scheduler rules; only node handler
  execution differs.

Serial foreach keeps its current behavior through clearer runtime mechanics:
the foreach parent blocks on one iteration child, the child runs until
completion/interruption/failure, and child completion wakes the parent to create
the next item or finish.

## Rejected Alternatives

**Patch concurrent execution directly into foreach.**
This would keep the stack-cursor model and force foreach to own scheduling,
interrupt, merge, and failure policy. It would not help native subgraphs.

**Infer runnable frames by scanning `RunState.frames`.**
This hides scheduling order in dictionary iteration, cleanup behavior, and frame
creation side effects. A ready queue makes the policy testable and preserves
resume/checkpoint order.

**Treat `RUNNING` as async-in-flight immediately.**
The first pass is deterministic and serial. `RUNNING` means the scheduler has
selected the frame for the current step. Actual async task orchestration can
refine this later.

**Upgrade `JoinNode` into a real merge barrier now.**
The future barrier semantics are larger than the current join node. `JoinNode`
may be repurposed later, but not silently in this scheduler pass.

## Future Concurrent Semantics

Concurrent foreach and graph-level convergence require additional semantics before
they are enabled:

- A future foreach policy must define admission limits, item failure handling,
  and quiescence behavior.
- Concurrent foreach should be bounded by default so workflows do not spawn
  unbounded MCP, HTTP, browser, or external service calls.
- A future barrier must wait for multiple child or upstream frames and merge
  their lineage patches.
- Missing reducers mean replace only within one serial lineage. At a barrier,
  multiple writes to the same path without a reducer are conflicts.
- Future concurrent child frames should produce state patches that the scheduler
  commits atomically.
- Concurrent branches require lineage isolation: a child frame sees parent-visible
  state plus its own ancestor writes, not sibling branch writes.
- Public trace remains append-only chronological execution history. Scheduler
  block/wake events are not added to public trace in this pass.

## Interrupt Semantics

A run has at most one outstanding interrupt. If any frame interrupts, the whole
run pauses and scheduling stops. Only the requesting frame becomes
`INTERRUPTED`; ancestors waiting on that frame remain `BLOCKED`.

Resume wakes the interrupted frame, clears the run interrupt, and places the
resumed frame at the front of the ready queue. Ready sibling frames are preserved
but do not run while the interrupt is outstanding.

Future concurrent execution should not assume in-flight node calls can be safely
cancelled. If one frame interrupts while sibling jobs are already started, the
runtime should stop scheduling new work, let started jobs drain to pending
results, and defer sibling state commits until resume/commit policy decides what
becomes visible. User-facing control should return only at a quiescent pause
point: no new work is scheduled, already-started jobs have drained, and their
results have been captured without unsafe commits. The first scheduler pass has
no simultaneous node jobs, so an interrupt still stops scheduling immediately.

## Failure Semantics

A runtime failure is distinct from a node returning an `error` outcome. Outcomes
are graph control flow. Runtime failures stop scheduling unless a future policy
explicitly handles them.

In the first scheduler pass, any runtime failure fails the whole run. Future
concurrent foreach policies may support `collect` or `skip`, but those are policy
features and should not be treated as implemented until runtime support exists.

## Compatibility Requirements

The first scheduler pass should be additive and regression-safe:

- Existing valid serial workflows keep the same final state, output, node
  execution order, foreach item count, interrupt resume behavior, and meaningful
  trace order.
- `RunState.to_dict()` may grow `ready_frame_ids`, but existing run fields and
  trace entry shape should not churn.
- Completed frames remain in the frame set for the lifetime of an in-memory run.
  Persistence can add explicit compaction later.
- Frame identifiers remain strings for now, but construction should be
  centralized so structural frame IDs can replace the string format later.

## Test Boundary

The scheduler foundation is not complete without tests for:

- root frame initialization in the ready queue
- selecting a frame pops it and marks it `RUNNING`
- normal step advance re-enqueues the same frame
- duplicate frame creation rejection
- duplicate enqueue prevention and priority move-to-front
- blocked frames not being selectable
- empty ready queue resolution into completed, interrupted, failed, or deadlock
- serial foreach regression and parent block/child wake behavior
- interrupt resume priority and single outstanding interrupt behavior