lda-wf/docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md

# Concurrent Foreach Policy and Barrier Commits

Status: accepted

Concurrent foreach will use bounded scheduling, lineage-local pending state, and
barrier-buffered commits. This keeps item execution deterministic, resumable,
and safe around reducers, interrupts, and external tool calls.

## Context

The scheduler foundation makes multiple runnable frames possible, but concurrent
foreach needs more than multiple node calls. Each item lineage may execute
several nodes, read its own pending writes, fail independently, block, or
interrupt the whole run. Sibling item writes must not leak into each other, and
state commits need deterministic merge behavior.

## Decision

Concurrent foreach is a workflow mode and must be bounded by policy.

The future foreach model should separate item error behavior from concurrency
behavior:

- `item_error` is foreach-wide and applies to serial and concurrent foreach.
- `concurrent` is a nested policy object used only when
  `mode="concurrent"`.
- `concurrent.max_active` defaults to `4`.
- `concurrent.max_outstanding` defaults to `20`.
- Validation requires `max_outstanding >= max_active`.
- Ready or running item frames consume active capacity.
- Blocked item frames consume outstanding capacity but not active capacity.

`foreach(mode="concurrent")` is executable by the sync runtime as deterministic
frame interleaving: one admitted node handler call at a time. The async runtime
may additionally run admitted async node handler calls simultaneously. Sync
handlers are not pushed into thread/process parallelism by default.

Current async execution batches ready concurrent-foreach item frames that are
about to execute node handlers for the same foreach barrier. Input resolution
happens before each handler is awaited, handler awaits may overlap, and state
patch finalization/tracing happens sequentially afterward. This keeps `RunState`
mutation deterministic while allowing async I/O overlap.

## Item Error Policy

Item error policy handles runtime failures inside an item frame, not normal graph
control-flow outcomes. A node returning an `error` outcome is still ordinary
graph routing if the graph declares that edge.

Supported policy actions:

- `fail`: stop scheduling new items, drain already-started jobs to a quiescent
  point, capture results for trace/observability, then fail the run.
- `skip`: mark the failed item frame failed, record no hidden state, continue,
  and emit `completed_with_errors` if any item was skipped.
- `collect`: mark the failed item frame failed, collect a structured item error
  record, continue, and emit `completed_with_errors` if any item was collected.

`collect` must declare an explicit state array destination. The foreach barrier
writes the full ordered error list once. On clean completion, `collect` writes an
empty list and emits `done`.

Current sync concurrent foreach supports `skip` and `collect`. Item node
exceptions under those policies mark the child item frame `FAILED`, store a
pending failed item result on the parent barrier, and wake the parent to refill
or finish. The failed frame stays visible for observability; the parent foreach
is responsible for turning handled item failures into aggregate control flow.

Collected error records include item index, frame id, failing node id, error
type/message, and the item value when it can be represented safely.

## Aggregate Outcomes

Foreach aggregate outcomes are policy-derived:

- `done`: all items completed cleanly.
- `completed_with_errors`: all items completed but one or more item failures
  were handled by `skip` or `collect`.

Validation should require a routable `completed_with_errors` edge whenever the
policy can emit it. Users may route it to the same target as `done` explicitly.

Aggregate result vocabulary should be shared with future barrier-like graph
steps. `fail` is a policy action; if a future aggregate node emits graph-level
failure as control flow, the outcome should be `failed`.

## Barrier-Buffered Commits

Concurrent foreach item writes are buffered as pending state patches/results, not
committed directly to `RunState.state`.

The state model is:

- `RunState.state` remains committed parent/global state.
- Each item lineage reads a visible state view built from committed state plus
  its own lineage-local patch overlay.
- Sibling item patches are invisible to each other.
- Pending barrier results live in resumable run state/frame metadata, not trace.
- Trace `state_changes` means committed state changes only.

The barrier commits item results in deterministic item-index order by default.
Completion-order merging may exist later as an explicit barrier merge policy,
not reducer behavior.

Patch creation and commit must extract/reuse the existing node output
validation, output binding, and reducer logic. Concurrent foreach must not
create a second write system.

Current sync execution supports item-local read overlays for concurrent foreach
item frames. `RunState.state` remains committed parent state, while
`state_view_for_frame` overlays the current item's buffered writes for reads by
later nodes in the same item lineage. Sibling overlays remain invisible until
the foreach barrier commits.

## Merge and Reducer Rules

At a barrier, missing reducer means default replace only for single-writer
paths. Multiple sibling lineages writing the same state path require a reducer
whose metadata declares it `mergeable`. `replace` is an explicit reducer, but it
is `exclusive`, so it is rejected for same-path sibling writes. Ancestor/
descendant overlapping writes across lineages are conflicts unless an explicit
future merge strategy covers them.

Reducers apply incrementally in deterministic lineage order. For foreach, that
means item index order.

Current barrier validation enforces this policy for sibling foreach item
lineages. Same-path sibling writes require a `mergeable` reducer on the exact
destination state path. Ancestor/descendant sibling writes are rejected until a
future explicit deep merge policy exists.

## Interrupt and Failure Quiescence

Concurrent execution should not assume in-flight node calls can be safely
cancelled.

If an interrupt or fail policy trips while sibling jobs are already started, the
runtime should:

1. stop scheduling new work
2. let already-started jobs drain to a quiescent point
3. capture their results safely
4. defer or discard commits according to the policy boundary
5. return control only after no hidden background jobs continue mutating state

For `fail`, drained sibling results are for observability/cleanup only and
should not commit normal state progress after the failure boundary.

Current async concurrent foreach implements the interrupt part of this by
prioritizing item frames that route into an `InterruptNode`. If a batched async
node result sends one item to an interrupt while a sibling completes, the
interrupt-bound frame is placed at the front of the ready queue before the
parent foreach can refill capacity. Already-started async handler calls from
the batch are awaited first, then their results are finalized sequentially.
The foreach barrier does not commit while an item frame remains interrupted.

## Capacity and Runtime Limits

Foreach capacity is local correctness policy, not total process protection.
Future runtime should also support a basic global node-call budget in `wf_core`.
Source-, account-, or tool-specific limits belong in the platform layer.

Global node-call limits count admitted node handler calls, not all frames.
Control-flow frames are scheduler work; node calls are the expensive boundary.
Node calls waiting on platform/source/tool semaphores still count as active
node calls, and those waits keep the frame `RUNNING` rather than `BLOCKED`.

## Context and Lineage

Concurrent item frames need lineage-local pending state. Later nodes in the same
item lineage can read earlier pending patches from that item, while siblings
cannot.

Patch overlays conceptually belong to lineage tokens, not execution frames. A
scoped foreach implementation may begin by storing them in foreach metadata, but
future Fork/Gather needs first-class lineage ownership.

Future foreach metadata should evolve into inherited structured lineage context:

- core runtime context is structural, not alias-first
- foreach structured context keys are foreach node ids
- `as_` remains authoring sugar and human-readable trace/docs context
- Python `RuntimeContext` should eventually expose typed context such as
  `ctx.foreach["docs"].index`
- normal node authors should receive foreach values through mapped input;
  inspecting runtime context is an advanced escape hatch

## Deferred Work

Explicit Fork/Gather is deferred. A future `GatherNode` should expose explicit
barrier semantics, but arbitrary graph convergence needs lineage tokens first.
Forks produce branch lineage tokens; gathers consume declared token sets and
produce merged tokens. This supports partial gathers such as merging `a+b`
before later merging with `c`.

Concurrent foreach remains the nearer target because its implicit lineage tokens
are item indexes owned by one foreach activation.