docs

CONTEXT.md

@@ -69,6 +69,14 @@ _Avoid_: Global committed state
 A merge boundary that waits for multiple child or upstream frames and combines their lineage patches before continuation.
 _Avoid_: Noop join
 
+**Gather Node**:
+A future graph step that exposes an explicit barrier for waiting on multiple branches and merging their results.
+_Avoid_: Promise.all node, converge node
+
+**Lineage Token**:
+A future runtime marker for one branch lineage that can be consumed and merged by gather-style barriers.
+_Avoid_: Edge id
+
 **Run**:
 One execution attempt of a workflow with input, state, frames, trace, and final output.
 _Avoid_: Job, invocation
@@ -87,6 +95,170 @@ _Avoid_: Job, invocation
 - Parallel foreach and native subgraphs depend on the **Scheduler Foundation**.
 - Parallel foreach requires a **Foreach Policy** before public support is
   enabled.
+- Future parallel foreach should be bounded by default. `max_active` and
+  `max_outstanding` prevent one workflow from spawning unbounded MCP, HTTP,
+  browser, or external service calls.
+- Future concurrency-specific foreach settings should live in a nested parallel
+  policy object rather than expanding `ForeachNode` with many top-level fields.
+- Item error handling is foreach-wide, not parallel-only. Serial and parallel
+  foreach can both profit from `fail`, `skip`, or `collect` item failure policy.
+- `collect` item error policy must declare an explicit destination for
+  structured item errors. Collected errors should be ordered by item index, not
+  async completion order.
+- Item error policy handles runtime failures inside an item frame, such as
+  thrown handler errors, schema failures, runtime-error utility nodes, or
+  invalid graph execution. It does not intercept normal graph outcomes like
+  `error` when those outcomes are routed.
+- A collected or skipped item failure should leave that item frame `FAILED`.
+  The foreach parent/barrier decides whether that failed child is handled and
+  whether the run may continue.
+- `skip` item error policy writes no hidden state. Failed frame status and
+  trace/observability are the record; use `collect` when structured error state
+  is needed.
+- `collect` writes structured item error records before emitting an aggregate
+  error outcome. Records include item index, frame id, failing node id, error
+  type/message, and the item value when it can be represented safely.
+- `collect` destinations must be state array fields. The foreach barrier writes
+  the full ordered error list once, so users should not expect per-item append
+  writes. Authoring and MCP validation should surface this clearly.
+- `completed_with_errors` is the canonical foreach aggregate outcome when all
+  items have finished but one or more item failures were handled by `collect` or
+  `skip`.
+- `completed_with_errors` is barrier aggregate vocabulary, not foreach-only.
+  Future explicit barriers may emit it when handled branch failures occurred.
+- Aggregate control-flow outcomes use result nouns/adjectives such as `done`,
+  `completed_with_errors`, and optionally `failed`. Policy actions use verbs
+  such as `fail`.
+- If a foreach policy can emit `completed_with_errors`, validation should treat
+  it as a required routable outcome. Users may route it to the same target as
+  `done` explicitly.
+- `collect` writes an empty list to its destination when all items succeed and
+  emits `done`. It emits `completed_with_errors` only when at least one item
+  failure was collected.
+- `skip` emits `completed_with_errors` when one or more item failures were
+  skipped. It emits `done` only when all items succeed.
+- Parallel `fail` item policy should stop scheduling new items, drain already
+  started jobs to a quiescent point, capture their results safely, and then fail
+  the run. It should not assume hard cancellation is safe.
+- After `fail` trips, drained sibling results are for trace/observability and
+  cleanup only. They should not commit normal state progress after the failure
+  boundary.
+- Foreach modes that continue after item failure, such as `collect` and `skip`,
+  should buffer item state patches until the foreach barrier completes. Future
+  parallel foreach should always use barrier-buffered commits.
+- Serial `fail` may keep immediate commits for compatibility. Commit strategy
+  should be extracted into runtime helpers instead of being smeared through
+  foreach execution code.
+- Barrier-buffered commits should store pending state patches/results, not full
+  state snapshots. Patch creation and commit must extract/reuse the existing
+  node output validation, output binding, and reducer logic rather than creating
+  a second write system.
+- Foreach barrier commits should merge item results in item index order, not
+  completion order. Reducer behavior such as list appends should therefore be
+  deterministic.
+- Successful item results may exist as pending barrier results before commit,
+  but they are not visible as workflow state until the barrier commits.
+- Pending barrier results must live in resumable `RunState`/frame metadata, not
+  only in trace. Trace records history; runtime state is what resume/checkpoint
+  uses.
+- Parallel item frames need lineage-local pending state: later nodes in the same
+  item lineage can read earlier pending patches from that item, while sibling
+  items cannot. The exact aggregate output API for committing item results to
+  parent state is deferred.
+- Lineage-local state should be represented as patch overlays over parent-visible
+  state, not deep-copied full state snapshots.
+- Patch overlays conceptually belong to lineage tokens, not execution frames.
+  A scoped foreach implementation may start by storing them in item/parent
+  metadata, but future Fork/Gather needs first-class lineage ownership.
+- `RunState.state` remains committed parent/global state. Future frame execution
+  should resolve reads against a frame-specific visible state view built from
+  committed state plus visible lineage overlays.
+- At a barrier, missing reducer means default replace only for single-writer
+  paths. Multiple sibling lineages writing the same path require an explicit
+  reducer; otherwise barrier commit raises a runtime error with writer details.
+- Barrier conflict detection should use the same write-overlap rules as normal
+  state writes. Ancestor/descendant writes from different lineages are conflicts
+  unless an explicit merge strategy covers them.
+- Reducers at barriers apply incrementally in deterministic lineage order by
+  default: item index order for foreach, declared branch token order for future
+  gather. Completion-order merging is a possible explicit barrier policy, not
+  reducer behavior.
+- Collected error records follow the same barrier merge order policy. The
+  default is deterministic lineage order.
+- Trace `state_changes` should mean committed state changes only. Do not encode
+  pending barrier patches there unless a future typed trace field is added.
+- Foreach may keep implicit barrier/iteration state on the foreach parent frame
+  because it owns item spawning and refill. General branch convergence should
+  become an explicit **Gather Node** later. Shared barrier merge/result helpers
+  should prevent foreach and future barriers from duplicating patch ordering,
+  conflict checks, and failure aggregation.
+- Future Fork/Gather needs **Lineage Tokens**, not raw edge ids. A fork produces
+  branch lineage tokens; a gather consumes a declared set of tokens and produces
+  a new merged token. This supports partial gathers such as merging branches
+  `a+b` before later merging with `c`.
+- Explicit Fork/Gather is deferred until lineage tokens are designed. Parallel
+  foreach remains the nearer target because its implicit lineage tokens are item
+  indexes owned by one foreach activation.
+- Future foreach metadata should evolve into inherited structured lineage
+  context. Alias lookup such as `context.document` can remain authoring sugar,
+  but runtime metadata should preserve nested foreach lineage without flat key
+  collisions.
+- Nested active foreach aliases should not shadow each other. Alias collisions
+  in inherited context scope should be validation errors.
+- Core runtime context should be structural, not alias-first. Foreach lineage
+  should be addressable through paths such as `context.foreach.<id>.index`;
+  `wf_authoring` can provide ergonomic alias helpers such as
+  `context_path(foreach_ref("docs").index)`.
+- Python `RuntimeContext` should eventually expose typed structured foreach
+  context, such as `ctx.foreach["docs"].index`, while serialized frame metadata
+  remains JSON-compatible dictionaries.
+- Structured foreach context keys should be foreach node ids. The `as_` alias is
+  authoring sugar for current item access, not the canonical runtime key.
+- `wf_authoring.WorkflowBuilder.foreach(...)` should eventually return a richer
+  ref exposing context selectors such as `.item` and `.index`, so users do not
+  hand-write `context.foreach.<id>...` paths.
+- Normal node authors should receive foreach values through mapped input.
+  Inspecting `RuntimeContext.foreach` is an advanced escape hatch for nodes that
+  genuinely need index/frame/lineage context.
+- `as_` remains useful ergonomic sugar and human-readable trace/docs context,
+  but structured foreach refs should be preferred for non-trivial authoring.
+- Future foreach policy shape should validate cross-field rules: `collect`
+  requires `collect_to`, non-collect actions forbid `collect_to`,
+  `mode="parallel"` requires a parallel policy, and `mode="serial"` forbids a
+  parallel policy. Deprecated top-level `on_item_error` may parse into the
+  nested item error policy, but canonical dumps should use the nested shape.
+- `ForeachParallelPolicy` should split limits into `max_active` and
+  `max_outstanding`. Defaults are `max_active=4` and `max_outstanding=20`;
+  validation requires `max_outstanding >= max_active`. Ready or running item
+  frames consume active capacity; blocked item frames consume outstanding
+  capacity but not active capacity.
+- A blocked non-interrupt item frame frees active capacity and may let foreach
+  start another item when `max_outstanding` also has room. A run-level interrupt
+  still stops scheduling.
+- The foreach parent frame owns refill decisions. Scheduler wakes/schedules
+  frames; foreach-specific policy decides whether to start more children,
+  finish, or fail.
+- When an item frame becomes blocked, it frees active capacity only for its
+  nearest foreach capacity owner. Future item metadata should identify that
+  owner rather than waking arbitrary ancestors.
+- Foreach capacity is local correctness policy, not total process protection.
+  Future runtime should also have basic global run limits in `wf_core`; source-
+  or tool-specific limits belong in the platform layer.
+- A future global `wf_core` runtime limit should count active node handler calls,
+  not all active frames. Control-flow frames are scheduler work; node calls are
+  the expensive external/user-code execution boundary.
+- Global node-call limits do not replace foreach caps. Foreach caps bound local
+  scheduling fairness, outstanding frame count, memory, and pending results;
+  global node-call limits bound expensive handler execution across the run.
+- Platform source/tool/account limits should be enforced at the node-handler
+  boundary, before invoking the external call. They layer on top of core global
+  node-call admission instead of replacing it.
+- Node calls waiting on platform source/tool/account limits still count as
+  active node calls. Core global node-call admission should happen before
+  platform-specific semaphore acquisition.
+- Waiting on a source/tool/account semaphore keeps the frame `RUNNING`; it is
+  backpressure inside the admitted node-call boundary, not workflow-level
+  `BLOCKED` state that frees foreach active capacity.
 - An **Interrupt** pauses the whole **Run**, even if the interrupted frame is a
   child of future parallel work.
 - A **Runtime Failure** is distinct from a node returning an `error` outcome.

docs/adr/0001-scheduler-foundation-before-parallel-foreach.md

@@ -81,6 +81,8 @@ they are enabled:
 
 - A future foreach policy must define batching, `max_concurrency`, item failure
   handling, and cancellation/drain behavior.
+- Parallel foreach should be bounded by default so workflows do not spawn
+  unbounded MCP, HTTP, browser, or external service calls.
 - A future barrier must wait for multiple child or upstream frames and merge
   their lineage patches.
 - Missing reducers mean replace only within one serial lineage. At a barrier,

docs/adr/0002-parallel-foreach-policy-and-barrier-commits.md

@@ -0,0 +1,161 @@
+# Parallel Foreach Policy and Barrier Commits
+
+Status: accepted
+
+Parallel foreach will use bounded scheduling, lineage-local pending state, and
+barrier-buffered commits. This keeps parallel item execution deterministic,
+resumable, and safe around reducers, interrupts, and external tool calls.
+
+## Context
+
+The scheduler foundation makes multiple runnable frames possible, but parallel
+foreach needs more than concurrent node calls. Each item lineage may execute
+several nodes, read its own pending writes, fail independently, block, or
+interrupt the whole run. Sibling item writes must not leak into each other, and
+state commits need deterministic merge behavior.
+
+## Decision
+
+Parallel foreach will be async-runtime-only and bounded by policy.
+
+The future foreach model should separate item error behavior from concurrency
+behavior:
+
+- `item_error` is foreach-wide and applies to serial and parallel foreach.
+- `parallel` is a nested policy object used only when `mode="parallel"`.
+- `parallel.max_active` defaults to `4`.
+- `parallel.max_outstanding` defaults to `20`.
+- Validation requires `max_outstanding >= max_active`.
+- Ready or running item frames consume active capacity.
+- Blocked item frames consume outstanding capacity but not active capacity.
+
+`foreach(mode="parallel")` requires async execution. Sync execution should reject
+it clearly rather than inventing thread/process semantics.
+
+## Item Error Policy
+
+Item error policy handles runtime failures inside an item frame, not normal graph
+control-flow outcomes. A node returning an `error` outcome is still ordinary
+graph routing if the graph declares that edge.
+
+Supported policy actions:
+
+- `fail`: stop scheduling new items, drain already-started jobs to a quiescent
+  point, capture results for trace/observability, then fail the run.
+- `skip`: mark the failed item frame failed, record no hidden state, continue,
+  and emit `completed_with_errors` if any item was skipped.
+- `collect`: mark the failed item frame failed, collect a structured item error
+  record, continue, and emit `completed_with_errors` if any item was collected.
+
+`collect` must declare an explicit state array destination. The foreach barrier
+writes the full ordered error list once. On clean completion, `collect` writes an
+empty list and emits `done`.
+
+Collected error records include item index, frame id, failing node id, error
+type/message, and the item value when it can be represented safely.
+
+## Aggregate Outcomes
+
+Foreach aggregate outcomes are policy-derived:
+
+- `done`: all items completed cleanly.
+- `completed_with_errors`: all items completed but one or more item failures
+  were handled by `skip` or `collect`.
+
+Validation should require a routable `completed_with_errors` edge whenever the
+policy can emit it. Users may route it to the same target as `done` explicitly.
+
+Aggregate result vocabulary should be shared with future barrier-like graph
+steps. `fail` is a policy action; if a future aggregate node emits graph-level
+failure as control flow, the outcome should be `failed`.
+
+## Barrier-Buffered Commits
+
+Parallel foreach item writes are buffered as pending state patches/results, not
+committed directly to `RunState.state`.
+
+The state model is:
+
+- `RunState.state` remains committed parent/global state.
+- Each item lineage reads a visible state view built from committed state plus
+  its own lineage-local patch overlay.
+- Sibling item patches are invisible to each other.
+- Pending barrier results live in resumable run state/frame metadata, not trace.
+- Trace `state_changes` means committed state changes only.
+
+The barrier commits item results in deterministic item-index order by default.
+Completion-order merging may exist later as an explicit barrier merge policy,
+not reducer behavior.
+
+Patch creation and commit must extract/reuse the existing node output
+validation, output binding, and reducer logic. Parallel foreach must not create
+a second write system.
+
+## Merge and Reducer Rules
+
+At a barrier, missing reducer means default replace only for single-writer
+paths. Multiple sibling lineages writing the same state path require an explicit
+reducer. Ancestor/descendant overlapping writes across lineages are conflicts
+unless an explicit merge strategy covers them.
+
+Reducers apply incrementally in deterministic lineage order. For foreach, that
+means item index order.
+
+## Interrupt and Failure Quiescence
+
+Future parallel execution should not assume in-flight node calls can be safely
+cancelled.
+
+If an interrupt or fail policy trips while sibling jobs are already started, the
+runtime should:
+
+1. stop scheduling new work
+2. let already-started jobs drain to a quiescent point
+3. capture their results safely
+4. defer or discard commits according to the policy boundary
+5. return control only after no hidden background jobs continue mutating state
+
+For `fail`, drained sibling results are for observability/cleanup only and
+should not commit normal state progress after the failure boundary.
+
+## Capacity and Runtime Limits
+
+Foreach capacity is local correctness policy, not total process protection.
+Future runtime should also support a basic global node-call budget in `wf_core`.
+Source-, account-, or tool-specific limits belong in the platform layer.
+
+Global node-call limits count admitted node handler calls, not all frames.
+Control-flow frames are scheduler work; node calls are the expensive boundary.
+Node calls waiting on platform/source/tool semaphores still count as active
+node calls, and those waits keep the frame `RUNNING` rather than `BLOCKED`.
+
+## Context and Lineage
+
+Parallel item frames need lineage-local pending state. Later nodes in the same
+item lineage can read earlier pending patches from that item, while siblings
+cannot.
+
+Patch overlays conceptually belong to lineage tokens, not execution frames. A
+scoped foreach implementation may begin by storing them in foreach metadata, but
+future Fork/Gather needs first-class lineage ownership.
+
+Future foreach metadata should evolve into inherited structured lineage context:
+
+- core runtime context is structural, not alias-first
+- foreach structured context keys are foreach node ids
+- `as_` remains authoring sugar and human-readable trace/docs context
+- Python `RuntimeContext` should eventually expose typed context such as
+  `ctx.foreach["docs"].index`
+- normal node authors should receive foreach values through mapped input;
+  inspecting runtime context is an advanced escape hatch
+
+## Deferred Work
+
+Explicit Fork/Gather is deferred. A future `GatherNode` should expose explicit
+barrier semantics, but arbitrary graph convergence needs lineage tokens first.
+Forks produce branch lineage tokens; gathers consume declared token sets and
+produce merged tokens. This supports partial gathers such as merging `a+b`
+before later merging with `c`.
+
+Parallel foreach remains the nearer target because its implicit lineage tokens
+are item indexes owned by one foreach activation.

docs/current_roadmap.md

@@ -41,6 +41,8 @@ implementation state.
 
 - Scheduler foundation decision record:
   [ADR 0001](./adr/0001-scheduler-foundation-before-parallel-foreach.md).
+- Parallel foreach policy decision record:
+  [ADR 0002](./adr/0002-parallel-foreach-policy-and-barrier-commits.md).
 - **Native subgraphs / graph-as-node**: add child run state, child trace
   preservation, interrupt bubbling, and resume back into the child workflow.
   Wrapper artifacts currently execute as deployments and return run status;

docs/superpowers/plans/2026-05-22-parallel-foreach-roadmap.md

@@ -0,0 +1,565 @@
+# Parallel Foreach Roadmap Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Implement parallel foreach incrementally without breaking serial workflows or duplicating state-write logic.
+
+**Architecture:** The work is split into four independently shippable layers: policy models, state patch extraction, barrier runtime state, and async parallel execution. Each layer preserves current serial behavior and adds tests before implementation. `foreach(mode="parallel")` remains unsupported until the final layer.
+
+**Tech Stack:** Python 3.14, Pydantic v2, dataclasses, pytest, basedpyright, ruff, existing `wf_core` scheduler/runtime modules.
+
+---
+
+## Phase 1: Foreach Policy Models
+
+**Goal:** Add the future policy shape while keeping runtime behavior serial-only.
+
+**Files:**
+- Modify: `src/wf_core/models/steps.py`
+- Modify: `src/wf_core/validation/outcomes.py`
+- Modify: `src/wf_core/validation/steps.py`
+- Modify: `src/wf_authoring/builder/core.py`
+- Test: `tests/core/test_foreach_policy.py`
+- Test: `tests/authoring/test_builder.py`
+
+- [ ] **Step 1: Add failing model tests**
+
+Create `tests/core/test_foreach_policy.py`:
+
+```python
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from wf_core.models.steps import ForeachNode
+
+
+def test_serial_foreach_defaults_to_fail_item_policy() -> None:
+    node = ForeachNode.model_validate(
+        {
+            "id": "each",
+            "type": "foreach",
+            "over": {"root": "state", "parts": ["items"]},
+            "as": "item",
+        }
+    )
+
+    assert node.mode == "serial"
+    assert node.item_error.action == "fail"
+    assert node.item_error.collect_to is None
+    assert node.parallel is None
+
+
+def test_collect_item_policy_requires_collect_to() -> None:
+    with pytest.raises(ValidationError, match="collect_to"):
+        ForeachNode.model_validate(
+            {
+                "id": "each",
+                "type": "foreach",
+                "over": {"root": "state", "parts": ["items"]},
+                "as": "item",
+                "item_error": {"action": "collect"},
+            }
+        )
+
+
+def test_parallel_policy_requires_parallel_mode() -> None:
+    with pytest.raises(ValidationError, match="parallel policy"):
+        ForeachNode.model_validate(
+            {
+                "id": "each",
+                "type": "foreach",
+                "over": {"root": "state", "parts": ["items"]},
+                "as": "item",
+                "parallel": {"max_active": 4, "max_outstanding": 20},
+            }
+        )
+
+
+def test_parallel_policy_validates_capacity_order() -> None:
+    with pytest.raises(ValidationError, match="max_outstanding"):
+        ForeachNode.model_validate(
+            {
+                "id": "each",
+                "type": "foreach",
+                "over": {"root": "state", "parts": ["items"]},
+                "as": "item",
+                "mode": "parallel",
+                "parallel": {"max_active": 10, "max_outstanding": 4},
+            }
+        )
+```
+
+- [ ] **Step 2: Add policy models**
+
+In `src/wf_core/models/steps.py`, add:
+
+```python
+from typing import Self
+
+
+class ForeachItemErrorPolicy(BaseModel):
+    """Policy for runtime failures inside one foreach item lineage."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    action: Literal["fail", "skip", "collect"] = "fail"
+    collect_to: StatePath | None = None
+
+    @model_validator(mode="after")
+    def _validate_collect_to(self) -> Self:
+        if self.action == "collect" and self.collect_to is None:
+            raise ValueError("collect item error policy requires collect_to")
+        if self.action != "collect" and self.collect_to is not None:
+            raise ValueError("collect_to is only valid when action='collect'")
+        return self
+
+
+class ForeachParallelPolicy(BaseModel):
+    """Concurrency policy for async parallel foreach execution."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    max_active: int = Field(default=4, ge=1)
+    max_outstanding: int = Field(default=20, ge=1)
+    interrupt: Literal["quiesce"] = "quiesce"
+
+    @model_validator(mode="after")
+    def _validate_capacity(self) -> Self:
+        if self.max_outstanding < self.max_active:
+            raise ValueError("max_outstanding must be >= max_active")
+        return self
+```
+
+Update `ForeachNode`:
+
+```python
+item_error: ForeachItemErrorPolicy = Field(default_factory=ForeachItemErrorPolicy)
+parallel: ForeachParallelPolicy | None = None
+on_item_error: Literal["fail", "collect", "skip"] | None = Field(
+    default=None,
+    exclude=True,
+    description="Deprecated parse-only shorthand; use item_error.action.",
+)
+```
+
+Add a `model_validator(mode="before")` that converts old `on_item_error` into `item_error.action`.
+
+Add a `model_validator(mode="after")` that enforces:
+
+```python
+if self.mode == "parallel" and self.parallel is None:
+    raise ValueError("parallel foreach requires parallel policy")
+if self.mode == "serial" and self.parallel is not None:
+    raise ValueError("parallel policy is only valid when mode='parallel'")
+```
+
+- [ ] **Step 3: Update derived outcomes**
+
+In `src/wf_core/validation/outcomes.py`, update foreach outcome derivation:
+
+```python
+if step.type == "foreach":
+    outcomes = {"loop", "done"}
+    if step.item_error.action in {"skip", "collect"}:
+        outcomes.add("completed_with_errors")
+    return outcomes
+```
+
+- [ ] **Step 4: Validate collect destination schema**
+
+In `src/wf_core/validation/steps.py`, when `node.item_error.action == "collect"`:
+
+```python
+destination_root = _state_destination_root(node.item_error.collect_to)
+if destination_root is None or destination_root not in state_root_fields:
+    report.add(...)
+```
+
+Add a follow-up test that collect-to unknown state root reports a validation issue.
+
+- [ ] **Step 5: Keep runtime unsupported**
+
+In `src/wf_core/runtime/ops/foreach.py`, keep:
+
+```python
+if step.mode != "serial":
+    raise WorkflowExecutionError("parallel foreach execution is not implemented yet")
+```
+
+Add a comment:
+
+```python
+# Policy models are accepted before execution support so saved workflows can
+# validate shape, but runtime must reject parallel until barrier commits exist.
+```
+
+- [ ] **Step 6: Verify phase**
+
+Run:
+
+```bash
+uv run pytest tests/core/test_foreach_policy.py tests/authoring/test_builder.py -q
+uvx ruff check src tests
+uv run basedpyright --level error
+```
+
+Expected: all pass.
+
+---
+
+## Phase 2: State Patch Extraction
+
+**Goal:** Split current node output writes into reusable “build patch” and “commit patch” operations without changing current serial behavior.
+
+**Files:**
+- Modify: `src/wf_core/runtime/ops/state.py`
+- Modify: `src/wf_core/runtime/ops/nodes.py`
+- Test: `tests/core/test_atomic_state_patches.py`
+- Test: `tests/core/test_nested_state_paths.py`
+
+- [ ] **Step 1: Add state patch model**
+
+In `src/wf_core/runtime/ops/state.py`, add:
+
+```python
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass(slots=True)
+class StatePatch:
+    """Validated state writes produced by one step before commit."""
+
+    changes: dict[str, Any] = field(default_factory=dict)
+```
+
+- [ ] **Step 2: Extract patch builder**
+
+Refactor existing `apply_output_bindings(...)` into:
+
+```python
+def build_output_patch(
+    workflow: Workflow,
+    bindings: Sequence[OutputBinding],
+    output: Mapping[str, Any],
+    state: MutableMapping[str, Any],
+    *,
+    reducers: Mapping[str, ReducerDefinition] | None = None,
+    missing_field_message: str = "node output is missing required field {field}",
+) -> StatePatch:
+    ...
+```
+
+This function should:
+- validate source paths
+- validate destination paths
+- calculate reducer-aware changes
+- not mutate `state`
+
+- [ ] **Step 3: Extract patch committer**
+
+Add:
+
+```python
+def commit_state_patch(
+    state: MutableMapping[str, Any],
+    patch: StatePatch,
+) -> dict[str, Any]:
+    """Commit a validated patch to state and return committed changes."""
+    for path, value in patch.changes.items():
+        set_nested_value(state, split_state_path(path), value)
+    return dict(patch.changes)
+```
+
+Use the existing typed path helpers; do not reintroduce dotted-string parsing if a typed path helper exists.
+
+- [ ] **Step 4: Preserve old API**
+
+Keep `apply_output_bindings(...)` as a wrapper:
+
+```python
+patch = build_output_patch(...)
+return commit_state_patch(state, patch)
+```
+
+Existing callers should keep working.
+
+- [ ] **Step 5: Add equivalence tests**
+
+Add tests that compare:
+
+```python
+old_changes = apply_output_bindings(...)
+patch = build_output_patch(...)
+new_changes = commit_state_patch(state2, patch)
+assert old_changes["state.some_path"] == new_changes["state.some_path"]
+assert state1["some_path"] == state2["some_path"]
+```
+
+Do not assert whole dict equality unless the test intentionally owns the full structure.
+
+- [ ] **Step 6: Verify phase**
+
+Run:
+
+```bash
+uv run pytest tests/core/test_atomic_state_patches.py tests/core/test_nested_state_paths.py tests/authoring/test_demo_workflow.py -q
+uvx ruff check src tests
+uv run basedpyright --level error
+```
+
+Expected: all pass; full suite should still pass before moving on.
+
+---
+
+## Phase 3: Foreach Barrier Runtime State
+
+**Goal:** Add resumable barrier metadata and pending result structures without enabling async parallel execution.
+
+**Files:**
+- Modify: `src/wf_core/runtime/scheduler.py`
+- Create: `src/wf_core/runtime/foreach_state.py`
+- Modify: `src/wf_core/runtime/ops/foreach.py`
+- Test: `tests/core/test_foreach_barrier_state.py`
+
+- [ ] **Step 1: Add pending result dataclasses**
+
+Create `src/wf_core/runtime/foreach_state.py`:
+
+```python
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Literal
+
+from wf_core.runtime.ops.state import StatePatch
+
+
+@dataclass(slots=True)
+class ItemErrorRecord:
+    """Structured runtime failure record for one foreach item."""
+
+    index: int
+    frame_id: str
+    node_id: str
+    error_type: str
+    message: str
+    item: Any = None
+
+
+@dataclass(slots=True)
+class PendingItemResult:
+    """Buffered item result waiting for foreach barrier commit."""
+
+    index: int
+    frame_id: str
+    status: Literal["succeeded", "failed"]
+    patch: StatePatch = field(default_factory=StatePatch)
+    error: ItemErrorRecord | None = None
+```
+
+- [ ] **Step 2: Add foreach barrier state**
+
+In the same file:
+
+```python
+@dataclass(slots=True)
+class ForeachBarrierState:
+    """Resumable state owned by one foreach parent frame."""
+
+    next_index: int = 0
+    active_frame_ids: tuple[str, ...] = ()
+    outstanding_frame_ids: tuple[str, ...] = ()
+    pending_results: dict[int, PendingItemResult] = field(default_factory=dict)
+```
+
+Add `to_metadata()` / `from_frame()` helpers. Wrong frame kind returns `None`; malformed metadata for a foreach parent raises `WorkflowExecutionError`.
+
+- [ ] **Step 3: Move serial progress into typed state**
+
+Current serial foreach uses:
+
+```python
+progress_map = frame.metadata.setdefault("foreach_progress", {})
+```
+
+Replace with typed barrier state, but keep behavior equivalent:
+
+```python
+barrier = ForeachBarrierState.from_frame(frame) or ForeachBarrierState()
+loop_index = barrier.next_index
+barrier.next_index += 1
+frame.metadata["foreach_barrier"] = barrier.to_metadata()
+```
+
+- [ ] **Step 4: Add serialization tests**
+
+Test:
+
+```python
+def test_foreach_barrier_state_round_trips_through_frame_metadata() -> None:
+    ...
+```
+
+Assert specific fields:
+
+```python
+assert loaded.next_index == 2
+assert loaded.outstanding_frame_ids == ("child-1",)
+```
+
+- [ ] **Step 5: Keep serial behavior passing**
+
+Run:
+
+```bash
+uv run pytest tests/core/test_foreach_barrier_state.py tests/authoring/test_demo_workflow.py -q
+```
+
+Expected: pass.
+
+---
+
+## Phase 4: Async Parallel Foreach
+
+**Goal:** Enable `foreach(mode="parallel")` in async execution only, using policy limits, pending results, and barrier commits.
+
+**Files:**
+- Modify: `src/wf_core/runtime/ops/foreach.py`
+- Modify: `src/wf_core/runtime/step.py`
+- Modify: `src/wf_core/runtime/engine.py`
+- Modify: `src/wf_core/runtime/ops/nodes.py`
+- Modify: `src/wf_core/runtime/foreach_state.py`
+- Test: `tests/core/test_parallel_foreach.py`
+
+- [ ] **Step 1: Add async-only rejection tests**
+
+Create `tests/core/test_parallel_foreach.py` with:
+
+```python
+def test_sync_runtime_rejects_parallel_foreach() -> None:
+    ...
+
+
+async def test_async_runtime_accepts_parallel_foreach() -> None:
+    ...
+```
+
+Expected before implementation: async test fails because runtime still rejects parallel.
+
+- [ ] **Step 2: Add capacity tests**
+
+Use async node handlers that record start/completion order and block on `asyncio.Event`.
+
+Test:
+
+```python
+async def test_parallel_foreach_respects_max_active() -> None:
+    ...
+    assert max_seen_active == 2
+```
+
+Use `max_active=2`.
+
+- [ ] **Step 3: Add outstanding tests**
+
+Use a node that blocks internally through a future block helper or controlled async wait.
+
+Test:
+
+```python
+async def test_blocked_items_count_against_max_outstanding_not_active() -> None:
+    ...
+```
+
+This may require a small test-only node that blocks through the runtime-supported internal wait. If internal blocking is not implemented yet, defer this test to subgraph/internal-wait work and keep `max_outstanding` tested through queued children.
+
+- [ ] **Step 4: Add collect/skip tests**
+
+Tests:
+
+```python
+async def test_parallel_collect_writes_ordered_errors_and_emits_completed_with_errors() -> None:
+    ...
+
+
+async def test_parallel_skip_emits_completed_with_errors_without_hidden_state() -> None:
+    ...
+```
+
+Assert:
+
+```python
+assert run.state["document_errors"][0]["index"] == 1
+assert run.trace[-1].outcome == "completed_with_errors"
+```
+
+- [ ] **Step 5: Add barrier commit ordering test**
+
+Use nodes that complete out of order but write list-like results.
+
+Assert committed state is ordered by item index, not completion order.
+
+- [ ] **Step 6: Implement async parallel child scheduling**
+
+In `step_foreach`, branch by mode:
+
+```python
+if step.mode == "serial":
+    return step_foreach_serial(...)
+return step_foreach_parallel(...)
+```
+
+`step_foreach_parallel` should:
+- inspect `ForeachBarrierState`
+- start children while `active < max_active` and `outstanding < max_outstanding`
+- block parent when waiting for children
+- finish when all items terminal
+- commit barrier patches in item index order
+- emit `done` or `completed_with_errors`
+
+- [ ] **Step 7: Add async node-call budget seam**
+
+Add execution option shape only if needed by implementation:
+
+```python
+@dataclass(slots=True)
+class RuntimeLimits:
+    max_active_node_calls: int = 16
+```
+
+If this is too large for the first async parallel pass, leave global node-call budget as follow-up and rely on foreach `max_active`.
+
+- [ ] **Step 8: Verify phase**
+
+Run:
+
+```bash
+uv run pytest tests/core/test_parallel_foreach.py tests/authoring/test_demo_workflow.py -q
+uv run pytest -q
+uvx ruff check src tests
+uv run basedpyright --level error
+```
+
+Expected: all pass.
+
+---
+
+## Implementation Order Recommendation
+
+Ship these as separate commits/PRs:
+
+1. Phase 1: policy shape and validation
+2. Phase 2: patch extraction with no behavior change
+3. Phase 3: barrier metadata with serial behavior unchanged
+4. Phase 4: async parallel execution
+
+Do not start Phase 4 until Phase 2 and Phase 3 are stable. Parallel foreach depends on patch extraction and resumable barrier state.
+
+## Self-Review
+
+- Spec coverage: ADR 0002 decisions are represented across the four phases.
+- Intentional gaps: explicit Fork/Gather, lineage-token graph nodes, OpenTelemetry, platform source/tool caps, and full run persistence are not included.
+- Risk control: phases 1-3 preserve serial behavior and keep `mode="parallel"` unsupported until phase 4.

docs/superpowers/plans/2026-05-22-scheduler-foundation.md

@@ -26,7 +26,7 @@
 - Modify: `src/wf_core/runtime/ops/frames.py`
   - Keep context helpers; demote stack-collapse usage or leave compatibility wrappers that call scheduler helpers.
 - Modify: `src/wf_core/runtime/preparation.py`
-  - Stop relying on `collapse_completed_frames()` for selection.
+  - Stop relying on stack-style frame collapse for selection.
   - Resume interrupt by waking/enqueueing the resumed frame.
 - Modify: `src/wf_core/runtime/engine.py`
   - Use scheduler loop for sync and async resume.
@@ -527,9 +527,9 @@ while True:
 
 Use the same selection flow in the async loop before `await step_workflow_async(...)`.
 
-- [ ] **Step 3: Remove stack collapse from step preparation**
+- [ ] **Step 3: Remove stack-style frame collapse from step preparation**
 
-In `prepare_step`, remove `collapse_completed_frames(run)` and keep it focused on resolving the selected frame’s node. It should still return `None` for interrupted/end states.
+In `prepare_step`, remove old stack-style frame collapse and keep it focused on resolving the selected frame’s node. It should still return `None` for interrupted/end states.
 
 - [ ] **Step 4: Verify serial workflows**
 

src/wf_core/runtime/ops/frames.py

@@ -1,23 +1,6 @@
 from __future__ import annotations
 
-from wf_core.run_state import ExecutionFrame, FrameStatus, RunState
-from wf_core.tokens import END
-
-
-def collapse_completed_frames(run: RunState) -> None:
-    while run.current_frame_id is not None:
-        frame = run.current_frame()
-        if frame.node_id == END and frame.status != FrameStatus.COMPLETED:
-            frame.status = FrameStatus.COMPLETED
-            frame.finished_at_node_id = END
-        if frame.status != FrameStatus.COMPLETED or frame.parent_frame_id is None:
-            run.sync_from_current_frame()
-            return
-        run.current_frame_id = frame.parent_frame_id
-        parent = run.current_frame()
-        if parent.status == FrameStatus.PENDING:
-            parent.status = FrameStatus.RUNNING
-        run.sync_from_current_frame()
+from wf_core.run_state import ExecutionFrame
 
 
 def frame_context_values(frame: ExecutionFrame) -> dict[str, object | None]: