lda-wf/docs/historical/superpowers/plans/2026-05-22-concurrent-foreach-phase4-roadmap.md

# Concurrent Foreach Phase 4 Roadmap

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Split concurrent foreach execution into safe, independently testable runtime slices.

**Architecture:** `foreach(mode="concurrent")` is the workflow mode. Sync runtime should interleave admitted item frames one node call at a time; async runtime may later run admitted async node handlers simultaneously. All item writes must flow through `StatePatch` and commit at a foreach barrier, not directly from child frames into shared state.

**Tech Stack:** Python 3.14, dataclasses, Pydantic v2, pytest, `wf_core.runtime.scheduler`, `wf_core.runtime.foreach_state`, `wf_core.runtime.ops.state`.

---

## Current State

Already implemented:

- `ForeachNode.mode` accepts canonical `"concurrent"`.
- `ForeachConcurrentPolicy` exists with `max_active`, `max_outstanding`, and `interrupt="quiesce"`.
- Legacy `mode="parallel"` / `parallel={...}` parse into canonical concurrent shape.
- `ForeachItemErrorPolicy` exists with `fail`, `skip`, and `collect`.
- `completed_with_errors` is derived for `skip` and `collect`.
- `collect_to` is validated as a declared array state field.
- `StatePatch`, `build_output_patch(...)`, and `commit_state_patch(...)` exist.
- `ForeachBarrierState`, `PendingItemResult`, and `ItemErrorRecord` exist.
- Serial foreach progress now uses `ForeachBarrierState`.
- Sync `foreach(mode="concurrent")` runs with fail-only item policy, bounded
  admission, deterministic interleaving, item-local overlays, and barrier
  commits.
- Multi-step concurrent item bodies are supported for fail-only item policy.
- Barrier write validation rejects ambiguous sibling writes: same-path sibling
  writes require a `mergeable` reducer, and ancestor/descendant
  sibling writes are rejected.
- Concurrent `item_error.action="skip"` and `"collect"` are supported.
- `collect` writes ordered item error records to `collect_to`, writes an empty
  list on clean success, and emits `completed_with_errors` only when failures
  were collected.
- Async runtime batches ready concurrent-foreach item node handlers so handler
  awaits may overlap up to admitted `max_active` work. State finalization and
  traces still happen sequentially after handler results return.

## Non-Goals For Phase 4

- Do not implement Fork/Gather graph nodes.
- Do not turn `JoinNode` into a real barrier.
- Do not run sync node handlers in threads or processes.
- Do not add OpenTelemetry.
- Do not add platform-level source/tool semaphores.
- Do not add persistent run storage.

## Slice 1: Sync Concurrent Foreach, Fail-Only

Implement first because it proves the scheduler and frame admission model without async task orchestration or handled item failures.

Scope:

- `foreach(mode="concurrent", item_error.action="fail")` runs in sync runtime.
- Parent foreach admits up to `concurrent.max_active` item frames.
- Scheduler interleaves item frames one step at a time.
- Child output writes are buffered as per-item `StatePatch` objects.
- Barrier commits all successful item patches only when every item succeeds.
- Any item runtime failure fails the whole run.
- `skip` and `collect` remain runtime-unsupported for concurrent mode.

Plan:

- See [`2026-05-22-concurrent-foreach-v1-sync-fail-only.md`](2026-05-22-concurrent-foreach-v1-sync-fail-only.md).

## Slice 2: Item-Local Overlays

Implemented after Slice 1 because fail-only concurrent foreach needed
lineage-local reads before multi-step item bodies could be supported. Overlays
let later nodes in one item read earlier buffered writes from the same item
without exposing those writes to siblings.

Scope:

- `state_view_for_frame(...)` returns committed parent state plus current item
  overlay for concurrent foreach item frames.
- Parent `RunState.state` remains unchanged until the barrier commits.
- Item patches accumulate across multiple nodes in the same item lineage.
- Multi-step concurrent item bodies are supported.
- Sibling item overlays remain isolated.
- Sibling write conflict policy remains deferred to Slice 3.

Plan:

- See [`2026-05-22-concurrent-foreach-item-overlays.md`](2026-05-22-concurrent-foreach-item-overlays.md).

## Slice 3: Barrier Commit Conflict Semantics

Implement after Slice 2 so conflict checks operate on real item-local overlays
and multi-step item patches.

Scope:

- Detect sibling lineage writes to the same state path.
- If exactly one lineage writes a destination path, default replace is allowed.
- If multiple sibling lineages write the same destination path, a declared
  `mergeable` reducer is required.
- Ancestor/descendant writes across sibling lineages are conflicts unless an explicit future merge strategy covers them.
- Commit order is item index order, never completion order.

Files likely touched:

- `src/wf_core/runtime/foreach_state.py`
- `src/wf_core/runtime/ops/state.py`
- `src/wf_core/runtime/ops/foreach.py`
- `tests/core/test_concurrent_foreach.py`

Key tests:

- `test_concurrent_foreach_rejects_sibling_writes_without_reducer`
- `test_concurrent_foreach_applies_reducer_in_item_index_order`
- `test_concurrent_foreach_rejects_ancestor_descendant_write_conflict`

Plan:

- See [`2026-05-22-concurrent-foreach-barrier-write-semantics.md`](2026-05-22-concurrent-foreach-barrier-write-semantics.md).

## Slice 4: Item Error Policies

Implemented after barrier success commits became correct.

Scope:

- `item_error.action="skip"` continues after item runtime failures.
- `item_error.action="collect"` continues and writes structured errors to `collect_to`.
- Both emit `completed_with_errors` if at least one item failed.
- `collect` writes an empty list and emits `done` if all items succeed.
- Failed item frames remain `FAILED`; parent foreach decides whether the failure is handled.

Files likely touched:

- `src/wf_core/runtime/foreach_state.py`
- `src/wf_core/runtime/ops/foreach.py`
- `tests/core/test_concurrent_foreach_errors.py`

Key tests:

- `test_concurrent_foreach_skip_emits_completed_with_errors`
- `test_concurrent_foreach_collect_writes_ordered_error_records`
- `test_concurrent_foreach_collect_writes_empty_list_on_clean_success`

## Slice 5: Async Concurrent Foreach

Implemented after sync semantics stabilized.

Scope:

- Async runtime may have multiple async node handler calls in flight.
- `concurrent.max_active` caps admitted/running item work.
- Sync handlers are still called normally; no thread/process executor.
- Trace remains append-only chronological execution history.
- Barrier commit order remains item index order.

Files likely touched:

- `src/wf_core/runtime/engine.py`
- `src/wf_core/runtime/step.py`
- `src/wf_core/runtime/ops/nodes.py`
- `src/wf_core/runtime/ops/foreach.py`
- `tests/core/test_concurrent_foreach_async.py`

Key tests:

- `test_async_concurrent_foreach_respects_max_active`
- `test_async_concurrent_foreach_commits_in_item_index_order`

## Slice 6: Interrupt Quiescence

Implemented after async execution exists.

Scope:

- If any concurrent item interrupts, the whole run pauses.
- No new item frames are admitted after the interrupt.
- Already-started async node calls drain to pending results.
- The caller gets control only at a quiescent point.
- Pending results do not commit until resume/commit policy allows it.
- Item frames that route into an `InterruptNode` are prioritized before the
  parent foreach can refill capacity.
- Already-started async handler calls drain at the batch boundary; state
  finalization remains sequential.

Files likely touched:

- `src/wf_core/runtime/engine.py`
- `src/wf_core/runtime/preparation.py`
- `src/wf_core/runtime/ops/interrupts.py`
- `src/wf_core/runtime/ops/foreach.py`
- `tests/core/test_concurrent_foreach_interrupts.py`

Key tests:

- `test_concurrent_foreach_interrupt_returns_before_refill`
- `test_resume_prioritizes_interrupted_item_frame_before_siblings`

## Execution Order

1. Sync concurrent foreach, fail-only.
2. Item-local overlays for multi-step item bodies.
3. Barrier conflict semantics.
4. `skip` / `collect` item error policies.
5. Async concurrent foreach.
6. Interrupt quiescence.

## Self-Review

- Spec coverage: the roadmap covers scheduler admission, barrier commits, reducer conflicts, handled item failures, async handler execution, and interrupt quiescence.
- Placeholder scan: each slice has scope, likely files, and named tests; concrete code lives in the slice-specific plan.
- Type consistency: the roadmap uses canonical `concurrent`, `ForeachConcurrentPolicy`, `ForeachBarrierState`, `PendingItemResult`, and `StatePatch`.