use lineages for foreach now

docs/current_roadmap.md

@@ -57,9 +57,11 @@ implementation state.
   lineage progress includes ordered `StateWrite` records, `LineageStateView`,
   foreach item `lineage_id`s, nested foreach lineage identity, root
   `RuntimeScope` / `LineageState` storage, scope-aware reads, and non-root write
-  buffering. Current direct commits are still root-frame-only via an explicit
+  buffering. New concurrent foreach item writes are stored in
-  helper; native subgraph completion should replace that shortcut with an
+  `RunState.lineages`, while `ForeachBarrierState` keeps scheduling/result
-  explicit scope/lineage commit target.
+  metadata and compatibility patches. Current direct commits are still
+  root-frame-only via an explicit helper; native subgraph completion should
+  replace that shortcut with an explicit scope/lineage commit target.
 - **Persistent run history**: add a run store before adding stable `run_id`,
   `inspect_run`, or `read_run_trace(run_id, range)` APIs. Current traces are
   returned directly from immediate run responses.

docs/superpowers/plans/2026-05-24-lineage-state-runtime.md

@@ -29,6 +29,9 @@ the compatibility subset needed before native subgraphs:
   including nested foreach frames.
 - `RunState` has root scope/lineage storage, scope-aware state views, and
   generic non-root node writes buffer into `RunState.lineages`.
+- New concurrent foreach item writes are stored in `RunState.lineages`.
+  `ForeachBarrierState` now keeps scheduling/result metadata plus compatibility
+  patches for old serialized barrier data.
 
 Direct commits currently go through `is_root_lineage_frame(frame)`, which is the
 migration shortcut for root scope/root lineage. The eventual better shape is an
@@ -36,10 +39,9 @@ explicit scope/lineage commit target, feasible once native subgraph completion
 can declare whether child writes commit to child scope, parent lineage, or only
 through boundary output bindings.
 
-Remaining work should avoid jumping straight to native subgraphs. The next
+Remaining work should avoid jumping straight into a broad rewrite. The next
-small slice is to migrate foreach pending patch storage from
+small slice can start native subgraph scaffolding using the current
-`ForeachBarrierState` into `RunState.lineages`, or defer that and start native
+scope/lineage primitives.
-subgraph scaffolding using the current scope/lineage primitives.
 
 ---
 
@@ -556,10 +558,10 @@ Expected: pass.
 
 ## Task 5: Migrate Concurrent Foreach to Lineages
 
-Status: partially implemented. Concurrent foreach child frames now have lineage
+Status: implemented for new concurrent foreach results. Concurrent foreach
-ids, nested item lineages are tested, and pending item results persist
+child frames have lineage ids, nested item lineages are tested, item writes are
-`lineage_id`. Patch ownership still lives in `ForeachBarrierState`, not in a
+stored in `RunState.lineages`, and pending item results persist `lineage_id`.
-global lineage store.
+`ForeachBarrierState.patch` remains as a compatibility fallback.
 
 **Files:**
 

docs/superpowers/specs/2026-05-24-lineage-state-runtime-design.md

@@ -26,11 +26,13 @@ The first compatibility slices are implemented:
 - Frames and runtime context carry `scope_id`, `lineage_id`, and
   `parent_lineage_id`.
 - Generic non-root frame writes are buffered into `RunState.lineages`.
+- New concurrent foreach item writes are stored in `RunState.lineages`;
+  `ForeachBarrierState` keeps item result metadata plus compatibility patches
+  for old serialized barrier data.
 
-The full `RuntimeScope` / `LineageState` store is not implemented yet.
+The full native subgraph use of `RuntimeScope` is not implemented yet.
-Currently, foreach still owns pending write storage through
+`ForeachBarrierState` still owns scheduling/barrier metadata, but no longer has
-`ForeachBarrierState`; the lineage ids are identity and diagnostics, not yet the
+to be the primary write store for new concurrent foreach item results.
-primary storage key.
 
 Direct node commits currently use the explicit root-frame helper
 `is_root_lineage_frame(frame)`. That helper still means "root scope plus root

src/wf_core/runtime/foreach_state.py

@@ -66,7 +66,12 @@ class ItemErrorRecord:
 
 @dataclass(slots=True)
 class PendingItemResult:
-    """Buffered item result waiting for a future foreach barrier commit."""
+    """Buffered item result waiting for a future foreach barrier commit.
+
+    New concurrent foreach execution stores item writes in `RunState.lineages`
+    and records `lineage_id` here. `patch` remains for old serialized barrier
+    metadata and direct unit tests that still construct pending patches.
+    """
 
     index: int
     frame_id: str
@@ -266,10 +271,10 @@ class ForeachBarrierState:
     ) -> None:
         """Buffer or extend successful item patches by item index.
 
-        A multi-step item body can produce multiple node patches. They are
+        New runtime paths pass an empty patch and use `lineage_id`; legacy
-        accumulated for the same item lineage and replayed by the barrier in
+        callers may still accumulate patches here and replay them at the
-        item index order. Do not merge `_prepared_writes` here: the barrier
+        barrier. Do not merge `_prepared_writes`: the barrier replays public
-        intentionally replays public changes against one staged parent state.
+        write records against one staged parent state.
         """
         existing = self.pending_results.get(index)
         if existing is None:

src/wf_core/runtime/ops/foreach.py

@@ -7,7 +7,17 @@ from wf_core.errors import WorkflowExecutionError
 from wf_core.models.steps import ForeachNode
 from wf_core.models.workflow import Workflow
 from wf_core.run_state import ExecutionFrame, FrameStatus, RunState, StepExecutionResult
-from wf_core.runtime.foreach_state import ForeachBarrierState, ItemErrorRecord
+from wf_core.runtime.foreach_state import (
+    ForeachBarrierState,
+    ItemErrorRecord,
+    PendingItemResult,
+)
+from wf_core.runtime.lineage import (
+    add_lineage,
+    append_lineage_writes,
+    is_root_lineage_frame,
+    lineage_patch,
+)
 from wf_core.runtime.ops.flow import advance_frame, append_step_result_trace
 from wf_core.runtime.ops.frames import frame_context_values
 from wf_core.runtime.ops.index import WorkflowIndex
@@ -248,6 +258,12 @@ def _admit_concurrent_children(
         item = iterable[loop_index]
         child_id = f"{frame.id}:{step.id}:{loop_index}"
         child_lineage_id = _child_lineage_id(frame, step, loop_index)
+        add_lineage(
+            run,
+            scope_id=frame.scope_id,
+            lineage_id=child_lineage_id,
+            parent_id=frame.lineage_id,
+        )
         active_count = len(barrier.active_frame_ids)
         barrier.next_index = loop_index + 1
         barrier.start_child(child_id)
@@ -310,7 +326,7 @@ def _finish_concurrent_foreach(
     outcome = "completed_with_errors" if error_records else "done"
     next_node_id = index.next_node_id(frame.node_id, outcome)
     success_patches = [
-        result.patch
+        _patch_for_successful_item(run, frame, result)
         for result in (
             barrier.pending_results[item_index]
             for item_index in sorted(barrier.pending_results)
@@ -331,7 +347,16 @@ def _finish_concurrent_foreach(
         run.state,
         reducers=reducers,
     )
-    state_changes = commit_state_patch(run.state, combined)
+    if is_root_lineage_frame(frame):
+        state_changes = commit_state_patch(run.state, combined)
+    else:
+        append_lineage_writes(
+            run,
+            scope_id=frame.scope_id,
+            lineage_id=frame.lineage_id,
+            writes=combined.writes,
+        )
+        state_changes = {}
     append_step_result_trace(
         run,
         frame_id=frame.id,
@@ -361,3 +386,23 @@ def _child_lineage_id(frame: ExecutionFrame, step: ForeachNode, loop_index: int)
     full id, not parse it; future structured lineage refs can replace this.
     """
     return f"{frame.lineage_id}/{step.id}[{loop_index}]"
+
+
+def _patch_for_successful_item(
+    run: RunState,
+    frame: ExecutionFrame,
+    result: PendingItemResult,
+) -> StatePatch:
+    """Return the replayable patch for a completed foreach item.
+
+    New concurrent foreach results store writes in `RunState.lineages` and keep
+    only lineage metadata in the barrier. Old serialized barrier metadata may
+    still carry `result.patch`, so keep that as the compatibility fallback.
+    """
+    if result.lineage_id is not None and result.lineage_id in run.lineages:
+        return lineage_patch(
+            run,
+            scope_id=frame.scope_id,
+            lineage_id=result.lineage_id,
+        )
+    return result.patch

src/wf_core/runtime/ops/nodes.py

@@ -23,7 +23,7 @@ from wf_core.runtime.ops.frames import frame_context_values
 from wf_core.runtime.ops.merges import ReducerDefinition
 from wf_core.runtime.ops.overlays import state_view_for_frame
 from wf_core.runtime.ops.schemas import validate_payload_against_schema
-from wf_core.runtime.ops.state import build_output_patch, commit_state_patch
+from wf_core.runtime.ops.state import StatePatch, build_output_patch, commit_state_patch
 
 NodeHandler = Callable[[dict[str, Any], RuntimeContext], NodeResult | dict[str, Any]]
 AsyncNodeHandler = Callable[
@@ -136,10 +136,16 @@ def _finalize_node_execution(
         parent_frame = run.frames[parent_frame_id]
         barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
         if barrier is not None and barrier.mode == "concurrent":
+            append_lineage_writes(
+                run,
+                scope_id=frame.scope_id,
+                lineage_id=frame.lineage_id,
+                writes=patch.writes,
+            )
             barrier.add_success_patch(
                 index=item_index,
                 frame_id=frame.id,
-                patch=patch,
+                patch=StatePatch(),
                 lineage_id=frame.lineage_id,
             )
             barrier.save_to_frame(parent_frame, foreach_node_id)

tests/core/test_concurrent_foreach.py

@@ -19,6 +19,7 @@ from wf_core import (
     execute_workflow,
 )
 from wf_core.run_state import ExecutionFrame, RunState, RuntimeContext
+from wf_core.runtime.foreach_state import ForeachBarrierState
 from wf_core.runtime.scheduler import ForeachIterationMetadata
 
 
@@ -263,6 +264,12 @@ def test_sync_concurrent_foreach_barrier_replays_add_reducer_inputs() -> None:
 
     assert run.state["number"] == 6
     assert run.output["number"] == 6
+    assert run.lineages["root/each[0]"].writes[0].incoming_value == 3
+    assert run.lineages["root/each[1]"].writes[0].incoming_value == 1
+    barrier = ForeachBarrierState.from_frame(run.frames["root"], "each")
+    assert barrier is not None
+    assert barrier.pending_results[0].lineage_id == "root/each[0]"
+    assert barrier.pending_results[0].patch.writes == []
     foreach_entries = [entry for entry in run.trace if entry.step_type == "foreach"]
     assert foreach_entries[-1].state_changes["state.number"] == 6