lda-wf/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md

Foreach Back-Edge Design

Status

Approved in conversation on 2026-09-04. This document specifies canonical foreach body-return semantics. It does not include the separately planned ergonomic Python DSL or authorize fork/gather implementation.

Purpose

Make the persisted workflow graph tell the truth about foreach control flow. An iteration body returns to its owning ForeachNode through an ordinary back-edge. It no longer routes to the workflow terminal END merely to make an internal child frame finish.

The canonical shape is:

a -> foreach
     foreach.loop -> body
     body.ok -> foreach
     foreach.done -> c
c -> END

This preserves a flat, inspectable control-flow graph. The runtime may continue to implement each item with a child execution frame and a parent barrier, but that scheduler detail no longer changes the visible meaning of END.

Decisions

• ForeachNode.loop enters one iteration body.
• An edge from an iteration body to that iteration frame's owning ForeachNode returns the item to the controller.
• Returning to the owner completes the item frame and wakes the blocked parent foreach frame. The child does not execute the ForeachNode again.
• The parent foreach frame remains positioned at the controller. It admits the next serial or concurrent item, or emits done/completed_with_errors after its barrier finishes.
• A foreach item path may not target END or an explicit EndNode. Workflow terminal routes are outside an iteration body.
• END remains the workflow/subgraph terminal shorthand for workflow outcome ok; explicit EndNode remains the terminal for other workflow outcomes.
• Existing fail, skip, and collect item-error policies are unchanged.
• The core graph stays flat. Foreach does not gain a nested serialized body.
• The existing manual WorkflowBuilder.foreach() and connect() interface remains the only authoring surface in this slice; no new authoring interface is added.

Runtime Semantics

The parent frame owns the foreach controller and resumable barrier. Taking its loop outcome creates an item frame whose metadata names that parent foreach. The item frame begins at the loop edge target and follows ordinary outcome edges.

Before ordinary frame advancement, runtime flow checks whether an item frame's next target is its recorded owning foreach. If so, it records a foreach return:

owner = foreach_item_owner(frame)
if owner is not None and next_node_id == owner.foreach_node_id:
    complete_item_frame(run, frame)
    wake_parent_for_child_progress(run, frame.id)
    return

This is an ownership check, not generic cycle detection. A root or unrelated frame targeting the same foreach node enters it normally. An item frame targeting a different foreach node also enters that node normally. Only a return to the frame's recorded owner completes the current item.

The completed child records the owning foreach as its terminal graph location for trace and checkpoint inspection. Existing barrier code remains responsible for consuming its lineage-local writes, applying item-error policy, and waking or completing the parent.

Serial and concurrent modes share the same return meaning:

• serial mode has at most one live item frame and can admit the next item after its return;
• concurrent mode may receive several independently identified item returns, buffers them by item identity, and completes only after its existing barrier policy is satisfied.

Nested Foreach

Nested foreach remains structured by frame ownership:

outer.loop -> inner
inner.loop -> work
work -> inner
inner.done -> after_inner
after_inner -> outer
outer.done -> after_outer

The work frame returns only to inner, its immediate owner. Once the inner controller emits done, execution resumes in the outer item frame. Returning after_inner -> outer then completes the outer item.

Runtime metadata already carries the immediate foreach owner. Static context analysis must evolve from a single active foreach identifier to an ownership stack so inner completion restores the outer item context.

Validation and Analysis

Validation performs structured abstract traversal over (node_id, foreach_owner_stack) rather than merely looking for graph cycles:

• a loop edge from a foreach pushes that foreach onto the stack;
• an edge targeting the top owner is an item return and ends that child traversal;
• a done or completed_with_errors edge stays in the outer context;
• END or EndNode reached with a non-empty foreach stack is invalid;
• nested returns must target the immediate owner before an outer owner;
• traversal memoizes node plus owner stack so valid cycles terminate analysis.

A node may remain reachable in more than one execution context. Existing context-contract analysis can report fields as conditional in that case. The runtime must never infer edge meaning from graph history alone: frame ownership always decides whether targeting a foreach is entry or return.

General unreachable-node detection is useful but is not part of this change. It cannot establish foreach ownership because any node reached from a body is connected by definition. General termination is also out of scope; finite structural checks cannot prove that data-dependent cycles eventually exit.

State and Failure Behavior

Back-edge return changes control representation, not state semantics. Iteration writes remain buffered in the item lineage. Serial behavior and the concurrent barrier continue to commit or merge those writes according to the accepted concurrent-foreach ADR and declared reducers.

An ordinary node outcome named error remains domain control. An exception remains a runtime item failure handled by fail, skip, or collect. Neither kind of failure is encoded by the foreach back-edge itself.

Migration

This is a clean canonical migration. Repository tests, examples, user-facing docs, and generated fixtures change from:

foreach.loop -> body
body -> END

to:

foreach.loop -> body
body -> foreach

Configured local artifact stores contain no persisted foreach workflows at the time of design. There is therefore no demonstrated persisted-data requirement for keeping body -> END as a compatibility behavior. Validation rejects that old shape rather than silently preserving two canonical return forms. If real external persisted data is identified before implementation lands, it requires an explicit migration decision rather than a permanent implicit shim.

Checkpoints from an in-progress old foreach execution are likewise not given a speculative compatibility path without real data. The implementation must not add parse-old behavior solely because repository fixtures previously used the old topology.

Testing

Focused tests must prove:

• serial item return wakes the parent and admits the next item;
• concurrent item returns preserve item identity and deterministic barrier commits;
• a child targeting its owner returns instead of executing the foreach node;
• a root frame targeting the same foreach enters it normally;
• an item frame can enter a different nested foreach;
• nested foreach restores the outer item context after inner completion;
• item paths to END and explicit EndNode fail validation;
• traces and serialized checkpoints retain an inspectable return location;
• existing fail, skip, collect, interrupt, reducer-conflict, sync, and async behavior remains intact after fixture migration.

Deferred Work

• Ergonomic/context-manager Python authoring syntax.
• break, continue nodes, target ports, or iteration return dispositions.
• First-completed, first-error, first-success, cancellation, or race policies.
• General unreachable-node and non-terminating-cycle diagnostics.
• Fork/gather nodes and activation-token persistence.
• Nested serialized workflow blocks.