docs: specify foreach back edge semantics
This commit is contained in:
@@ -281,6 +281,12 @@ The end of one execution frame. Its owner determines whether this completes a
|
|||||||
workflow invocation, returns from a subgraph, or completes one foreach item.
|
workflow invocation, returns from a subgraph, or completes one foreach item.
|
||||||
_Avoid_: Always completing the run, domain outcome, implicit break
|
_Avoid_: Always completing the run, domain outcome, implicit break
|
||||||
|
|
||||||
|
**Foreach Return**:
|
||||||
|
Completion of one foreach item through a graph back-edge targeting the item's
|
||||||
|
owning foreach. The owner resumes its serial controller or concurrent barrier;
|
||||||
|
the item does not execute the foreach node again.
|
||||||
|
_Avoid_: Workflow end, generic graph cycle, implicit break
|
||||||
|
|
||||||
**Fork**:
|
**Fork**:
|
||||||
An explicit workflow control step that creates several concurrent branch
|
An explicit workflow control step that creates several concurrent branch
|
||||||
activations. A fork is distinct from an ordinary outcome, which selects exactly
|
activations. A fork is distinct from an ordinary outcome, which selects exactly
|
||||||
@@ -567,6 +573,8 @@ _Avoid_: Job, invocation
|
|||||||
its runtime scope through that lineage.
|
its runtime scope through that lineage.
|
||||||
- Frame ancestry expresses scheduling ownership; lineage ancestry expresses
|
- Frame ancestry expresses scheduling ownership; lineage ancestry expresses
|
||||||
state visibility.
|
state visibility.
|
||||||
|
- A **Foreach Return** completes an item frame and returns control to its owning
|
||||||
|
foreach without ending the workflow scope.
|
||||||
- A **Frame Set** is the source of truth for runtime cursors.
|
- A **Frame Set** is the source of truth for runtime cursors.
|
||||||
- The **Scheduler Foundation** selects one **Runnable Frame** at a time in the
|
- The **Scheduler Foundation** selects one **Runnable Frame** at a time in the
|
||||||
first pass.
|
first pass.
|
||||||
|
|||||||
@@ -114,6 +114,8 @@ docs as the active references:
|
|||||||
concurrent foreach policy and barrier commit semantics.
|
concurrent foreach policy and barrier commit semantics.
|
||||||
- [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md):
|
- [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md):
|
||||||
proposed explicit fork, gather-slot, and activation-token semantics.
|
proposed explicit fork, gather-slot, and activation-token semantics.
|
||||||
|
- [`superpowers/specs/2026-09-04-foreach-back-edge-design.md`](superpowers/specs/2026-09-04-foreach-back-edge-design.md):
|
||||||
|
approved canonical foreach body-return and validation semantics.
|
||||||
- [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md):
|
- [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md):
|
||||||
native subgraph design.
|
native subgraph design.
|
||||||
- [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md):
|
- [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md):
|
||||||
|
|||||||
@@ -185,6 +185,29 @@ Future foreach metadata should evolve into inherited structured lineage context:
|
|||||||
- normal node authors should receive foreach values through mapped input;
|
- normal node authors should receive foreach values through mapped input;
|
||||||
inspecting runtime context is an advanced escape hatch
|
inspecting runtime context is an advanced escape hatch
|
||||||
|
|
||||||
|
## Iteration Body Return
|
||||||
|
|
||||||
|
An iteration body returns through an ordinary edge targeting its owning
|
||||||
|
`ForeachNode`. The runtime recognizes that target using the item frame's typed
|
||||||
|
owner metadata, completes the child frame, and wakes the parent barrier. It does
|
||||||
|
not execute the foreach controller inside the child frame.
|
||||||
|
|
||||||
|
`END` and explicit `EndNode` are workflow/subgraph terminals, not foreach item
|
||||||
|
returns. Validation rejects either terminal while traversing an active foreach
|
||||||
|
item context. This replaces the earlier implementation convention in which an
|
||||||
|
item reached `END` to wake its parent.
|
||||||
|
|
||||||
|
The canonical graph therefore uses:
|
||||||
|
|
||||||
|
```text
|
||||||
|
foreach.loop -> body
|
||||||
|
body -> foreach
|
||||||
|
foreach.done -> continuation
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact runtime, validation, migration, and nested-context rules are specified
|
||||||
|
in the [foreach back-edge design](../superpowers/specs/2026-09-04-foreach-back-edge-design.md).
|
||||||
|
|
||||||
## Deferred Work
|
## Deferred Work
|
||||||
|
|
||||||
Explicit Fork/Gather is deferred. A future `GatherNode` should expose explicit
|
Explicit Fork/Gather is deferred. A future `GatherNode` should expose explicit
|
||||||
|
|||||||
@@ -97,12 +97,12 @@ overlap in the asynchronous runtime. Both modes must produce equivalent graph
|
|||||||
semantics. Scheduler order decides when compatible work progresses, never which
|
semantics. Scheduler order decides when compatible work progresses, never which
|
||||||
arrivals belong together.
|
arrivals belong together.
|
||||||
|
|
||||||
`END` means completion of the current execution frame, not necessarily
|
`END` and explicit `EndNode` represent workflow/subgraph termination, not a
|
||||||
completion of the whole run. The frame owner determines the consequence: a
|
generic way to complete any child frame. A foreach item returns through a
|
||||||
root frame completes its workflow invocation, a subgraph root returns to its
|
back-edge targeting its owning `ForeachNode`; the runtime completes that item
|
||||||
parent boundary, and a foreach item reports completion to its parent barrier.
|
frame and wakes its parent barrier without executing the controller inside the
|
||||||
For a foreach, normal item-frame completion therefore allows the controller to
|
child. Future fork branches likewise converge through explicit gathers rather
|
||||||
admit or resume the next item without introducing a separate continue node.
|
than independently terminating the containing workflow scope.
|
||||||
|
|
||||||
General break and race behavior are not part of the first fork/gather design.
|
General break and race behavior are not part of the first fork/gather design.
|
||||||
The existing foreach `fail`, `skip`, and `collect` policies describe the
|
The existing foreach `fail`, `skip`, and `collect` policies describe the
|
||||||
@@ -141,10 +141,10 @@ those relationships are canonical on the lineage and duplicated frame fields
|
|||||||
can disagree with them. Runtime operations should resolve and validate a
|
can disagree with them. Runtime operations should resolve and validate a
|
||||||
frame's lineage and scope together.
|
frame's lineage and scope together.
|
||||||
|
|
||||||
**Model normal foreach completion as a continue node.** Rejected because
|
**Model normal foreach completion as a continue node.** Rejected because an
|
||||||
completion of an item frame already returns control to the owning foreach.
|
ordinary back-edge to the owning foreach already expresses item return in the
|
||||||
Break and race semantics remain separate future policies rather than additional
|
canonical graph. Break and race semantics remain separate future policies
|
||||||
meanings assigned to ordinary outcomes.
|
rather than additional meanings assigned to ordinary outcomes.
|
||||||
|
|
||||||
**Reuse or rename `JoinNode`.** Rejected as the default because the existing
|
**Reuse or rename `JoinNode`.** Rejected as the default because the existing
|
||||||
node is a pass-through marker with no barrier contract.
|
node is a pass-through marker with no barrier contract.
|
||||||
|
|||||||
@@ -795,7 +795,10 @@ stable.
|
|||||||
|
|
||||||
- Native subgraph polish: optional per-use-site child deployment overrides and
|
- Native subgraph polish: optional per-use-site child deployment overrides and
|
||||||
clearer child trace inspection.
|
clearer child trace inspection.
|
||||||
- Concurrent foreach polish: reuse barrier/lineage machinery for future
|
- Active concurrent foreach correction: replace item-body `END` routes with
|
||||||
|
canonical back-edges to the owning foreach. The approved semantics are in the
|
||||||
|
[`foreach back-edge design`](superpowers/specs/2026-09-04-foreach-back-edge-design.md).
|
||||||
|
- After that correction, reuse the foreach barrier/lineage machinery for
|
||||||
fork/gather. The proposed control semantics are recorded in
|
fork/gather. The proposed control semantics are recorded in
|
||||||
[`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md).
|
[`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md).
|
||||||
- Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only
|
- Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only
|
||||||
|
|||||||
@@ -0,0 +1,193 @@
|
|||||||
|
# Foreach Back-Edge Design
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Approved in conversation on 2026-09-04. This document specifies canonical
|
||||||
|
foreach body-return semantics. It does not include the separately planned
|
||||||
|
ergonomic Python DSL or authorize fork/gather implementation.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Make the persisted workflow graph tell the truth about foreach control flow.
|
||||||
|
An iteration body returns to its owning `ForeachNode` through an ordinary
|
||||||
|
back-edge. It no longer routes to the workflow terminal `END` merely to make an
|
||||||
|
internal child frame finish.
|
||||||
|
|
||||||
|
The canonical shape is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
a -> foreach
|
||||||
|
foreach.loop -> body
|
||||||
|
body.ok -> foreach
|
||||||
|
foreach.done -> c
|
||||||
|
c -> END
|
||||||
|
```
|
||||||
|
|
||||||
|
This preserves a flat, inspectable control-flow graph. The runtime may continue
|
||||||
|
to implement each item with a child execution frame and a parent barrier, but
|
||||||
|
that scheduler detail no longer changes the visible meaning of `END`.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
- `ForeachNode.loop` enters one iteration body.
|
||||||
|
- An edge from an iteration body to that iteration frame's owning
|
||||||
|
`ForeachNode` returns the item to the controller.
|
||||||
|
- Returning to the owner completes the item frame and wakes the blocked parent
|
||||||
|
foreach frame. The child does not execute the `ForeachNode` again.
|
||||||
|
- The parent foreach frame remains positioned at the controller. It admits the
|
||||||
|
next serial or concurrent item, or emits `done`/`completed_with_errors` after
|
||||||
|
its barrier finishes.
|
||||||
|
- A foreach item path may not target `END` or an explicit `EndNode`. Workflow
|
||||||
|
terminal routes are outside an iteration body.
|
||||||
|
- `END` remains the workflow/subgraph terminal shorthand for workflow outcome
|
||||||
|
`ok`; explicit `EndNode` remains the terminal for other workflow outcomes.
|
||||||
|
- Existing `fail`, `skip`, and `collect` item-error policies are unchanged.
|
||||||
|
- The core graph stays flat. Foreach does not gain a nested serialized body.
|
||||||
|
- The existing manual `WorkflowBuilder.foreach()` and `connect()` interface
|
||||||
|
remains the only authoring surface in this slice; no new authoring interface
|
||||||
|
is added.
|
||||||
|
|
||||||
|
## Runtime Semantics
|
||||||
|
|
||||||
|
The parent frame owns the foreach controller and resumable barrier. Taking its
|
||||||
|
`loop` outcome creates an item frame whose metadata names that parent foreach.
|
||||||
|
The item frame begins at the `loop` edge target and follows ordinary outcome
|
||||||
|
edges.
|
||||||
|
|
||||||
|
Before ordinary frame advancement, runtime flow checks whether an item frame's
|
||||||
|
next target is its recorded owning foreach. If so, it records a foreach return:
|
||||||
|
|
||||||
|
```python
|
||||||
|
owner = foreach_item_owner(frame)
|
||||||
|
if owner is not None and next_node_id == owner.foreach_node_id:
|
||||||
|
complete_item_frame(run, frame)
|
||||||
|
wake_parent_for_child_progress(run, frame.id)
|
||||||
|
return
|
||||||
|
```
|
||||||
|
|
||||||
|
This is an ownership check, not generic cycle detection. A root or unrelated
|
||||||
|
frame targeting the same foreach node enters it normally. An item frame
|
||||||
|
targeting a different foreach node also enters that node normally. Only a
|
||||||
|
return to the frame's recorded owner completes the current item.
|
||||||
|
|
||||||
|
The completed child records the owning foreach as its terminal graph location
|
||||||
|
for trace and checkpoint inspection. Existing barrier code remains responsible
|
||||||
|
for consuming its lineage-local writes, applying item-error policy, and waking
|
||||||
|
or completing the parent.
|
||||||
|
|
||||||
|
Serial and concurrent modes share the same return meaning:
|
||||||
|
|
||||||
|
- serial mode has at most one live item frame and can admit the next item after
|
||||||
|
its return;
|
||||||
|
- concurrent mode may receive several independently identified item returns,
|
||||||
|
buffers them by item identity, and completes only after its existing barrier
|
||||||
|
policy is satisfied.
|
||||||
|
|
||||||
|
## Nested Foreach
|
||||||
|
|
||||||
|
Nested foreach remains structured by frame ownership:
|
||||||
|
|
||||||
|
```text
|
||||||
|
outer.loop -> inner
|
||||||
|
inner.loop -> work
|
||||||
|
work -> inner
|
||||||
|
inner.done -> after_inner
|
||||||
|
after_inner -> outer
|
||||||
|
outer.done -> after_outer
|
||||||
|
```
|
||||||
|
|
||||||
|
The `work` frame returns only to `inner`, its immediate owner. Once the inner
|
||||||
|
controller emits `done`, execution resumes in the outer item frame. Returning
|
||||||
|
`after_inner -> outer` then completes the outer item.
|
||||||
|
|
||||||
|
Runtime metadata already carries the immediate foreach owner. Static context
|
||||||
|
analysis must evolve from a single active foreach identifier to an ownership
|
||||||
|
stack so inner completion restores the outer item context.
|
||||||
|
|
||||||
|
## Validation and Analysis
|
||||||
|
|
||||||
|
Validation performs structured abstract traversal over `(node_id,
|
||||||
|
foreach_owner_stack)` rather than merely looking for graph cycles:
|
||||||
|
|
||||||
|
- a `loop` edge from a foreach pushes that foreach onto the stack;
|
||||||
|
- an edge targeting the top owner is an item return and ends that child
|
||||||
|
traversal;
|
||||||
|
- a `done` or `completed_with_errors` edge stays in the outer context;
|
||||||
|
- `END` or `EndNode` reached with a non-empty foreach stack is invalid;
|
||||||
|
- nested returns must target the immediate owner before an outer owner;
|
||||||
|
- traversal memoizes node plus owner stack so valid cycles terminate analysis.
|
||||||
|
|
||||||
|
A node may remain reachable in more than one execution context. Existing
|
||||||
|
context-contract analysis can report fields as conditional in that case. The
|
||||||
|
runtime must never infer edge meaning from graph history alone: frame ownership
|
||||||
|
always decides whether targeting a foreach is entry or return.
|
||||||
|
|
||||||
|
General unreachable-node detection is useful but is not part of this change.
|
||||||
|
It cannot establish foreach ownership because any node reached from a body is
|
||||||
|
connected by definition. General termination is also out of scope; finite
|
||||||
|
structural checks cannot prove that data-dependent cycles eventually exit.
|
||||||
|
|
||||||
|
## State and Failure Behavior
|
||||||
|
|
||||||
|
Back-edge return changes control representation, not state semantics.
|
||||||
|
Iteration writes remain buffered in the item lineage. Serial behavior and the
|
||||||
|
concurrent barrier continue to commit or merge those writes according to the
|
||||||
|
accepted concurrent-foreach ADR and declared reducers.
|
||||||
|
|
||||||
|
An ordinary node outcome named `error` remains domain control. An exception
|
||||||
|
remains a runtime item failure handled by `fail`, `skip`, or `collect`. Neither
|
||||||
|
kind of failure is encoded by the foreach back-edge itself.
|
||||||
|
|
||||||
|
## Migration
|
||||||
|
|
||||||
|
This is a clean canonical migration. Repository tests, examples, user-facing
|
||||||
|
docs, and generated fixtures change from:
|
||||||
|
|
||||||
|
```text
|
||||||
|
foreach.loop -> body
|
||||||
|
body -> END
|
||||||
|
```
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
```text
|
||||||
|
foreach.loop -> body
|
||||||
|
body -> foreach
|
||||||
|
```
|
||||||
|
|
||||||
|
Configured local artifact stores contain no persisted foreach workflows at the
|
||||||
|
time of design. There is therefore no demonstrated persisted-data requirement
|
||||||
|
for keeping `body -> END` as a compatibility behavior. Validation rejects that
|
||||||
|
old shape rather than silently preserving two canonical return forms. If real
|
||||||
|
external persisted data is identified before implementation lands, it requires
|
||||||
|
an explicit migration decision rather than a permanent implicit shim.
|
||||||
|
|
||||||
|
Checkpoints from an in-progress old foreach execution are likewise not given a
|
||||||
|
speculative compatibility path without real data. The implementation must not
|
||||||
|
add parse-old behavior solely because repository fixtures previously used the
|
||||||
|
old topology.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Focused tests must prove:
|
||||||
|
|
||||||
|
- serial item return wakes the parent and admits the next item;
|
||||||
|
- concurrent item returns preserve item identity and deterministic barrier
|
||||||
|
commits;
|
||||||
|
- a child targeting its owner returns instead of executing the foreach node;
|
||||||
|
- a root frame targeting the same foreach enters it normally;
|
||||||
|
- an item frame can enter a different nested foreach;
|
||||||
|
- nested foreach restores the outer item context after inner completion;
|
||||||
|
- item paths to `END` and explicit `EndNode` fail validation;
|
||||||
|
- traces and serialized checkpoints retain an inspectable return location;
|
||||||
|
- existing `fail`, `skip`, `collect`, interrupt, reducer-conflict, sync, and
|
||||||
|
async behavior remains intact after fixture migration.
|
||||||
|
|
||||||
|
## Deferred Work
|
||||||
|
|
||||||
|
- Ergonomic/context-manager Python authoring syntax.
|
||||||
|
- `break`, `continue` nodes, target ports, or iteration return dispositions.
|
||||||
|
- First-completed, first-error, first-success, cancellation, or race policies.
|
||||||
|
- General unreachable-node and non-terminating-cycle diagnostics.
|
||||||
|
- Fork/gather nodes and activation-token persistence.
|
||||||
|
- Nested serialized workflow blocks.
|
||||||
Reference in New Issue
Block a user