From 9aaf1b4d780752bb948142ac9169f9677af38b82 Mon Sep 17 00:00:00 2001 From: lda Date: Fri, 4 Sep 2026 01:44:30 +0700 Subject: [PATCH] docs: specify foreach back edge semantics --- CONTEXT.md | 8 + docs/README.md | 2 + ...rent-foreach-policy-and-barrier-commits.md | 23 +++ ...xplicit-fork-and-topology-driven-gather.md | 20 +- docs/current_roadmap.md | 5 +- .../2026-09-04-foreach-back-edge-design.md | 193 ++++++++++++++++++ 6 files changed, 240 insertions(+), 11 deletions(-) create mode 100644 docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md diff --git a/CONTEXT.md b/CONTEXT.md index 140e5bc5..00ab9264 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -281,6 +281,12 @@ The end of one execution frame. Its owner determines whether this completes a workflow invocation, returns from a subgraph, or completes one foreach item. _Avoid_: Always completing the run, domain outcome, implicit break +**Foreach Return**: +Completion of one foreach item through a graph back-edge targeting the item's +owning foreach. The owner resumes its serial controller or concurrent barrier; +the item does not execute the foreach node again. +_Avoid_: Workflow end, generic graph cycle, implicit break + **Fork**: An explicit workflow control step that creates several concurrent branch activations. A fork is distinct from an ordinary outcome, which selects exactly @@ -567,6 +573,8 @@ _Avoid_: Job, invocation its runtime scope through that lineage. - Frame ancestry expresses scheduling ownership; lineage ancestry expresses state visibility. +- A **Foreach Return** completes an item frame and returns control to its owning + foreach without ending the workflow scope. - A **Frame Set** is the source of truth for runtime cursors. - The **Scheduler Foundation** selects one **Runnable Frame** at a time in the first pass. diff --git a/docs/README.md b/docs/README.md index 8c06aef9..b636aa80 100644 --- a/docs/README.md +++ b/docs/README.md @@ -114,6 +114,8 @@ docs as the active references: concurrent foreach policy and barrier commit semantics. - [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md): proposed explicit fork, gather-slot, and activation-token semantics. +- [`superpowers/specs/2026-09-04-foreach-back-edge-design.md`](superpowers/specs/2026-09-04-foreach-back-edge-design.md): + approved canonical foreach body-return and validation semantics. - [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md): native subgraph design. - [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md): diff --git a/docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md b/docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md index 8f80ec60..070c6bfb 100644 --- a/docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md +++ b/docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md @@ -185,6 +185,29 @@ Future foreach metadata should evolve into inherited structured lineage context: - normal node authors should receive foreach values through mapped input; inspecting runtime context is an advanced escape hatch +## Iteration Body Return + +An iteration body returns through an ordinary edge targeting its owning +`ForeachNode`. The runtime recognizes that target using the item frame's typed +owner metadata, completes the child frame, and wakes the parent barrier. It does +not execute the foreach controller inside the child frame. + +`END` and explicit `EndNode` are workflow/subgraph terminals, not foreach item +returns. Validation rejects either terminal while traversing an active foreach +item context. This replaces the earlier implementation convention in which an +item reached `END` to wake its parent. + +The canonical graph therefore uses: + +```text +foreach.loop -> body +body -> foreach +foreach.done -> continuation +``` + +The exact runtime, validation, migration, and nested-context rules are specified +in the [foreach back-edge design](../superpowers/specs/2026-09-04-foreach-back-edge-design.md). + ## Deferred Work Explicit Fork/Gather is deferred. A future `GatherNode` should expose explicit diff --git a/docs/adr/0006-explicit-fork-and-topology-driven-gather.md b/docs/adr/0006-explicit-fork-and-topology-driven-gather.md index 0070f373..b7727acf 100644 --- a/docs/adr/0006-explicit-fork-and-topology-driven-gather.md +++ b/docs/adr/0006-explicit-fork-and-topology-driven-gather.md @@ -97,12 +97,12 @@ overlap in the asynchronous runtime. Both modes must produce equivalent graph semantics. Scheduler order decides when compatible work progresses, never which arrivals belong together. -`END` means completion of the current execution frame, not necessarily -completion of the whole run. The frame owner determines the consequence: a -root frame completes its workflow invocation, a subgraph root returns to its -parent boundary, and a foreach item reports completion to its parent barrier. -For a foreach, normal item-frame completion therefore allows the controller to -admit or resume the next item without introducing a separate continue node. +`END` and explicit `EndNode` represent workflow/subgraph termination, not a +generic way to complete any child frame. A foreach item returns through a +back-edge targeting its owning `ForeachNode`; the runtime completes that item +frame and wakes its parent barrier without executing the controller inside the +child. Future fork branches likewise converge through explicit gathers rather +than independently terminating the containing workflow scope. General break and race behavior are not part of the first fork/gather design. The existing foreach `fail`, `skip`, and `collect` policies describe the @@ -141,10 +141,10 @@ those relationships are canonical on the lineage and duplicated frame fields can disagree with them. Runtime operations should resolve and validate a frame's lineage and scope together. -**Model normal foreach completion as a continue node.** Rejected because -completion of an item frame already returns control to the owning foreach. -Break and race semantics remain separate future policies rather than additional -meanings assigned to ordinary outcomes. +**Model normal foreach completion as a continue node.** Rejected because an +ordinary back-edge to the owning foreach already expresses item return in the +canonical graph. Break and race semantics remain separate future policies +rather than additional meanings assigned to ordinary outcomes. **Reuse or rename `JoinNode`.** Rejected as the default because the existing node is a pass-through marker with no barrier contract. diff --git a/docs/current_roadmap.md b/docs/current_roadmap.md index d59c1778..902f526c 100644 --- a/docs/current_roadmap.md +++ b/docs/current_roadmap.md @@ -795,7 +795,10 @@ stable. - Native subgraph polish: optional per-use-site child deployment overrides and clearer child trace inspection. -- Concurrent foreach polish: reuse barrier/lineage machinery for future +- Active concurrent foreach correction: replace item-body `END` routes with + canonical back-edges to the owning foreach. The approved semantics are in the + [`foreach back-edge design`](superpowers/specs/2026-09-04-foreach-back-edge-design.md). +- After that correction, reuse the foreach barrier/lineage machinery for fork/gather. The proposed control semantics are recorded in [`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md). - Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only diff --git a/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md b/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md new file mode 100644 index 00000000..40c9dd4c --- /dev/null +++ b/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md @@ -0,0 +1,193 @@ +# Foreach Back-Edge Design + +## Status + +Approved in conversation on 2026-09-04. This document specifies canonical +foreach body-return semantics. It does not include the separately planned +ergonomic Python DSL or authorize fork/gather implementation. + +## Purpose + +Make the persisted workflow graph tell the truth about foreach control flow. +An iteration body returns to its owning `ForeachNode` through an ordinary +back-edge. It no longer routes to the workflow terminal `END` merely to make an +internal child frame finish. + +The canonical shape is: + +```text +a -> foreach + foreach.loop -> body + body.ok -> foreach + foreach.done -> c +c -> END +``` + +This preserves a flat, inspectable control-flow graph. The runtime may continue +to implement each item with a child execution frame and a parent barrier, but +that scheduler detail no longer changes the visible meaning of `END`. + +## Decisions + +- `ForeachNode.loop` enters one iteration body. +- An edge from an iteration body to that iteration frame's owning + `ForeachNode` returns the item to the controller. +- Returning to the owner completes the item frame and wakes the blocked parent + foreach frame. The child does not execute the `ForeachNode` again. +- The parent foreach frame remains positioned at the controller. It admits the + next serial or concurrent item, or emits `done`/`completed_with_errors` after + its barrier finishes. +- A foreach item path may not target `END` or an explicit `EndNode`. Workflow + terminal routes are outside an iteration body. +- `END` remains the workflow/subgraph terminal shorthand for workflow outcome + `ok`; explicit `EndNode` remains the terminal for other workflow outcomes. +- Existing `fail`, `skip`, and `collect` item-error policies are unchanged. +- The core graph stays flat. Foreach does not gain a nested serialized body. +- The existing manual `WorkflowBuilder.foreach()` and `connect()` interface + remains the only authoring surface in this slice; no new authoring interface + is added. + +## Runtime Semantics + +The parent frame owns the foreach controller and resumable barrier. Taking its +`loop` outcome creates an item frame whose metadata names that parent foreach. +The item frame begins at the `loop` edge target and follows ordinary outcome +edges. + +Before ordinary frame advancement, runtime flow checks whether an item frame's +next target is its recorded owning foreach. If so, it records a foreach return: + +```python +owner = foreach_item_owner(frame) +if owner is not None and next_node_id == owner.foreach_node_id: + complete_item_frame(run, frame) + wake_parent_for_child_progress(run, frame.id) + return +``` + +This is an ownership check, not generic cycle detection. A root or unrelated +frame targeting the same foreach node enters it normally. An item frame +targeting a different foreach node also enters that node normally. Only a +return to the frame's recorded owner completes the current item. + +The completed child records the owning foreach as its terminal graph location +for trace and checkpoint inspection. Existing barrier code remains responsible +for consuming its lineage-local writes, applying item-error policy, and waking +or completing the parent. + +Serial and concurrent modes share the same return meaning: + +- serial mode has at most one live item frame and can admit the next item after + its return; +- concurrent mode may receive several independently identified item returns, + buffers them by item identity, and completes only after its existing barrier + policy is satisfied. + +## Nested Foreach + +Nested foreach remains structured by frame ownership: + +```text +outer.loop -> inner +inner.loop -> work +work -> inner +inner.done -> after_inner +after_inner -> outer +outer.done -> after_outer +``` + +The `work` frame returns only to `inner`, its immediate owner. Once the inner +controller emits `done`, execution resumes in the outer item frame. Returning +`after_inner -> outer` then completes the outer item. + +Runtime metadata already carries the immediate foreach owner. Static context +analysis must evolve from a single active foreach identifier to an ownership +stack so inner completion restores the outer item context. + +## Validation and Analysis + +Validation performs structured abstract traversal over `(node_id, +foreach_owner_stack)` rather than merely looking for graph cycles: + +- a `loop` edge from a foreach pushes that foreach onto the stack; +- an edge targeting the top owner is an item return and ends that child + traversal; +- a `done` or `completed_with_errors` edge stays in the outer context; +- `END` or `EndNode` reached with a non-empty foreach stack is invalid; +- nested returns must target the immediate owner before an outer owner; +- traversal memoizes node plus owner stack so valid cycles terminate analysis. + +A node may remain reachable in more than one execution context. Existing +context-contract analysis can report fields as conditional in that case. The +runtime must never infer edge meaning from graph history alone: frame ownership +always decides whether targeting a foreach is entry or return. + +General unreachable-node detection is useful but is not part of this change. +It cannot establish foreach ownership because any node reached from a body is +connected by definition. General termination is also out of scope; finite +structural checks cannot prove that data-dependent cycles eventually exit. + +## State and Failure Behavior + +Back-edge return changes control representation, not state semantics. +Iteration writes remain buffered in the item lineage. Serial behavior and the +concurrent barrier continue to commit or merge those writes according to the +accepted concurrent-foreach ADR and declared reducers. + +An ordinary node outcome named `error` remains domain control. An exception +remains a runtime item failure handled by `fail`, `skip`, or `collect`. Neither +kind of failure is encoded by the foreach back-edge itself. + +## Migration + +This is a clean canonical migration. Repository tests, examples, user-facing +docs, and generated fixtures change from: + +```text +foreach.loop -> body +body -> END +``` + +to: + +```text +foreach.loop -> body +body -> foreach +``` + +Configured local artifact stores contain no persisted foreach workflows at the +time of design. There is therefore no demonstrated persisted-data requirement +for keeping `body -> END` as a compatibility behavior. Validation rejects that +old shape rather than silently preserving two canonical return forms. If real +external persisted data is identified before implementation lands, it requires +an explicit migration decision rather than a permanent implicit shim. + +Checkpoints from an in-progress old foreach execution are likewise not given a +speculative compatibility path without real data. The implementation must not +add parse-old behavior solely because repository fixtures previously used the +old topology. + +## Testing + +Focused tests must prove: + +- serial item return wakes the parent and admits the next item; +- concurrent item returns preserve item identity and deterministic barrier + commits; +- a child targeting its owner returns instead of executing the foreach node; +- a root frame targeting the same foreach enters it normally; +- an item frame can enter a different nested foreach; +- nested foreach restores the outer item context after inner completion; +- item paths to `END` and explicit `EndNode` fail validation; +- traces and serialized checkpoints retain an inspectable return location; +- existing `fail`, `skip`, `collect`, interrupt, reducer-conflict, sync, and + async behavior remains intact after fixture migration. + +## Deferred Work + +- Ergonomic/context-manager Python authoring syntax. +- `break`, `continue` nodes, target ports, or iteration return dispositions. +- First-completed, first-error, first-success, cancellation, or race policies. +- General unreachable-node and non-terminating-cycle diagnostics. +- Fork/gather nodes and activation-token persistence. +- Nested serialized workflow blocks.