diff --git a/CONTEXT.md b/CONTEXT.md index 00ab9264..af84fa11 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -276,6 +276,11 @@ ancestry describes scheduling ownership and block/wake behavior, not state ancestry. _Avoid_: State lineage, workflow invocation, operating-system thread +**Control Region**: +The one static foreach-owner stack assigned to a workflow node use from graph +entry. A node use cannot belong to several control regions. +_Avoid_: Execution frame, state lineage, dynamic call stack + **Frame Completion**: The end of one execution frame. Its owner determines whether this completes a workflow invocation, returns from a subgraph, or completes one foreach item. @@ -287,6 +292,11 @@ owning foreach. The owner resumes its serial controller or concurrent barrier; the item does not execute the foreach node again. _Avoid_: Workflow end, generic graph cycle, implicit break +**Foreach Activation**: +One dynamic visit to a foreach controller, owning that visit's barrier and item +executions. Re-entering the same node use creates another activation. +_Avoid_: Foreach node use, control region, item frame + **Fork**: An explicit workflow control step that creates several concurrent branch activations. A fork is distinct from an ordinary outcome, which selects exactly @@ -573,6 +583,9 @@ _Avoid_: Job, invocation its runtime scope through that lineage. - Frame ancestry expresses scheduling ownership; lineage ancestry expresses state visibility. +- Every workflow node use belongs to exactly one static **Control Region**. +- Each dynamic visit to a foreach node use creates a distinct **Foreach + Activation** whose item frames share that activation identity. - A **Foreach Return** completes an item frame and returns control to its owning foreach without ending the workflow scope. - A **Frame Set** is the source of truth for runtime cursors. diff --git a/docs/current_roadmap.md b/docs/current_roadmap.md index 902f526c..27578115 100644 --- a/docs/current_roadmap.md +++ b/docs/current_roadmap.md @@ -798,6 +798,9 @@ stable. - Active concurrent foreach correction: replace item-body `END` routes with canonical back-edges to the owning foreach. The approved semantics are in the [`foreach back-edge design`](superpowers/specs/2026-09-04-foreach-back-edge-design.md). + This slice also makes foreach control regions fail-closed, rejects unreachable + workflow nodes, and gives repeated visits to one foreach node distinct + persisted activation identities. - After that correction, reuse the foreach barrier/lineage machinery for fork/gather. The proposed control semantics are recorded in [`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md). diff --git a/docs/superpowers/specs/2026-08-14-workflow-console-contract-graph-design.md b/docs/superpowers/specs/2026-08-14-workflow-console-contract-graph-design.md index 214c58f5..7f15e97e 100644 --- a/docs/superpowers/specs/2026-08-14-workflow-console-contract-graph-design.md +++ b/docs/superpowers/specs/2026-08-14-workflow-console-contract-graph-design.md @@ -149,8 +149,10 @@ execution-frame semantics: - always-available frame keys may be suggested for every executable step; - foreach keys are suggested when the selected step is proven to execute in the child frame entered through a foreach `loop` route; -- a node reachable both inside and outside that frame receives conditional - entries with a warning rather than a false guarantee; +- the approved [foreach back-edge design](2026-09-04-foreach-back-edge-design.md) + rejects a node reachable both inside and outside that frame as a + control-region conflict, rather than representing two owner stacks as + conditional context; - ambiguous or malformed control flow never produces a guaranteed scoped key; and - nested foreach scopes expose only the keys actually carried by the current @@ -162,7 +164,8 @@ This analysis is intentionally conservative and advisory. Existing validation accepts syntactically valid custom `context.*` paths because the runtime may provide extensions that static analysis does not know. Runtime resolution therefore remains authoritative; this slice does not claim strict static -context-scope enforcement. +context-path enforcement. Structural foreach control-region validation is the +separate responsibility specified by the later foreach back-edge design. ## Source And Target Picker @@ -278,7 +281,7 @@ The implementation must include: - unit tests for schema flattening and normalized inventory entries; - graph-scope tests for common context, serial/concurrent foreach bodies, - aliases, nested foreach, mixed reachability, and malformed graphs; + aliases, nested foreach, control-region conflicts, and malformed graphs; - operation-model, protocol dispatch, generated-schema parity, and browser allowlist tests for inventory inspection; - picker tests for grouping, search, nested fields, compatibility, conditional diff --git a/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md b/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md index 40c9dd4c..aeeb58da 100644 --- a/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md +++ b/docs/superpowers/specs/2026-09-04-foreach-back-edge-design.md @@ -37,6 +37,9 @@ that scheduler detail no longer changes the visible meaning of `END`. - The parent foreach frame remains positioned at the controller. It admits the next serial or concurrent item, or emits `done`/`completed_with_errors` after its barrier finishes. +- Each dynamic entry into a foreach controller creates a fresh persisted + foreach activation. Its barrier and item frames belong to that activation, + so revisiting the same node use cannot reuse completed iteration state. - A foreach item path may not target `END` or an explicit `EndNode`. Workflow terminal routes are outside an iteration body. - `END` remains the workflow/subgraph terminal shorthand for workflow outcome @@ -49,10 +52,11 @@ that scheduler detail no longer changes the visible meaning of `END`. ## Runtime Semantics -The parent frame owns the foreach controller and resumable barrier. Taking its -`loop` outcome creates an item frame whose metadata names that parent foreach. -The item frame begins at the `loop` edge target and follows ordinary outcome -edges. +The parent frame owns the foreach controller and resumable barrier. On first +entry for one visit, it creates a persisted foreach activation identity. Taking +its `loop` outcome creates an item frame whose metadata names both the owning +foreach node use and that activation. The item frame begins at the `loop` edge +target and follows ordinary outcome edges. Before ordinary frame advancement, runtime flow checks whether an item frame's next target is its recorded owning foreach. If so, it records a foreach return: @@ -60,21 +64,34 @@ next target is its recorded owning foreach. If so, it records a foreach return: ```python owner = foreach_item_owner(frame) if owner is not None and next_node_id == owner.foreach_node_id: - complete_item_frame(run, frame) + complete_item_frame(run, frame, owner.activation_id) wake_parent_for_child_progress(run, frame.id) return ``` This is an ownership check, not generic cycle detection. A root or unrelated -frame targeting the same foreach node enters it normally. An item frame -targeting a different foreach node also enters that node normally. Only a -return to the frame's recorded owner completes the current item. +frame targeting the same foreach node enters it normally. An item frame may +enter a different foreach node only when that node is not already in its active +owner stack; that is an ordinary nested foreach entry. Targeting a non-immediate +ancestor foreach is an invalid non-local return and must fail closed rather than +re-entering the ancestor controller. Only a return to the frame's immediate +recorded owner completes the current item. The completed child records the owning foreach as its terminal graph location for trace and checkpoint inspection. Existing barrier code remains responsible for consuming its lineage-local writes, applying item-error policy, and waking or completing the parent. +The activation identity is dynamic; it is separate from the static owner stack +of foreach node-use identifiers. It must remain stable across checkpoints and +interrupts. Barrier lookup, child-frame identity, item result ownership, and +wake-up checks include the activation identity. When the parent emits `done` or +`completed_with_errors`, it closes that activation before following the +outgoing edge. A later visit to the same foreach node use in the same parent +frame creates a new activation with fresh barrier state and child identities. +Completed frames and traces may remain as history without colliding with the +new visit. + Serial and concurrent modes share the same return meaning: - serial mode has at most one live item frame and can admit the next item after @@ -104,6 +121,66 @@ Runtime metadata already carries the immediate foreach owner. Static context analysis must evolve from a single active foreach identifier to an ownership stack so inner completion restores the outer item context. +The owner stack describes control, not flat alias inheritance. This slice keeps +the current context contract: an inner item body exposes the innermost +`loop_item`, `loop_index`, and configured alias; after the inner foreach emits +`done`, the surrounding item frame exposes the outer values again. Inherited +structured foreach context remains separate future work. + +## Static Control Regions and Dynamic Executions + +A node use has exactly one static control region, represented by its foreach +owner stack: + +```python +control_region(node_use) == ("outer_foreach", "inner_foreach") +``` + +That same node use may execute dynamically many times: + +```text +same node use +├── item 0 frame / lineage +├── item 1 frame / lineage +└── item 2 frame / lineage +``` + +The invariant is therefore one static program location with many possible +dynamic executions. When one reusable capability is needed at several program +locations, authoring creates distinct node uses: + +```python +before = builder.use(clean_document, id="clean_before") +each = builder.use(clean_document, id="clean_each") +after = builder.use(clean_document, id="clean_after") +``` + +Ordinary cycles remain valid when every node in the cycle belongs to the same +control region: + +```text +a -> b +^ | +└────┘ +``` + +Both nodes have the empty owner stack. This is an ordinary graph loop, like +`goto a`. Whether it exits is the author's responsibility. The runtime does not +currently promise a general step limit, so this design must not claim one. + +A cycle within a foreach body is valid for the same reason when it has a +possible return to the owner: + +```text +f.loop -> a -> b + ^ | + └────┘ +b.exit -> f +``` + +Here both `a` and `b` belong to `("f",)`. The cycle may repeat before returning +the current iteration to `f`. + ## Validation and Analysis Validation performs structured abstract traversal over `(node_id, @@ -114,18 +191,242 @@ foreach_owner_stack)` rather than merely looking for graph cycles: traversal; - a `done` or `completed_with_errors` edge stays in the outer context; - `END` or `EndNode` reached with a non-empty foreach stack is invalid; -- nested returns must target the immediate owner before an outer owner; +- an edge targeting a non-top foreach already in the stack is an invalid + non-local return; +- re-entering any active ancestor foreach as a nested controller is invalid; +- every node use has exactly one static foreach-owner stack. Reaching the same + node use under another stack is an invalid control-region crossing; +- every node reached inside a foreach body has at least one structural path + back to its immediate owner without leaving that owner context; - traversal memoizes node plus owner stack so valid cycles terminate analysis. -A node may remain reachable in more than one execution context. Existing -context-contract analysis can report fields as conditional in that case. The -runtime must never infer edge meaning from graph history alone: frame ownership -always decides whether targeting a foreach is entry or return. +A node use therefore belongs to one static control region, while remaining free +to execute in any number of dynamic frames, lineages, or items. When the same +capability is needed at two program locations, authoring creates two node uses +with distinct identifiers. Existing context-contract analysis may still report +fields as conditional because multiple paths can reach a node within its one +region; it must not use multiple owner stacks to represent that case. -General unreachable-node detection is useful but is not part of this change. -It cannot establish foreach ownership because any node reached from a body is -connected by definition. General termination is also out of scope; finite -structural checks cannot prove that data-dependent cycles eventually exit. +The unique-owner rule rejects both ways of crossing a foreach boundary. An +outside edge into a body node reaches that node under both the outer and item +stacks. A body edge into an outside continuation reaches that continuation +under both the item and outer stacks. Both graphs are invalid rather than +silently annexing nodes into or out of the body. + +The structural-return check is intentionally weaker than proving termination. +A data-dependent cycle is valid when some graph path can return to the immediate +owner; it may still run forever for particular inputs, just like an ordinary +program loop. A closed body cycle with no return path is invalid. The initially +ambiguous empty-body shape `foreach.loop -> foreach` is also invalid; an +iteration body must contain at least one distinct node use. + +The runtime must never infer edge meaning from graph history alone: validated +static ownership establishes the legal regions, and current frame ownership +decides whether targeting a foreach is entry or immediate return. + +Validation rejects every workflow node that is unreachable from the workflow +start. An unreachable node has no derivable control region, so accepting it +would contradict the one-region-per-node-use invariant and leave malformed +disconnected foreach structures unchecked. This applies to all workflow node +types, not only foreach bodies. + +General termination remains out of scope. Finite structural checks cannot prove +that data-dependent cycles eventually exit. The narrower foreach rule only +requires a possible graph path from every body node back to its immediate +owner, because otherwise the enclosing controller is structurally unable to +finish that item. + +The validation line is semantic ambiguity or structural impossibility. Reject +a graph when its control region cannot be derived uniquely, when it crosses a +structured foreach boundary illegally, or when an item has no possible return. +Accept a graph with one coherent meaning when runtime data alone determines +whether an available exit is taken. Non-termination in that accepted case is an +authoring error, not something static validation can honestly predict. + +### Transition Classification + +For every transition, validation applies these rules in order: + +| Transition | Meaning | +| --- | --- | +| Source and target remain in the same region | Ordinary edge | +| Target is the immediate foreach owner | Item return | +| Target is a new, inactive foreach | Enter nested foreach | +| Target is an older ancestor owner | Invalid non-local return | +| Target already belongs to another region | Invalid region crossing | +| Target is `END` while inside a foreach | Invalid workflow termination | + +### Pressure Cases + +These examples are normative validation cases rather than illustrative syntax +alone. + +#### External Entry Into a Body + +```text +start.true -> f +start.false -> b +f.loop -> b +b -> f +``` + +`b` is reachable under both `()` and `("f",)`. Validation rejects the graph. +Use two distinct node uses when both executions are intentional. + +#### Body Escape Into a Post-Loop Continuation + +```text +f.loop -> b -> after +f.done ------> after +``` + +`after` is reachable under both `("f",)` and `()`, so validation rejects the +graph. The correct structure is: + +```text +f.loop -> b -> f +f.done -> after +``` + +#### Legal Nested Return + +```text +f1.loop -> f2 +f2.loop -> work +work -> f2 +f2.done -> tail +tail -> f1 +f1.done -> after +``` + +The static owner stacks are: + +```text +f2: ("f1",) +work: ("f1", "f2") +tail: ("f1",) +after: () +``` + +#### Skipping an Inner Owner + +```text +f1.loop -> f2 +f2.loop -> work +work -> f1 +``` + +At `work`, the owner stack is `("f1", "f2")`. Returning directly to `f1` +would skip `f2`, so validation rejects the graph as a non-local return. + +#### Entering a Sibling Body + +```text +f1.loop -> b1 -> b2 +f2.loop -------> b2 +``` + +`b2` would belong to both `("f1",)` and `("f2",)`. Validation rejects the +graph. + +#### Empty Body + +```text +f.loop -> f +``` + +This is ambiguous because the item frame would begin on its owner. Validation +rejects it. An empty foreach has no useful item-level state or output effect. + +#### Conditional Body Returns + +```text +f.loop -> condition +condition.true -> work -> f +condition.false -------> f +``` + +This is valid. Both outcomes return the current item normally. + +#### Body Cycle With No Possible Return + +```text +f.loop -> a -> b -> a +``` + +Although every node has the correct region, the body has no path back to `f`. +Validation rejects this structurally stuck graph. + +The following graph remains valid but may run forever for some runtime data: + +```text +f.loop -> a +a.again -> a +a.done -> f +``` + +The graph contains a possible return. Runtime data decides whether it happens. + +#### Unreachable Node or Component + +```text +start -> work -> END + +detached_a -> detached_b -> detached_a +``` + +Validation rejects both detached nodes. Starting a second abstract traversal at +an arbitrary default owner stack would invent a control region rather than +derive one from program entry. + +#### Re-entering the Same Foreach + +```text +again.true -> f +f.loop -> work -> f +f.done -> again +again.false -> END +``` + +This is valid. Every visit to `f` creates a fresh foreach activation, even when +the same parent frame visited `f` before. Each activation starts its barrier at +item zero and gives its child frames distinct identities. Runtime data decides +how many visits occur. + +#### Subgraph Inside Foreach + +```text +f.loop -> child_workflow -> f +``` + +This is valid. The child's own `END` completes its child workflow scope. The +parent `SubgraphNode` then returns to `f`, completing the foreach item: + +```text +outer workflow scope +└── foreach item frame / lineage + └── child workflow scope +``` + +#### Interrupt Inside Foreach + +```text +f.loop -> ask_user -> work -> f +``` + +The item frame suspends while the parent foreach remains blocked. Resume must +restore the same frame, lineage, item identity, and owner stack. + +#### Future Fork Inside Foreach + +```text +f.loop -> fork -> left/right -> gather -> f +``` + +This design does not add fork/gather, but it fixes the future constraint. Both +branch activations inherit the foreach activation identity, item identity, and +static owner stack. They must gather before returning to `f`; neither branch +may independently return and complete the item. ## State and Failure Behavior @@ -169,17 +470,58 @@ old topology. ## Testing -Focused tests must prove: +Every pressure case maps to at least one focused test. Invalid cases assert the +diagnostic category and offending node or edge path, not just `report.ok is +False`. Valid cases assert clean validation; cases with observable execution +semantics also run through the runtime. + +| Case | Validation test | Runtime test | +| --- | --- | --- | +| Closed root cycle | Accept; author owns termination | Step; remain running | +| Root cycle with an exit | Accept | Take a finite loop and exit | +| Foreach cycle with a return | Accept | Repeat, then return the item | +| External body entry | Reject region conflict | N/A | +| Body escape | Reject region conflict | N/A | +| Legal nested return | Accept | Complete inner, restore outer, then return | +| Skip inner owner | Reject non-local return | Defensive invariant failure | +| Sibling-body entry | Reject region conflict | N/A | +| Empty `loop -> owner` body | Reject empty body | N/A | +| Conditional body returns | Accept | Exercise both return paths | +| Closed body cycle | Reject missing owner return | N/A | +| Unreachable nodes | Reject each node | N/A | +| Re-enter foreach after `done` | Accept | Fresh activation and children | +| Subgraph inside foreach | Accept | Child `END`, then item return | +| Interrupt inside foreach | Accept | Resume the same item activation | +| Future fork in foreach | Deferred with fork/gather | Gather before return | + +The future-fork row is an acceptance test owned by the later fork/gather slice; +this slice cannot instantiate a graph node type that does not yet exist. It +must remain visible here so that implementation cannot weaken foreach ownership +when fork/gather lands. + +Additional focused tests must prove: - serial item return wakes the parent and admits the next item; - concurrent item returns preserve item identity and deterministic barrier commits; - a child targeting its owner returns instead of executing the foreach node; - a root frame targeting the same foreach enters it normally; -- an item frame can enter a different nested foreach; +- an item frame can enter a different, inactive nested foreach; - nested foreach restores the outer item context after inner completion; +- nested foreach keeps the existing innermost-only context contract inside the + inner body; +- targeting a non-immediate ancestor foreach fails validation and at runtime; +- one node use reached under multiple owner stacks fails validation, covering + both outside entry into a body and escape from a body to an outside node; +- ordinary cycles within one owner stack remain valid when they have a + structural return path; +- a closed body cycle with no path back to its owner fails validation; +- an empty `loop` self-return fails validation; - item paths to `END` and explicit `EndNode` fail validation; - traces and serialized checkpoints retain an inspectable return location; +- foreach activation identity survives checkpoint serialization and interrupt; +- a completed activation cannot consume a later activation's item result or + wake its parent; - existing `fail`, `skip`, `collect`, interrupt, reducer-conflict, sync, and async behavior remains intact after fixture migration. @@ -188,6 +530,7 @@ Focused tests must prove: - Ergonomic/context-manager Python authoring syntax. - `break`, `continue` nodes, target ports, or iteration return dispositions. - First-completed, first-error, first-success, cancellation, or race policies. -- General unreachable-node and non-terminating-cycle diagnostics. +- General non-terminating-cycle diagnostics beyond foreach-body returnability. - Fork/gather nodes and activation-token persistence. +- Inherited structured context for all active foreach owners. - Nested serialized workflow blocks.