docs: record proposed fork gather semantics
This commit is contained in:
+27
-6
@@ -260,6 +260,33 @@ state, records trace, and persists runs predictably even when individual
|
|||||||
capabilities call nondeterministic tools, APIs, Python code, or LLMs.
|
capabilities call nondeterministic tools, APIs, Python code, or LLMs.
|
||||||
_Avoid_: Deterministic external tools, deterministic LLM output
|
_Avoid_: Deterministic external tools, deterministic LLM output
|
||||||
|
|
||||||
|
**Fork**:
|
||||||
|
An explicit workflow control step that creates several concurrent branch
|
||||||
|
activations. A fork is distinct from an ordinary outcome, which selects exactly
|
||||||
|
one continuation.
|
||||||
|
_Avoid_: Multiple matching edges, implicit broadcast, async handler call
|
||||||
|
|
||||||
|
**Gather**:
|
||||||
|
An explicit workflow control step that waits for compatible activation tokens
|
||||||
|
at every declared local input slot, merges their lineage-local state according
|
||||||
|
to reducers and gather policy, and emits one continuation. Several alternative
|
||||||
|
edges may satisfy one slot.
|
||||||
|
_Avoid_: Pass-through join marker, arbitrary graph convergence, reference to one
|
||||||
|
originating fork
|
||||||
|
|
||||||
|
**Activation Token**:
|
||||||
|
A runtime value identifying one control-flow arrival, its workflow scope,
|
||||||
|
lineage worldview, activation context, and merge provenance. Compatible tokens
|
||||||
|
can rendezvous at a gather without mixing loop iterations, subgraph
|
||||||
|
invocations, or repeated fork activations.
|
||||||
|
_Avoid_: Static edge, node output payload, scheduler position
|
||||||
|
|
||||||
|
**Gather Slot**:
|
||||||
|
A named local rendezvous requirement on a gather. A gather requires all of its
|
||||||
|
slots, while any one compatible incoming edge assigned to a slot can satisfy
|
||||||
|
that slot.
|
||||||
|
_Avoid_: Fork branch reference, positional input, executable edge predicate
|
||||||
|
|
||||||
**Planner Efficiency**:
|
**Planner Efficiency**:
|
||||||
The degree to which the platform reduces LLM trial-and-error when creating or
|
The degree to which the platform reduces LLM trial-and-error when creating or
|
||||||
running workflows. Typed schemas, source catalogs, validation errors, dry
|
running workflows. Typed schemas, source catalogs, validation errors, dry
|
||||||
@@ -443,12 +470,6 @@ Persisted deployments and server-side execution make scheduling plausible, but
|
|||||||
scheduling itself is not implemented yet.
|
scheduling itself is not implemented yet.
|
||||||
_Avoid_: Claiming production background scheduling exists today
|
_Avoid_: Claiming production background scheduling exists today
|
||||||
|
|
||||||
**Fork/Gather Workflow Control**:
|
|
||||||
Future workflow control for parallel branches and explicit gather/join behavior.
|
|
||||||
The current product should not claim general fork/gather orchestration yet.
|
|
||||||
_Avoid_: Treating current foreach or serial graph execution as full parallel
|
|
||||||
workflow orchestration
|
|
||||||
|
|
||||||
**Scheduler Foundation**:
|
**Scheduler Foundation**:
|
||||||
The runtime model that selects runnable frames and advances workflow execution without assuming there is only one active cursor.
|
The runtime model that selects runnable frames and advances workflow execution without assuming there is only one active cursor.
|
||||||
_Avoid_: Foreach feature, concurrent foreach implementation
|
_Avoid_: Foreach feature, concurrent foreach implementation
|
||||||
|
|||||||
@@ -112,6 +112,8 @@ docs as the active references:
|
|||||||
scheduler foundation decision.
|
scheduler foundation decision.
|
||||||
- [`adr/0002-concurrent-foreach-policy-and-barrier-commits.md`](adr/0002-concurrent-foreach-policy-and-barrier-commits.md):
|
- [`adr/0002-concurrent-foreach-policy-and-barrier-commits.md`](adr/0002-concurrent-foreach-policy-and-barrier-commits.md):
|
||||||
concurrent foreach policy and barrier commit semantics.
|
concurrent foreach policy and barrier commit semantics.
|
||||||
|
- [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md):
|
||||||
|
proposed explicit fork, gather-slot, and activation-token semantics.
|
||||||
- [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md):
|
- [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md):
|
||||||
native subgraph design.
|
native subgraph design.
|
||||||
- [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md):
|
- [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md):
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
---
|
||||||
|
status: proposed
|
||||||
|
---
|
||||||
|
|
||||||
|
# Explicit Fork and Topology-Driven Gather
|
||||||
|
|
||||||
|
General graph concurrency will use explicit fork and gather steps. Outcomes
|
||||||
|
continue to select one transition; forks create several branch activations;
|
||||||
|
gathers rendezvous compatible activation tokens, merge their lineage-local
|
||||||
|
state patches, and create one continuation. This preserves topology-only edges
|
||||||
|
without making multiple matching edges silently mean broadcast.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The scheduler, ready queue, blocked frames, lineage-local state views, and
|
||||||
|
reducer-aware barrier commits already support concurrent foreach and native
|
||||||
|
subgraphs. They do not yet define general graph-level fork/gather. The existing
|
||||||
|
`JoinNode` is only a day-one marker: it immediately emits `done` and neither
|
||||||
|
waits nor merges. Renaming it would falsely preserve semantics it never had.
|
||||||
|
|
||||||
|
A cross-system semantics review supported keeping `NodeResult.output` separate
|
||||||
|
from its named domain `outcome`, keeping operational failure outside that
|
||||||
|
outcome namespace, and representing concurrency with explicit control steps.
|
||||||
|
The decisive design pressure came from partial and repeated gathers rather than
|
||||||
|
from output typing.
|
||||||
|
|
||||||
|
Consider a fork producing `a`, `b`, and `c`. One gather may merge `a+b` into
|
||||||
|
`d`, while a later gather merges `d+c`. The first gather must not know which
|
||||||
|
fork originally produced its inputs, and the second must accept the merged
|
||||||
|
`d` continuation like any other arrival. Inside a loop, it must combine
|
||||||
|
`a1+b1`, never `a1+b2`.
|
||||||
|
|
||||||
|
Conditional paths add another requirement. If one branch continues through
|
||||||
|
either `d` or `e`, a later gather may require `b AND (d OR e)`. Waiting for
|
||||||
|
every incoming edge would deadlock because only one alternative can arrive.
|
||||||
|
|
||||||
|
Cross-merging raises the same issue at larger scale. If one activated region
|
||||||
|
produces `b,c` and another produces `e,f`, independent gathers may consume
|
||||||
|
`b+e` and `c+f`. Correctness cannot depend on whether the scheduler happens to
|
||||||
|
run `b,e,c,f` or `c,f,b,e` first.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The graph model distinguishes three control meanings:
|
||||||
|
|
||||||
|
- an ordinary node outcome selects exactly one continuation;
|
||||||
|
- a fork creates one child activation for every declared branch;
|
||||||
|
- a gather consumes compatible arrivals and creates one continuation.
|
||||||
|
|
||||||
|
A gather is independent of the construct that produced its inputs. It knows
|
||||||
|
its required local input slots from the graph and owns a merge policy. Edges
|
||||||
|
targeting a gather identify the slot they can satisfy. A gather requires every
|
||||||
|
slot, while several incoming edges may be alternatives for one slot. This gives
|
||||||
|
`b AND (d OR e)` without placing executable predicates on edges.
|
||||||
|
|
||||||
|
Runtime arrivals carry activation identity, lineage identity, and provenance.
|
||||||
|
The gather buckets arrivals by compatible activation context, accepts at most
|
||||||
|
one token per slot for an activation, waits until all slots are present, then
|
||||||
|
merges their lineage-local patches. Its result is a new continuation token that
|
||||||
|
preserves combined provenance and can enter another gather.
|
||||||
|
|
||||||
|
Lineage remains a virtual worldview: committed scope state plus writes visible
|
||||||
|
to one branch. Gathering several lineages does not require turning lineage
|
||||||
|
ancestry into a multi-parent graph. A partial gather can create an intermediate
|
||||||
|
lineage under the branches' common parent, retaining multi-input provenance in
|
||||||
|
activation-token metadata. A final gather can merge that lineage with remaining
|
||||||
|
siblings and resume the blocked parent continuation.
|
||||||
|
|
||||||
|
The first gather merge policy is fail-closed:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class GatherMergePolicy:
|
||||||
|
conflicts: Literal["error"] = "error"
|
||||||
|
```
|
||||||
|
|
||||||
|
State-field reducers remain the source of truth for legitimate concurrent
|
||||||
|
merges. The gather policy determines what happens when patches cannot be
|
||||||
|
merged; the initial behavior is to fail rather than choose a last writer.
|
||||||
|
|
||||||
|
Branch execution order may be deterministic in the synchronous runtime and
|
||||||
|
overlap in the asynchronous runtime. Both modes must produce equivalent graph
|
||||||
|
semantics. Scheduler order decides when compatible work progresses, never which
|
||||||
|
arrivals belong together.
|
||||||
|
|
||||||
|
The current `JoinNode` will not be silently upgraded. Before implementation we
|
||||||
|
will verify whether real persisted artifacts use it. With no real compatibility
|
||||||
|
obligation, remove it and introduce `GatherNode` cleanly. If persisted callers
|
||||||
|
exist, define an explicit migration rather than assigning barrier semantics to
|
||||||
|
old `join` payloads.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
**Multiple ordinary edges broadcast.** Rejected because edge cardinality would
|
||||||
|
silently change an outcome from selection to spawning and make ordinary graph
|
||||||
|
convergence ambiguous.
|
||||||
|
|
||||||
|
**Gather references its originating fork.** Rejected because partial gathers,
|
||||||
|
staged gathers, conditional paths, and cross-merges consume tokens based on
|
||||||
|
their local rendezvous contract, not one producer.
|
||||||
|
|
||||||
|
**Gather waits for every incoming edge.** Rejected because alternative paths
|
||||||
|
such as `d OR e` would require mutually exclusive arrivals. Named slots provide
|
||||||
|
AND across slots and OR within a slot.
|
||||||
|
|
||||||
|
**Lineage becomes a multi-parent DAG.** Not currently required. Merge provenance
|
||||||
|
belongs to activation tokens; an intermediate merged worldview can remain a
|
||||||
|
child of the compatible inputs' common lineage parent.
|
||||||
|
|
||||||
|
**Reuse or rename `JoinNode`.** Rejected as the default because the existing
|
||||||
|
node is a pass-through marker with no barrier contract.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Edge identity gains gather-slot significance only when its target is a
|
||||||
|
gather; ordinary edge semantics stay unchanged.
|
||||||
|
- Workflow validation must prove that every gather slot has an incoming edge,
|
||||||
|
reject slots on non-gather targets, and preserve one successor per ordinary
|
||||||
|
`(node, outcome)` pair.
|
||||||
|
- Checkpoints must persist pending gather arrivals and activation provenance so
|
||||||
|
interruption/resume cannot mix loop iterations or subgraph invocations.
|
||||||
|
- Trace output must make fork activation, branch identity, gather waiting, and
|
||||||
|
merged continuation inspectable without treating scheduler bookkeeping as
|
||||||
|
ordinary node output.
|
||||||
|
- Fork/gather should generalize concurrent-foreach lineage and barrier helpers,
|
||||||
|
not create a second state-patch system.
|
||||||
|
- Runtime branch failures remain execution failures in the first version. Skip,
|
||||||
|
collect, race, first-success, cancellation, and timeout policies are deferred.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- The exact serialized edge field and authoring name for a gather slot.
|
||||||
|
- The minimal activation-context and provenance representation that supports
|
||||||
|
loops, nested forks, subgraphs, partial gathers, and cross-merges.
|
||||||
|
- Whether a gather resumes an existing blocked frame or creates a dedicated
|
||||||
|
continuation frame in each topology shape.
|
||||||
|
- The trace representation for waiting and merging without excessive internal
|
||||||
|
scheduler noise.
|
||||||
|
- Whether any real persisted artifact requires migration from `JoinNode`.
|
||||||
|
|
||||||
|
This ADR extends the lineage and barrier direction established by
|
||||||
|
[ADR-0002](0002-concurrent-foreach-policy-and-barrier-commits.md). It remains
|
||||||
|
proposed until the open runtime-state questions are resolved in the fork/gather
|
||||||
|
design specification.
|
||||||
@@ -796,7 +796,8 @@ stable.
|
|||||||
- Native subgraph polish: optional per-use-site child deployment overrides and
|
- Native subgraph polish: optional per-use-site child deployment overrides and
|
||||||
clearer child trace inspection.
|
clearer child trace inspection.
|
||||||
- Concurrent foreach polish: reuse barrier/lineage machinery for future
|
- Concurrent foreach polish: reuse barrier/lineage machinery for future
|
||||||
fork/gather.
|
fork/gather. The proposed control semantics are recorded in
|
||||||
|
[`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md).
|
||||||
- Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only
|
- Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only
|
||||||
after polling `wf run watch` proves insufficient.
|
after polling `wf run watch` proves insufficient.
|
||||||
- OpenAPI sources: continue from [`openapi capability sources`](openapi_capability_source.md)
|
- OpenAPI sources: continue from [`openapi capability sources`](openapi_capability_source.md)
|
||||||
|
|||||||
Reference in New Issue
Block a user