docs: record proposed fork gather semantics

2026-09-03 22:42:50 +07:00
parent 327a7b24b0
commit d9559bfbd2
4 changed files with 174 additions and 7 deletions
@@ -260,6 +260,33 @@ state, records trace, and persists runs predictably even when individual
 capabilities call nondeterministic tools, APIs, Python code, or LLMs.
 _Avoid_: Deterministic external tools, deterministic LLM output

+**Fork**:
+An explicit workflow control step that creates several concurrent branch
+activations. A fork is distinct from an ordinary outcome, which selects exactly
+one continuation.
+_Avoid_: Multiple matching edges, implicit broadcast, async handler call
+
+**Gather**:
+An explicit workflow control step that waits for compatible activation tokens
+at every declared local input slot, merges their lineage-local state according
+to reducers and gather policy, and emits one continuation. Several alternative
+edges may satisfy one slot.
+_Avoid_: Pass-through join marker, arbitrary graph convergence, reference to one
+originating fork
+
+**Activation Token**:
+A runtime value identifying one control-flow arrival, its workflow scope,
+lineage worldview, activation context, and merge provenance. Compatible tokens
+can rendezvous at a gather without mixing loop iterations, subgraph
+invocations, or repeated fork activations.
+_Avoid_: Static edge, node output payload, scheduler position
+
+**Gather Slot**:
+A named local rendezvous requirement on a gather. A gather requires all of its
+slots, while any one compatible incoming edge assigned to a slot can satisfy
+that slot.
+_Avoid_: Fork branch reference, positional input, executable edge predicate
+
 **Planner Efficiency**:
 The degree to which the platform reduces LLM trial-and-error when creating or
 running workflows. Typed schemas, source catalogs, validation errors, dry
@@ -443,12 +470,6 @@ Persisted deployments and server-side execution make scheduling plausible, but
 scheduling itself is not implemented yet.
 _Avoid_: Claiming production background scheduling exists today

-**Fork/Gather Workflow Control**:
-Future workflow control for parallel branches and explicit gather/join behavior.
-The current product should not claim general fork/gather orchestration yet.
-_Avoid_: Treating current foreach or serial graph execution as full parallel
-workflow orchestration
-
 **Scheduler Foundation**:
 The runtime model that selects runnable frames and advances workflow execution without assuming there is only one active cursor.
 _Avoid_: Foreach feature, concurrent foreach implementation
@@ -112,6 +112,8 @@ docs as the active references:
  scheduler foundation decision.
 - [`adr/0002-concurrent-foreach-policy-and-barrier-commits.md`](adr/0002-concurrent-foreach-policy-and-barrier-commits.md):
  concurrent foreach policy and barrier commit semantics.
+- [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md):
+  proposed explicit fork, gather-slot, and activation-token semantics.
 - [`superpowers/specs/2026-05-24-native-subgraphs-design.md`](superpowers/specs/2026-05-24-native-subgraphs-design.md):
  native subgraph design.
 - [`superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md`](superpowers/specs/2026-05-26-durable-workflow-runs-and-resume-design.md):
@@ -0,0 +1,143 @@
+---
+status: proposed
+---
+
+# Explicit Fork and Topology-Driven Gather
+
+General graph concurrency will use explicit fork and gather steps. Outcomes
+continue to select one transition; forks create several branch activations;
+gathers rendezvous compatible activation tokens, merge their lineage-local
+state patches, and create one continuation. This preserves topology-only edges
+without making multiple matching edges silently mean broadcast.
+
+## Context
+
+The scheduler, ready queue, blocked frames, lineage-local state views, and
+reducer-aware barrier commits already support concurrent foreach and native
+subgraphs. They do not yet define general graph-level fork/gather. The existing
+`JoinNode` is only a day-one marker: it immediately emits `done` and neither
+waits nor merges. Renaming it would falsely preserve semantics it never had.
+
+A cross-system semantics review supported keeping `NodeResult.output` separate
+from its named domain `outcome`, keeping operational failure outside that
+outcome namespace, and representing concurrency with explicit control steps.
+The decisive design pressure came from partial and repeated gathers rather than
+from output typing.
+
+Consider a fork producing `a`, `b`, and `c`. One gather may merge `a+b` into
+`d`, while a later gather merges `d+c`. The first gather must not know which
+fork originally produced its inputs, and the second must accept the merged
+`d` continuation like any other arrival. Inside a loop, it must combine
+`a1+b1`, never `a1+b2`.
+
+Conditional paths add another requirement. If one branch continues through
+either `d` or `e`, a later gather may require `b AND (d OR e)`. Waiting for
+every incoming edge would deadlock because only one alternative can arrive.
+
+Cross-merging raises the same issue at larger scale. If one activated region
+produces `b,c` and another produces `e,f`, independent gathers may consume
+`b+e` and `c+f`. Correctness cannot depend on whether the scheduler happens to
+run `b,e,c,f` or `c,f,b,e` first.
+
+## Decision
+
+The graph model distinguishes three control meanings:
+
+- an ordinary node outcome selects exactly one continuation;
+- a fork creates one child activation for every declared branch;
+- a gather consumes compatible arrivals and creates one continuation.
+
+A gather is independent of the construct that produced its inputs. It knows
+its required local input slots from the graph and owns a merge policy. Edges
+targeting a gather identify the slot they can satisfy. A gather requires every
+slot, while several incoming edges may be alternatives for one slot. This gives
+`b AND (d OR e)` without placing executable predicates on edges.
+
+Runtime arrivals carry activation identity, lineage identity, and provenance.
+The gather buckets arrivals by compatible activation context, accepts at most
+one token per slot for an activation, waits until all slots are present, then
+merges their lineage-local patches. Its result is a new continuation token that
+preserves combined provenance and can enter another gather.
+
+Lineage remains a virtual worldview: committed scope state plus writes visible
+to one branch. Gathering several lineages does not require turning lineage
+ancestry into a multi-parent graph. A partial gather can create an intermediate
+lineage under the branches' common parent, retaining multi-input provenance in
+activation-token metadata. A final gather can merge that lineage with remaining
+siblings and resume the blocked parent continuation.
+
+The first gather merge policy is fail-closed:
+
+```python
+class GatherMergePolicy:
+    conflicts: Literal["error"] = "error"
+```
+
+State-field reducers remain the source of truth for legitimate concurrent
+merges. The gather policy determines what happens when patches cannot be
+merged; the initial behavior is to fail rather than choose a last writer.
+
+Branch execution order may be deterministic in the synchronous runtime and
+overlap in the asynchronous runtime. Both modes must produce equivalent graph
+semantics. Scheduler order decides when compatible work progresses, never which
+arrivals belong together.
+
+The current `JoinNode` will not be silently upgraded. Before implementation we
+will verify whether real persisted artifacts use it. With no real compatibility
+obligation, remove it and introduce `GatherNode` cleanly. If persisted callers
+exist, define an explicit migration rather than assigning barrier semantics to
+old `join` payloads.
+
+## Considered Options
+
+**Multiple ordinary edges broadcast.** Rejected because edge cardinality would
+silently change an outcome from selection to spawning and make ordinary graph
+convergence ambiguous.
+
+**Gather references its originating fork.** Rejected because partial gathers,
+staged gathers, conditional paths, and cross-merges consume tokens based on
+their local rendezvous contract, not one producer.
+
+**Gather waits for every incoming edge.** Rejected because alternative paths
+such as `d OR e` would require mutually exclusive arrivals. Named slots provide
+AND across slots and OR within a slot.
+
+**Lineage becomes a multi-parent DAG.** Not currently required. Merge provenance
+belongs to activation tokens; an intermediate merged worldview can remain a
+child of the compatible inputs' common lineage parent.
+
+**Reuse or rename `JoinNode`.** Rejected as the default because the existing
+node is a pass-through marker with no barrier contract.
+
+## Consequences
+
+- Edge identity gains gather-slot significance only when its target is a
+  gather; ordinary edge semantics stay unchanged.
+- Workflow validation must prove that every gather slot has an incoming edge,
+  reject slots on non-gather targets, and preserve one successor per ordinary
+  `(node, outcome)` pair.
+- Checkpoints must persist pending gather arrivals and activation provenance so
+  interruption/resume cannot mix loop iterations or subgraph invocations.
+- Trace output must make fork activation, branch identity, gather waiting, and
+  merged continuation inspectable without treating scheduler bookkeeping as
+  ordinary node output.
+- Fork/gather should generalize concurrent-foreach lineage and barrier helpers,
+  not create a second state-patch system.
+- Runtime branch failures remain execution failures in the first version. Skip,
+  collect, race, first-success, cancellation, and timeout policies are deferred.
+
+## Open Questions
+
+- The exact serialized edge field and authoring name for a gather slot.
+- The minimal activation-context and provenance representation that supports
+  loops, nested forks, subgraphs, partial gathers, and cross-merges.
+- Whether a gather resumes an existing blocked frame or creates a dedicated
+  continuation frame in each topology shape.
+- The trace representation for waiting and merging without excessive internal
+  scheduler noise.
+- Whether any real persisted artifact requires migration from `JoinNode`.
+
+This ADR extends the lineage and barrier direction established by
+[ADR-0002](0002-concurrent-foreach-policy-and-barrier-commits.md). It remains
+proposed until the open runtime-state questions are resolved in the fork/gather
+design specification.
@@ -796,7 +796,8 @@ stable.
 - Native subgraph polish: optional per-use-site child deployment overrides and
  clearer child trace inspection.
 - Concurrent foreach polish: reuse barrier/lineage machinery for future
-  fork/gather.
+  fork/gather. The proposed control semantics are recorded in
+  [`ADR-0006`](adr/0006-explicit-fork-and-topology-driven-gather.md).
 - Protocol-native progress: investigate MCP tasks/progress or WebSocket/SSE only
  after polling `wf run watch` proves insufficient.
 - OpenAPI sources: continue from [`openapi capability sources`](openapi_capability_source.md)