docs: capture fork-gather research and verification plan

This commit is contained in:
lda
2026-09-07 01:34:28 +07:00 Verified
parent e36e462fd6
commit bb361b7f89
16 changed files with 8892 additions and 15 deletions
+15
View File
@@ -5,6 +5,21 @@ This context defines the core workflow runtime language used by `wf_core`,
## Language ## Language
**Fork Control Claim**:
The outstanding branch obligations carried by a fork/gather continuation in the
proposed concurrency model; complete sibling obligations restore their parent
claim.
_Avoid_: Historical trace, state lineage
**Referenced Fork Visit**:
An unresolved fork occurrence on which at least one component of a control
claim depends.
_Avoid_: Assuming it encloses the entire claim
**Enclosing Fork Visit**:
An unresolved fork occurrence encompassing every component of a control claim.
_Avoid_: Any fork mentioned in historical provenance
**lda.chat**: **lda.chat**:
An AI agent platform for authoring and executing workspace workflows. An AI agent platform for authoring and executing workspace workflows.
_Avoid_: Chatbot, MCP server _Avoid_: Chatbot, MCP server
+4
View File
@@ -114,6 +114,10 @@ docs as the active references:
concurrent foreach policy and barrier commit semantics. concurrent foreach policy and barrier commit semantics.
- [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md): - [`adr/0006-explicit-fork-and-topology-driven-gather.md`](adr/0006-explicit-fork-and-topology-driven-gather.md):
proposed explicit fork, gather-slot, and activation-token semantics. proposed explicit fork, gather-slot, and activation-token semantics.
- [`superpowers/specs/2026-09-06-fork-gather-design.md`](superpowers/specs/2026-09-06-fork-gather-design.md):
draft fork/gather contract, unresolved decisions, and verification gates.
- [`historical/research/2026-09-06-fork-gather/README.md`](historical/research/2026-09-06-fork-gather/README.md):
copied research reports and runnable reference artifacts, with known defects.
- [`superpowers/specs/2026-09-04-foreach-back-edge-design.md`](superpowers/specs/2026-09-04-foreach-back-edge-design.md): - [`superpowers/specs/2026-09-04-foreach-back-edge-design.md`](superpowers/specs/2026-09-04-foreach-back-edge-design.md):
approved canonical foreach body-return and validation semantics. approved canonical foreach body-return and validation semantics.
- [`superpowers/specs/2026-09-04-structured-runtime-context-design.md`](superpowers/specs/2026-09-04-structured-runtime-context-design.md): - [`superpowers/specs/2026-09-04-structured-runtime-context-design.md`](superpowers/specs/2026-09-04-structured-runtime-context-design.md):
@@ -10,6 +10,12 @@ gathers rendezvous compatible activation tokens, merge their lineage-local
state patches, and create one continuation. This preserves topology-only edges state patches, and create one continuation. This preserves topology-only edges
without making multiple matching edges silently mean broadcast. without making multiple matching edges silently mean broadcast.
The [draft design contract](../superpowers/specs/2026-09-06-fork-gather-design.md)
records the current requirements, open decisions, and verification gates.
The [research archive](../historical/research/2026-09-06-fork-gather/README.md)
preserves supporting evidence and known reference-model defects. This ADR is
architectural direction, not an executable implementation plan.
## Context ## Context
The scheduler, ready queue, blocked frames, lineage-local state views, and The scheduler, ready queue, blocked frames, lineage-local state views, and
@@ -83,8 +89,25 @@ most one deepest shared lineage: the lowest common ancestor is the deterministic
merge base. Different scopes or no shared ancestor fail before state mutation. merge base. Different scopes or no shared ancestor fail before state mutation.
A partial gather creates its intermediate lineage under that merge base and A partial gather creates its intermediate lineage under that merge base and
retains multi-input provenance in activation-token metadata. A final gather can retains multi-input provenance in activation-token metadata. A final gather can
merge that lineage with remaining siblings and resume the blocked parent merge that lineage with remaining siblings and create a continuation token.
continuation. Completion ownership survives token/frame replacement; foreach and subgraph
parents still wait for the corresponding item or call to finish.
The lineage tree is not sufficient to deduplicate shared pending history after
re-forking, cross-gathering, and reconvergence. Supporting that topology requires
stable original contribution identities or an equivalent demonstrated scheme;
otherwise the topology must be explicitly rejected. New node executions create
new contributions even when node and token identities are unchanged. Merging
existing contributions preserves their identities.
Control obligations also need two distinct ancestry queries. Referenced fork
visits cover any component of a claim and govern unresolved re-entry checks.
Enclosing visits cover every component and govern gather correlation. For a
mixed claim `{k.x, r.c}` with `k` nested under `r`, only `r` encloses the whole
claim. The proposed V1 matches a compiled anchor to that enclosing dynamic
visit, without requiring an authored originating-fork field. One gather may
fire at most once per owner/gather/anchor occurrence. Validation must account
for past firings as well as currently parked arrivals.
The first gather merge policy is fail-closed: The first gather merge policy is fail-closed:
@@ -98,9 +121,12 @@ merges. The gather policy determines what happens when patches cannot be
merged; the initial behavior is to fail rather than choose a last writer. merged; the initial behavior is to fail rather than choose a last writer.
Gather slots have declaration order, and that order is the canonical reducer Gather slots have declaration order, and that order is the canonical reducer
replay order. After choosing the merge base, the runtime applies each selected replay order for independent histories, never arrival, scheduler, or frame-id
lineage's writes after that base in declared-slot order, never arrival, order. Whether an earlier partial gather's serialization becomes a permanent
scheduler, or frame-id order. A bucket accepts exactly one token for each slot; ordering constraint at later reconvergence remains open. Opposite histories
`[A,B]` and `[B,A]` can arise in legitimate authored graphs; treating their
combination as a merge-order conflict is a policy choice, not a corruption
check. A bucket accepts exactly one token for each slot;
a second token for the same activation and slot fails the activation instead of a second token for the same activation and slot fails the activation instead of
making an alternative-path race decide the result. Order-sensitive reducers making an alternative-path race decide the result. Order-sensitive reducers
such as append are therefore deterministic in synchronous and asynchronous such as append are therefore deterministic in synchronous and asynchronous
@@ -155,9 +181,10 @@ their local rendezvous contract, not one producer.
such as `d OR e` would require mutually exclusive arrivals. Named slots provide such as `d OR e` would require mutually exclusive arrivals. Named slots provide
AND across slots and OR within a slot. AND across slots and OR within a slot.
**Lineage becomes a multi-parent DAG.** Not currently required. Merge provenance **Lineage becomes a multi-parent DAG.** Not currently required. An intermediate
belongs to activation tokens; an intermediate merged worldview can remain a merged worldview can remain a child of the inputs' common lineage parent, but
child of the compatible inputs' common lineage parent. control provenance alone is not proof of state-write uniqueness. Shared-write
identity and ordering require the separate contract described above.
**Store scope and parent-lineage identity on every frame.** Rejected because **Store scope and parent-lineage identity on every frame.** Rejected because
those relationships are canonical on the lineage and duplicated frame fields those relationships are canonical on the lineage and duplicated frame fields
@@ -178,10 +205,13 @@ the node was a pass-through marker with no barrier contract.
gather; ordinary edge semantics stay unchanged. gather; ordinary edge semantics stay unchanged.
- Workflow validation must require every gather-target edge to name exactly one - Workflow validation must require every gather-target edge to name exactly one
declared slot, reject missing or unknown gather slots, prove that every slot declared slot, reject missing or unknown gather slots, prove that every slot
has an incoming edge, reject slots on non-gather targets, and preserve one has an incoming edge, preserve but ignore ports on ordinary targets, and keep
successor per ordinary `(node, outcome)` pair. one successor per ordinary `(node, outcome)` pair.
- Checkpoints must persist pending gather arrivals and activation provenance so - Checkpoints must persist pending gather arrivals and activation provenance so
interruption/resume cannot mix loop iterations or subgraph invocations. interruption/resume cannot mix loop iterations or subgraph invocations.
- Supported checkpoints must not expose partially applied transitions, including
when an exception produces a failed stopped run. This does not require a
durable transition journal or promise exactly-once external effects.
- Runtime operations should resolve a frame, its lineage, and its scope through - Runtime operations should resolve a frame, its lineage, and its scope through
one internal interface instead of accepting several independently supplied one internal interface instead of accepting several independently supplied
identifiers. identifiers.
@@ -199,11 +229,12 @@ the node was a pass-through marker with no barrier contract.
## Open Questions ## Open Questions
- The exact serialized edge field and authoring name for a gather slot. - The supported correlation grammar and production validation algorithm for
- The minimal activation-correlation and provenance representation that loops, nested forks, partial gathers, and cross-merges. Destination edges use
supports loops, nested forks, subgraphs, partial gathers, and cross-merges. `target_port`; ordinary destinations ignore that metadata.
- Whether a gather resumes an existing blocked frame or creates a dedicated - Whether prior gather ordering is permanent history or a local state view.
continuation frame in each topology shape. - The concrete owner/occurrence representation and integration with blocked
foreach and subgraph completion, including failure and checkpoint handling.
- The trace representation for waiting and merging without excessive internal - The trace representation for waiting and merging without excessive internal
scheduler noise. scheduler noise.
+11
View File
@@ -56,6 +56,17 @@ independently or repeat ownership walks.
Reuse the scheduler, activation, lineage, and reducer-aware barrier machinery: Reuse the scheduler, activation, lineage, and reducer-aware barrier machinery:
- [`ADR-0006: explicit fork and topology-driven gather`](adr/0006-explicit-fork-and-topology-driven-gather.md) - [`ADR-0006: explicit fork and topology-driven gather`](adr/0006-explicit-fork-and-topology-driven-gather.md)
- [`Draft contract and verification gates`](superpowers/specs/2026-09-06-fork-gather-design.md)
Production planning remains gated on correlation analysis and merge-order
semantics. The draft records two reproduced reference-model defects and points
to archived runnable research; passing that prototype's tests is not approval
to copy it into the runtime.
The isolated
[`reference verification plan`](superpowers/plans/2026-09-07-fork-gather-reference-verification.md)
closes occurrence regressions and gathers executable decision evidence first;
it does not authorize production fork/gather changes.
Outcomes continue to choose one transition. Forks create concurrent branch Outcomes continue to choose one transition. Forks create concurrent branch
activations. Gathers wait on declared incoming topology, merge compatible activations. Gathers wait on declared incoming topology, merge compatible
@@ -0,0 +1,97 @@
# Fork/gather research evidence — 2026-09-06
These are historical research inputs, not approved specifications or production
code. The current contract is the
[draft fork/gather spec](../../../superpowers/specs/2026-09-06-fork-gather-design.md).
The [ADR](../../../adr/0006-explicit-fork-and-topology-driven-gather.md) records
the architectural direction. Both remain proposed.
## Provenance
Eight files were copied, not moved, from `random shit/research-chatgpt/`.
Every copy was checked against its original with SHA-256. The redundant ZIP,
older citation-less exports, and other unrelated research were not imported.
| Archived file under `imported/` | Original relative path |
| --- | --- |
| `round-2-report.md` | `deep-research-report060926 copy 3.md` |
| `round-2-citations.md` | `deep-research-report060926 copy 2.md` |
| `audit-1.md` | `research-audit060926.md` |
| `audit-2.md` | `research-audit060926-002.md` |
The remaining four files retain their original basenames and were copied from
the `research-audit060926-002/` subdirectory:
- `fork_gather_reference.py`
- `test_fork_gather_reference.py`
- `README_fork_gather_reference.md`
- `verification_test_results.txt`
The report's inline citation numbers can be matched to the citations export.
Uploaded-file citations concern an older repository snapshot, not necessarily
current code. Original `sandbox:` download links are inert historical links;
the actual supplied files are alongside the reports here.
Keep `imported/` byte-identical to these source snapshots. Its local Markdown
lint configuration disables style rules solely to preserve the imported text.
This README and the live documents use normal lint rules. Future experiments
belong outside `imported/`; do not silently repair the evidence.
## Independent verification and limitations
On 2026-09-06, the 25 supplied tests passed locally. Reproduce from the repo root:
```powershell
Push-Location 'docs/historical/research/2026-09-06-fork-gather/imported'
try { python -B -m unittest -v test_fork_gather_reference.py }
finally { Pop-Location }
```
Audit 2 explicitly labels its implementation a reconstruction: the scratch
code claimed in audit 1 was not retained. The captured output is the author's
record; the local test run is independent verification of the supplied code.
Neither is a test of `wf_core`.
The reference uses a single owner, append-only example contributions,
deep-copied transitions, and trusted in-process pickle snapshots. It does not
establish production codec validation, durable storage atomicity, arbitrary
reducers, external-effect idempotency, foreach, subgraphs, or cancellation.
Never load an untrusted pickle checkpoint.
## Known defects reproduced locally
### Repeated write execution collides
```text
w: append X -> pick
pick.again -> w
pick.done -> END
```
`SymbolicAnalyzer` accepts this graph. Execute `w`, choose `again`, then execute
`w` using the same token. `_write()` raises
`RuntimeError: duplicate semantic contribution w@T0` because its ID uses only
node and token identity, not the execution occurrence. This is a valid loop
that the simulator must eventually execute correctly.
### Repeated gather occurrence is not rejected by analysis
```text
g.a -> h.only -> pick
pick.again -> h.only
pick.done -> final.left
g.b -> final.right
final -> END
```
The supplied model permits one-port gathers. Analysis accepts the graph with
both gather anchors set to `g`. Execute `g`, fire `h`, choose `again`, then fire
`h` again: `_fire_gather_impl()` raises
`RuntimeError: duplicate merged lineage merge:h@g@T0`.
The analyzer tracks current tokens but not past firings under an unresolved
anchor. The proposed one-fire-per-occurrence contract requires validation to
reject this case. Banning one-port gathers would not prove the general rule.
Both reproductions were executed independently during review. They are not
included in the unchanged imported 25-test suite. Their regression requirements
and the next verification gates live in the draft spec, not in this archive.
@@ -0,0 +1,3 @@
{
"default": false
}
@@ -0,0 +1,44 @@
# Fork/Gather Reference Verification Artifacts
These files are a **reconstruction**, created during the focused verification round. The scratch code implicitly referred to in the earlier audit was not available as a persisted artifact.
Files:
- `fork_gather_reference.py` — independent graph model, runtime simulator, checkpoint/recovery, merge policies, symbolic correlation/discharge analyzer, finite re-entry checker, and demo graph builders.
- `test_fork_gather_reference.py` — executable `unittest` suite.
No production repository code is imported.
## Requirements
Python 3.11+; standard library only.
## Run all executed tests
```bash
cd /path/to/files
python -m unittest -v test_fork_gather_reference.py
```
## Run focused demos
```bash
python fork_gather_reference.py ordering-strict
python fork_gather_reference.py ordering-port
python fork_gather_reference.py checkpoint
python fork_gather_reference.py analysis
```
`ordering-strict` demonstrates that the authored `hx=[A,B]`, `hy=[B,A]` graph is control-valid but the final merge has a precedence cycle.
`ordering-port` uses final-gather port order plus stable first-occurrence deduplication and completes as `ROOT,A,B`.
`checkpoint` snapshots a stable runtime with a ready gather, reconstructs it with `pickle`, then completes the run.
`analysis` runs the finite symbolic analyzer across the requested graph classes.
## Persistence contract modeled
A runtime semantic transition mutates a deep-copied candidate state. The candidate replaces the externally visible state only after the transition succeeds and runtime invariants pass. `checkpoint()` is allowed only between completed semantic transitions. Consequently, an exception during a transition cannot expose its partially mutated candidate to checkpointing.
This models consistent snapshots and recovery from the last durable checkpoint. It does **not** model recovery from partially persisted internal transitions and does **not** claim exactly-once external side effects.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,700 @@
The exact scratch code behind my previous audit was **not persisted and is no longer accessible**. The artifacts below are a **reconstruction** of those semantics, written and executed in this verification round. They do not import or implement against the production repository.
### Runnable artifacts
[Download the complete artifact bundle](sandbox:/mnt/data/fork_gather_reference_artifacts.zip)
Individual files: [simulator + analyzer](sandbox:/mnt/data/fork_gather_reference.py), [25-test suite](sandbox:/mnt/data/test_fork_gather_reference.py), [README / commands](sandbox:/mnt/data/README_fork_gather_reference.md), and [captured test output](sandbox:/mnt/data/verification_test_results.txt).
Run them with:
```bash
python -m unittest -v test_fork_gather_reference.py
python fork_gather_reference.py ordering-strict
python fork_gather_reference.py ordering-port
python fork_gather_reference.py checkpoint
python fork_gather_reference.py analysis
```
I executed all four code paths represented by the tests/demos. The final suite result was:
```text
Ran 25 tests in 0.024s
OK
```
Those are **actual executions of the reconstructed reference implementation**. Nothing below relies on the earlier claimed execution.
## 1. The ordering counterexample breaks one conclusion from the prior audit
Your graph is accepted by the reconstructed control analyzer.
It derives:
```text
hx anchor = r
hy anchor = r
final anchor = r
hx discharges: {}
hy discharges: {}
final discharges: {fa, fb, r}
```
After the two inner gathers, the actual stable contribution IDs were:
```text
hx = [
A@r@T0/a,
B@r@T0/b,
]
hy = [
B@r@T0/b,
A@r@T0/a,
]
```
or by labels:
```text
hx = [A, B]
hy = [B, A]
```
There is no existing control rule in the reconstructed model that rejects this. Every branch obligation is consumed exactly once, and the final normalization restores the owner root.
So my previous statement that opposite histories imply "corruption" was wrong.
The correct classification is:
> **This is a legitimate authored control graph that produces incompatible merge-order constraints under one possible definition of merge-history preservation.**
The metadata itself is not malformed.
### Causal order versus gather-imposed order
This distinction matters.
In your graph:
```text
r.a -> A -> fa
r.b -> B -> fb
```
`A` and `B` are concurrent sibling writes. There is no causal dependency:
```text
A -> B
```
or:
```text
B -> A
```
`hx` later imposes:
```text
A < B
```
because its ports are `(left,right)`.
`hy` independently imposes:
```text
B < A
```
because its left input happens to originate from `B`.
Those are **merge serialization orders**, not original happens-before dependencies.
The architectural question is therefore whether an earlier gather's serialization order becomes a permanent constraint on later reconvergence.
That is a policy decision, not a provenance-integrity invariant.
---
# 2. Policy A: preserve previous merge orders, error on cycles
The implemented `STRICT_PRECEDENCE` policy treats each materialized history as an ordering constraint.
Thus:
```text
hx: A, B => A < B
hy: B, A => B < A
```
At `final`, their union contains a cycle:
```text
A < B < A
```
The actual simulator result is:
```text
MergeOrderConflict:
incompatible contribution-order constraints;
cycle involves A@r@T0/a, B@r@T0/b
```
Importantly, `hx` and `hy` both succeed. **Only `final` fails.**
This policy preserves:
* exactly-once contribution identity;
* every previously established ordering when merging succeeds;
* schedule independence;
* order-sensitive reducer semantics as a coherent accumulated history;
* the property that a later merge never silently reverses an order already visible in an earlier merged lineage.
It sacrifices composability: a completely legitimate control graph can reach a deterministic merge conflict because independently serialized concurrent histories are incompatible.
Given your existing preference `conflicts="error"`, I think this is the cleaner first policy. The error should be called something like:
```text
merge-order conflict
```
not corruption.
## Global cycle detection is necessary
Checking histories pairwise is insufficient.
I executed:
```text
H1 = [A, B]
H2 = [B, C]
H3 = [C, A]
```
A straightforward pairwise shared-subsequence agreement predicate returns:
```text
True
```
because each pair shares only one contribution:
```text
H1 ∩ H2 = {B}
H2 ∩ H3 = {C}
H3 ∩ H1 = {A}
```
There is no pairwise disagreement to observe.
But their combined constraints are:
```text
A < B
B < C
C < A
```
and the implemented global precedence check produces:
```text
MergeOrderConflict:
cycle involves A, B, C
```
Therefore:
> Pairwise agreement on shared IDs is not sufficient. Strict preservation requires building the union precedence graph and checking it globally for cycles.
The implementation uses adjacent sequence pairs:
```text
[A,B,C] -> A→B, B→C
```
rather than every ordered pair. Transitivity supplies the rest. It then performs deterministic topological sorting.
---
# 3. Policy B: final gather port order wins
The implemented `FINAL_PORT_ORDER` policy performs stable first-occurrence deduplication in the declared final port order.
For:
```text
final.left = hx = [A, B]
final.right = hy = [B, A]
```
the scan is:
```text
left:
A -> retain
B -> retain
right:
B -> already present
A -> already present
```
so:
```text
final = [A, B]
```
The actual simulator commits:
```text
["ROOT", "A", "B"]
```
Reversing the histories supplied to the same primitive produces `[B,A]`, as expected.
This policy preserves:
* one occurrence per contribution ID;
* deterministic behavior;
* schedule independence;
* declared **current gather** port precedence;
* very simple runtime semantics.
It sacrifices a stronger property:
> An ordering already observable in an earlier merged state is not necessarily preserved after later reconvergence.
A node downstream of `hy` could have observed:
```text
ROOT, B, A
```
while the final continuation observes:
```text
ROOT, A, B
```
That is not a causal contradiction—A and B were originally concurrent—but it means a lineage history is no longer interpretable as a monotonically accumulated global ordering.
For an order-sensitive reducer such as append, that is a substantial semantic choice.
### Recommendation between A and B
For the prototype, I would choose **A: strict precedence + deterministic merge error**.
That matches `conflicts="error"` without confusing a legitimate authored conflict with malformed metadata. It also avoids making partial-gather ordering retroactively disposable.
If later experience shows that gather order should be a purely local projection rather than historical ordering, policy B can be introduced explicitly. I would not make that interpretation implicit.
---
# 4. Correlation inference is now independently executable
The reconstructed `SymbolicAnalyzer` does **not** receive manually supplied gather anchors or fork-discharge tables.
Its supported graph class is intentionally narrow:
* finite flat graph;
* one initial owner token;
* concurrency arises only from explicit `ForkNode`;
* a fork simultaneously emits all declared branches;
* ordinary outcomes have exactly one successor;
* `ChoiceNode` alternatives are explored nondeterministically;
* gathers have fixed statically declared ports;
* each gather firing consumes exactly one arrival per port;
* no unresolved re-entry of the same static fork along a token's referenced ancestry;
* no foreach/subgraph/failure/interrupt semantics in this verifier.
That is enough to test the correlation issue in isolation.
## Abstract state
A symbolic activation is:
```python
SActivation(
fork_id,
parent_claim,
)
```
A branch obligation is:
```python
SBranch(
activation,
branch,
)
```
So two sibling activations of the same static fork are distinguishable.
For example:
```text
p.left -> g
p.right -> g
```
produces abstract activations equivalent to:
```text
g<[p.left]>
g<[p.right]>
```
rather than merely:
```text
g
g
```
A marking is a multiset of symbolic tokens located either:
```text
at node N
```
or:
```text
parked at gather H.port
```
with their complete current control claims.
## Fork transfer
Executing `g` with claim `C` creates:
```text
activation = g<C>
```
and emits:
```text
g<C>.x
g<C>.y
...
```
Before doing so, the analyzer recursively examines the incoming claim. If static `g` already occurs in any referenced unresolved obligation, that transition is rejected as:
```text
unresolved_reentry:g
```
It does not unroll another `g`.
## Gather inference
When one arrival exists for every port, the analyzer enumerates complete port combinations.
For each combination it computes the **deepest common enclosing symbolic activation**.
That dynamically observed activation is the candidate correlation anchor.
The analyzer does not need to know in advance what the gather discharges.
It then unions the input claims and repeatedly performs:
```text
all branches of activation X present
->
replace them with X.parent_claim
```
The activations removed by that normalization are the observed discharges for this firing.
Only after exploring all reachable markings does it require that each static gather have one static anchor fork across every possible firing.
That avoids the previous circular construction:
```text
need anchor to infer firing
need firing to infer discharge
need discharge to infer anchor
```
For this supported class, firing combinations are enumerated from the actual symbolic marking first; anchor and discharge are observations of each combination.
---
# 5. Actual analyzer results
| Graph | Result | Derived anchor(s) | Observed discharge |
| ----------------------------------------------------------- | ---------- | ------------------------- | ---------------------------- |
| Ordinary `g -> a,b -> h` | accept | `h → g` | `h: {g}` |
| Direct `g -> h.left,h.right` | accept | `h → g` | `h: {g}` |
| Partial `a+b`, then result+`c` | accept | both → `g` | partial `{}`, final `{g}` |
| Sibling nested cross-gathers | accept | `h1,h2,final → r` | final `{fa,fd,r}` |
| Your A/B ordering graph | **accept** | `hx,hy,final → r` | final `{fa,fb,r}` |
| Conditional `d OR e → h.right` | accept | `h → g` | `{g}` |
| Simultaneous two arrivals to `h.right` | **reject** | `h → g` observed | duplicate compatible arrival |
| Completed round `g→h→repeat→g` | accept | `h → g` | `{g}` |
| Unresolved `g...→g` recursion | **reject** | — | unresolved re-entry |
| Two sibling activations of same static `g` feeding same `h` | **reject** | both `g` and `p` observed | ambiguous correlation |
The completed-round analysis reached a fixed point after **8 symbolic markings**. The unresolved-recursion graph rejected after **9**.
The separate simpler `(node, open_static_forks)` re-entry checker examined six states in each of its two executed cases:
```text
completed round:
accepted
6 states
unresolved recursion:
rejected: unresolved_reentry:g
6 states
```
---
# 6. The two-sibling-same-static-fork case reveals the real conservative boundary
Consider:
```text
Fork p(branches=l,r)
p.l -> g
p.r -> g
Fork g(branches=x,y)
g.x -> h.left
g.y -> h.right
```
The symbolic state contains:
```text
g<[p.l]>.x
g<[p.l]>.y
g<[p.r]>.x
g<[p.r]>.y
```
There are sensible same-activation combinations:
```text
g<[p.l]>.x + g<[p.l]>.y
anchor = g<[p.l]>
g<[p.r]>.x + g<[p.r]>.y
anchor = g<[p.r]>
```
But ports alone also permit:
```text
g<[p.l]>.x + g<[p.r]>.y
```
whose deepest common encloser is:
```text
p
```
and similarly for the opposite cross-pair.
The executed analyzer therefore observes:
```text
h anchors = {g, p}
```
and rejects:
```text
non_unique_static_anchor:h:['g', 'p']
```
It also observes duplicate compatible arrivals under the `p` interpretation.
This is a useful result because it demonstrates that:
> Static port identity + provenance + deepest-common-encloser does **not** solve general correlation.
For this prototype grammar, the exact restriction is:
> A gather is accepted only if every reachable complete port combination yields the same static anchor fork, and each dynamic anchor can have at most one compatible arrival per port.
An explicit correlation declaration could make more graphs expressible later. The prototype does not need one yet.
---
# 7. Why the symbolic analysis terminates
This is a demonstrated property of the implemented restricted analysis, not a claim about unrestricted workflow graphs.
An activation is recursively:
```text
(static fork ID, parent claim)
```
and unresolved re-entry of an already referenced static fork is rejected.
Therefore a single obligation ancestry cannot contain the same static fork twice, so nesting depth is bounded by the finite number of fork nodes.
Branch sets and static node positions are finite. Forking can create exponentially many combinations, but without same-fork unresolved recursion it cannot create an unbounded activation ancestry. Ordinary cycles eventually revisit an existing marking. Fully converged rounds similarly restore their parent claim, so another round produces the same symbolic abstract state and hits the visited set.
Thus the state space is finite for this supported class.
The practical cost is still poor in the worst case: explicit concurrent marking exploration is combinatorial. The implementation has a `max_markings` guard. I would treat this as a correctness prototype, not the intended production validator algorithm.
---
# 8. Durability correction: no journal is justified by the stated contract
My previous recommendation overreached here.
The current persistence contract you supplied is:
> externally stopped states are persisted: interrupted, completed, failed.
It does **not** promise that arbitrary internal fork/gather mutations are individually durable.
Under that contract, a gather transition journal is not required merely because an internal transition has several logical substeps.
The reconstructed runtime instead models:
```python
candidate = deepcopy(stable_state)
perform_complete_semantic_transition(candidate)
validate(candidate)
# publish only after successful completion
stable_state = candidate
```
`checkpoint()` serializes only `stable_state`.
That is deliberately illustrative rather than a production implementation prescription.
### What was actually tested
The 25-test run includes:
**Checkpoint round trip.** A complete state containing live tokens, a ready gather, fork visits, lineages, contributions, and gather buckets was serialized and reconstructed exactly. This is actually *stronger* than your currently promised checkpoint surface because the reference test snapshots an internally runnable state.
**Recovery from last durable checkpoint.** The simulator checkpointed, performed additional uncheckpointed gather/write progress, discarded that process state, reconstructed the old checkpoint, re-executed from there, and completed as:
```text
ROOT, S, C, D
```
**Exception isolation.** Under strict ordering, `final` raises `MergeOrderConflict`. Because it executes against a candidate copy, the published runtime state after the exception is byte-for-semantic-state identical to the state immediately before attempting the merge.
**Shared-write recovery.** After checkpoint/recovery, reconvergence still produced:
```text
S, C, D
```
rather than:
```text
S, C, S, D
```
because stable contribution IDs survived the checkpoint.
### Four durability questions should remain separate
| Problem | Required here? | Mechanism |
| ----------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------- |
| Consistent supported checkpoint | **Yes** | snapshot only a stable semantic state |
| Process crash after uncheckpointed progress | **Yes** | recover last durable state; later work may rerun |
| Recovery from half-persisted internal gather mutation | **No, given stated contract** | don't persist such states; use atomic snapshot storage |
| Exactly-once external side effect | **Not solved here** | requires idempotency/dedup/activity semantics at effect boundary |
There is one implementation consequence worth retaining.
If an operational exception causes the runtime to persist a **failed** stopped run, that failed snapshot should be derived from:
```text
last stable pre-transition state
+ structured failure information
```
not from an in-place `RunState` that may have been partially mutated before the exception.
That can be achieved with a transition patch, copy-on-write, rollback, or another commit boundary. It does **not** imply a durable transition journal.
Exactly-once external effects are different. If:
```text
checkpoint
-> call payment API
-> process crashes before next checkpoint
```
recovery can invoke the API again. Fork/gather contribution IDs cannot undo or intrinsically deduplicate that external effect.
---
# 9. What the artifacts now demonstrate—and what they don't
Demonstrated by execution:
* direct fork→gather preserves destination ports;
* partial gathers can retain unresolved obligations and later complete them;
* cross-gathers can be correlated for the tested sibling-nested topology;
* your ordering graph is control-valid;
* `[A,B]` and `[B,A]` are legitimate intermediate histories;
* strict precedence rejects their final reconvergence;
* final-port-order deterministically accepts it;
* global precedence cycles can exist despite pairwise agreement;
* stable contribution IDs deduplicate shared history;
* two alternate schedules of the shared-write adversarial graph both commit `ROOT,S,C,D`;
* checkpoint/recovery preserves tokens, gathers, lineages, contributions and correlation state;
* unresolved same-static-fork re-entry can be rejected with a terminating finite analysis;
* completed rounds are not falsely rejected;
* two sibling activations of one static fork expose a real correlation ambiguity.
Still assumptions or intentionally unsupported:
* scalability of explicit marking exploration;
* foreach/subgraph owner correlation;
* interrupts and cancellation;
* operational failures during concurrent branches;
* correlation for graphs outside the stated grammar;
* general workflow soundness or termination;
* exactly-once external effects;
* whether earlier gather serialization should semantically constrain all later reconvergences.
---
# 10. Narrow recommendation for the next prototype
I would prototype exactly this much:
1. **Keep explicit forks, destination gather ports, and token replacement.**
2. Represent symbolic/runtime fork activations by fork identity plus the consumed parent control claim; don't use static fork ID alone for correlation.
3. Initially accept a gather only when finite analysis proves one static deepest-common-encloser across all reachable firing combinations and at most one compatible arrival per port.
4. Reject unresolved same-static-fork re-entry. Permit loops after normalization fully discharges that fork.
5. Treat the two-sibling-same-static-fork/shared-gather topology as unsupported rather than inventing a correlation heuristic.
6. If shared pending histories can reconverge, retain stable contribution identities.
7. For noncommutative reducers, prototype **strict global precedence checking** and report cycles as ordinary merge conflicts.
8. Keep persistence at atomic, consistent externally supported stop boundaries. **Do not add a transition journal** for fork/gather at this stage.
9. Keep exactly-once external effect semantics out of this prototype.
The principal remaining semantic decision is now much narrower than before: **does a partial gather's deterministic ordering become durable history, or is it merely that continuation's local state view?** Policy A and B correspond almost exactly to those two interpretations. The control/correlation model no longer needs to answer that question for you.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,289 @@
from __future__ import annotations
import unittest
from fork_gather_reference import (
MergeOrderConflict,
MergePolicy,
Runtime,
SymbolicAnalyzer,
finite_reentry_check,
graph_adversarial_shared_reconvergence,
graph_basic,
graph_completed_round_loop,
graph_conditional_alternative,
graph_cross_gather,
graph_direct_fork_gather,
graph_duplicate_arrival,
graph_ordering_counterexample,
graph_partial,
graph_three_history_order_cycle,
graph_two_sibling_same_fork_analysis_only,
graph_unresolved_reentry,
precedence_merge,
pairwise_shared_order_agrees,
stable_ordered_dedup,
)
class CorrelationAnalysisTests(unittest.TestCase):
def assertAcceptedAnchor(self, graph, expected):
result = SymbolicAnalyzer(graph).analyze()
self.assertTrue(result.accepted, result.errors)
self.assertEqual(result.unique_anchor_plan(), expected)
return result
def test_ordinary_fork_gather(self):
r = self.assertAcceptedAnchor(graph_basic(), {"h": "g"})
self.assertEqual(r.always_discharged_by("h"), {"g"})
def test_direct_fork_to_gather(self):
r = self.assertAcceptedAnchor(graph_direct_fork_gather(), {"h": "g"})
self.assertEqual(r.always_discharged_by("h"), {"g"})
def test_partial_gather_then_final(self):
r = self.assertAcceptedAnchor(graph_partial(), {"h_ab": "g", "h_final": "g"})
self.assertEqual(r.always_discharged_by("h_ab"), set())
self.assertEqual(r.always_discharged_by("h_final"), {"g"})
def test_cross_gathers(self):
r = self.assertAcceptedAnchor(graph_cross_gather(), {"h1": "r", "h2": "r", "final": "r"})
self.assertEqual(r.always_discharged_by("h1"), set())
self.assertEqual(r.always_discharged_by("h2"), set())
self.assertEqual(r.always_discharged_by("final"), {"fa", "fd", "r"})
def test_ordering_counterexample_control_is_accepted(self):
r = self.assertAcceptedAnchor(
graph_ordering_counterexample(),
{"hx": "r", "hy": "r", "final": "r"},
)
self.assertEqual(r.always_discharged_by("final"), {"fa", "fb", "r"})
def test_mutually_exclusive_alternatives_same_port(self):
self.assertAcceptedAnchor(graph_conditional_alternative(), {"h": "g"})
def test_simultaneous_duplicate_arrivals_same_port_are_rejected(self):
r = SymbolicAnalyzer(graph_duplicate_arrival()).analyze()
self.assertFalse(r.accepted)
self.assertTrue(any(e.startswith("duplicate_compatible_arrivals:h.right") for e in r.errors), r.errors)
def test_completed_round_loop_terminates_and_is_accepted(self):
r = self.assertAcceptedAnchor(graph_completed_round_loop(), {"h": "g"})
self.assertLess(r.explored_markings, 100)
def test_unresolved_same_fork_reentry_is_rejected(self):
r = SymbolicAnalyzer(graph_unresolved_reentry()).analyze()
self.assertFalse(r.accepted)
self.assertIn("unresolved_reentry:g", r.errors)
self.assertLess(r.explored_markings, 100)
def test_two_sibling_activations_same_static_fork_are_ambiguous(self):
r = SymbolicAnalyzer(graph_two_sibling_same_fork_analysis_only()).analyze()
self.assertFalse(r.accepted)
# Same-activation pairs correlate to g; cross-pairs correlate to p.
self.assertIn("non_unique_static_anchor:h:['g', 'p']", r.errors)
class ReentryCheckerTests(unittest.TestCase):
def test_finite_checker_accepts_completed_round_loop(self):
g = graph_completed_round_loop()
analysis = SymbolicAnalyzer(g).analyze()
discharge = {"h": analysis.always_discharged_by("h")}
r = finite_reentry_check(g, discharge)
self.assertTrue(r.accepted, r.errors)
self.assertLess(r.explored_states, 100)
def test_finite_checker_rejects_unresolved_reentry(self):
g = graph_unresolved_reentry()
# h would discharge g if reached normally; recurse reaches g before h.
r = finite_reentry_check(g, {"h": {"g"}})
self.assertFalse(r.accepted)
self.assertIn("unresolved_reentry:g", r.errors)
self.assertLess(r.explored_states, 100)
class MergeOrderingTests(unittest.TestCase):
def test_counterexample_has_legitimate_opposite_intermediate_orders(self):
graph = graph_ordering_counterexample()
analysis = SymbolicAnalyzer(graph).analyze()
self.assertTrue(analysis.accepted, analysis.errors)
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.FINAL_PORT_ORDER)
rt.step_token("T0")
rt.step_token(rt.token_at("A")[0])
rt.step_token(rt.token_at("B")[0])
rt.step_token(rt.token_at("fa")[0])
rt.step_token(rt.token_at("fb")[0])
rt.fire_gather("hx")
rt.fire_gather("hy")
hx = [tid for tid in rt.state.tokens if tid.startswith("out:hx@")][0]
hy = [tid for tid in rt.state.tokens if tid.startswith("out:hy@")][0]
self.assertEqual([rt.state.contributions[c].value for c in rt.token_history(hx)], ["A", "B"])
self.assertEqual([rt.state.contributions[c].value for c in rt.token_history(hy)], ["B", "A"])
def test_policy_a_rejects_opposite_order_constraints(self):
with self.assertRaises(MergeOrderConflict):
precedence_merge([["A", "B"], ["B", "A"]])
def test_policy_b_final_port_order_resolves_by_first_occurrence(self):
self.assertEqual(stable_ordered_dedup([["A", "B"], ["B", "A"]]), ["A", "B"])
self.assertEqual(stable_ordered_dedup([["B", "A"], ["A", "B"]]), ["B", "A"])
def test_pairwise_shared_pair_agreement_is_not_sufficient_collectively(self):
histories, exc = graph_three_history_order_cycle()
self.assertEqual(histories, [["A", "B"], ["B", "C"], ["C", "A"]])
self.assertTrue(pairwise_shared_order_agrees(histories))
self.assertIsNotNone(exc)
def test_counterexample_strict_runtime_raises_at_final_not_intermediate(self):
graph = graph_ordering_counterexample()
analysis = SymbolicAnalyzer(graph).analyze()
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.STRICT_PRECEDENCE)
rt.step_token("T0")
for node in ("A", "B", "fa", "fb"):
rt.step_token(rt.token_at(node)[0])
rt.fire_gather("hx")
rt.fire_gather("hy")
with self.assertRaises(MergeOrderConflict):
rt.fire_gather("final")
def test_counterexample_port_order_runtime_commits_A_B(self):
graph = graph_ordering_counterexample()
analysis = SymbolicAnalyzer(graph).analyze()
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.FINAL_PORT_ORDER)
rt.step_token("T0")
for node in ("A", "B", "fa", "fb"):
rt.step_token(rt.token_at(node)[0])
rt.fire_gather("hx")
rt.fire_gather("hy")
rt.fire_gather("final")
end = rt.token_at("END")[0]
self.assertEqual([rt.state.contributions[c].value for c in rt.token_history(end)], ["A", "B"])
rt.step_token(end)
self.assertEqual(rt.state.committed_values, ["ROOT", "A", "B"])
class RuntimeScheduleTests(unittest.TestCase):
def test_direct_fork_to_gather_preserves_ports_at_runtime(self):
graph = graph_direct_fork_gather()
analysis = SymbolicAnalyzer(graph).analyze()
rt = Runtime(graph, analysis.unique_anchor_plan())
rt.step_token("T0")
self.assertEqual(rt.token_at("h"), [])
self.assertEqual(len(rt.ready_gathers()), 1)
key = rt.ready_gathers()[0]
bucket = rt.state.gather_buckets[key]
self.assertEqual(set(bucket.arrivals), {"left", "right"})
def _finish_adversarial(self, schedule):
graph = graph_adversarial_shared_reconvergence()
analysis = SymbolicAnalyzer(graph).analyze()
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.FINAL_PORT_ORDER)
rt.step_token("T0")
for action in schedule:
if action in {"S", "C", "D", "k"}:
rt.step_token(rt.token_at(action)[0])
else:
rt.fire_gather(action)
rt.step_token(rt.token_at("END")[0])
return rt
def test_adversarial_two_schedules_same_semantic_result(self):
first = self._finish_adversarial(["S", "k", "C", "hx", "D", "hy", "final"])
second = self._finish_adversarial(["D", "C", "S", "k", "hy", "hx", "final"])
self.assertEqual(first.state.committed_values, ["ROOT", "S", "C", "D"])
self.assertEqual(second.state.committed_values, ["ROOT", "S", "C", "D"])
self.assertEqual(first.state.committed_values, second.state.committed_values)
class CheckpointTests(unittest.TestCase):
def _prepared_runtime(self):
graph = graph_adversarial_shared_reconvergence()
analysis = SymbolicAnalyzer(graph).analyze()
self.assertTrue(analysis.accepted, analysis.errors)
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.FINAL_PORT_ORDER)
rt.step_token("T0")
rt.step_token(rt.token_at("S")[0])
rt.step_token(rt.token_at("k")[0])
rt.step_token(rt.token_at("C")[0])
return graph, analysis, rt
def test_checkpoint_round_trip_ready_gather(self):
graph, analysis, rt = self._prepared_runtime()
before = rt.snapshot_dict()
blob = rt.checkpoint()
recovered = Runtime.recover(
graph,
analysis.unique_anchor_plan(),
blob,
merge_policy=MergePolicy.FINAL_PORT_ORDER,
)
self.assertEqual(recovered.snapshot_dict(), before)
self.assertEqual([g for g, _ in recovered.ready_gathers()], ["hx"])
def test_process_crash_discards_uncheckpointed_progress_and_recovers_last_snapshot(self):
graph, analysis, rt = self._prepared_runtime()
blob = rt.checkpoint()
# Progress after the durable checkpoint is intentionally lost.
rt.fire_gather("hx")
rt.step_token(rt.token_at("D")[0])
recovered = Runtime.recover(
graph,
analysis.unique_anchor_plan(),
blob,
merge_policy=MergePolicy.FINAL_PORT_ORDER,
)
# Re-execute from the last durable semantic snapshot.
recovered.fire_gather("hx")
recovered.step_token(recovered.token_at("D")[0])
recovered.fire_gather("hy")
recovered.fire_gather("final")
recovered.step_token(recovered.token_at("END")[0])
self.assertEqual(recovered.state.committed_values, ["ROOT", "S", "C", "D"])
def test_resume_from_checkpoint_matches_no_checkpoint_run(self):
graph, analysis, rt = self._prepared_runtime()
blob = rt.checkpoint()
recovered = Runtime.recover(
graph,
analysis.unique_anchor_plan(), blob,
merge_policy=MergePolicy.FINAL_PORT_ORDER,
)
for current in (rt, recovered):
current.fire_gather("hx")
current.step_token(current.token_at("D")[0])
current.fire_gather("hy")
current.fire_gather("final")
current.step_token(current.token_at("END")[0])
self.assertEqual(recovered.state.committed_values, rt.state.committed_values)
self.assertEqual(recovered.snapshot_dict(), rt.snapshot_dict())
def test_exception_inside_transition_does_not_publish_partial_mutation(self):
graph = graph_ordering_counterexample()
analysis = SymbolicAnalyzer(graph).analyze()
rt = Runtime(graph, analysis.unique_anchor_plan(), merge_policy=MergePolicy.STRICT_PRECEDENCE)
rt.step_token("T0")
for node in ("A", "B", "fa", "fb"):
rt.step_token(rt.token_at(node)[0])
rt.fire_gather("hx")
rt.fire_gather("hy")
before = rt.snapshot_dict()
with self.assertRaises(MergeOrderConflict):
rt.fire_gather("final")
self.assertEqual(rt.snapshot_dict(), before)
def test_shared_write_is_deduplicated_after_checkpoint(self):
graph, analysis, rt = self._prepared_runtime()
blob = rt.checkpoint()
rt = Runtime.recover(graph, analysis.unique_anchor_plan(), blob, merge_policy=MergePolicy.FINAL_PORT_ORDER)
rt.fire_gather("hx")
rt.step_token(rt.token_at("D")[0])
rt.fire_gather("hy")
rt.fire_gather("final")
end = rt.token_at("END")[0]
self.assertEqual([rt.state.contributions[c].value for c in rt.token_history(end)], ["S", "C", "D"])
if __name__ == "__main__":
unittest.main(verbosity=2)
@@ -0,0 +1,30 @@
test_checkpoint_round_trip_ready_gather (test_fork_gather_reference.CheckpointTests.test_checkpoint_round_trip_ready_gather) ... ok
test_exception_inside_transition_does_not_publish_partial_mutation (test_fork_gather_reference.CheckpointTests.test_exception_inside_transition_does_not_publish_partial_mutation) ... ok
test_process_crash_discards_uncheckpointed_progress_and_recovers_last_snapshot (test_fork_gather_reference.CheckpointTests.test_process_crash_discards_uncheckpointed_progress_and_recovers_last_snapshot) ... ok
test_resume_from_checkpoint_matches_no_checkpoint_run (test_fork_gather_reference.CheckpointTests.test_resume_from_checkpoint_matches_no_checkpoint_run) ... ok
test_shared_write_is_deduplicated_after_checkpoint (test_fork_gather_reference.CheckpointTests.test_shared_write_is_deduplicated_after_checkpoint) ... ok
test_completed_round_loop_terminates_and_is_accepted (test_fork_gather_reference.CorrelationAnalysisTests.test_completed_round_loop_terminates_and_is_accepted) ... ok
test_cross_gathers (test_fork_gather_reference.CorrelationAnalysisTests.test_cross_gathers) ... ok
test_direct_fork_to_gather (test_fork_gather_reference.CorrelationAnalysisTests.test_direct_fork_to_gather) ... ok
test_mutually_exclusive_alternatives_same_port (test_fork_gather_reference.CorrelationAnalysisTests.test_mutually_exclusive_alternatives_same_port) ... ok
test_ordering_counterexample_control_is_accepted (test_fork_gather_reference.CorrelationAnalysisTests.test_ordering_counterexample_control_is_accepted) ... ok
test_ordinary_fork_gather (test_fork_gather_reference.CorrelationAnalysisTests.test_ordinary_fork_gather) ... ok
test_partial_gather_then_final (test_fork_gather_reference.CorrelationAnalysisTests.test_partial_gather_then_final) ... ok
test_simultaneous_duplicate_arrivals_same_port_are_rejected (test_fork_gather_reference.CorrelationAnalysisTests.test_simultaneous_duplicate_arrivals_same_port_are_rejected) ... ok
test_two_sibling_activations_same_static_fork_are_ambiguous (test_fork_gather_reference.CorrelationAnalysisTests.test_two_sibling_activations_same_static_fork_are_ambiguous) ... ok
test_unresolved_same_fork_reentry_is_rejected (test_fork_gather_reference.CorrelationAnalysisTests.test_unresolved_same_fork_reentry_is_rejected) ... ok
test_counterexample_has_legitimate_opposite_intermediate_orders (test_fork_gather_reference.MergeOrderingTests.test_counterexample_has_legitimate_opposite_intermediate_orders) ... ok
test_counterexample_port_order_runtime_commits_A_B (test_fork_gather_reference.MergeOrderingTests.test_counterexample_port_order_runtime_commits_A_B) ... ok
test_counterexample_strict_runtime_raises_at_final_not_intermediate (test_fork_gather_reference.MergeOrderingTests.test_counterexample_strict_runtime_raises_at_final_not_intermediate) ... ok
test_pairwise_shared_pair_agreement_is_not_sufficient_collectively (test_fork_gather_reference.MergeOrderingTests.test_pairwise_shared_pair_agreement_is_not_sufficient_collectively) ... ok
test_policy_a_rejects_opposite_order_constraints (test_fork_gather_reference.MergeOrderingTests.test_policy_a_rejects_opposite_order_constraints) ... ok
test_policy_b_final_port_order_resolves_by_first_occurrence (test_fork_gather_reference.MergeOrderingTests.test_policy_b_final_port_order_resolves_by_first_occurrence) ... ok
test_finite_checker_accepts_completed_round_loop (test_fork_gather_reference.ReentryCheckerTests.test_finite_checker_accepts_completed_round_loop) ... ok
test_finite_checker_rejects_unresolved_reentry (test_fork_gather_reference.ReentryCheckerTests.test_finite_checker_rejects_unresolved_reentry) ... ok
test_adversarial_two_schedules_same_semantic_result (test_fork_gather_reference.RuntimeScheduleTests.test_adversarial_two_schedules_same_semantic_result) ... ok
test_direct_fork_to_gather_preserves_ports_at_runtime (test_fork_gather_reference.RuntimeScheduleTests.test_direct_fork_to_gather_preserves_ports_at_runtime) ... ok
----------------------------------------------------------------------
Ran 25 tests in 0.024s
OK
@@ -0,0 +1,298 @@
# Fork/Gather Reference Verification Implementation Plan
> **For agentic workers:** Use `subagent-driven-development` or
> `executing-plans` to implement this plan task-by-task. OpenCode should use
> one fresh implementer per task, sequentially, with separate spec and quality
> reviews before proceeding. Do not launch production implementation.
**Goal:** Close two reproduced reference-model defects and deliver executable
evidence for the next fork/gather design decision.
**Architecture:** Copy the archived reference into an isolated experimental
directory. Preserve the original evidence, repair occurrence tracking, and
exercise compiler/runtime agreement. This produces a verification prototype,
not a new production runtime or approval of the entire graph model.
**Tech Stack:** Repository Python 3.14, standard-library `unittest`, existing
Ruff tooling, Markdown lint. No new dependencies or lockfile changes.
**Spec:** [Draft fork/gather contract](../specs/2026-09-06-fork-gather-design.md).
Only its reference-verification gates are implemented here. Production owner
integration, step-budget dispatch, and merge-policy selection remain open.
## Global constraints
- Do not modify `src/`, production tests, public APIs, or dependencies.
- Do not modify `docs/historical/research/2026-09-06-fork-gather/imported/`.
- Do not choose strict precedence or final-port order as product policy.
- Do not add a transition journal, external-effect guarantees, or compatibility
for this throwaway prototype's old checkpoints.
- New writes get new occurrence identities; merges retain original identities.
- One gather firing per owner/gather/anchor visit; resolved rounds may repeat.
- Report analysis bounds as inability to prove, never acceptance.
- Work in an isolated worktree; no main-worktree edits, merge, push, or squash.
- Do not remove untracked files or reset somebody else's changes.
## Handoff preflight
The planning inputs may still be uncommitted in the primary workspace. A new
worktree does not inherit those files automatically. Before dispatching a
worker, copy the following planning inputs from the primary workspace into the
isolated worktree if they are absent or differ from the clean branch baseline:
- `CONTEXT.md`, `docs/README.md`, and `docs/current_roadmap.md`;
- `docs/adr/0006-explicit-fork-and-topology-driven-gather.md`;
- `docs/superpowers/specs/2026-09-06-fork-gather-design.md`;
- this plan;
- `docs/historical/research/2026-09-06-fork-gather/` in full.
Use read-only access to the primary workspace and copy, never move. Verify
SHA-256 equality before starting work. Do not overwrite unrelated dirty files
in the destination; stop on a conflict or if the primary source cannot be
identified. Commit transferred inputs separately on the isolated branch and
record the resulting task base. Do not substitute older docs from branch HEAD.
Record the worktree path, branch, initial commit, and initial dirty-file list.
Read root `AGENTS.md`, `docs/AGENTS.md`, the spec, and the archive README. The
archive README is enough to understand the prior findings; reading both long
reports is optional unless resolving a specific discrepancy.
## File ownership
Create `experiments/fork_gather/` with these files:
- `fork_gather_reference.py`: editable copy of the reference simulator/analyzer.
- `test_fork_gather_reference.py`: copy of the original 25 tests.
- `test_occurrences.py`: repeated writes, gather firings, recovery regressions.
- `test_agreement.py`: bounded schedules, inference agreement, merge evidence.
- `README.md`: commands, supported grammar, experimental status, limitations.
- `VERIFICATION.md`: actual commands/results, counterexamples, decision report.
The copied flat module is an explicitly disposable research artifact, not a
production package-design precedent. Avoid a gratuitous module split in this
verification wave; document non-obvious changes near their implementation.
## Task 1: Distinguish write executions from token identity
**Files:** Create the experimental directory, copy the two Python files above,
create `test_occurrences.py` and `README.md`. Edit only the experimental copies.
**Consumes:** Archived `Graph`, `Runtime`, `RuntimeState`, `_write`, `_atomic`,
`checkpoint`, and `recover` interfaces.
**Produces:** Existing public interfaces unchanged; persisted per-token
execution counters in `RuntimeState`. Write identity includes the node-execution
occurrence, not only the static node and token.
- [ ] Copy the two Python sources with `Copy-Item`, not move. Record SHA-256
hashes of the originals before starting. Run the copied 25-test baseline:
```powershell
Push-Location experiments/fork_gather
try { python -B -m unittest -v test_fork_gather_reference.py }
finally { Pop-Location }
```
- [ ] Add this graph and a regression that executes two writes, then ends:
```python
graph = Graph("w", {
"w": WriteNode("w", "X"),
"pick": ChoiceNode("pick", ("again", "done")),
"end": EndNode("end"),
}, (
Edge("w", "ok", "pick"),
Edge("pick", "again", "w"),
Edge("pick", "done", "end"),
))
rt = Runtime(graph, {})
rt.step_token("T0")
rt.step_token("T0", outcome="again")
rt.step_token("T0")
rt.step_token("T0", outcome="done")
rt.step_token("T0")
assert rt.state.committed_values == ["ROOT", "X", "X"]
assert len(rt.state.contributions) == 2
```
- [ ] Run `python -B -m unittest -v test_occurrences.py` from the experimental
directory. Confirm failure is `duplicate semantic contribution w@T0`.
- [ ] Add `execution_counts: dict[str, int]` with a default factory to
`RuntimeState`. Increment only inside the candidate-state transaction for a
valid node dispatch; use the token-local count in each fresh write ID:
```python
occurrence = s.execution_counts.get(token.id, 0) + 1
s.execution_counts[token.id] = occurrence
# _write reads this count after _step_token_impl admits the execution.
contribution_id = f"{node.id}@{token.id}:execution:{occurrence}:write:0"
```
The reference has one write per WriteNode execution. Document that limitation;
do not invent multi-write capabilities. Failed candidate transitions must
leave the published counter unchanged. Do not use a global scheduling counter
that changes contribution IDs when sibling execution order changes.
- [ ] Add checkpoint coverage after the first write and before the second.
Continue both original and recovered states through the same choices. Assert
equal contribution ID sets and two final X values. Keep shared-history
deduplication tests passing so fresh IDs are not minted during a merge.
- [ ] Run the baseline and occurrence suites. Document commands and the identity
rule in the experiment README. Commit only Task 1 files, then obtain separate
spec and quality reviews. Close concrete findings before Task 2.
## Task 2: Track fired gathers through unresolved activations
**Files:** Modify the experimental reference and `test_occurrences.py`.
**Consumes:** Existing `SActivation`, `SToken`, claim normalization/enclosure,
`SymbolicAnalyzer.analyze()`, and runtime gather transitions.
**Produces:** Same `AnalysisResult` interface, with a structured diagnostic
string prefix `repeated_gather:` for the forbidden repeated occurrence.
Runtime raises `RepeatedGatherError`, not incidental duplicate-lineage errors.
- [ ] Add the following fixture to `test_occurrences.py`:
```python
def repeated_gather_graph():
nodes = (
ForkNode("g", ("a", "b")),
GatherNode("h", ("only",)),
ChoiceNode("pick", ("again", "done")),
GatherNode("final", ("left", "right")), EndNode("end"),
)
edges = (
Edge("g", "a", "h", "only"),
Edge("g", "b", "final", "right"),
Edge("h", "ok", "pick"),
Edge("pick", "again", "h", "only"),
Edge("pick", "done", "final", "left"),
Edge("final", "ok", "end"),
)
return Graph("g", {n.id: n for n in nodes}, edges)
```
Assert analysis rejects with `repeated_gather:`. Independently supply runtime
anchors `{"h": "g", "final": "g"}` to exercise runtime defense despite
invalid authoring. Fire `h`, checkpoint, recover, choose `again`; assert
`RepeatedGatherError` and unchanged published state. It is acceptable to
reject on deposit before the second firing; test that explicit boundary.
- [ ] Confirm the rejection test fails against the Task 1 analyzer, which
incorrectly accepts. Do not make it pass by banning one-port gathers.
- [ ] Introduce immutable symbolic marking state:
```python
@dataclass(frozen=True)
class SMarking:
tokens: tuple[SToken, ...]
fired: frozenset[tuple[str, SActivation]] = frozenset()
```
Preserve the canonical token ordering from `_marking`. Queue and visited
membership must include `fired`. An inferred firing key is `(gather_id,
anchor_activation)`. If already present, record `repeated_gather:` and do not
explore that invalid transition. Otherwise insert it before continuation.
- [ ] Keep firing keys only while their anchor remains referenced by at least
one token in the resulting marking, including references through saved parent
claims. Add a helper returning symbolic activation objects, not merely static
fork IDs. This garbage collection lets fully resolved rounds reach the same
abstract state again. Test retention through partial claims and cleanup only
after complete discharge. Do not drop markers based on enclosure alone.
- [ ] Add runtime `fired_gathers: set[tuple[str, str]]` to `RuntimeState`, using
the existing single-owner `(gather_id, dynamic_anchor_id)` keys. Add
`RepeatedGatherError(RuntimeError)`. Check before deposit and firing mutation;
insert the marker within the successful candidate transition before routing
its continuation. Persist the set in reference checkpoints. Distinct dynamic
visits must remain distinct; this is not a serialized production format.
- [ ] Verify the completed-round loop remains accepted within a fixed bound
(100 markings for the supplied tiny graph). Execute two complete rounds in
the runtime and assert distinct visit and continuation IDs. Force a tiny
`max_markings` bound and assert rejection with `analysis_limit_exceeded:`.
- [ ] Run all copied and new tests. Explain the revised finite-state argument,
including token multiplicity and retained markers, without claiming general
workflow soundness. If the argument fails, report a counterexample and stop
that claim; do not waive it because the sample tests pass. Commit Task 2 and
obtain separate spec and quality reviews before Task 3.
## Task 3: Adversarial agreement and decision handoff
**Files:** Create `test_agreement.py` and `VERIFICATION.md`; update experiment
README and only evidence-backed reference gates in the live spec.
**Consumes:** Repaired occurrence tracking and existing graph fixtures,
`AnalysisResult.unique_anchor_plan()`, runtime transitions, and both merge
policies. **Produces:** Executable bounded agreement tests and a short report;
no selected production policy and no production implementation plan.
- [ ] For the supplied basic, direct, partial, cross, conditional, and ordering
graph fixtures, derive plans through the analyzer. Do not hardcode anchors
in happy-path tests. Execute accepted acyclic examples to completion under
all schedules within an explicit per-fixture bound, branching over ready
token/gather actions and declared choices. Fail the test if the bound is
exceeded rather than silently omitting paths. Keep loops in separate bounded
tests so this is not a termination test.
- [ ] Use semantic comparisons, not trace order or counts alone:
```python
semantic_result = (
tuple(rt.state.committed_values),
frozenset(rt.state.committed_contribution_ids),
rt.state.completed,
)
assert not rt.live_token_ids()
assert not rt.parked_token_ids()
assert not rt.state.gather_buckets
```
For choice fixtures compare schedules for the same outcome choices. Treat
deliberate strict-policy merge conflicts as expected results, not completion.
Report schedule counts and bounds by fixture.
- [ ] Exercise the two new occurrence regressions alongside simultaneous
duplicate ports, unresolved re-entry, and sibling-same-static-fork ambiguity.
Record conservative unsupported cases separately from invalid runtime state.
Add a missing-arrival/deadlock probe and report what the analyzer does; do
not relabel `accepted` as complete workflow soundness.
- [ ] Preserve the A/B opposite-history and three-history precedence-cycle
tests under both policies. Show the concrete final values or merge error.
Add a pure contribution-replay example with numeric addition to demonstrate
that deduplication is distinct from append ordering. This is evidence for
one more reducer, not proof of generic state-path merge semantics.
- [ ] For a ready-gather snapshot, resume and compare semantic results with
uninterrupted execution. For a failing strict-order transition, compare the
complete published dataclass state (not only `snapshot_dict`, which omits
fields) before/after; include occurrence counters and fired markers.
- [ ] Run `python -B -m unittest discover -v` inside the experimental directory.
Run Ruff check and format-check on the exact changed experimental Python
files. Document inherited diagnostics before fixing; keep formatting changes
confined to editable copies. Lint the exact changed Markdown files and run
`git diff --check`. Recheck archived hashes and forbidden-path diffs.
- [ ] Write `VERIFICATION.md` with the exact base/HEAD, commands, pass/fail
counts, explored-state/schedule counts, unsupported cases, and new findings.
Include a short user-facing choice: preserve previous merge orders and fail
on cycles, or let final-port order define the local merged view. Recommend
one with tradeoffs, but leave the spec's policy undecided.
- [ ] Update only closed spec gates. Leave ordering selection, owner integration,
and production-validator selection open. Once this plan's work is complete,
move it under `docs/historical/superpowers/plans/` and repair its relative
links and live references per `docs/AGENTS.md`. Commit Task 3 and obtain a
final whole-branch review. Stop without merging or pushing.
## Controller and review contract
The controller owns scope, baseline, integration, and verdict triage; it should
not implement tasks behind its reviewers. Use a fresh implementer per task and
separate spec/quality reviewers. Give each worker exact file ownership and the
preceding commit. Do not run concurrent writers against the same files.
Reviewers must execute the two regression cases and inspect occurrence state,
not merely repeat the implementer's test counts. Distinguish a demonstrated
bug, a conservative limitation, and a policy preference. Reproduce findings
before fixing; commission focused fix waves without expanding into production.
If subagents are unavailable, report that fact and request a workflow decision;
do not invent review verdicts. The model-coordination guide is optional context,
not a dependency on its ignored filesystem path or its model availability claims.
Final handoff: branch/worktree, commits, verification evidence, outstanding
findings, and the one merge-order policy choice in plain language. Do not label
fork/gather implemented or production-ready because this plan is complete.
@@ -0,0 +1,154 @@
# Explicit fork/gather design
Status: draft, not implemented or approved for production execution.
This is the current fork/gather contract under development. It refines
[ADR-0006](../../adr/0006-explicit-fork-and-topology-driven-gather.md).
The [research archive](../../historical/research/2026-09-06-fork-gather/README.md)
preserves the reports, reconstructed reference code, and independent review.
Research recommendations are not requirements unless adopted below.
## Settled direction
### Emission and destination ports
Ordinary outcomes select exactly one transition. An explicit fork emits one
branch token for every statically declared branch. Gathers declare ordered,
named input ports; every incoming edge must name a declared `target_port`.
Several edges may be mutually exclusive alternatives for the same port.
Ports may be derived from node configuration before execution, not invented
by runtime data. On ordinary destinations, `target_port` is preserved metadata
and is ignored for execution, not rejected.
All emissions use the same destination handling, including direct fork-to-gather
edges. An arrival retains the destination port; a cursor located only at the
gather node is insufficient. A ready gather's execution must respect the
existing run-wide step budget before publishing its transition.
### Control obligations and correlation
A fork consumes its incoming token, saves its parent claim, and creates branch
obligations. A partial gather consumes its arrivals and produces one token with
their combined obligations. Complete sibling obligations normalize to their
saved parent claim. Re-forking a partial claim saves that claim on the new visit;
it does not duplicate ownership of that claim onto every new branch.
Keep these relations distinct:
- Referenced visits: unresolved visits on which any part of the claim depends.
Use this relation to reject re-entry of an unresolved static fork.
- Enclosing visits: unresolved visits encompassing every part of the claim.
Use this relation for gather correlation, not the union of referenced visits.
For `{k.x, r.c}`, where `k` is inside `r`, referenced visits are `{k, r}` but
enclosing visits are only `{r}`. The mixed token cannot be correlated under `k`.
Gather declarations do not name the originating fork. The proposed V1 uses a
compiled static anchor resolved to a unique enclosing dynamic visit. Matching
also requires the same completion owner and runtime scope; scheduling order
must never decide matching. The supported inference grammar remains open.
### Occurrences and completion
The proposed V1 permits at most one arrival per port and one gather firing per
`(owner, gather, anchor visit)`. Validation must account for earlier firings,
not only the current marking of runnable and parked tokens. Runtime checks
remain necessary. Do not turn a known illegal second firing into an incidental
duplicate-lineage-ID exception. Fully converged rounds may loop again under
a fresh dynamic visit; ordinary same-region loops remain valid.
Token identity is not node-execution occurrence identity. A repeated execution
of a writing node creates fresh contribution identities even when it uses the
same token. Replaying or merging an existing contribution preserves its
identity. A recovered occurrence and a genuinely new execution must not be
conflated. The concrete ID encoding is not selected here.
Completion ownership must survive token/frame replacement. Existing foreach
and subgraph blocking responsibilities do not disappear: the parent waits for
item/call completion, not merely the original cursor's retirement. Branches
must converge before owner completion or foreach return. No branch may use
`END` to silently discard outstanding siblings.
### State and ordering
A single-parent lineage tree may remain the physical state-view structure.
Its lowest common ancestor supplies a merge base, not proof that pending
contributions are unique. Accepting shared-history re-fork/cross/reconvergence
requires stable original contribution identity or an equivalent demonstrated
deduplication mechanism. Otherwise that topology must be rejected explicitly.
Reducers and `conflicts="error"` remain the merge foundation. Results must not
depend on arrival or scheduling order. Declared port order determines replay
for independent histories. It does not yet settle conflicting earlier merge
orders at later reconvergence:
```text
r.a -> write A -> fa(x,y)
r.b -> write B -> fb(x,y)
hx(left=fa.x, right=fb.x) -> [A, B]
hy(left=fb.y, right=fa.y) -> [B, A]
final(left=hx, right=hy)
```
This is a legitimate authored control graph, not inherently corrupt metadata.
Strict preservation of previous orders would fail at `final`; final-port
precedence would choose `[A, B]`. Neither policy is adopted yet. Strict ordering
would require global cycle detection: `[A,B]`, `[B,C]`, `[C,A]` can pass pairwise
shared-subsequence checks while jointly imposing a cycle.
### Persistence boundary
Supported stopped checkpoints must contain a consistent semantic state,
including any pending arrivals and occurrence information needed on resume.
An exception must not publish a partially applied transition as the failed
checkpoint: retain stable state plus failure information, or provide an
equivalent rollback/commit boundary. Whole-run deepcopy is a reference-model
technique, not a production requirement.
Do not add a transition journal merely because a transition has several
in-memory steps. Arbitrary mid-transition persistence and exactly-once external
effects are not added by this design. A process crash can discard progress since
the last durable checkpoint; contribution IDs alone do not deduplicate an
external API call. Existing store transaction contracts still apply.
## Open decisions before production planning
1. Does an earlier gather's serialization order constrain later merges, or is
it a local state view? Select and document the reducer conflict semantics.
2. Which correlation grammar can be validated soundly with acceptable cost and
conservative rejection? The imported explicit marking explorer is evidence,
not an approved production algorithm or complete soundness validator.
3. How are owner completion, failure cleanup, scope boundaries, occurrence
state, and checkpoint validation integrated with foreach and subgraphs?
The single-owner reference does not verify these integrations.
## Verification gates before an executable implementation plan
The [reference verification plan](../plans/2026-09-07-fork-gather-reference-verification.md)
executes the experimental subset below. A production implementation plan remains
gated on the open decisions; completing the reference plan does not close them.
Do not copy the imported simulator into production or modify the archived
evidence. Develop any revised experiment separately. Gates remain unchecked
until new executable evidence closes them.
- [ ] Add a repeated-write regression: `w -> pick.again -> w`, then exit.
Two executions append twice with distinct IDs; merging the same contribution
through two descendants still applies it once. Include checkpoint/resume.
- [ ] Add the repeated-gather regression: `g.a -> h.only -> pick`,
`pick.again -> h.only`, `pick.done -> final.left`, `g.b -> final.right`.
Reject the second firing under the same anchor during validation. Preserve
acceptance of fully resolved rounds that revisit the same static nodes.
- [ ] Compare inferred plans with runtime behavior for basic, direct, partial,
cross, conditional-alternative, duplicate-port, unresolved-reentry, and
sibling-same-static-fork graphs. Define unsupported cases explicitly.
- [ ] Include already-fired occurrence state in the analysis argument without
preventing completed rounds from reaching a finite abstract fixed point.
Bound exploration and report inability to prove, not silent acceptance.
- [ ] Resolve ordering with the A/B graph and the three-history cycle above.
Test more than append-only examples before claiming general reducer support.
- [ ] Test schedule permutations and consistent stopped checkpoint/resume,
including transition failures. Distinguish this from external-effect safety.
- [ ] Design and test owner integration, then produce a scoped production plan
against the chosen grammar and merge policy. Keep its code/test paths and
compatibility decisions grounded in current runtime seams.