lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

state write

Browse Source
This commit is contained in:
lda
2026-05-24 22:08:00 +07:00 Verified
parent 2dc00518b9
commit 39b473ea25
9 changed files with 1889 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current_roadmap.md
+10 -8
View File
@@ -43,15 +43,17 @@ implementation state.
[ADR 0001](./adr/0001-scheduler-foundation-before-concurrent-foreach.md).
- Concurrent foreach policy decision record:
[ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md).
- **Native subgraphs / graph-as-node**: add child run state, child trace
- Native subgraph design spec:
[2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md).
- **Native subgraphs / graph-as-node**: next major runtime feature. Add a
first-class subgraph step with child run/frame identity, child trace
preservation, interrupt bubbling, and resume back into the child workflow.
Wrapper artifacts currently execute as deployments and return run status;
true graph-as-node outcome propagation belongs here.
- **Concurrent foreach**: add explicit scheduling, reducer/merge semantics, and
failure policy. Sync runtime can interleave admitted item frames one node at a
time; async runtime can additionally run admitted async node handlers
simultaneously. Do not model this as plain `asyncio.gather` over sync
handlers.
Wrapper helpers currently run child workflows as ordinary nodes; true
graph-as-node behavior belongs here.
- **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of
its barrier/lineage machinery by native subgraphs and fork/gather.
- **Persistent run history**: add a run store before adding stable `run_id`,
`inspect_run`, or `read_run_trace(run_id, range)` APIs. Current traces are
returned directly from immediate run responses.
docs/superpowers/plans/2026-05-24-lineage-state-runtime.md
+688
View File
@@ -0,0 +1,688 @@
# Lineage State Runtime Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add runtime scopes and lineages before native subgraphs so sibling branches/items can own isolated replayable state writes and later merge through barriers.
**Architecture:** Implements [`../specs/2026-05-24-lineage-state-runtime-design.md`](../specs/2026-05-24-lineage-state-runtime-design.md). Keep `RunState.state` as the committed root-scope compatibility state, add `RuntimeScope`, `LineageState`, and `StateWrite`, and migrate concurrent foreach from foreach-specific overlays to lineage-backed state views. A frame says where execution is; a scope says which workflow state root execution belongs to; a lineage says which pending writes that execution can see.
**Tech Stack:** Python 3.14, dataclasses, existing `StatePatch`, `StatePath`, `ReducerRef`, `ForeachBarrierState`, pytest, basedpyright, ruff.
---
## File Structure
- Modify: `src/wf_core/run_state.py`
- Add `RuntimeScope`, `StateWrite`, `LineageState`.
- Add `ExecutionFrame.scope_id` and `ExecutionFrame.lineage_id`.
- Add `RunState.scopes` and `RunState.lineages`.
- Modify: `src/wf_core/runtime/ops/state.py`
- Change `StatePatch` from only path-value maps to ordered `StateWrite` records while preserving `changes` as a compatibility/trace view.
- Create: `src/wf_core/runtime/lineage.py`
- Own scope/lineage lookup, state view materialization, append writes, and conversion of completed lineage writes into barrier patches.
- Modify: `src/wf_core/runtime/ops/runs.py`
- Initialize root scope and root lineage.
- Modify: `src/wf_core/runtime/ops/nodes.py`
- Resolve node input from frame scope/lineage view and buffer non-root writes into lineage records.
- Modify: `src/wf_core/runtime/ops/foreach.py`
- Create concurrent item lineages and commit completed lineage writes through the barrier.
- Modify: `src/wf_core/runtime/foreach_state.py`
- Store completed lineage ids in pending item results; keep old patch metadata parse-compatible.
- Modify: `src/wf_core/runtime/ops/overlays.py`
- Reduce to a compatibility facade over lineage state views.
- Test: `tests/core/test_lineage_state.py`
- Unit tests for root scope/lineage, state views, write records, and non-root write buffering.
- Test: `tests/core/test_atomic_state_patches.py`
- Tests for ordered `StateWrite` records and compatibility `changes`.
- Test: `tests/core/test_concurrent_foreach.py`
- Regression tests for sibling isolation, same-item visibility, and deterministic barrier commits.
- Docs: `docs/wf_core_architecture.md`, `docs/current_roadmap.md`, `docs/superpowers/specs/2026-05-24-native-subgraphs-design.md`
- Document scope/lineage as the prerequisite for native subgraphs.
---
## Core Shape
Target runtime state types:
```python
@dataclass(slots=True)
class StateWrite:
"""One reducer-aware write record owned by a lineage or patch."""
path: StatePath
incoming_value: Any
visible_value: Any
reducer: ReducerRef
@dataclass(slots=True)
class RuntimeScope:
"""Committed state root for one workflow activation."""
id: str
workflow_name: str
committed_state: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class LineageState:
"""Pending ordered writes visible to frames in one lineage."""
id: str
scope_id: str
parent_id: str | None = None
writes: list[StateWrite] = field(default_factory=list)
```
Target patch shape:
```python
@dataclass(slots=True)
class StatePatch:
"""Validated state writes produced by one step before commit."""
writes: list[StateWrite] = dataclass_field(default_factory=list)
_staged_state: dict[str, Any] = dataclass_field(default_factory=dict, repr=False)
@property
def changes(self) -> dict[str, Any]:
return {str(write.path): write.incoming_value for write in self.writes}
@property
def visible_values(self) -> dict[str, Any]:
return {str(write.path): write.visible_value for write in self.writes}
```
Compatibility requirement: existing tests and callers that read `patch.changes`
should continue to work. New lineage code must use ordered `writes`, not
flattened final values.
---
## Task 1: Add Ordered StateWrite Records to StatePatch
**Files:**
- Modify: `src/wf_core/runtime/ops/state.py`
- Test: `tests/core/test_atomic_state_patches.py`
- [ ] **Step 1: Add failing tests**
Append:
```python
def test_output_patch_records_incoming_and_visible_values() -> None:
workflow = _workflow_with_state_field(
path="state.count",
schema={"type": "integer"},
reducer="wf.std.add",
)
state = {"count": 2}
patch = build_output_patch(
workflow,
[OutputBinding.model_validate({"source": "delta", "target": "state.count"})],
{"delta": 3},
state,
)
assert patch.changes["state.count"] == 3
assert patch.visible_values["state.count"] == 5
assert patch.writes[0].incoming_value == 3
assert patch.writes[0].visible_value == 5
def test_barrier_replays_incoming_values_not_lineage_visible_values() -> None:
workflow = _workflow_with_state_field(
path="state.number",
schema={"type": "integer"},
reducer="wf.std.add",
)
patch = build_barrier_patch(
workflow,
[
StatePatch(
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
)
]
),
StatePatch(
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=1,
visible_value=3,
reducer=ReducerRef(name="wf.std.add"),
)
]
),
],
{"number": 2},
)
assert patch.changes["state.number"] == 6
assert patch.visible_values["state.number"] == 6
```
Add imports:
```python
from wf_core.models.reducers import ReducerRef
from wf_core.paths import StatePath
from wf_core.run_state import StateWrite
```
- [ ] **Step 2: Run expected failing tests**
Run:
```bash
uv run pytest tests/core/test_atomic_state_patches.py::test_output_patch_records_incoming_and_visible_values tests/core/test_atomic_state_patches.py::test_barrier_replays_incoming_values_not_lineage_visible_values -q
```
Expected: failure because `StateWrite`, `StatePatch.writes`, and `visible_values`
do not exist.
- [ ] **Step 3: Implement `StateWrite` and patch views**
In `src/wf_core/run_state.py`, add `StateWrite` near runtime dataclasses:
```python
@dataclass(slots=True)
class StateWrite:
path: StatePath
incoming_value: Any
visible_value: Any
reducer: ReducerRef
```
Import `ReducerRef` and `StatePath`.
In `src/wf_core/runtime/ops/state.py`, change `StatePatch` to store ordered
writes and expose `changes` / `visible_values` properties. Keep `_staged_state`.
- [ ] **Step 4: Build write records in `build_output_patch`**
When `prepare_state_value(...)` returns a merged value, also capture the reducer
used for the destination. If needed, extract reducer lookup from
`prepare_state_value(...)` into a helper:
```python
def reducer_for_state_path(
path: StatePath,
state_fields: Mapping[StatePath, StateFieldDecl],
) -> ReducerRef:
field = state_fields.get(path)
return field.reducer if field and field.reducer else ReducerRef(name="wf.std.replace")
```
Create:
```python
StateWrite(
path=destination_path,
incoming_value=value,
visible_value=merged_value,
reducer=reducer,
)
```
- [ ] **Step 5: Replay incoming values in `build_barrier_patch`**
Update `build_barrier_patch(...)` to iterate over `item_patch.writes`, not over
`item_patch.changes.items()`. Replay `write.incoming_value` against the staged
state. The resulting barrier patch should contain one `StateWrite` per final
destination with both incoming and visible values set to the final committed
aggregate value, because the barrier is the public commit point.
- [ ] **Step 6: Run atomic patch tests**
Run:
```bash
uv run pytest tests/core/test_atomic_state_patches.py -q
```
Expected: pass.
---
## Task 2: Add Runtime Scopes and Root Lineage
**Files:**
- Modify: `src/wf_core/run_state.py`
- Modify: `src/wf_core/runtime/ops/runs.py`
- Test: `tests/core/test_lineage_state.py`
- [ ] **Step 1: Add root initialization test**
Create `tests/core/test_lineage_state.py` with a local minimal workflow helper
that builds a tiny core `Workflow`. Then add:
```python
def test_create_run_state_initializes_root_scope_and_lineage() -> None:
workflow = minimal_workflow()
run = create_run_state(workflow, {"value": "seed"})
assert run.scopes["root"].id == "root"
assert run.scopes["root"].workflow_name == workflow.name
assert run.scopes["root"].committed_state["value"] == "seed"
assert run.lineages["root"].id == "root"
assert run.lineages["root"].scope_id == "root"
assert run.lineages["root"].parent_id is None
assert run.lineages["root"].writes == []
assert run.frames["root"].scope_id == "root"
assert run.frames["root"].lineage_id == "root"
```
- [ ] **Step 2: Run expected failing test**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py::test_create_run_state_initializes_root_scope_and_lineage -q
```
Expected: failure because `scopes`, `lineages`, `scope_id`, and `lineage_id`
do not exist.
- [ ] **Step 3: Add dataclasses and fields**
In `src/wf_core/run_state.py`, add `RuntimeScope` and `LineageState`. Add
`scope_id: str = "root"` and `lineage_id: str = "root"` to `ExecutionFrame`.
Add `scopes` and `lineages` to `RunState`.
- [ ] **Step 4: Initialize root scope and lineage**
In `src/wf_core/runtime/ops/runs.py`, initialize:
```python
run = RunState(
workflow_name=workflow.name,
status=RunStatus.PENDING,
workflow_input=dict(workflow_input),
state=state,
scopes={
"root": RuntimeScope(
id="root",
workflow_name=workflow.name,
committed_state=state,
)
},
lineages={"root": LineageState(id="root", scope_id="root")},
current_frame_id="root",
current_node_id=workflow.start,
)
```
The root scope may share the same dict object as `RunState.state` during this
migration.
- [ ] **Step 5: Run focused test**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py -q
```
Expected: pass.
---
## Task 3: Add Lineage Runtime Helpers
**Files:**
- Create: `src/wf_core/runtime/lineage.py`
- Test: `tests/core/test_lineage_state.py`
- [ ] **Step 1: Add helper tests**
Append:
```python
def test_lineage_state_view_applies_visible_values_only_for_reads() -> None:
workflow = minimal_workflow()
run = create_run_state(workflow, {"number": 2})
add_lineage(run, scope_id="root", lineage_id="branch", parent_id="root")
append_lineage_writes(
run,
scope_id="root",
lineage_id="branch",
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
)
],
)
view = lineage_state_view(run, scope_id="root", lineage_id="branch")
assert view["number"] == 5
assert run.state["number"] == 2
def test_lineage_write_patch_preserves_incoming_values_for_barrier_replay() -> None:
workflow = minimal_workflow()
run = create_run_state(workflow, {"number": 2})
add_lineage(run, scope_id="root", lineage_id="branch", parent_id="root")
append_lineage_writes(
run,
scope_id="root",
lineage_id="branch",
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
)
],
)
patch = lineage_patch(run, scope_id="root", lineage_id="branch")
assert patch.writes[0].incoming_value == 3
assert patch.writes[0].visible_value == 5
```
- [ ] **Step 2: Run expected failing tests**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py -q
```
Expected: import failure for `wf_core.runtime.lineage`.
- [ ] **Step 3: Implement `runtime.lineage`**
Create helpers:
```python
def add_lineage(
run: RunState, *, scope_id: str, lineage_id: str, parent_id: str
) -> None: ...
def append_lineage_writes(
run: RunState,
*,
scope_id: str,
lineage_id: str,
writes: Sequence[StateWrite],
) -> None: ...
def lineage_patch(run: RunState, *, scope_id: str, lineage_id: str) -> StatePatch: ...
def lineage_state_view(
run: RunState, *, scope_id: str, lineage_id: str
) -> dict[str, Any]: ...
```
`lineage_state_view(...)` should deep-copy `run.scopes[scope_id].committed_state`
and apply `write.visible_value` from ancestor/current lineage writes in order.
`lineage_patch(...)` should return ordered writes with incoming values intact.
- [ ] **Step 4: Run focused tests**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py -q
```
Expected: pass.
---
## Task 4: Route Node Reads and Non-Root Writes Through Lineage
**Files:**
- Modify: `src/wf_core/runtime/ops/overlays.py`
- Modify: `src/wf_core/runtime/ops/nodes.py`
- Test: `tests/core/test_lineage_state.py`
- [ ] **Step 1: Add non-root write buffering test**
Add a test with a one-node workflow that reads `state.value`, writes
`state.value`, and runs the frame with `lineage_id="child"`. Assert:
```python
assert result.state_changes == {}
assert run.state["value"] == "root"
assert run.lineages["child"].writes[0].incoming_value == "root-child"
assert lineage_state_view(run, scope_id="root", lineage_id="child")["value"] == "root-child"
```
- [ ] **Step 2: Run expected failing test**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py::test_non_root_lineage_node_writes_are_buffered_not_committed -q
```
Expected: failure because node execution still commits or cannot read through
lineage.
- [ ] **Step 3: Update overlay facade**
`state_view_for_frame(run, frame)` should call:
```python
lineage_state_view(run, scope_id=frame.scope_id, lineage_id=frame.lineage_id)
```
For root scope/root lineage it may return `run.state` directly as an optimization.
- [ ] **Step 4: Update node finalization**
In `_finalize_node_execution(...)`:
- if `frame.scope_id == "root"` and `frame.lineage_id == "root"`, commit patch
to `run.state`
- otherwise append `patch.writes` to the frame lineage and return empty committed
`state_changes`
- [ ] **Step 5: Run focused tests**
Run:
```bash
uv run pytest tests/core/test_lineage_state.py tests/core/test_atomic_state_patches.py -q
```
Expected: pass.
---
## Task 5: Migrate Concurrent Foreach to Lineages
**Files:**
- Modify: `src/wf_core/runtime/ops/foreach.py`
- Modify: `src/wf_core/runtime/foreach_state.py`
- Test: `tests/core/test_concurrent_foreach.py`
- Test: `tests/core/test_foreach_barrier_state.py`
- [ ] **Step 1: Add item lineage regression**
Add:
```python
def test_concurrent_foreach_item_frames_use_distinct_lineages() -> None:
workflow = _workflow(mode="concurrent", concurrent={"max_active": 2})
run = execute_workflow(workflow, {"items": ["a", "b"]}, {"record": _record_handler})
item_frames = [frame for frame in run.frames.values() if frame.kind == "foreach_iteration"]
assert len(item_frames) == 2
assert item_frames[0].lineage_id != "root"
assert item_frames[1].lineage_id != "root"
assert item_frames[0].lineage_id != item_frames[1].lineage_id
```
- [ ] **Step 2: Add same-item read regression**
Add or keep a multi-step concurrent foreach test where item node 1 writes
`state.scratch`, item node 2 reads `state.scratch`, and siblings do not see each
other's scratch.
- [ ] **Step 3: Store lineage id on pending item result**
Add `lineage_id: str | None = None` to `PendingItemResult`, parse it from
metadata, and serialize it back. Keep old `patch` parse compatibility.
- [ ] **Step 4: Create item lineages on admission**
In `_admit_concurrent_children(...)`, before adding the child frame:
```python
child_lineage_id = child_id
add_lineage(
run,
scope_id=frame.scope_id,
lineage_id=child_lineage_id,
parent_id=frame.lineage_id,
)
```
Pass `scope_id=frame.scope_id` and `lineage_id=child_lineage_id` to the child
`ExecutionFrame`.
- [ ] **Step 5: Record completed lineage ids**
When a child completes, record `child.lineage_id` on the barrier pending result.
Do not copy flattened visible values into the barrier.
- [ ] **Step 6: Build barrier from lineage patches**
In `_finish_concurrent_foreach(...)`, construct `success_patches` from
`lineage_patch(run, scope_id=frame.scope_id, lineage_id=result.lineage_id)` for
new results. Keep existing `result.patch` fallback for old metadata.
- [ ] **Step 7: Run foreach tests**
Run:
```bash
uv run pytest tests/core/test_concurrent_foreach.py tests/core/test_concurrent_foreach_async.py tests/core/test_foreach_barrier_state.py -q
```
Expected: pass.
---
## Task 6: Remove Foreach-Specific Overlay Coupling
**Files:**
- Modify: `src/wf_core/runtime/ops/overlays.py`
- Modify: `src/wf_core/runtime/ops/nodes.py`
- Test: `tests/core`
- [ ] **Step 1: Remove foreach imports from node/overlay state path**
Ensure `ops/nodes.py` and `ops/overlays.py` do not import
`ForeachBarrierState` or `item_frame_owner`.
- [ ] **Step 2: Run core tests**
Run:
```bash
uv run pytest tests/core -q
```
Expected: pass.
---
## Task 7: Update Docs
**Files:**
- Modify: `docs/wf_core_architecture.md`
- Modify: `docs/current_roadmap.md`
- Modify: `docs/superpowers/specs/2026-05-24-native-subgraphs-design.md`
- [ ] **Step 1: Document scope/lineage**
Add a section explaining:
- scope is workflow state root
- frame is scheduler position
- lineage is pending write ownership
- concurrent foreach uses child lineages
- native subgraphs require child scopes
- [ ] **Step 2: Update native subgraph spec**
Ensure it says native subgraphs depend on child runtime scopes plus lineages,
not lineage alone.
- [ ] **Step 3: Run doc red-flag scan**
Run:
```bash
rg -n "U[N]RESOLVED|I[N]COMPLETE|F[I]LL_ME|D[E]CIDE_ME" docs/wf_core_architecture.md docs/current_roadmap.md docs/superpowers/specs/2026-05-24-native-subgraphs-design.md
```
Expected: no output.
---
## Task 8: Full Verification
- [ ] **Step 1: Run tests**
```bash
uv run pytest -q
```
- [ ] **Step 2: Run type check**
```bash
uv run basedpyright --level error src tests examples
```
- [ ] **Step 3: Run lint**
```bash
uvx ruff check src tests examples
```
- [ ] **Step 4: Run format check**
```bash
uvx ruff format --check src tests examples
```
---
## Self-Review
Spec coverage:
- Scope is represented explicitly and is available for native subgraph state
roots.
- Lineage stores ordered replayable writes, not full state and not only visible
values.
- `StatePatch` preserves incoming values for gather/barrier replay.
- Same-lineage reads use visible values.
- Concurrent foreach is the first migration target.
Type consistency:
- `StateWrite.incoming_value` is replay/trace input.
- `StateWrite.visible_value` is same-lineage read value.
- `RuntimeScope.committed_state` is scope-local committed state.
- `ExecutionFrame.scope_id` and `lineage_id` select visibility.
docs/superpowers/specs/2026-05-24-lineage-state-runtime-design.md
+393
View File
@@ -0,0 +1,393 @@
# Lineage State Runtime Design
Status: proposed
Lineage is the missing primitive between the scheduler frame model and future
native subgraphs / fork-gather. A frame says where execution is. A scope says
which workflow state root execution belongs to. A lineage says what pending
writes execution can see inside that scope.
The current runtime stores committed state directly on `RunState.state` and uses
foreach-specific barrier metadata to emulate item-local overlays. That worked
for concurrent foreach, but native subgraphs and future fork/gather need the
same state-visibility rule in a reusable core concept.
## Problem
`RunState` currently owns too many meanings:
- run status
- scheduler cursor
- frame set
- trace
- interrupt request
- committed workflow state
- temporary state visibility hacks through frame/barrier metadata
This creates pressure when a workflow branches:
```text
root
fork -> A
-> B
```
Branch `A` must not see branch `B` writes before an explicit merge point.
Branch `B` must not see branch `A` writes either. A future gather then decides
how to merge both branches back into a parent state view.
The same problem already exists in concurrent foreach item frames. Each item is
a sibling lineage. Current foreach solves it locally with barrier pending
patches. That solution should become a general runtime primitive.
Native subgraphs add one more missing concept: a child workflow has its own
committed state root. A lineage alone is not enough for subgraphs because a
child workflow initializes state from child input/defaults, not from the
parent's state dictionary.
## Vocabulary
```text
Scope = workflow state root and workflow identity
Frame = scheduler/control-flow position inside a scope
Lineage = pending write ownership and read overlay inside a scope
```
A frame owns execution lifecycle:
```python
ExecutionFrame(
id="root:each:0",
scope_id="root",
kind="foreach_iteration",
node_id="work",
status="pending",
parent_frame_id="root",
lineage_id="root:each:0",
)
```
A lineage owns ordered pending writes:
```python
LineageState(
id="root:each:0",
scope_id="root",
parent_id="root",
writes=[
StateWrite(
path=StatePath(("count",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
),
],
)
```
The frame, scope, and lineage ids may match in simple cases, but they are not
the same concept and runtime logic must not depend on parsing any of them.
## Core Invariants
- Every executable frame belongs to exactly one scope.
- Every executable frame points at exactly one lineage in that scope.
- The root frame points at the root scope and root lineage.
- `RunState.state` remains the committed root scope state during the migration.
- A non-root lineage stores replayable write records, not a full copied state
snapshot and not only final visible values.
- A frame reads through its lineage state view.
- A non-root frame write updates its lineage writes and does not mutate
`RunState.state`.
- A barrier/gather commits lineage write records into a parent lineage or scope
root state by replaying incoming values through reducers.
- Trace `state_changes` means committed changes only.
- Buffered lineage writes may be visible to later frames in the same lineage,
but they are not public committed state changes until a barrier commits them.
## Why Write Records Instead of Full State
Lineages should store write records, not full state snapshots.
Full snapshots make reads easy but make merges difficult. At gather time the
runtime would have to diff branch snapshots against a base snapshot to discover
what changed. That becomes fragile with nested objects, defaults, reducers,
missing fields, and JSON Schema validation.
Write-record-owned lineages make the merge boundary explicit:
```text
lineage A wrote incoming value X to state.person.name
lineage B wrote incoming value Y to state.person.email
```
The barrier can then validate conflicts and apply reducers deterministically.
Runtime may cache or materialize state views for performance later. The source
of truth should still be ordered lineage write records.
## Incoming Values vs Visible Values
Existing `StatePatch.changes` is trace-facing. For normal node patches it stores
the incoming values from node output bindings. That is useful for trace and for
barrier replay. It is not enough by itself for same-lineage reads when reducers
are involved.
Example:
```text
committed state.count = 2
node output delta = 3
reducer = add
visible state.count after write = 5
```
Trace should be able to say the node emitted `3`. A later node in the same
lineage must see `5`. A future gather must still replay `3` into the parent,
not replay the visible value `5`.
So the runtime needs ordered writes:
```python
StateWrite(
path=StatePath(("count",)),
incoming_value=3, # trace and barrier replay value
visible_value=5, # same-lineage read value
reducer=ReducerRef(name="wf.std.add"),
)
```
`StatePatch` should preserve ordered `StateWrite` records. Convenience views can
derive incoming trace changes and same-lineage visible values, but lineage must
not discard incoming values.
This matters for fork/gather:
```text
root state.number = 2
fork A and B
A applies add(3), visible in A is 5
B applies add(1), visible in B is 3
gather must commit 2 + 3 + 1 = 6
```
If lineage stored only visible values, gather would incorrectly replay `5` and
`3` instead of `3` and `1`.
## State View
The visible state for a frame is:
```text
committed scope root state
+ ancestor lineage visible values
+ current lineage visible values
```
For v1, materializing this with a deep copy is acceptable because correctness is
more important than performance. The implementation should keep this behind one
helper so copy-on-write or structural sharing can replace it later.
```python
def lineage_state_view(run: RunState, scope_id: str, lineage_id: str) -> dict[str, Any]:
...
```
## Relationship to Concurrent Foreach
Concurrent foreach is the first real user of lineage.
Current behavior:
- parent foreach frame stores pending item patches in barrier metadata
- `state_view_for_frame` knows about foreach metadata
- node finalization knows about foreach item ownership
Target behavior:
- each concurrent item frame gets its own lineage
- `state_view_for_frame` delegates to lineage helpers
- node finalization only decides root commit vs lineage buffer
- foreach barrier records completed lineage ids
- barrier commit replays completed lineage write records into the existing
reducer/conflict validation path
This removes foreach-specific state overlay logic from node execution.
## Relationship to Native Subgraphs
Native subgraphs should build on scopes plus lineage.
A subgraph creates a child scope because the child workflow has a separate
committed state root initialized from child workflow input and child state
defaults. The subgraph root frame runs in that child scope and receives the
child scope's root lineage. Child workflow internal frames may create further
child lineages for foreach or future fork/gather.
Parent state changes only happen when the subgraph boundary completes and
applies explicit output bindings. Child internal writes remain inside the child
scope until that boundary.
This prevents child workflow internals from leaking into parent state and gives
interrupt/resume a stable state-visibility boundary.
## Relationship to Future Fork/Gather
Fork creates sibling lineages within the same scope. Gather consumes declared
lineage ids and commits their writes into a parent lineage or scope root.
Conceptually:
```text
fork parent lineage P
-> child lineage A
-> child lineage B
gather A+B into P
```
The gather operation should reuse the same conflict validation and reducer rules
as concurrent foreach barrier commits:
- same-path sibling writes require a mergeable reducer
- ancestor/descendant sibling writes are rejected until a deeper merge policy is
explicitly designed
- reducer application order is deterministic
## RunState Shape
Initial additive shape:
```python
@dataclass(slots=True)
class StateWrite:
path: StatePath
incoming_value: Any
visible_value: Any
reducer: ReducerRef
@dataclass(slots=True)
class RuntimeScope:
id: str
workflow_name: str
committed_state: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class LineageState:
id: str
scope_id: str
parent_id: str | None = None
writes: list[StateWrite] = field(default_factory=list)
@dataclass(slots=True)
class ExecutionFrame:
...
scope_id: str = "root"
lineage_id: str = "root"
@dataclass(slots=True)
class RunState:
...
state: dict[str, Any]
scopes: dict[str, RuntimeScope] = field(default_factory=dict)
lineages: dict[str, LineageState] = field(default_factory=dict)
```
`RunState.state` remains committed root scope state for compatibility. The root
scope may initially mirror `RunState.state`; native subgraphs should use
`RuntimeScope.committed_state` for child scopes instead of writing child state
into `RunState.state`.
## Explicit END and Gather
An explicit `EndNode` is best understood as a degenerate gather:
```text
END = gather exactly one active lineage and finalize output/outcome
```
That does not mean `EndNode` should secretly merge siblings. It means `EndNode`
and future `GatherNode` should eventually share completion machinery.
For native subgraphs, explicit child end nodes would make it possible for a
child graph to dispatch both:
- final child output
- child completion outcome
For v1 lineage work, no explicit end node is required. The lineage migration
should only avoid blocking that future design.
## Serialization
Scopes and lineages must be serializable with `RunState.to_dict()`. This is
required for future run stores and resume. Lineage state should avoid storing
unserializable objects. Values in `StateWrite` are expected to be
JSON-compatible because they come from validated node outputs and state writes.
If a future node output can produce non-JSON Python objects, that should be
handled by schema/runtime validation before it reaches lineage state.
## Error Handling
Malformed scope or lineage state is runtime corruption and should fail fast with
`WorkflowExecutionError`.
Examples:
- frame references an unknown scope id
- frame references an unknown lineage id
- lineage references an unknown parent lineage id
- lineage belongs to a different scope than its frame
- a lineage path is not a valid `StatePath`
- a barrier tries to commit a lineage that is still active or failed
- a barrier sees sibling writes without a mergeable reducer
## Migration Strategy
Do not rewrite the whole runtime at once.
1. Add scope and lineage models while preserving current root-state behavior.
2. Add ordered state write records to `StatePatch`.
3. Route state reads through lineage helpers.
4. Buffer non-root frame writes in lineage state.
5. Move concurrent foreach item overlays onto lineages.
6. Remove foreach-specific overlay coupling from node execution.
7. Document native subgraphs as depending on scopes plus lineage.
This gives the project a stable primitive before adding native subgraphs.
## Out of Scope
- Native subgraph implementation.
- Fork/gather implementation.
- Persistent run store.
- Copy-on-write state view optimization.
- Recursive workflow detection.
- Multiple subgraph outcome schemas.
## Open Design Questions
- Should serial foreach item frames also get item lineages immediately, or only
concurrent foreach item frames in the first migration?
- Should v1 implement non-root lineage commits into parent lineage, or can v1
only commit to root scope through foreach barriers?
- Should `TraceEntry` gain explicit `scope_id` and `lineage_id` fields, or
should those ids stay inspectable through frames and run state?
- Should `LineageState` store all ordered `StateWrite` records indefinitely, or
compact same-lineage writes by path while preserving enough incoming values for
future gather replay?
## Recommendation
Implement lineage before native subgraphs, and include runtime scopes in the
design now even if root scope is the only implemented scope in the first code
slice.
Start with concurrent foreach because it already has the same semantics in a
localized form. Once foreach no longer owns custom overlay logic, native
subgraph design gets much cleaner: child workflow state becomes another scope
with its own lineage tree instead of another special case in `RunState`.
docs/superpowers/specs/2026-05-24-native-subgraphs-design.md
+403
View File
@@ -0,0 +1,403 @@
# Native Subgraphs Design
Status: proposed
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
`wf_authoring.subgraph_node` and `async_subgraph_node` helpers are useful
compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`.
This design defines the core runtime shape before implementation.
## Goals
- Add a first-class core step for running a child workflow inside a parent
workflow.
- Preserve child trace information in a way that can be inspected without
pretending child nodes are parent nodes.
- Bubble child interrupts to the parent run and resume back into the child.
- Reuse existing input/output binding, state write, reducer, and schema
validation machinery.
- Keep saved workflow / deployment resolution outside `wf_core`.
- Leave room for future fork/gather and saved-workflow-as-node execution.
## Non-Goals
- Do not make arbitrary graph convergence or fork/gather in this pass.
- Do not dynamically load saved artifacts inside `wf_core`.
- Do not support multiple simultaneous child workflow activations from one
subgraph node until the frame identity model is explicit.
- Do not hide async behavior behind sync helpers or `asyncio.run()`.
- Do not expose child internal state as parent state except through explicit
output bindings.
## Current Wrapper Problem
The wrapper helpers convert a child workflow into a `NodeSpec` by calling
`execute_workflow` or `execute_workflow_async` from inside a node handler. That
means:
- the parent trace sees one node call
- the child trace is not embedded in the parent run state
- child interrupts cannot bubble cleanly into the parent
- resume cannot re-enter the child workflow
- child workflow identity/version is not part of the core graph
That is acceptable as a temporary compatibility path, but it is not native
subgraph execution.
## Model Shape
Add a core step model:
```python
class SubgraphNode(BaseModel):
id: str
type: Literal["subgraph"]
workflow: WorkflowRef
input: list[InputBinding] = Field(default_factory=list)
output: list[OutputBinding] = Field(default_factory=list)
outcomes: list[str] = Field(default_factory=lambda: ["ok"])
```
`WorkflowRef` should be structural, not a dotted string parser:
```python
class WorkflowRef(BaseModel):
source: str | None = None
artifact_id: str | None = None
version: int | None = None
inline_name: str | None = None
```
The exact reference model can be smaller in v1, but it must not derive meaning
from formatted display names. Higher layers may resolve saved artifacts,
deployments, or local builders into an executable child workflow before the
core runtime starts.
`SubgraphNode.workflow` identifies the child workflow. It does not carry Python
handlers. Handler registries remain runtime dependencies, not graph schema.
## Runtime Dependencies
`wf_core` should execute only already-resolved child workflows. The platform or
authoring layer should prepare a runtime dependency object such as:
```python
SubgraphRuntime(
workflows: Mapping[WorkflowRef, Workflow],
registries: Mapping[WorkflowRef, Mapping[str, NodeHandler]],
reducers: Mapping[WorkflowRef, Mapping[str, ReducerDefinition]],
)
```
The exact type can evolve, but the boundary matters:
- `wf_core` owns execution semantics.
- `wf_artifacts` owns saved workflow artifact models.
- `wf_mcp` / platform layers own source binding, auth, deployment resolution,
and capability availability checks.
## Frame Model
A subgraph step creates a child frame tree owned by the parent subgraph frame.
The parent frame blocks until the child workflow completes, interrupts, or
fails.
Recommended frame metadata:
```python
SubgraphFrameMetadata(
parent_step_id: str,
workflow_ref: WorkflowRef,
child_root_frame_id: str,
child_run_id: str | None = None,
)
```
The child root frame should have:
- `kind="subgraph_root"` or another typed kind
- `parent_frame_id` set to the parent subgraph frame
- `node_id` set to the child workflow start node
- metadata identifying the child workflow
Child frames created by foreach inside the child workflow remain descendants of
the child root, not siblings of the parent graph.
Frame ids should be centrally constructed. The display format may be stringy for
now, but logic should not parse frame ids for workflow semantics.
## Child State
A child workflow has its own input, state, output, frames, ready queue, and trace
semantics.
For v1, the parent `RunState` can store child runtime data in typed subgraph
metadata rather than a fully nested `RunState` object. However, the design
should preserve this invariant:
> Child workflow state is not parent state.
Parent state changes only happen when the subgraph node completes and applies
its explicit `output` bindings.
This prevents child internal keys from leaking into the parent and keeps reducer
behavior local to the parent output boundary.
## Input and Output Mapping
Subgraph input uses the same `InputBinding` model as `NodeUse` and
`InterruptNode.request`:
- read from parent `input`, `state`, or `context`
- build the child workflow input payload
- validate against child workflow `input_schema`
Subgraph output uses the same `OutputBinding` model as `NodeUse`:
- read from child workflow output
- write to parent workflow state
- validate against parent state schema
- apply parent reducers only at the parent write boundary
Child workflow internal reducers are applied only inside child execution.
## Trace Shape
Do not flatten child trace entries into parent trace as if they were parent
nodes. That loses ownership and makes frame ids misleading.
Recommended trace representation:
- Parent trace gets a `subgraph` step entry for the parent step.
- Child trace entries keep their child frame ids and child node ids.
- Each child trace entry should be inspectable through parent run state with
structural ownership fields, not a generic metadata bag.
Potential future shape:
```python
TraceEntry(
scope_id="subgraph:run_child",
lineage_id="subgraph:run_child:root",
parent_trace_id="trace:root:run_child",
frame_id="root:child_demo",
node_id="classify",
step_type="node",
...
)
```
Current `TraceEntry` has no scope, lineage, or parent-trace fields. The first
implementation can either add explicit optional fields or store child traces in
a separate typed child-trace structure and expose an inspection helper. The
design preference is structural fields or typed containers, not `metadata`
dictionaries and not overloaded `node_id` strings.
## Interrupt Bubbling
If a child frame reaches an `InterruptNode`:
1. The child frame becomes `INTERRUPTED`.
2. The parent subgraph frame remains `BLOCKED`.
3. The whole parent run status becomes `INTERRUPTED`.
4. `RunState.interrupt` points to the child interrupt, with enough route data
to resume into the child.
The parent-facing interrupt request should include:
- parent frame id
- subgraph parent step id
- child workflow reference
- child frame id
- child interrupt node id
- interrupt kind
- payload
Current `InterruptRequest` only has `id`, `frame_id`, `node_id`, `kind`,
`payload`, and `resumable`. Native subgraphs need either:
- explicit structural route fields on `InterruptRequest`, such as `scope_id`,
`lineage_id`, `parent_frame_id`, and `workflow_ref`, or
- a typed nested route object, such as `InterruptRoute`.
The preferred direction is explicit route structure. A generic metadata field
would recreate the ad hoc frame metadata problem, while string parsing is
exactly what the project has been moving away from.
## Resume Semantics
Resume should target the interrupted child frame, not the parent subgraph step.
On resume:
1. Validate that the outstanding interrupt belongs to a live child frame.
2. Apply the child interrupt `resume` bindings to child state.
3. Advance the child frame through its resume outcome.
4. Put the child frame at the front of the ready queue.
5. Continue scheduling.
The parent subgraph frame wakes only when the child workflow reaches a terminal
workflow output state.
This mirrors the current scheduler rule: ancestors blocked on child work do not
become runnable until the child boundary is actually done.
## Completion Semantics
When the child workflow completes:
1. Validate child workflow output against child output schema.
2. Apply the subgraph step `output` bindings from child output into parent
state.
3. Record a parent `subgraph` trace entry with committed parent state changes.
4. Advance the parent subgraph frame through outcome `ok`.
For v1, a child workflow completion maps to one parent outcome: `ok`.
Later, saved workflow artifacts may declare multiple outcomes, but core
`Workflow.output_schema` is currently one output shape. Outcome-per-child-graph
needs a separate design if we want a subgraph to behave exactly like a
multi-outcome node.
## Failure Semantics
Runtime failure inside a child workflow fails the parent run unless future
policy explicitly handles child failures.
Do not turn child runtime failures into normal graph outcomes by default.
Normal outcomes are graph control flow; runtime failures are execution failures.
If a child node returns an `error` outcome and the child graph routes it, that is
ordinary child workflow behavior. If the child graph reaches a runtime error,
that is a run failure.
## Validation
`validate_workflow` should add subgraph checks:
- subgraph step has a resolvable child workflow reference in the runtime
environment or validation context
- subgraph input bindings target child input paths
- subgraph output bindings source child output paths and target parent state
paths
- `outcomes` are declared and outgoing edges match them
- native subgraphs cannot be recursive unless explicit cycle detection exists
- child workflow structural validation runs before parent execution
Pure model validation should still avoid executing or resolving external
artifacts. Runtime/deployment validation can perform stronger checks with a
resolved dependency set.
## Authoring Layer
`wf_authoring` should expose native subgraph use separately from wrapper-node
composition.
Possible API:
```python
child = parent.subgraph(
workflow=child_builder.compile(),
id="run_child",
input=[input_from(state_path("request"), "request")],
output=[output_to("summary", state_path("child_summary"))],
)
parent.connect(child, "ok", END)
```
For saved artifacts:
```python
child = parent.subgraph_ref(
workflow=WorkflowCapabilityRef(artifact_id="demo_child", version=1),
...
)
```
`subgraph_node` and `async_subgraph_node` should remain compatibility helpers
until native subgraphs cover the same use cases. They should keep warning in
docs that they are wrapper nodes.
## MCP and Artifact Layer
Saved workflows should be reusable through the same native subgraph boundary,
but `wf_core` should not know how to load them.
The platform layer should:
- resolve workflow artifact refs to concrete workflows
- resolve capability/source bindings for that artifact
- provide node registries and reducers for the child workflow
- validate dependency availability before run
- expose clear diagnostics when a child workflow is unrunnable
This keeps auth, source availability, deployment binding, and MCP account
selection out of `wf_core`.
## Implementation Slices
### Slice 1: Non-Interrupting Inline Subgraph
- Add `SubgraphNode` to the core `Step` union.
- Add minimal `WorkflowRef` / inline child workflow dependency resolution.
- Execute child workflow to completion through child frames.
- Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code.
- Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent.
### Slice 2: Interrupt Bubbling and Resume
- Extend `InterruptRequest` with explicit route structure.
- Bubble child interrupts to the parent run.
- Resume into the child frame.
- Tests: child interrupt pauses parent, resume continues child, parent completes,
wrong resume target fails clearly.
### Slice 3: Saved Workflow References
- Add platform-level resolution for saved workflow artifacts.
- Validate dependencies and source bindings before execution.
- Tests: saved child workflow runs through a deployment binding, missing child
artifact reports an unrunnable dependency.
### Slice 4: Outcome and Policy Expansion
- Decide whether subgraphs can expose multiple outcomes.
- Decide child failure handling policy, if any.
- Keep default behavior strict until the use case is clear.
## Risks
- Trace shape can become confusing if child entries are flattened too early.
- Interrupt resume can become string-parsing-heavy if `InterruptRequest` is not
extended structurally.
- Storing nested run state directly may bloat persisted runs unless inspection
APIs paginate trace/state detail.
- Recursive saved workflows need explicit cycle detection.
- Multiple child workflow dependency registries can make runtime dependencies
complex; keep the boundary typed early.
## Open Questions
- Should child runtime state be stored as a nested `RunState`, or as typed child
frame metadata plus shared parent `RunState.frames`?
- Should `TraceEntry` gain explicit `scope_id`, `lineage_id`, and parent-trace
fields, or should child traces live in a separate inspectable structure?
- Is v1 allowed to reference only inline/compiled child workflows, or should it
immediately accept artifact refs resolved by the platform?
- Should subgraph completion always emit `ok` initially, or should child
workflow artifacts declare outcomes before native subgraphs ship?
## Recommendation
Start with Slice 1 as a non-interrupting inline subgraph. It gives us native
trace/frame semantics without taking on the hardest resume problem immediately.
Do not delete the wrapper-node helpers yet; use them as compatibility and
examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but
break at exactly the moment users need persistence and resume.
src/wf_core/run_state.py
+18
View File
@@ -4,6 +4,9 @@ from dataclasses import asdict, dataclass, field
from enum import StrEnum
from typing import Any
from wf_core.models.reducers import ReducerRef
from wf_core.paths import StatePath
class RunStatus(StrEnum):
PENDING = "pending"
@@ -22,6 +25,21 @@ class FrameStatus(StrEnum):
INTERRUPTED = "interrupted"
@dataclass(slots=True)
class StateWrite:
"""One reducer-aware state write.
`incoming_value` is the value contributed by the node or lineage and is the
value barriers/gathers must replay. `visible_value` is what later steps in
the same lineage should read after reducer application.
"""
path: StatePath
incoming_value: Any
visible_value: Any
reducer: ReducerRef
@dataclass(slots=True)
class ExecutionFrame:
id: str
src/wf_core/runtime/foreach_state.py
+1 -1
View File
@@ -260,7 +260,7 @@ class ForeachBarrierState:
f"foreach item result for index {index!r} belongs to frame "
f"{existing.frame_id!r}, got {frame_id!r}"
)
existing.patch.changes.update(patch.changes)
existing.patch.extend(patch)
def add_failure(self, *, error: ItemErrorRecord) -> None:
"""Buffer one handled item failure for the foreach barrier.
src/wf_core/runtime/ops/state.py
+73 -8
View File
@@ -19,6 +19,7 @@ from wf_core.paths import (
set_nested_value,
split_graph_path,
)
from wf_core.run_state import StateWrite
from wf_core.runtime.ops.merges import (
ReducerDefinition,
apply_reducer,
@@ -33,19 +34,52 @@ _MISSING = object()
class StatePatch:
"""Validated state writes produced by one step before commit.
`changes` is the public trace-facing view: the incoming values keyed by
state path. `_prepared_writes` and `_staged_state` are the executor internals
needed to commit reducer-aware values atomically without recomputing the
patch.
`changes` is the public trace-facing incoming-value view. `writes` preserves
the reducer-aware records needed by lineage overlays and future gathers:
barriers replay `incoming_value`, while same-lineage reads use
`visible_value`.
"""
changes: dict[str, Any] = dataclass_field(default_factory=dict)
writes: list[StateWrite] = dataclass_field(default_factory=list)
_prepared_writes: dict[StatePath, tuple[list[str], Any]] = dataclass_field(
default_factory=dict,
repr=False,
)
_staged_state: dict[str, Any] = dataclass_field(default_factory=dict, repr=False)
def __post_init__(self) -> None:
"""Keep legacy `StatePatch(changes=...)` usable during migration."""
if not self.changes and self.writes:
self.changes = {
str(write.path): write.incoming_value for write in self.writes
}
if not self.writes and self.changes:
self.writes = [
StateWrite(
path=StatePath.parse(destination),
incoming_value=value,
visible_value=value,
reducer=ReducerRef(name="wf.std.replace"),
)
for destination, value in self.changes.items()
]
@property
def visible_values(self) -> dict[str, Any]:
"""Final values visible to later reads in the same lineage."""
return {str(write.path): write.visible_value for write in self.writes}
def extend(self, patch: StatePatch) -> None:
"""Append another patch from the same lineage.
Multi-step foreach item bodies accumulate several node patches before a
barrier sees them. Both the legacy trace view and the ordered reducer
write records must be preserved.
"""
self.changes.update(patch.changes)
self.writes.extend(patch.writes)
@dataclass(slots=True, frozen=True)
class _BarrierWrite:
@@ -139,6 +173,15 @@ def build_output_patch(
state_fields=state_fields,
)
prepared_patch[destination_path] = (key_path, merged_value)
writes = [
StateWrite(
path=destination_path,
incoming_value=resolved_patch[destination_path],
visible_value=merged_value,
reducer=reducer_for_state_path(destination_path, state_fields),
)
for destination_path, (_key_path, merged_value) in prepared_patch.items()
]
# Stage writes on a copy so commit-time path errors cannot partially mutate state.
staged_state = deepcopy(state)
@@ -147,6 +190,7 @@ def build_output_patch(
validate_staged_state_patch(staged_state, prepared_patch, state_fields)
return StatePatch(
changes={str(path): value for path, value in resolved_patch.items()},
writes=writes,
_prepared_writes=prepared_patch,
_staged_state=staged_state,
)
@@ -184,22 +228,32 @@ def build_barrier_patch(
prepared_patch: dict[StatePath, tuple[list[str], Any]] = {}
committed_changes: dict[str, Any] = {}
for item_patch in item_patches:
for destination, incoming_value in item_patch.changes.items():
destination_path = StatePath.parse(destination)
for write in item_patch.writes:
destination_path = write.path
key_path, merged_value = prepare_state_value(
workflow,
staged_state,
destination_path,
incoming_value,
write.incoming_value,
reducers=reducers,
state_fields=state_fields,
)
safe_set_nested_value(staged_state, key_path, merged_value)
prepared_patch[destination_path] = (key_path, merged_value)
committed_changes[destination] = merged_value
committed_changes[str(destination_path)] = merged_value
writes = [
StateWrite(
path=destination_path,
incoming_value=merged_value,
visible_value=merged_value,
reducer=reducer_for_state_path(destination_path, state_fields),
)
for destination_path, (_key_path, merged_value) in prepared_patch.items()
]
validate_staged_state_patch(staged_state, prepared_patch, state_fields)
return StatePatch(
changes=committed_changes,
writes=writes,
_prepared_writes=prepared_patch,
_staged_state=staged_state,
)
@@ -356,6 +410,17 @@ def prepare_state_value(
return key_path, merged_value
def reducer_for_state_path(
path: StatePath,
state_fields: Mapping[StatePath, StateFieldDecl],
) -> ReducerRef:
"""Return the reducer declared for one exact state path."""
declared_field = state_fields.get(path)
return (
declared_field.reducer if declared_field else ReducerRef(name="wf.std.replace")
)
def project_output(workflow: Workflow, state: dict[str, Any]) -> dict[str, Any]:
return {
key: state[key] for key in workflow.output_schema.properties if key in state
tests/core/test_atomic_state_patches.py
+66
View File
@@ -16,6 +16,8 @@ from wf_core import (
Workflow,
WorkflowExecutionError,
)
from wf_core.paths import StatePath
from wf_core.run_state import StateWrite
from wf_core.models.steps import OutputBinding
from wf_core.runtime.engine import resume_workflow
from wf_core.runtime.ops.merges import ReducerDefinition
@@ -200,6 +202,70 @@ def test_build_output_patch_does_not_mutate_until_commit() -> None:
assert state["person"]["name"] == "Ada"
def test_output_patch_records_incoming_and_visible_values() -> None:
workflow = _workflow(
fields={
"count": StateField(
type="integer",
reducer=ReducerRef(name="wf.std.add"),
)
}
)
state = {"count": 2}
patch = build_output_patch(
workflow,
[_binding("delta", "state.count")],
{"delta": 3},
state,
)
assert patch.changes["state.count"] == 3
assert patch.visible_values["state.count"] == 5
assert patch.writes[0].incoming_value == 3
assert patch.writes[0].visible_value == 5
def test_barrier_replays_incoming_values_not_lineage_visible_values() -> None:
workflow = _workflow(
fields={
"number": StateField(
type="integer",
reducer=ReducerRef(name="wf.std.add"),
)
}
)
patch = build_barrier_patch(
workflow,
[
StatePatch(
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
)
]
),
StatePatch(
writes=[
StateWrite(
path=StatePath(("number",)),
incoming_value=1,
visible_value=3,
reducer=ReducerRef(name="wf.std.add"),
)
]
),
],
{"number": 2},
)
assert patch.changes["state.number"] == 6
assert patch.visible_values["state.number"] == 6
def test_build_and_commit_patch_matches_apply_output_bindings() -> None:
workflow = _workflow(fields={"person.name": StateField(type="string")})
state_from_apply = {"person": {"name": "old"}}
tests/core/test_concurrent_foreach.py
+237
View File
@@ -170,6 +170,63 @@ def test_sync_concurrent_foreach_sibling_overlays_do_not_leak() -> None:
assert run.state["seen"] == ["a", "b"]
def test_sync_concurrent_foreach_barrier_replays_add_reducer_inputs() -> None:
workflow = _sum_items_workflow()
run = execute_workflow(
workflow,
{"items": [3, 1]},
{
"add_item": lambda payload, _ctx: {
"outcome": "ok",
"output": {"number": payload["value"]},
}
},
)
assert run.state["number"] == 6
assert run.output["number"] == 6
foreach_entries = [entry for entry in run.trace if entry.step_type == "foreach"]
assert foreach_entries[-1].state_changes["state.number"] == 6
@pytest.mark.xfail(
reason=(
"Current foreach overlays use StatePatch.changes, which stores incoming "
"reducer values; lineage StateWrite.visible_value should make this pass."
),
strict=True,
)
def test_sync_concurrent_foreach_same_item_reads_add_reducer_visible_value() -> None:
workflow = _same_item_reducer_visibility_workflow()
run = execute_workflow(
workflow,
{"items": [3]},
{
"add_item": lambda payload, _ctx: {
"outcome": "ok",
"output": {"number": payload["value"]},
},
"read_number": lambda payload, _ctx: {
"outcome": "ok",
"output": {"seen_number": payload["number"]},
},
},
)
stage_entry = next(entry for entry in run.trace if entry.node_id == "add_item")
assert stage_entry.resolved_input["value"] == 3
assert stage_entry.resolved_input["current_number"] == 2
assert stage_entry.state_changes == {}
read_entry = next(entry for entry in run.trace if entry.node_id == "read_number")
# Current limitation: foreach overlays use StatePatch.changes, which stores
# the incoming reducer value. The future lineage StateWrite model should let
# this same item read the reducer-visible value 5 instead.
assert read_entry.resolved_input["number"] == 5
assert run.state["seen_number"] == [5]
def test_sync_concurrent_foreach_rejects_sibling_replace_writes() -> None:
workflow = _same_path_replace_workflow()
@@ -186,6 +243,186 @@ def test_sync_concurrent_foreach_rejects_sibling_replace_writes() -> None:
)
def _sum_items_workflow() -> Workflow:
foreach = ForeachNode.model_validate(
{
"id": "each",
"type": "foreach",
"over": "state.items",
"as": "item",
"mode": "concurrent",
"concurrent": {"max_active": 2, "max_outstanding": 2},
}
)
return Workflow(
name="concurrent_foreach_sum",
input_schema=SchemaRef(
type="object",
properties={"items": {"type": "array"}},
),
state_schema=StateSchema.from_field_map(
{
"items": StateField(type="array"),
"number": StateField(
type="integer",
default=2,
reducer=ReducerRef(name="wf.std.add"),
),
}
),
output_schema=SchemaRef(
type="object",
properties={"number": {"type": "integer"}},
),
node_defs=[
NodeDef(
name="add_item",
input_schema=SchemaRef(
type="object",
properties={
"value": {"type": "integer"},
"current_number": {"type": "integer"},
},
required=["value", "current_number"],
),
output_schema=SchemaRef(
type="object",
properties={"number": {"type": "integer"}},
required=["number"],
),
outcomes=["ok"],
)
],
start="each",
nodes=[
foreach,
NodeUse.model_validate(
{
"id": "add_item",
"type": "node",
"node": "add_item",
"input": [
{"target": "value", "path": "context.item"},
{"target": "current_number", "path": "state.number"},
],
"output": [{"source": "number", "target": "state.number"}],
}
),
],
edges=[
Edge.model_validate({"from": "each", "outcome": "loop", "to": "add_item"}),
Edge.model_validate({"from": "add_item", "outcome": "ok", "to": END}),
Edge.model_validate({"from": "each", "outcome": "done", "to": END}),
],
)
def _same_item_reducer_visibility_workflow() -> Workflow:
foreach = ForeachNode.model_validate(
{
"id": "each",
"type": "foreach",
"over": "state.items",
"as": "item",
"mode": "concurrent",
"concurrent": {"max_active": 1, "max_outstanding": 1},
}
)
return Workflow(
name="concurrent_foreach_same_item_reducer_visibility",
input_schema=SchemaRef(
type="object",
properties={"items": {"type": "array"}},
),
state_schema=StateSchema.from_field_map(
{
"items": StateField(type="array"),
"number": StateField(
type="integer",
default=2,
reducer=ReducerRef(name="wf.std.add"),
),
"seen_number": StateField(
type="array",
reducer=ReducerRef(name="wf.std.append"),
),
}
),
output_schema=SchemaRef(
type="object",
properties={"seen_number": {"type": "array"}},
),
node_defs=[
NodeDef(
name="add_item",
input_schema=SchemaRef(
type="object",
properties={
"value": {"type": "integer"},
"current_number": {"type": "integer"},
},
required=["value", "current_number"],
),
output_schema=SchemaRef(
type="object",
properties={"number": {"type": "integer"}},
required=["number"],
),
outcomes=["ok"],
),
NodeDef(
name="read_number",
input_schema=SchemaRef(
type="object",
properties={"number": {"type": "integer"}},
required=["number"],
),
output_schema=SchemaRef(
type="object",
properties={"seen_number": {"type": "integer"}},
required=["seen_number"],
),
outcomes=["ok"],
),
],
start="each",
nodes=[
foreach,
NodeUse.model_validate(
{
"id": "add_item",
"type": "node",
"node": "add_item",
"input": [
{"target": "value", "path": "context.item"},
{"target": "current_number", "path": "state.number"},
],
"output": [{"source": "number", "target": "state.number"}],
}
),
NodeUse.model_validate(
{
"id": "read_number",
"type": "node",
"node": "read_number",
"input": [{"target": "number", "path": "state.number"}],
"output": [
{"source": "seen_number", "target": "state.seen_number"}
],
}
),
],
edges=[
Edge.model_validate({"from": "each", "outcome": "loop", "to": "add_item"}),
Edge.model_validate(
{"from": "add_item", "outcome": "ok", "to": "read_number"}
),
Edge.model_validate({"from": "read_number", "outcome": "ok", "to": END}),
Edge.model_validate({"from": "each", "outcome": "done", "to": END}),
],
)
def _workflow(
*,
state_schema: StateSchema,