lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

state write

Browse Source
This commit is contained in:
lda
2026-05-24 22:08:00 +07:00 Verified
parent 2dc00518b9
commit 39b473ea25
9 changed files with 1889 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-05-24-lineage-state-runtime-design.md
+393
View File
@@ -0,0 +1,393 @@
# Lineage State Runtime Design
Status: proposed
Lineage is the missing primitive between the scheduler frame model and future
native subgraphs / fork-gather. A frame says where execution is. A scope says
which workflow state root execution belongs to. A lineage says what pending
writes execution can see inside that scope.
The current runtime stores committed state directly on `RunState.state` and uses
foreach-specific barrier metadata to emulate item-local overlays. That worked
for concurrent foreach, but native subgraphs and future fork/gather need the
same state-visibility rule in a reusable core concept.
## Problem
`RunState` currently owns too many meanings:
- run status
- scheduler cursor
- frame set
- trace
- interrupt request
- committed workflow state
- temporary state visibility hacks through frame/barrier metadata
This creates pressure when a workflow branches:
```text
root
fork -> A
-> B
```
Branch `A` must not see branch `B` writes before an explicit merge point.
Branch `B` must not see branch `A` writes either. A future gather then decides
how to merge both branches back into a parent state view.
The same problem already exists in concurrent foreach item frames. Each item is
a sibling lineage. Current foreach solves it locally with barrier pending
patches. That solution should become a general runtime primitive.
Native subgraphs add one more missing concept: a child workflow has its own
committed state root. A lineage alone is not enough for subgraphs because a
child workflow initializes state from child input/defaults, not from the
parent's state dictionary.
## Vocabulary
```text
Scope = workflow state root and workflow identity
Frame = scheduler/control-flow position inside a scope
Lineage = pending write ownership and read overlay inside a scope
```
A frame owns execution lifecycle:
```python
ExecutionFrame(
id="root:each:0",
scope_id="root",
kind="foreach_iteration",
node_id="work",
status="pending",
parent_frame_id="root",
lineage_id="root:each:0",
)
```
A lineage owns ordered pending writes:
```python
LineageState(
id="root:each:0",
scope_id="root",
parent_id="root",
writes=[
StateWrite(
path=StatePath(("count",)),
incoming_value=3,
visible_value=5,
reducer=ReducerRef(name="wf.std.add"),
),
],
)
```
The frame, scope, and lineage ids may match in simple cases, but they are not
the same concept and runtime logic must not depend on parsing any of them.
## Core Invariants
- Every executable frame belongs to exactly one scope.
- Every executable frame points at exactly one lineage in that scope.
- The root frame points at the root scope and root lineage.
- `RunState.state` remains the committed root scope state during the migration.
- A non-root lineage stores replayable write records, not a full copied state
snapshot and not only final visible values.
- A frame reads through its lineage state view.
- A non-root frame write updates its lineage writes and does not mutate
`RunState.state`.
- A barrier/gather commits lineage write records into a parent lineage or scope
root state by replaying incoming values through reducers.
- Trace `state_changes` means committed changes only.
- Buffered lineage writes may be visible to later frames in the same lineage,
but they are not public committed state changes until a barrier commits them.
## Why Write Records Instead of Full State
Lineages should store write records, not full state snapshots.
Full snapshots make reads easy but make merges difficult. At gather time the
runtime would have to diff branch snapshots against a base snapshot to discover
what changed. That becomes fragile with nested objects, defaults, reducers,
missing fields, and JSON Schema validation.
Write-record-owned lineages make the merge boundary explicit:
```text
lineage A wrote incoming value X to state.person.name
lineage B wrote incoming value Y to state.person.email
```
The barrier can then validate conflicts and apply reducers deterministically.
Runtime may cache or materialize state views for performance later. The source
of truth should still be ordered lineage write records.
## Incoming Values vs Visible Values
Existing `StatePatch.changes` is trace-facing. For normal node patches it stores
the incoming values from node output bindings. That is useful for trace and for
barrier replay. It is not enough by itself for same-lineage reads when reducers
are involved.
Example:
```text
committed state.count = 2
node output delta = 3
reducer = add
visible state.count after write = 5
```
Trace should be able to say the node emitted `3`. A later node in the same
lineage must see `5`. A future gather must still replay `3` into the parent,
not replay the visible value `5`.
So the runtime needs ordered writes:
```python
StateWrite(
path=StatePath(("count",)),
incoming_value=3, # trace and barrier replay value
visible_value=5, # same-lineage read value
reducer=ReducerRef(name="wf.std.add"),
)
```
`StatePatch` should preserve ordered `StateWrite` records. Convenience views can
derive incoming trace changes and same-lineage visible values, but lineage must
not discard incoming values.
This matters for fork/gather:
```text
root state.number = 2
fork A and B
A applies add(3), visible in A is 5
B applies add(1), visible in B is 3
gather must commit 2 + 3 + 1 = 6
```
If lineage stored only visible values, gather would incorrectly replay `5` and
`3` instead of `3` and `1`.
## State View
The visible state for a frame is:
```text
committed scope root state
+ ancestor lineage visible values
+ current lineage visible values
```
For v1, materializing this with a deep copy is acceptable because correctness is
more important than performance. The implementation should keep this behind one
helper so copy-on-write or structural sharing can replace it later.
```python
def lineage_state_view(run: RunState, scope_id: str, lineage_id: str) -> dict[str, Any]:
...
```
## Relationship to Concurrent Foreach
Concurrent foreach is the first real user of lineage.
Current behavior:
- parent foreach frame stores pending item patches in barrier metadata
- `state_view_for_frame` knows about foreach metadata
- node finalization knows about foreach item ownership
Target behavior:
- each concurrent item frame gets its own lineage
- `state_view_for_frame` delegates to lineage helpers
- node finalization only decides root commit vs lineage buffer
- foreach barrier records completed lineage ids
- barrier commit replays completed lineage write records into the existing
reducer/conflict validation path
This removes foreach-specific state overlay logic from node execution.
## Relationship to Native Subgraphs
Native subgraphs should build on scopes plus lineage.
A subgraph creates a child scope because the child workflow has a separate
committed state root initialized from child workflow input and child state
defaults. The subgraph root frame runs in that child scope and receives the
child scope's root lineage. Child workflow internal frames may create further
child lineages for foreach or future fork/gather.
Parent state changes only happen when the subgraph boundary completes and
applies explicit output bindings. Child internal writes remain inside the child
scope until that boundary.
This prevents child workflow internals from leaking into parent state and gives
interrupt/resume a stable state-visibility boundary.
## Relationship to Future Fork/Gather
Fork creates sibling lineages within the same scope. Gather consumes declared
lineage ids and commits their writes into a parent lineage or scope root.
Conceptually:
```text
fork parent lineage P
-> child lineage A
-> child lineage B
gather A+B into P
```
The gather operation should reuse the same conflict validation and reducer rules
as concurrent foreach barrier commits:
- same-path sibling writes require a mergeable reducer
- ancestor/descendant sibling writes are rejected until a deeper merge policy is
explicitly designed
- reducer application order is deterministic
## RunState Shape
Initial additive shape:
```python
@dataclass(slots=True)
class StateWrite:
path: StatePath
incoming_value: Any
visible_value: Any
reducer: ReducerRef
@dataclass(slots=True)
class RuntimeScope:
id: str
workflow_name: str
committed_state: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class LineageState:
id: str
scope_id: str
parent_id: str | None = None
writes: list[StateWrite] = field(default_factory=list)
@dataclass(slots=True)
class ExecutionFrame:
...
scope_id: str = "root"
lineage_id: str = "root"
@dataclass(slots=True)
class RunState:
...
state: dict[str, Any]
scopes: dict[str, RuntimeScope] = field(default_factory=dict)
lineages: dict[str, LineageState] = field(default_factory=dict)
```
`RunState.state` remains committed root scope state for compatibility. The root
scope may initially mirror `RunState.state`; native subgraphs should use
`RuntimeScope.committed_state` for child scopes instead of writing child state
into `RunState.state`.
## Explicit END and Gather
An explicit `EndNode` is best understood as a degenerate gather:
```text
END = gather exactly one active lineage and finalize output/outcome
```
That does not mean `EndNode` should secretly merge siblings. It means `EndNode`
and future `GatherNode` should eventually share completion machinery.
For native subgraphs, explicit child end nodes would make it possible for a
child graph to dispatch both:
- final child output
- child completion outcome
For v1 lineage work, no explicit end node is required. The lineage migration
should only avoid blocking that future design.
## Serialization
Scopes and lineages must be serializable with `RunState.to_dict()`. This is
required for future run stores and resume. Lineage state should avoid storing
unserializable objects. Values in `StateWrite` are expected to be
JSON-compatible because they come from validated node outputs and state writes.
If a future node output can produce non-JSON Python objects, that should be
handled by schema/runtime validation before it reaches lineage state.
## Error Handling
Malformed scope or lineage state is runtime corruption and should fail fast with
`WorkflowExecutionError`.
Examples:
- frame references an unknown scope id
- frame references an unknown lineage id
- lineage references an unknown parent lineage id
- lineage belongs to a different scope than its frame
- a lineage path is not a valid `StatePath`
- a barrier tries to commit a lineage that is still active or failed
- a barrier sees sibling writes without a mergeable reducer
## Migration Strategy
Do not rewrite the whole runtime at once.
1. Add scope and lineage models while preserving current root-state behavior.
2. Add ordered state write records to `StatePatch`.
3. Route state reads through lineage helpers.
4. Buffer non-root frame writes in lineage state.
5. Move concurrent foreach item overlays onto lineages.
6. Remove foreach-specific overlay coupling from node execution.
7. Document native subgraphs as depending on scopes plus lineage.
This gives the project a stable primitive before adding native subgraphs.
## Out of Scope
- Native subgraph implementation.
- Fork/gather implementation.
- Persistent run store.
- Copy-on-write state view optimization.
- Recursive workflow detection.
- Multiple subgraph outcome schemas.
## Open Design Questions
- Should serial foreach item frames also get item lineages immediately, or only
concurrent foreach item frames in the first migration?
- Should v1 implement non-root lineage commits into parent lineage, or can v1
only commit to root scope through foreach barriers?
- Should `TraceEntry` gain explicit `scope_id` and `lineage_id` fields, or
should those ids stay inspectable through frames and run state?
- Should `LineageState` store all ordered `StateWrite` records indefinitely, or
compact same-lineage writes by path while preserving enough incoming values for
future gather replay?
## Recommendation
Implement lineage before native subgraphs, and include runtime scopes in the
design now even if root scope is the only implemented scope in the first code
slice.
Start with concurrent foreach because it already has the same semantics in a
localized form. Once foreach no longer owns custom overlay logic, native
subgraph design gets much cleaner: child workflow state becomes another scope
with its own lineage tree instead of another special case in `RunState`.
docs/superpowers/specs/2026-05-24-native-subgraphs-design.md
+403
View File
@@ -0,0 +1,403 @@
# Native Subgraphs Design
Status: proposed
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
`wf_authoring.subgraph_node` and `async_subgraph_node` helpers are useful
compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`.
This design defines the core runtime shape before implementation.
## Goals
- Add a first-class core step for running a child workflow inside a parent
workflow.
- Preserve child trace information in a way that can be inspected without
pretending child nodes are parent nodes.
- Bubble child interrupts to the parent run and resume back into the child.
- Reuse existing input/output binding, state write, reducer, and schema
validation machinery.
- Keep saved workflow / deployment resolution outside `wf_core`.
- Leave room for future fork/gather and saved-workflow-as-node execution.
## Non-Goals
- Do not make arbitrary graph convergence or fork/gather in this pass.
- Do not dynamically load saved artifacts inside `wf_core`.
- Do not support multiple simultaneous child workflow activations from one
subgraph node until the frame identity model is explicit.
- Do not hide async behavior behind sync helpers or `asyncio.run()`.
- Do not expose child internal state as parent state except through explicit
output bindings.
## Current Wrapper Problem
The wrapper helpers convert a child workflow into a `NodeSpec` by calling
`execute_workflow` or `execute_workflow_async` from inside a node handler. That
means:
- the parent trace sees one node call
- the child trace is not embedded in the parent run state
- child interrupts cannot bubble cleanly into the parent
- resume cannot re-enter the child workflow
- child workflow identity/version is not part of the core graph
That is acceptable as a temporary compatibility path, but it is not native
subgraph execution.
## Model Shape
Add a core step model:
```python
class SubgraphNode(BaseModel):
id: str
type: Literal["subgraph"]
workflow: WorkflowRef
input: list[InputBinding] = Field(default_factory=list)
output: list[OutputBinding] = Field(default_factory=list)
outcomes: list[str] = Field(default_factory=lambda: ["ok"])
```
`WorkflowRef` should be structural, not a dotted string parser:
```python
class WorkflowRef(BaseModel):
source: str | None = None
artifact_id: str | None = None
version: int | None = None
inline_name: str | None = None
```
The exact reference model can be smaller in v1, but it must not derive meaning
from formatted display names. Higher layers may resolve saved artifacts,
deployments, or local builders into an executable child workflow before the
core runtime starts.
`SubgraphNode.workflow` identifies the child workflow. It does not carry Python
handlers. Handler registries remain runtime dependencies, not graph schema.
## Runtime Dependencies
`wf_core` should execute only already-resolved child workflows. The platform or
authoring layer should prepare a runtime dependency object such as:
```python
SubgraphRuntime(
workflows: Mapping[WorkflowRef, Workflow],
registries: Mapping[WorkflowRef, Mapping[str, NodeHandler]],
reducers: Mapping[WorkflowRef, Mapping[str, ReducerDefinition]],
)
```
The exact type can evolve, but the boundary matters:
- `wf_core` owns execution semantics.
- `wf_artifacts` owns saved workflow artifact models.
- `wf_mcp` / platform layers own source binding, auth, deployment resolution,
and capability availability checks.
## Frame Model
A subgraph step creates a child frame tree owned by the parent subgraph frame.
The parent frame blocks until the child workflow completes, interrupts, or
fails.
Recommended frame metadata:
```python
SubgraphFrameMetadata(
parent_step_id: str,
workflow_ref: WorkflowRef,
child_root_frame_id: str,
child_run_id: str | None = None,
)
```
The child root frame should have:
- `kind="subgraph_root"` or another typed kind
- `parent_frame_id` set to the parent subgraph frame
- `node_id` set to the child workflow start node
- metadata identifying the child workflow
Child frames created by foreach inside the child workflow remain descendants of
the child root, not siblings of the parent graph.
Frame ids should be centrally constructed. The display format may be stringy for
now, but logic should not parse frame ids for workflow semantics.
## Child State
A child workflow has its own input, state, output, frames, ready queue, and trace
semantics.
For v1, the parent `RunState` can store child runtime data in typed subgraph
metadata rather than a fully nested `RunState` object. However, the design
should preserve this invariant:
> Child workflow state is not parent state.
Parent state changes only happen when the subgraph node completes and applies
its explicit `output` bindings.
This prevents child internal keys from leaking into the parent and keeps reducer
behavior local to the parent output boundary.
## Input and Output Mapping
Subgraph input uses the same `InputBinding` model as `NodeUse` and
`InterruptNode.request`:
- read from parent `input`, `state`, or `context`
- build the child workflow input payload
- validate against child workflow `input_schema`
Subgraph output uses the same `OutputBinding` model as `NodeUse`:
- read from child workflow output
- write to parent workflow state
- validate against parent state schema
- apply parent reducers only at the parent write boundary
Child workflow internal reducers are applied only inside child execution.
## Trace Shape
Do not flatten child trace entries into parent trace as if they were parent
nodes. That loses ownership and makes frame ids misleading.
Recommended trace representation:
- Parent trace gets a `subgraph` step entry for the parent step.
- Child trace entries keep their child frame ids and child node ids.
- Each child trace entry should be inspectable through parent run state with
structural ownership fields, not a generic metadata bag.
Potential future shape:
```python
TraceEntry(
scope_id="subgraph:run_child",
lineage_id="subgraph:run_child:root",
parent_trace_id="trace:root:run_child",
frame_id="root:child_demo",
node_id="classify",
step_type="node",
...
)
```
Current `TraceEntry` has no scope, lineage, or parent-trace fields. The first
implementation can either add explicit optional fields or store child traces in
a separate typed child-trace structure and expose an inspection helper. The
design preference is structural fields or typed containers, not `metadata`
dictionaries and not overloaded `node_id` strings.
## Interrupt Bubbling
If a child frame reaches an `InterruptNode`:
1. The child frame becomes `INTERRUPTED`.
2. The parent subgraph frame remains `BLOCKED`.
3. The whole parent run status becomes `INTERRUPTED`.
4. `RunState.interrupt` points to the child interrupt, with enough route data
to resume into the child.
The parent-facing interrupt request should include:
- parent frame id
- subgraph parent step id
- child workflow reference
- child frame id
- child interrupt node id
- interrupt kind
- payload
Current `InterruptRequest` only has `id`, `frame_id`, `node_id`, `kind`,
`payload`, and `resumable`. Native subgraphs need either:
- explicit structural route fields on `InterruptRequest`, such as `scope_id`,
`lineage_id`, `parent_frame_id`, and `workflow_ref`, or
- a typed nested route object, such as `InterruptRoute`.
The preferred direction is explicit route structure. A generic metadata field
would recreate the ad hoc frame metadata problem, while string parsing is
exactly what the project has been moving away from.
## Resume Semantics
Resume should target the interrupted child frame, not the parent subgraph step.
On resume:
1. Validate that the outstanding interrupt belongs to a live child frame.
2. Apply the child interrupt `resume` bindings to child state.
3. Advance the child frame through its resume outcome.
4. Put the child frame at the front of the ready queue.
5. Continue scheduling.
The parent subgraph frame wakes only when the child workflow reaches a terminal
workflow output state.
This mirrors the current scheduler rule: ancestors blocked on child work do not
become runnable until the child boundary is actually done.
## Completion Semantics
When the child workflow completes:
1. Validate child workflow output against child output schema.
2. Apply the subgraph step `output` bindings from child output into parent
state.
3. Record a parent `subgraph` trace entry with committed parent state changes.
4. Advance the parent subgraph frame through outcome `ok`.
For v1, a child workflow completion maps to one parent outcome: `ok`.
Later, saved workflow artifacts may declare multiple outcomes, but core
`Workflow.output_schema` is currently one output shape. Outcome-per-child-graph
needs a separate design if we want a subgraph to behave exactly like a
multi-outcome node.
## Failure Semantics
Runtime failure inside a child workflow fails the parent run unless future
policy explicitly handles child failures.
Do not turn child runtime failures into normal graph outcomes by default.
Normal outcomes are graph control flow; runtime failures are execution failures.
If a child node returns an `error` outcome and the child graph routes it, that is
ordinary child workflow behavior. If the child graph reaches a runtime error,
that is a run failure.
## Validation
`validate_workflow` should add subgraph checks:
- subgraph step has a resolvable child workflow reference in the runtime
environment or validation context
- subgraph input bindings target child input paths
- subgraph output bindings source child output paths and target parent state
paths
- `outcomes` are declared and outgoing edges match them
- native subgraphs cannot be recursive unless explicit cycle detection exists
- child workflow structural validation runs before parent execution
Pure model validation should still avoid executing or resolving external
artifacts. Runtime/deployment validation can perform stronger checks with a
resolved dependency set.
## Authoring Layer
`wf_authoring` should expose native subgraph use separately from wrapper-node
composition.
Possible API:
```python
child = parent.subgraph(
workflow=child_builder.compile(),
id="run_child",
input=[input_from(state_path("request"), "request")],
output=[output_to("summary", state_path("child_summary"))],
)
parent.connect(child, "ok", END)
```
For saved artifacts:
```python
child = parent.subgraph_ref(
workflow=WorkflowCapabilityRef(artifact_id="demo_child", version=1),
...
)
```
`subgraph_node` and `async_subgraph_node` should remain compatibility helpers
until native subgraphs cover the same use cases. They should keep warning in
docs that they are wrapper nodes.
## MCP and Artifact Layer
Saved workflows should be reusable through the same native subgraph boundary,
but `wf_core` should not know how to load them.
The platform layer should:
- resolve workflow artifact refs to concrete workflows
- resolve capability/source bindings for that artifact
- provide node registries and reducers for the child workflow
- validate dependency availability before run
- expose clear diagnostics when a child workflow is unrunnable
This keeps auth, source availability, deployment binding, and MCP account
selection out of `wf_core`.
## Implementation Slices
### Slice 1: Non-Interrupting Inline Subgraph
- Add `SubgraphNode` to the core `Step` union.
- Add minimal `WorkflowRef` / inline child workflow dependency resolution.
- Execute child workflow to completion through child frames.
- Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code.
- Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent.
### Slice 2: Interrupt Bubbling and Resume
- Extend `InterruptRequest` with explicit route structure.
- Bubble child interrupts to the parent run.
- Resume into the child frame.
- Tests: child interrupt pauses parent, resume continues child, parent completes,
wrong resume target fails clearly.
### Slice 3: Saved Workflow References
- Add platform-level resolution for saved workflow artifacts.
- Validate dependencies and source bindings before execution.
- Tests: saved child workflow runs through a deployment binding, missing child
artifact reports an unrunnable dependency.
### Slice 4: Outcome and Policy Expansion
- Decide whether subgraphs can expose multiple outcomes.
- Decide child failure handling policy, if any.
- Keep default behavior strict until the use case is clear.
## Risks
- Trace shape can become confusing if child entries are flattened too early.
- Interrupt resume can become string-parsing-heavy if `InterruptRequest` is not
extended structurally.
- Storing nested run state directly may bloat persisted runs unless inspection
APIs paginate trace/state detail.
- Recursive saved workflows need explicit cycle detection.
- Multiple child workflow dependency registries can make runtime dependencies
complex; keep the boundary typed early.
## Open Questions
- Should child runtime state be stored as a nested `RunState`, or as typed child
frame metadata plus shared parent `RunState.frames`?
- Should `TraceEntry` gain explicit `scope_id`, `lineage_id`, and parent-trace
fields, or should child traces live in a separate inspectable structure?
- Is v1 allowed to reference only inline/compiled child workflows, or should it
immediately accept artifact refs resolved by the platform?
- Should subgraph completion always emit `ok` initially, or should child
workflow artifacts declare outcomes before native subgraphs ship?
## Recommendation
Start with Slice 1 as a non-interrupting inline subgraph. It gives us native
trace/frame semantics without taking on the hardest resume problem immediately.
Do not delete the wrapper-node helpers yet; use them as compatibility and
examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but
break at exactly the moment users need persistence and resume.