docs + code review
This commit is contained in:
+22
-17
@@ -45,14 +45,17 @@ implementation state.
|
|||||||
[ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md).
|
[ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md).
|
||||||
- Native subgraph design spec:
|
- Native subgraph design spec:
|
||||||
[2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md).
|
[2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md).
|
||||||
- **Native subgraphs / graph-as-node**: next major runtime feature. A
|
- **Native subgraphs / graph-as-node**: next major runtime feature. The
|
||||||
first-class `SubgraphNode` placeholder exists and validates parent-side
|
scaffolding slice is complete: core has `SubgraphNode`, structural
|
||||||
bindings/outcomes. Core workflows now also declare workflow-level outcomes
|
`WorkflowRef`, workflow-level outcomes plus explicit `EndNode` termination,
|
||||||
and can terminate through explicit `EndNode` steps. Runtime subgraph execution
|
authoring helpers (`subgraph_ref` / `WorkflowBuilder.subgraph`), and artifact
|
||||||
still needs child run/frame identity, child trace preservation, interrupt
|
reference conversion helpers. Runtime subgraph execution is still absent.
|
||||||
bubbling, and resume back into the child workflow. Wrapper helpers currently
|
The next slice is non-interrupting child execution: resolve a prepared child
|
||||||
run child workflows as ordinary nodes; true graph-as-node behavior belongs
|
workflow, create a child scope/lineage, preserve child trace, map child
|
||||||
here.
|
output back through the subgraph boundary, and route by the child's terminal
|
||||||
|
outcome. Interrupt bubbling/resume and saved/deployed child resolution follow
|
||||||
|
after that. Wrapper helpers currently run child workflows as ordinary nodes;
|
||||||
|
true graph-as-node behavior belongs here.
|
||||||
- **Concurrent foreach**: implemented in core with explicit scheduling,
|
- **Concurrent foreach**: implemented in core with explicit scheduling,
|
||||||
reducer/merge semantics, item error policy, async handler batching, and
|
reducer/merge semantics, item error policy, async handler batching, and
|
||||||
quiescent interrupt behavior. Remaining work is polish and future reuse of
|
quiescent interrupt behavior. Remaining work is polish and future reuse of
|
||||||
@@ -77,13 +80,14 @@ implementation state.
|
|||||||
- **Dashboard/source controls**: future UI should consume the same source
|
- **Dashboard/source controls**: future UI should consume the same source
|
||||||
inventory and deployment metadata instead of reverse-engineering MCP tools.
|
inventory and deployment metadata instead of reverse-engineering MCP tools.
|
||||||
|
|
||||||
Frame stress points to solve before either feature:
|
Frame stress points remaining for native subgraphs and future fork/gather:
|
||||||
|
|
||||||
- `RunState.current_frame_id` currently models the selected execution cursor.
|
- `RunState.current_frame_id` remains the selected execution cursor even though
|
||||||
Concurrent foreach needs multiple runnable child frames.
|
concurrent foreach now schedules multiple child frames. Native subgraphs
|
||||||
- `ExecutionFrame.metadata` currently carries ad hoc foreach data. Subgraphs and
|
must preserve that cursor model while owning a nested child execution scope.
|
||||||
concurrent foreach should get typed frame payloads or strongly bounded helper
|
- `ExecutionFrame.metadata` has typed foreach access paths, but subgraphs still
|
||||||
accessors before metadata grows more meanings.
|
need typed child-workflow ownership and completion metadata rather than new
|
||||||
|
ad hoc dictionary fields.
|
||||||
- Subgraph frames need child workflow identity/version/deployment binding, not
|
- Subgraph frames need child workflow identity/version/deployment binding, not
|
||||||
just a generic metadata dictionary.
|
just a generic metadata dictionary.
|
||||||
- `RunState.current_node_id` duplicates the current frame's node id for
|
- `RunState.current_node_id` duplicates the current frame's node id for
|
||||||
@@ -94,6 +98,7 @@ Frame stress points to solve before either feature:
|
|||||||
|
|
||||||
The MCP workflow authoring path is now usable enough for real testing. The next
|
The MCP workflow authoring path is now usable enough for real testing. The next
|
||||||
bottleneck is runtime/platform correctness: resumable child execution,
|
bottleneck is runtime/platform correctness: resumable child execution,
|
||||||
concurrent scheduling, persistent run history, and protocol-native progress
|
native subgraph execution, persistent run history, and protocol-native
|
||||||
reporting. Those pieces should come before adding more high-level authoring
|
progress reporting. Concurrent foreach supplies scheduler/lineage precedent;
|
||||||
sugar.
|
native child graphs are now the missing runtime boundary. Those pieces should
|
||||||
|
come before adding more high-level authoring sugar.
|
||||||
|
|||||||
@@ -39,9 +39,12 @@ explicit scope/lineage commit target, feasible once native subgraph completion
|
|||||||
can declare whether child writes commit to child scope, parent lineage, or only
|
can declare whether child writes commit to child scope, parent lineage, or only
|
||||||
through boundary output bindings.
|
through boundary output bindings.
|
||||||
|
|
||||||
Remaining work should avoid jumping straight into a broad rewrite. The next
|
Remaining work should avoid jumping straight into a broad rewrite. Native
|
||||||
small slice can start native subgraph scaffolding using the current
|
subgraph scaffolding is now present (`SubgraphNode`, structural `WorkflowRef`,
|
||||||
scope/lineage primitives.
|
terminal workflow outcomes, and authoring helpers). The next runtime slice can
|
||||||
|
execute a non-interrupting prepared child graph using the current scope/lineage
|
||||||
|
primitives; interrupt bubbling and saved/deployed workflow resolution remain
|
||||||
|
later work.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Native Subgraphs Design
|
# Native Subgraphs Design
|
||||||
|
|
||||||
Status: proposed
|
Status: scaffolding implemented; runtime execution planned
|
||||||
|
|
||||||
Native subgraphs should make a workflow usable as a workflow step without
|
Native subgraphs should make a workflow usable as a workflow step without
|
||||||
collapsing the child run into one opaque Python node call. The current
|
collapsing the child run into one opaque Python node call. The current
|
||||||
@@ -8,7 +8,8 @@ collapsing the child run into one opaque Python node call. The current
|
|||||||
compatibility wrappers, but they hide the child trace, child frames, and child
|
compatibility wrappers, but they hide the child trace, child frames, and child
|
||||||
interrupt lifecycle from `wf_core`.
|
interrupt lifecycle from `wf_core`.
|
||||||
|
|
||||||
This design defines the core runtime shape before implementation.
|
This design defines the core runtime shape. The boundary model is implemented;
|
||||||
|
child execution, interruption, and saved-workflow resolution remain planned.
|
||||||
|
|
||||||
## Goals
|
## Goals
|
||||||
|
|
||||||
@@ -63,26 +64,32 @@ class SubgraphNode(BaseModel):
|
|||||||
outcomes: list[str] = Field(default_factory=lambda: ["ok"])
|
outcomes: list[str] = Field(default_factory=lambda: ["ok"])
|
||||||
```
|
```
|
||||||
|
|
||||||
Current implementation status: `wf_core` has a first placeholder
|
Current implementation status: the boundary scaffolding is implemented.
|
||||||
`SubgraphNode`. Its `workflow` field is a structural `WorkflowRef`: local
|
`wf_core` has `SubgraphNode`; its `workflow` field is a structural
|
||||||
compiled workflows use `{"name": "child"}`, while saved artifacts can use
|
`WorkflowRef`: local compiled workflows use `{"name": "child"}`, while saved
|
||||||
`{"artifact_id": "child", "version": 1}`. Legacy strings still parse as input,
|
artifacts can use `{"artifact_id": "child", "version": 1}`. Legacy strings
|
||||||
but saved graphs should persist the structural shape. The placeholder also
|
still parse as input, but saved graphs persist the structural shape. The
|
||||||
carries `input_schema` and `output_schema` so validation can check parent
|
placeholder carries input/output schemas and bindings so validation can check
|
||||||
bindings before native execution exists. Runtime execution intentionally raises
|
the parent boundary before native execution exists. Core workflows also
|
||||||
until a later slice adds child scope/frame execution.
|
declare terminal outcomes through `Workflow.outcomes` and `EndNode`.
|
||||||
|
`wf_authoring.subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` build the
|
||||||
|
native boundary, while artifact helpers convert saved/capability workflow
|
||||||
|
references into core `WorkflowRef` values.
|
||||||
|
|
||||||
|
Runtime execution is deliberately not implemented: stepping a `SubgraphNode`
|
||||||
|
fails explicitly until the next slice adds child scope/frame execution.
|
||||||
|
|
||||||
`WorkflowRef` should be structural, not a dotted string parser:
|
`WorkflowRef` should be structural, not a dotted string parser:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class WorkflowRef(BaseModel):
|
class WorkflowRef(BaseModel):
|
||||||
source: str | None = None
|
name: str | None = None
|
||||||
artifact_id: str | None = None
|
artifact_id: str | None = None
|
||||||
version: int | None = None
|
version: int | None = None
|
||||||
inline_name: str | None = None
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The exact reference model can be smaller in v1, but it must not derive meaning
|
The reference has two valid forms: local compiled `{"name": ...}` or saved
|
||||||
|
artifact `{"artifact_id": ..., "version": ...}`. It must not derive meaning
|
||||||
from formatted display names. Higher layers may resolve saved artifacts,
|
from formatted display names. Higher layers may resolve saved artifacts,
|
||||||
deployments, or local builders into an executable child workflow before the
|
deployments, or local builders into an executable child workflow before the
|
||||||
core runtime starts.
|
core runtime starts.
|
||||||
@@ -336,11 +343,13 @@ child = parent.subgraph(
|
|||||||
parent.connect(child, "ok", END)
|
parent.connect(child, "ok", END)
|
||||||
```
|
```
|
||||||
|
|
||||||
For saved artifacts:
|
For saved artifacts, use the lower-level helper with a structural core ref:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
child = parent.subgraph_ref(
|
child = subgraph_ref(
|
||||||
workflow=WorkflowCapabilityRef(artifact_id="demo_child", version=1),
|
id="run_child",
|
||||||
|
workflow=child_builder.compile(),
|
||||||
|
workflow_ref=WorkflowRef(artifact_id="demo_child", version=1),
|
||||||
...
|
...
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
@@ -367,13 +376,29 @@ selection out of `wf_core`.
|
|||||||
|
|
||||||
## Implementation Slices
|
## Implementation Slices
|
||||||
|
|
||||||
### Slice 1: Non-Interrupting Inline Subgraph
|
### Completed Scaffold: Typed Native Boundary
|
||||||
|
|
||||||
- Add `SubgraphNode` to the core `Step` union.
|
- `SubgraphNode` is part of the core `Step` union and validates its declared
|
||||||
- Add minimal `WorkflowRef` / inline child workflow dependency resolution.
|
parent-side boundary.
|
||||||
|
- `WorkflowRef` is structural and supports local compiled or saved artifact
|
||||||
|
references without requiring runtime string parsing.
|
||||||
|
- `Workflow.outcomes`, `EndNode`, and `RunState.outcome` define child terminal
|
||||||
|
outcome semantics before child execution exists.
|
||||||
|
- `subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` produce native
|
||||||
|
boundaries; wrapper-node helpers remain compatibility APIs.
|
||||||
|
- Artifact conversion helpers bridge saved workflow identities to core
|
||||||
|
`WorkflowRef` values.
|
||||||
|
|
||||||
|
### Slice 1: Non-Interrupting Inline Subgraph Runtime
|
||||||
|
|
||||||
|
- Resolve local/prepared child `WorkflowRef` dependencies at runtime; do not
|
||||||
|
load saved artifacts inside `wf_core`.
|
||||||
- Execute child workflow to completion through child frames.
|
- Execute child workflow to completion through child frames.
|
||||||
|
- Give the child an explicit runtime scope/lineage so child state is isolated
|
||||||
|
from parent state until boundary completion.
|
||||||
- Preserve child trace in a clearly-owned form.
|
- Preserve child trace in a clearly-owned form.
|
||||||
- Apply child output to parent state through existing output binding code.
|
- Apply child output to parent state through existing output binding code.
|
||||||
|
- Route the parent step through the child's terminal `RunState.outcome`.
|
||||||
- Tests: child output mapping, child internal trace visibility, parent trace
|
- Tests: child output mapping, child internal trace visibility, parent trace
|
||||||
shape, child runtime failure fails parent.
|
shape, child runtime failure fails parent.
|
||||||
|
|
||||||
@@ -387,16 +412,21 @@ selection out of `wf_core`.
|
|||||||
|
|
||||||
### Slice 3: Saved Workflow References
|
### Slice 3: Saved Workflow References
|
||||||
|
|
||||||
|
- Structural saved-workflow references and conversion helpers already exist;
|
||||||
|
this slice is execution resolution, not a new identity shape.
|
||||||
- Add platform-level resolution for saved workflow artifacts.
|
- Add platform-level resolution for saved workflow artifacts.
|
||||||
- Validate dependencies and source bindings before execution.
|
- Validate dependencies and source bindings before execution.
|
||||||
- Tests: saved child workflow runs through a deployment binding, missing child
|
- Tests: saved child workflow runs through a deployment binding, missing child
|
||||||
artifact reports an unrunnable dependency.
|
artifact reports an unrunnable dependency.
|
||||||
|
|
||||||
### Slice 4: Outcome and Policy Expansion
|
### Slice 4: Optional Policy Expansion
|
||||||
|
|
||||||
- Decide whether subgraphs can expose multiple outcomes.
|
- Workflow outcome propagation is settled: child `RunState.outcome` is the
|
||||||
- Decide child failure handling policy, if any.
|
parent-visible subgraph outcome; legacy `__end__` means `ok`, while explicit
|
||||||
- Keep default behavior strict until the use case is clear.
|
`EndNode` carries other declared outcomes.
|
||||||
|
- Keep child runtime failures as parent runtime failures by default.
|
||||||
|
- Only add configurable child-failure policy or richer boundary result
|
||||||
|
semantics when an actual use case requires it.
|
||||||
|
|
||||||
## Risks
|
## Risks
|
||||||
|
|
||||||
@@ -417,15 +447,14 @@ selection out of `wf_core`.
|
|||||||
fields, or should child traces live in a separate inspectable structure?
|
fields, or should child traces live in a separate inspectable structure?
|
||||||
- Is v1 allowed to reference only inline/compiled child workflows, or should it
|
- Is v1 allowed to reference only inline/compiled child workflows, or should it
|
||||||
immediately accept artifact refs resolved by the platform?
|
immediately accept artifact refs resolved by the platform?
|
||||||
- Should subgraph completion always emit `ok` initially, or should child
|
|
||||||
workflow artifacts declare outcomes before native subgraphs ship?
|
|
||||||
|
|
||||||
## Recommendation
|
## Recommendation
|
||||||
|
|
||||||
Start with Slice 1 as a non-interrupting inline subgraph. It gives us native
|
The typed boundary scaffold is complete. Start runtime work with Slice 1 as a
|
||||||
trace/frame semantics without taking on the hardest resume problem immediately.
|
non-interrupting inline/prepared subgraph. It gives us native trace/frame and
|
||||||
Do not delete the wrapper-node helpers yet; use them as compatibility and
|
scope/lineage semantics without taking on the hardest resume problem
|
||||||
examples while native subgraphs mature.
|
immediately. Do not delete the wrapper-node helpers yet; use them as
|
||||||
|
compatibility and examples while native subgraphs mature.
|
||||||
|
|
||||||
Then implement Slice 2 before exposing saved workflows as broadly reusable child
|
Then implement Slice 2 before exposing saved workflows as broadly reusable child
|
||||||
graphs. Saved workflows without nested interrupt support would look reusable but
|
graphs. Saved workflows without nested interrupt support would look reusable but
|
||||||
|
|||||||
@@ -324,7 +324,7 @@ class EndNode(BaseModel):
|
|||||||
|
|
||||||
id: str
|
id: str
|
||||||
type: Literal["end"]
|
type: Literal["end"]
|
||||||
outcome: str = "ok"
|
outcome: str = Field(default="ok", min_length=1)
|
||||||
|
|
||||||
|
|
||||||
class InterruptNode(BaseModel):
|
class InterruptNode(BaseModel):
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ from copy import deepcopy
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from wf_core.errors import WorkflowExecutionError
|
||||||
from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite
|
from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite
|
||||||
from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID
|
from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID
|
||||||
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
|
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
|
||||||
@@ -62,7 +63,12 @@ def lineage_writes_for_frame(
|
|||||||
if owner is None:
|
if owner is None:
|
||||||
return ()
|
return ()
|
||||||
parent_frame_id, foreach_node_id, item_index = owner
|
parent_frame_id, foreach_node_id, item_index = owner
|
||||||
parent_frame = run.frames[parent_frame_id]
|
parent_frame = run.frames.get(parent_frame_id)
|
||||||
|
if parent_frame is None:
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
"foreach lineage compatibility state references missing parent frame "
|
||||||
|
f"{parent_frame_id!r} for child frame {frame.id!r}"
|
||||||
|
)
|
||||||
barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
|
barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
|
||||||
if barrier is None or barrier.mode != "concurrent":
|
if barrier is None or barrier.mode != "concurrent":
|
||||||
return ()
|
return ()
|
||||||
|
|||||||
@@ -170,6 +170,31 @@ def test_lineage_writes_for_frame_reads_current_foreach_pending_result() -> None
|
|||||||
assert writes[0].visible_value == 5
|
assert writes[0].visible_value == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_lineage_writes_for_frame_rejects_missing_compatibility_parent_frame() -> None:
|
||||||
|
child = ExecutionFrame(
|
||||||
|
id="missing:each:0",
|
||||||
|
kind="foreach_iteration",
|
||||||
|
node_id="work",
|
||||||
|
parent_frame_id="missing",
|
||||||
|
metadata={
|
||||||
|
"foreach_node_id": "each",
|
||||||
|
"loop_index": 0,
|
||||||
|
"loop_item": "a",
|
||||||
|
"loop_alias": "item",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
run = RunState(
|
||||||
|
workflow_name="lineage",
|
||||||
|
status=RunStatus.PENDING,
|
||||||
|
workflow_input={},
|
||||||
|
state={},
|
||||||
|
frames={child.id: child},
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(WorkflowExecutionError, match="missing parent frame"):
|
||||||
|
lineage_writes_for_frame(run, child)
|
||||||
|
|
||||||
|
|
||||||
def test_foreach_barrier_state_returns_none_when_missing() -> None:
|
def test_foreach_barrier_state_returns_none_when_missing() -> None:
|
||||||
frame = ExecutionFrame(id="root", kind="root", node_id="each")
|
frame = ExecutionFrame(id="root", kind="root", node_id="each")
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,9 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from wf_core import END, Workflow
|
import pytest
|
||||||
|
from pydantic import ValidationError
|
||||||
|
|
||||||
|
from wf_core import END, EndNode, Workflow
|
||||||
from wf_core.runtime import execute_workflow
|
from wf_core.runtime import execute_workflow
|
||||||
from wf_core.validation.issues import ValidationIssueCode
|
from wf_core.validation.issues import ValidationIssueCode
|
||||||
|
|
||||||
@@ -63,6 +66,11 @@ def test_validation_rejects_legacy_end_without_ok_workflow_outcome() -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_end_node_rejects_empty_workflow_outcome() -> None:
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
EndNode(id="end_empty", type="end", outcome="")
|
||||||
|
|
||||||
|
|
||||||
def _finish(payload: dict[str, object], _ctx: object) -> dict[str, object]:
|
def _finish(payload: dict[str, object], _ctx: object) -> dict[str, object]:
|
||||||
return {"outcome": "done", "output": {"echoed": payload["text"]}}
|
return {"outcome": "done", "output": {"echoed": payload["text"]}}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user