docs + code review

This commit is contained in:
lda
2026-05-25 04:02:55 +07:00 Verified
parent 492226a596
commit 3880a86c26
7 changed files with 128 additions and 52 deletions
+22 -17
View File
@@ -45,14 +45,17 @@ implementation state.
[ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md).
- Native subgraph design spec:
[2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md).
- **Native subgraphs / graph-as-node**: next major runtime feature. A
first-class `SubgraphNode` placeholder exists and validates parent-side
bindings/outcomes. Core workflows now also declare workflow-level outcomes
and can terminate through explicit `EndNode` steps. Runtime subgraph execution
still needs child run/frame identity, child trace preservation, interrupt
bubbling, and resume back into the child workflow. Wrapper helpers currently
run child workflows as ordinary nodes; true graph-as-node behavior belongs
here.
- **Native subgraphs / graph-as-node**: next major runtime feature. The
scaffolding slice is complete: core has `SubgraphNode`, structural
`WorkflowRef`, workflow-level outcomes plus explicit `EndNode` termination,
authoring helpers (`subgraph_ref` / `WorkflowBuilder.subgraph`), and artifact
reference conversion helpers. Runtime subgraph execution is still absent.
The next slice is non-interrupting child execution: resolve a prepared child
workflow, create a child scope/lineage, preserve child trace, map child
output back through the subgraph boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution follow
after that. Wrapper helpers currently run child workflows as ordinary nodes;
true graph-as-node behavior belongs here.
- **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -77,13 +80,14 @@ implementation state.
- **Dashboard/source controls**: future UI should consume the same source
inventory and deployment metadata instead of reverse-engineering MCP tools.
Frame stress points to solve before either feature:
Frame stress points remaining for native subgraphs and future fork/gather:
- `RunState.current_frame_id` currently models the selected execution cursor.
Concurrent foreach needs multiple runnable child frames.
- `ExecutionFrame.metadata` currently carries ad hoc foreach data. Subgraphs and
concurrent foreach should get typed frame payloads or strongly bounded helper
accessors before metadata grows more meanings.
- `RunState.current_frame_id` remains the selected execution cursor even though
concurrent foreach now schedules multiple child frames. Native subgraphs
must preserve that cursor model while owning a nested child execution scope.
- `ExecutionFrame.metadata` has typed foreach access paths, but subgraphs still
need typed child-workflow ownership and completion metadata rather than new
ad hoc dictionary fields.
- Subgraph frames need child workflow identity/version/deployment binding, not
just a generic metadata dictionary.
- `RunState.current_node_id` duplicates the current frame's node id for
@@ -94,6 +98,7 @@ Frame stress points to solve before either feature:
The MCP workflow authoring path is now usable enough for real testing. The next
bottleneck is runtime/platform correctness: resumable child execution,
concurrent scheduling, persistent run history, and protocol-native progress
reporting. Those pieces should come before adding more high-level authoring
sugar.
native subgraph execution, persistent run history, and protocol-native
progress reporting. Concurrent foreach supplies scheduler/lineage precedent;
native child graphs are now the missing runtime boundary. Those pieces should
come before adding more high-level authoring sugar.
@@ -39,9 +39,12 @@ explicit scope/lineage commit target, feasible once native subgraph completion
can declare whether child writes commit to child scope, parent lineage, or only
through boundary output bindings.
Remaining work should avoid jumping straight into a broad rewrite. The next
small slice can start native subgraph scaffolding using the current
scope/lineage primitives.
Remaining work should avoid jumping straight into a broad rewrite. Native
subgraph scaffolding is now present (`SubgraphNode`, structural `WorkflowRef`,
terminal workflow outcomes, and authoring helpers). The next runtime slice can
execute a non-interrupting prepared child graph using the current scope/lineage
primitives; interrupt bubbling and saved/deployed workflow resolution remain
later work.
---
@@ -1,6 +1,6 @@
# Native Subgraphs Design
Status: proposed
Status: scaffolding implemented; runtime execution planned
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
@@ -8,7 +8,8 @@ collapsing the child run into one opaque Python node call. The current
compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`.
This design defines the core runtime shape before implementation.
This design defines the core runtime shape. The boundary model is implemented;
child execution, interruption, and saved-workflow resolution remain planned.
## Goals
@@ -63,26 +64,32 @@ class SubgraphNode(BaseModel):
outcomes: list[str] = Field(default_factory=lambda: ["ok"])
```
Current implementation status: `wf_core` has a first placeholder
`SubgraphNode`. Its `workflow` field is a structural `WorkflowRef`: local
compiled workflows use `{"name": "child"}`, while saved artifacts can use
`{"artifact_id": "child", "version": 1}`. Legacy strings still parse as input,
but saved graphs should persist the structural shape. The placeholder also
carries `input_schema` and `output_schema` so validation can check parent
bindings before native execution exists. Runtime execution intentionally raises
until a later slice adds child scope/frame execution.
Current implementation status: the boundary scaffolding is implemented.
`wf_core` has `SubgraphNode`; its `workflow` field is a structural
`WorkflowRef`: local compiled workflows use `{"name": "child"}`, while saved
artifacts can use `{"artifact_id": "child", "version": 1}`. Legacy strings
still parse as input, but saved graphs persist the structural shape. The
placeholder carries input/output schemas and bindings so validation can check
the parent boundary before native execution exists. Core workflows also
declare terminal outcomes through `Workflow.outcomes` and `EndNode`.
`wf_authoring.subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` build the
native boundary, while artifact helpers convert saved/capability workflow
references into core `WorkflowRef` values.
Runtime execution is deliberately not implemented: stepping a `SubgraphNode`
fails explicitly until the next slice adds child scope/frame execution.
`WorkflowRef` should be structural, not a dotted string parser:
```python
class WorkflowRef(BaseModel):
source: str | None = None
name: str | None = None
artifact_id: str | None = None
version: int | None = None
inline_name: str | None = None
```
The exact reference model can be smaller in v1, but it must not derive meaning
The reference has two valid forms: local compiled `{"name": ...}` or saved
artifact `{"artifact_id": ..., "version": ...}`. It must not derive meaning
from formatted display names. Higher layers may resolve saved artifacts,
deployments, or local builders into an executable child workflow before the
core runtime starts.
@@ -336,11 +343,13 @@ child = parent.subgraph(
parent.connect(child, "ok", END)
```
For saved artifacts:
For saved artifacts, use the lower-level helper with a structural core ref:
```python
child = parent.subgraph_ref(
workflow=WorkflowCapabilityRef(artifact_id="demo_child", version=1),
child = subgraph_ref(
id="run_child",
workflow=child_builder.compile(),
workflow_ref=WorkflowRef(artifact_id="demo_child", version=1),
...
)
```
@@ -367,13 +376,29 @@ selection out of `wf_core`.
## Implementation Slices
### Slice 1: Non-Interrupting Inline Subgraph
### Completed Scaffold: Typed Native Boundary
- Add `SubgraphNode` to the core `Step` union.
- Add minimal `WorkflowRef` / inline child workflow dependency resolution.
- `SubgraphNode` is part of the core `Step` union and validates its declared
parent-side boundary.
- `WorkflowRef` is structural and supports local compiled or saved artifact
references without requiring runtime string parsing.
- `Workflow.outcomes`, `EndNode`, and `RunState.outcome` define child terminal
outcome semantics before child execution exists.
- `subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` produce native
boundaries; wrapper-node helpers remain compatibility APIs.
- Artifact conversion helpers bridge saved workflow identities to core
`WorkflowRef` values.
### Slice 1: Non-Interrupting Inline Subgraph Runtime
- Resolve local/prepared child `WorkflowRef` dependencies at runtime; do not
load saved artifacts inside `wf_core`.
- Execute child workflow to completion through child frames.
- Give the child an explicit runtime scope/lineage so child state is isolated
from parent state until boundary completion.
- Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code.
- Route the parent step through the child's terminal `RunState.outcome`.
- Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent.
@@ -387,16 +412,21 @@ selection out of `wf_core`.
### Slice 3: Saved Workflow References
- Structural saved-workflow references and conversion helpers already exist;
this slice is execution resolution, not a new identity shape.
- Add platform-level resolution for saved workflow artifacts.
- Validate dependencies and source bindings before execution.
- Tests: saved child workflow runs through a deployment binding, missing child
artifact reports an unrunnable dependency.
### Slice 4: Outcome and Policy Expansion
### Slice 4: Optional Policy Expansion
- Decide whether subgraphs can expose multiple outcomes.
- Decide child failure handling policy, if any.
- Keep default behavior strict until the use case is clear.
- Workflow outcome propagation is settled: child `RunState.outcome` is the
parent-visible subgraph outcome; legacy `__end__` means `ok`, while explicit
`EndNode` carries other declared outcomes.
- Keep child runtime failures as parent runtime failures by default.
- Only add configurable child-failure policy or richer boundary result
semantics when an actual use case requires it.
## Risks
@@ -417,15 +447,14 @@ selection out of `wf_core`.
fields, or should child traces live in a separate inspectable structure?
- Is v1 allowed to reference only inline/compiled child workflows, or should it
immediately accept artifact refs resolved by the platform?
- Should subgraph completion always emit `ok` initially, or should child
workflow artifacts declare outcomes before native subgraphs ship?
## Recommendation
Start with Slice 1 as a non-interrupting inline subgraph. It gives us native
trace/frame semantics without taking on the hardest resume problem immediately.
Do not delete the wrapper-node helpers yet; use them as compatibility and
examples while native subgraphs mature.
The typed boundary scaffold is complete. Start runtime work with Slice 1 as a
non-interrupting inline/prepared subgraph. It gives us native trace/frame and
scope/lineage semantics without taking on the hardest resume problem
immediately. Do not delete the wrapper-node helpers yet; use them as
compatibility and examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but
+1 -1
View File
@@ -324,7 +324,7 @@ class EndNode(BaseModel):
id: str
type: Literal["end"]
outcome: str = "ok"
outcome: str = Field(default="ok", min_length=1)
class InterruptNode(BaseModel):
+7 -1
View File
@@ -5,6 +5,7 @@ from copy import deepcopy
from dataclasses import dataclass
from typing import Any
from wf_core.errors import WorkflowExecutionError
from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite
from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
@@ -62,7 +63,12 @@ def lineage_writes_for_frame(
if owner is None:
return ()
parent_frame_id, foreach_node_id, item_index = owner
parent_frame = run.frames[parent_frame_id]
parent_frame = run.frames.get(parent_frame_id)
if parent_frame is None:
raise WorkflowExecutionError(
"foreach lineage compatibility state references missing parent frame "
f"{parent_frame_id!r} for child frame {frame.id!r}"
)
barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
if barrier is None or barrier.mode != "concurrent":
return ()
+25
View File
@@ -170,6 +170,31 @@ def test_lineage_writes_for_frame_reads_current_foreach_pending_result() -> None
assert writes[0].visible_value == 5
def test_lineage_writes_for_frame_rejects_missing_compatibility_parent_frame() -> None:
child = ExecutionFrame(
id="missing:each:0",
kind="foreach_iteration",
node_id="work",
parent_frame_id="missing",
metadata={
"foreach_node_id": "each",
"loop_index": 0,
"loop_item": "a",
"loop_alias": "item",
},
)
run = RunState(
workflow_name="lineage",
status=RunStatus.PENDING,
workflow_input={},
state={},
frames={child.id: child},
)
with pytest.raises(WorkflowExecutionError, match="missing parent frame"):
lineage_writes_for_frame(run, child)
def test_foreach_barrier_state_returns_none_when_missing() -> None:
frame = ExecutionFrame(id="root", kind="root", node_id="each")
+9 -1
View File
@@ -1,6 +1,9 @@
from __future__ import annotations
from wf_core import END, Workflow
import pytest
from pydantic import ValidationError
from wf_core import END, EndNode, Workflow
from wf_core.runtime import execute_workflow
from wf_core.validation.issues import ValidationIssueCode
@@ -63,6 +66,11 @@ def test_validation_rejects_legacy_end_without_ok_workflow_outcome() -> None:
)
def test_end_node_rejects_empty_workflow_outcome() -> None:
with pytest.raises(ValidationError):
EndNode(id="end_empty", type="end", outcome="")
def _finish(payload: dict[str, object], _ctx: object) -> dict[str, object]:
return {"outcome": "done", "output": {"echoed": payload["text"]}}