docs + code review

This commit is contained in:
lda
2026-05-25 04:02:55 +07:00 Verified
parent 492226a596
commit 3880a86c26
7 changed files with 128 additions and 52 deletions
+22 -17
View File
@@ -45,14 +45,17 @@ implementation state.
[ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md). [ADR 0002](./adr/0002-concurrent-foreach-policy-and-barrier-commits.md).
- Native subgraph design spec: - Native subgraph design spec:
[2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md). [2026-05-24 native subgraphs](./superpowers/specs/2026-05-24-native-subgraphs-design.md).
- **Native subgraphs / graph-as-node**: next major runtime feature. A - **Native subgraphs / graph-as-node**: next major runtime feature. The
first-class `SubgraphNode` placeholder exists and validates parent-side scaffolding slice is complete: core has `SubgraphNode`, structural
bindings/outcomes. Core workflows now also declare workflow-level outcomes `WorkflowRef`, workflow-level outcomes plus explicit `EndNode` termination,
and can terminate through explicit `EndNode` steps. Runtime subgraph execution authoring helpers (`subgraph_ref` / `WorkflowBuilder.subgraph`), and artifact
still needs child run/frame identity, child trace preservation, interrupt reference conversion helpers. Runtime subgraph execution is still absent.
bubbling, and resume back into the child workflow. Wrapper helpers currently The next slice is non-interrupting child execution: resolve a prepared child
run child workflows as ordinary nodes; true graph-as-node behavior belongs workflow, create a child scope/lineage, preserve child trace, map child
here. output back through the subgraph boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution follow
after that. Wrapper helpers currently run child workflows as ordinary nodes;
true graph-as-node behavior belongs here.
- **Concurrent foreach**: implemented in core with explicit scheduling, - **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -77,13 +80,14 @@ implementation state.
- **Dashboard/source controls**: future UI should consume the same source - **Dashboard/source controls**: future UI should consume the same source
inventory and deployment metadata instead of reverse-engineering MCP tools. inventory and deployment metadata instead of reverse-engineering MCP tools.
Frame stress points to solve before either feature: Frame stress points remaining for native subgraphs and future fork/gather:
- `RunState.current_frame_id` currently models the selected execution cursor. - `RunState.current_frame_id` remains the selected execution cursor even though
Concurrent foreach needs multiple runnable child frames. concurrent foreach now schedules multiple child frames. Native subgraphs
- `ExecutionFrame.metadata` currently carries ad hoc foreach data. Subgraphs and must preserve that cursor model while owning a nested child execution scope.
concurrent foreach should get typed frame payloads or strongly bounded helper - `ExecutionFrame.metadata` has typed foreach access paths, but subgraphs still
accessors before metadata grows more meanings. need typed child-workflow ownership and completion metadata rather than new
ad hoc dictionary fields.
- Subgraph frames need child workflow identity/version/deployment binding, not - Subgraph frames need child workflow identity/version/deployment binding, not
just a generic metadata dictionary. just a generic metadata dictionary.
- `RunState.current_node_id` duplicates the current frame's node id for - `RunState.current_node_id` duplicates the current frame's node id for
@@ -94,6 +98,7 @@ Frame stress points to solve before either feature:
The MCP workflow authoring path is now usable enough for real testing. The next The MCP workflow authoring path is now usable enough for real testing. The next
bottleneck is runtime/platform correctness: resumable child execution, bottleneck is runtime/platform correctness: resumable child execution,
concurrent scheduling, persistent run history, and protocol-native progress native subgraph execution, persistent run history, and protocol-native
reporting. Those pieces should come before adding more high-level authoring progress reporting. Concurrent foreach supplies scheduler/lineage precedent;
sugar. native child graphs are now the missing runtime boundary. Those pieces should
come before adding more high-level authoring sugar.
@@ -39,9 +39,12 @@ explicit scope/lineage commit target, feasible once native subgraph completion
can declare whether child writes commit to child scope, parent lineage, or only can declare whether child writes commit to child scope, parent lineage, or only
through boundary output bindings. through boundary output bindings.
Remaining work should avoid jumping straight into a broad rewrite. The next Remaining work should avoid jumping straight into a broad rewrite. Native
small slice can start native subgraph scaffolding using the current subgraph scaffolding is now present (`SubgraphNode`, structural `WorkflowRef`,
scope/lineage primitives. terminal workflow outcomes, and authoring helpers). The next runtime slice can
execute a non-interrupting prepared child graph using the current scope/lineage
primitives; interrupt bubbling and saved/deployed workflow resolution remain
later work.
--- ---
@@ -1,6 +1,6 @@
# Native Subgraphs Design # Native Subgraphs Design
Status: proposed Status: scaffolding implemented; runtime execution planned
Native subgraphs should make a workflow usable as a workflow step without Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current collapsing the child run into one opaque Python node call. The current
@@ -8,7 +8,8 @@ collapsing the child run into one opaque Python node call. The current
compatibility wrappers, but they hide the child trace, child frames, and child compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`. interrupt lifecycle from `wf_core`.
This design defines the core runtime shape before implementation. This design defines the core runtime shape. The boundary model is implemented;
child execution, interruption, and saved-workflow resolution remain planned.
## Goals ## Goals
@@ -63,26 +64,32 @@ class SubgraphNode(BaseModel):
outcomes: list[str] = Field(default_factory=lambda: ["ok"]) outcomes: list[str] = Field(default_factory=lambda: ["ok"])
``` ```
Current implementation status: `wf_core` has a first placeholder Current implementation status: the boundary scaffolding is implemented.
`SubgraphNode`. Its `workflow` field is a structural `WorkflowRef`: local `wf_core` has `SubgraphNode`; its `workflow` field is a structural
compiled workflows use `{"name": "child"}`, while saved artifacts can use `WorkflowRef`: local compiled workflows use `{"name": "child"}`, while saved
`{"artifact_id": "child", "version": 1}`. Legacy strings still parse as input, artifacts can use `{"artifact_id": "child", "version": 1}`. Legacy strings
but saved graphs should persist the structural shape. The placeholder also still parse as input, but saved graphs persist the structural shape. The
carries `input_schema` and `output_schema` so validation can check parent placeholder carries input/output schemas and bindings so validation can check
bindings before native execution exists. Runtime execution intentionally raises the parent boundary before native execution exists. Core workflows also
until a later slice adds child scope/frame execution. declare terminal outcomes through `Workflow.outcomes` and `EndNode`.
`wf_authoring.subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` build the
native boundary, while artifact helpers convert saved/capability workflow
references into core `WorkflowRef` values.
Runtime execution is deliberately not implemented: stepping a `SubgraphNode`
fails explicitly until the next slice adds child scope/frame execution.
`WorkflowRef` should be structural, not a dotted string parser: `WorkflowRef` should be structural, not a dotted string parser:
```python ```python
class WorkflowRef(BaseModel): class WorkflowRef(BaseModel):
source: str | None = None name: str | None = None
artifact_id: str | None = None artifact_id: str | None = None
version: int | None = None version: int | None = None
inline_name: str | None = None
``` ```
The exact reference model can be smaller in v1, but it must not derive meaning The reference has two valid forms: local compiled `{"name": ...}` or saved
artifact `{"artifact_id": ..., "version": ...}`. It must not derive meaning
from formatted display names. Higher layers may resolve saved artifacts, from formatted display names. Higher layers may resolve saved artifacts,
deployments, or local builders into an executable child workflow before the deployments, or local builders into an executable child workflow before the
core runtime starts. core runtime starts.
@@ -336,11 +343,13 @@ child = parent.subgraph(
parent.connect(child, "ok", END) parent.connect(child, "ok", END)
``` ```
For saved artifacts: For saved artifacts, use the lower-level helper with a structural core ref:
```python ```python
child = parent.subgraph_ref( child = subgraph_ref(
workflow=WorkflowCapabilityRef(artifact_id="demo_child", version=1), id="run_child",
workflow=child_builder.compile(),
workflow_ref=WorkflowRef(artifact_id="demo_child", version=1),
... ...
) )
``` ```
@@ -367,13 +376,29 @@ selection out of `wf_core`.
## Implementation Slices ## Implementation Slices
### Slice 1: Non-Interrupting Inline Subgraph ### Completed Scaffold: Typed Native Boundary
- Add `SubgraphNode` to the core `Step` union. - `SubgraphNode` is part of the core `Step` union and validates its declared
- Add minimal `WorkflowRef` / inline child workflow dependency resolution. parent-side boundary.
- `WorkflowRef` is structural and supports local compiled or saved artifact
references without requiring runtime string parsing.
- `Workflow.outcomes`, `EndNode`, and `RunState.outcome` define child terminal
outcome semantics before child execution exists.
- `subgraph_ref(...)` and `WorkflowBuilder.subgraph(...)` produce native
boundaries; wrapper-node helpers remain compatibility APIs.
- Artifact conversion helpers bridge saved workflow identities to core
`WorkflowRef` values.
### Slice 1: Non-Interrupting Inline Subgraph Runtime
- Resolve local/prepared child `WorkflowRef` dependencies at runtime; do not
load saved artifacts inside `wf_core`.
- Execute child workflow to completion through child frames. - Execute child workflow to completion through child frames.
- Give the child an explicit runtime scope/lineage so child state is isolated
from parent state until boundary completion.
- Preserve child trace in a clearly-owned form. - Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code. - Apply child output to parent state through existing output binding code.
- Route the parent step through the child's terminal `RunState.outcome`.
- Tests: child output mapping, child internal trace visibility, parent trace - Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent. shape, child runtime failure fails parent.
@@ -387,16 +412,21 @@ selection out of `wf_core`.
### Slice 3: Saved Workflow References ### Slice 3: Saved Workflow References
- Structural saved-workflow references and conversion helpers already exist;
this slice is execution resolution, not a new identity shape.
- Add platform-level resolution for saved workflow artifacts. - Add platform-level resolution for saved workflow artifacts.
- Validate dependencies and source bindings before execution. - Validate dependencies and source bindings before execution.
- Tests: saved child workflow runs through a deployment binding, missing child - Tests: saved child workflow runs through a deployment binding, missing child
artifact reports an unrunnable dependency. artifact reports an unrunnable dependency.
### Slice 4: Outcome and Policy Expansion ### Slice 4: Optional Policy Expansion
- Decide whether subgraphs can expose multiple outcomes. - Workflow outcome propagation is settled: child `RunState.outcome` is the
- Decide child failure handling policy, if any. parent-visible subgraph outcome; legacy `__end__` means `ok`, while explicit
- Keep default behavior strict until the use case is clear. `EndNode` carries other declared outcomes.
- Keep child runtime failures as parent runtime failures by default.
- Only add configurable child-failure policy or richer boundary result
semantics when an actual use case requires it.
## Risks ## Risks
@@ -417,15 +447,14 @@ selection out of `wf_core`.
fields, or should child traces live in a separate inspectable structure? fields, or should child traces live in a separate inspectable structure?
- Is v1 allowed to reference only inline/compiled child workflows, or should it - Is v1 allowed to reference only inline/compiled child workflows, or should it
immediately accept artifact refs resolved by the platform? immediately accept artifact refs resolved by the platform?
- Should subgraph completion always emit `ok` initially, or should child
workflow artifacts declare outcomes before native subgraphs ship?
## Recommendation ## Recommendation
Start with Slice 1 as a non-interrupting inline subgraph. It gives us native The typed boundary scaffold is complete. Start runtime work with Slice 1 as a
trace/frame semantics without taking on the hardest resume problem immediately. non-interrupting inline/prepared subgraph. It gives us native trace/frame and
Do not delete the wrapper-node helpers yet; use them as compatibility and scope/lineage semantics without taking on the hardest resume problem
examples while native subgraphs mature. immediately. Do not delete the wrapper-node helpers yet; use them as
compatibility and examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but graphs. Saved workflows without nested interrupt support would look reusable but
+1 -1
View File
@@ -324,7 +324,7 @@ class EndNode(BaseModel):
id: str id: str
type: Literal["end"] type: Literal["end"]
outcome: str = "ok" outcome: str = Field(default="ok", min_length=1)
class InterruptNode(BaseModel): class InterruptNode(BaseModel):
+7 -1
View File
@@ -5,6 +5,7 @@ from copy import deepcopy
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any from typing import Any
from wf_core.errors import WorkflowExecutionError
from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite
from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
@@ -62,7 +63,12 @@ def lineage_writes_for_frame(
if owner is None: if owner is None:
return () return ()
parent_frame_id, foreach_node_id, item_index = owner parent_frame_id, foreach_node_id, item_index = owner
parent_frame = run.frames[parent_frame_id] parent_frame = run.frames.get(parent_frame_id)
if parent_frame is None:
raise WorkflowExecutionError(
"foreach lineage compatibility state references missing parent frame "
f"{parent_frame_id!r} for child frame {frame.id!r}"
)
barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id) barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
if barrier is None or barrier.mode != "concurrent": if barrier is None or barrier.mode != "concurrent":
return () return ()
+25
View File
@@ -170,6 +170,31 @@ def test_lineage_writes_for_frame_reads_current_foreach_pending_result() -> None
assert writes[0].visible_value == 5 assert writes[0].visible_value == 5
def test_lineage_writes_for_frame_rejects_missing_compatibility_parent_frame() -> None:
child = ExecutionFrame(
id="missing:each:0",
kind="foreach_iteration",
node_id="work",
parent_frame_id="missing",
metadata={
"foreach_node_id": "each",
"loop_index": 0,
"loop_item": "a",
"loop_alias": "item",
},
)
run = RunState(
workflow_name="lineage",
status=RunStatus.PENDING,
workflow_input={},
state={},
frames={child.id: child},
)
with pytest.raises(WorkflowExecutionError, match="missing parent frame"):
lineage_writes_for_frame(run, child)
def test_foreach_barrier_state_returns_none_when_missing() -> None: def test_foreach_barrier_state_returns_none_when_missing() -> None:
frame = ExecutionFrame(id="root", kind="root", node_id="each") frame = ExecutionFrame(id="root", kind="root", node_id="each")
+9 -1
View File
@@ -1,6 +1,9 @@
from __future__ import annotations from __future__ import annotations
from wf_core import END, Workflow import pytest
from pydantic import ValidationError
from wf_core import END, EndNode, Workflow
from wf_core.runtime import execute_workflow from wf_core.runtime import execute_workflow
from wf_core.validation.issues import ValidationIssueCode from wf_core.validation.issues import ValidationIssueCode
@@ -63,6 +66,11 @@ def test_validation_rejects_legacy_end_without_ok_workflow_outcome() -> None:
) )
def test_end_node_rejects_empty_workflow_outcome() -> None:
with pytest.raises(ValidationError):
EndNode(id="end_empty", type="end", outcome="")
def _finish(payload: dict[str, object], _ctx: object) -> dict[str, object]: def _finish(payload: dict[str, object], _ctx: object) -> dict[str, object]:
return {"outcome": "done", "output": {"echoed": payload["text"]}} return {"outcome": "done", "output": {"echoed": payload["text"]}}