native subgraph execution

This commit is contained in:
lda
2026-05-25 04:28:06 +07:00 Verified
parent 3880a86c26
commit a4eeb506be
20 changed files with 746 additions and 146 deletions
+10 -11
View File
@@ -49,13 +49,12 @@ implementation state.
scaffolding slice is complete: core has `SubgraphNode`, structural
`WorkflowRef`, workflow-level outcomes plus explicit `EndNode` termination,
authoring helpers (`subgraph_ref` / `WorkflowBuilder.subgraph`), and artifact
reference conversion helpers. Runtime subgraph execution is still absent.
The next slice is non-interrupting child execution: resolve a prepared child
workflow, create a child scope/lineage, preserve child trace, map child
output back through the subgraph boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution follow
after that. Wrapper helpers currently run child workflows as ordinary nodes;
true graph-as-node behavior belongs here.
reference conversion helpers. Core can now execute a prepared local child
workflow through an isolated child scope/lineage, preserve its trace entries,
map child output through the boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution remain
next. Wrapper helpers currently run child workflows as ordinary nodes; native
`SubgraphNode` is now the graph-as-node path for prepared children.
- **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -98,7 +97,7 @@ Frame stress points remaining for native subgraphs and future fork/gather:
The MCP workflow authoring path is now usable enough for real testing. The next
bottleneck is runtime/platform correctness: resumable child execution,
native subgraph execution, persistent run history, and protocol-native
progress reporting. Concurrent foreach supplies scheduler/lineage precedent;
native child graphs are now the missing runtime boundary. Those pieces should
come before adding more high-level authoring sugar.
saved/deployed child resolution, persistent run history, and protocol-native
progress reporting. Concurrent foreach and prepared native child execution now
supply scheduler/lineage precedent. Those remaining pieces should come before
adding more high-level authoring sugar.
@@ -33,18 +33,17 @@ the compatibility subset needed before native subgraphs:
`ForeachBarrierState` now keeps scheduling/result metadata plus compatibility
patches for old serialized barrier data.
Direct commits currently go through `is_root_lineage_frame(frame)`, which is the
migration shortcut for root scope/root lineage. The eventual better shape is an
explicit scope/lineage commit target, feasible once native subgraph completion
can declare whether child writes commit to child scope, parent lineage, or only
through boundary output bindings.
Direct commits now go through a scope-root commit decision: top-level frames
commit to root state, and prepared native-child root frames commit to their
child scope state. Descendant item/branch lineages still buffer writes until a
barrier or future gather commits them.
Remaining work should avoid jumping straight into a broad rewrite. Native
subgraph scaffolding is now present (`SubgraphNode`, structural `WorkflowRef`,
terminal workflow outcomes, and authoring helpers). The next runtime slice can
execute a non-interrupting prepared child graph using the current scope/lineage
primitives; interrupt bubbling and saved/deployed workflow resolution remain
later work.
subgraph scaffolding and non-interrupting prepared-child execution are now
present (`SubgraphNode`, structural `WorkflowRef`, terminal workflow outcomes,
authoring helpers, and `PreparedSubgraph`). Child graphs execute through their
own scope/lineage and map output back at completion. Interrupt bubbling and
saved/deployed workflow resolution remain later work.
---
@@ -34,11 +34,10 @@ The full native subgraph use of `RuntimeScope` is not implemented yet.
`ForeachBarrierState` still owns scheduling/barrier metadata, but no longer has
to be the primary write store for new concurrent foreach item results.
Direct node commits currently use the explicit root-frame helper
`is_root_lineage_frame(frame)`. That helper still means "root scope plus root
lineage" during migration. The better long-term shape becomes feasible when
native subgraph completion exists: direct commits should be decided by an
explicit scope/lineage commit target, not by root ids.
Direct node commits now use a scope-root decision: the top-level root commits
to `RunState.state`, while a prepared native-subgraph root commits to its child
scope state. Descendant item/branch lineages buffer writes until a barrier or
future gather commits them.
## Problem
@@ -1,6 +1,6 @@
# Native Subgraphs Design
Status: scaffolding implemented; runtime execution planned
Status: prepared-child execution implemented; interrupts/artifact resolution planned
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
@@ -8,8 +8,9 @@ collapsing the child run into one opaque Python node call. The current
compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`.
This design defines the core runtime shape. The boundary model is implemented;
child execution, interruption, and saved-workflow resolution remain planned.
This design defines the core runtime shape. The boundary model and
non-interrupting prepared-child execution are implemented; interruption and
saved-workflow resolution remain planned.
## Goals
@@ -76,8 +77,11 @@ declare terminal outcomes through `Workflow.outcomes` and `EndNode`.
native boundary, while artifact helpers convert saved/capability workflow
references into core `WorkflowRef` values.
Runtime execution is deliberately not implemented: stepping a `SubgraphNode`
fails explicitly until the next slice adds child scope/frame execution.
Runtime execution now accepts caller-supplied `PreparedSubgraph` dependencies
for local workflow refs. It creates a child scope/lineage, schedules child
frames in the parent run, retains child trace entries, and applies mapped
output only at boundary completion. Saved artifact refs are not loaded by
`wf_core`, and child interrupts fail explicitly until resume routing exists.
`WorkflowRef` should be structural, not a dotted string parser:
@@ -325,11 +329,11 @@ child = parent.subgraph(
```
This copies the compiled child workflow contract into a core `SubgraphNode`,
appends it to the builder, and returns the step for normal routing. It does not
make the child executable yet. The core `workflow` field is structural, but
higher layers still need dependency resolution before saved/deployed workflow
refs can run. The lower-level `subgraph_ref(...)` helper exists for code that
wants only the core step object.
appends it to the builder, and returns the step for normal routing. Runtime
execution requires the child graph and its handlers to be supplied as a
`PreparedSubgraph`; higher layers still need dependency resolution before
saved/deployed workflow refs can run. The lower-level `subgraph_ref(...)`
helper exists for code that wants only the core step object.
Possible API:
@@ -389,18 +393,18 @@ selection out of `wf_core`.
- Artifact conversion helpers bridge saved workflow identities to core
`WorkflowRef` values.
### Slice 1: Non-Interrupting Inline Subgraph Runtime
### Completed Slice 1: Non-Interrupting Prepared Subgraph Runtime
- Resolve local/prepared child `WorkflowRef` dependencies at runtime; do not
load saved artifacts inside `wf_core`.
- Execute child workflow to completion through child frames.
- Give the child an explicit runtime scope/lineage so child state is isolated
- Local/prepared child `WorkflowRef` dependencies resolve through
`PreparedSubgraph`; `wf_core` does not load saved artifacts.
- Child workflows execute through child frames in the parent scheduler.
- Each activation owns a child runtime scope/lineage so child state is isolated
from parent state until boundary completion.
- Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code.
- Route the parent step through the child's terminal `RunState.outcome`.
- Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent.
- Child trace entries remain in the parent run with child frame ids; completion
records the parent `subgraph` trace entry.
- Child output maps to parent state through existing output binding machinery.
- The parent step routes through the child's terminal workflow outcome.
- Child interrupts reject explicitly until structural resume routing exists.
### Slice 2: Interrupt Bubbling and Resume
@@ -450,12 +454,10 @@ selection out of `wf_core`.
## Recommendation
The typed boundary scaffold is complete. Start runtime work with Slice 1 as a
non-interrupting inline/prepared subgraph. It gives us native trace/frame and
scope/lineage semantics without taking on the hardest resume problem
immediately. Do not delete the wrapper-node helpers yet; use them as
compatibility and examples while native subgraphs mature.
The typed boundary scaffold and non-interrupting prepared-child runtime are
complete. Do not delete the wrapper-node helpers yet; use them as compatibility
and examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but
break at exactly the moment users need persistence and resume.
Implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable
but break at exactly the moment users need persistence and resume.
+14 -11
View File
@@ -127,30 +127,33 @@ limits and intended adapter seam.
- Foreach supports serial and concurrent execution. Concurrent foreach uses
explicit policy, typed barrier state, lineage-aware patch commits, item error
policy, and quiescent interrupt handling. Remaining gaps are higher-level
graph constructs such as native subgraphs, explicit fork/gather nodes, and
advanced conflict strategies beyond exact-path mergeable reducers.
graph constructs such as explicit fork/gather nodes and advanced conflict
strategies beyond exact-path mergeable reducers.
- Interrupt lifecycle is still node-level and run-state-level. Long-lived
external subscriptions or notification streams need a separate lifecycle
design. Interrupt `request` and `resume` are canonical binding lists; nested
child-workflow resume is still future work.
- Native subgraphs have a core model placeholder, `SubgraphNode`, but runtime
execution is not implemented yet. The placeholder carries a child workflow
reference, declared input/output schemas, binding lists, and declared
outcomes so parent graph structure can validate before execution support
lands.
- Native subgraphs use `SubgraphNode` plus caller-supplied `PreparedSubgraph`
dependencies. A prepared local child executes through a child runtime scope
and lineage; child output commits only through declared boundary bindings and
the parent routes by the child's terminal workflow outcome. Saved/deployed
workflow resolution remains outside core and is not implemented at this
boundary yet.
- Nested subgraph interruption is not first-class yet. The current
`wf_authoring` subgraph helpers wrap a child workflow as an ordinary sync or
async node and validate the child output; they do not preserve a child run
state that can interrupt, bubble to the parent, and later resume inside the
child. See `examples/authoring_workflow_as_node.py` for the current
wrapper-node shape.
child. Native `SubgraphNode` preserves prepared-child execution and trace,
but rejects child interrupts until route-aware resume is implemented. See
`examples/authoring_workflow_as_node.py` for the wrapper-node shape.
- Saved workflow-as-node execution with interrupts requires a core runtime
upgrade: nested run state, child-frame trace preservation, interrupt bubbling
with path metadata, and resume back into the child workflow.
- Frames are no longer only a serial execution stack: the runtime has a ready
queue, `BLOCKED` frame state, lineage isolation, barrier merge semantics, and
pending child results for concurrent foreach. Native subgraphs still need
explicit child workflow/deployment identity and child-scope execution.
pending child results for concurrent foreach. Native prepared subgraphs now
use child-scope execution; saved/deployed child resolution and nested
interruption remain outstanding.
Concurrent foreach is the primary current use case for async concurrent node
handler execution.
- Runtime errors are still ordinary exceptions plus failed run status. A richer