native subgraph execution

This commit is contained in:
lda
2026-05-25 04:28:06 +07:00 Verified
parent 3880a86c26
commit a4eeb506be
20 changed files with 746 additions and 146 deletions
+10 -11
View File
@@ -49,13 +49,12 @@ implementation state.
scaffolding slice is complete: core has `SubgraphNode`, structural
`WorkflowRef`, workflow-level outcomes plus explicit `EndNode` termination,
authoring helpers (`subgraph_ref` / `WorkflowBuilder.subgraph`), and artifact
reference conversion helpers. Runtime subgraph execution is still absent.
The next slice is non-interrupting child execution: resolve a prepared child
workflow, create a child scope/lineage, preserve child trace, map child
output back through the subgraph boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution follow
after that. Wrapper helpers currently run child workflows as ordinary nodes;
true graph-as-node behavior belongs here.
reference conversion helpers. Core can now execute a prepared local child
workflow through an isolated child scope/lineage, preserve its trace entries,
map child output through the boundary, and route by the child's terminal
outcome. Interrupt bubbling/resume and saved/deployed child resolution remain
next. Wrapper helpers currently run child workflows as ordinary nodes; native
`SubgraphNode` is now the graph-as-node path for prepared children.
- **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -98,7 +97,7 @@ Frame stress points remaining for native subgraphs and future fork/gather:
The MCP workflow authoring path is now usable enough for real testing. The next
bottleneck is runtime/platform correctness: resumable child execution,
native subgraph execution, persistent run history, and protocol-native
progress reporting. Concurrent foreach supplies scheduler/lineage precedent;
native child graphs are now the missing runtime boundary. Those pieces should
come before adding more high-level authoring sugar.
saved/deployed child resolution, persistent run history, and protocol-native
progress reporting. Concurrent foreach and prepared native child execution now
supply scheduler/lineage precedent. Those remaining pieces should come before
adding more high-level authoring sugar.
@@ -33,18 +33,17 @@ the compatibility subset needed before native subgraphs:
`ForeachBarrierState` now keeps scheduling/result metadata plus compatibility
patches for old serialized barrier data.
Direct commits currently go through `is_root_lineage_frame(frame)`, which is the
migration shortcut for root scope/root lineage. The eventual better shape is an
explicit scope/lineage commit target, feasible once native subgraph completion
can declare whether child writes commit to child scope, parent lineage, or only
through boundary output bindings.
Direct commits now go through a scope-root commit decision: top-level frames
commit to root state, and prepared native-child root frames commit to their
child scope state. Descendant item/branch lineages still buffer writes until a
barrier or future gather commits them.
Remaining work should avoid jumping straight into a broad rewrite. Native
subgraph scaffolding is now present (`SubgraphNode`, structural `WorkflowRef`,
terminal workflow outcomes, and authoring helpers). The next runtime slice can
execute a non-interrupting prepared child graph using the current scope/lineage
primitives; interrupt bubbling and saved/deployed workflow resolution remain
later work.
subgraph scaffolding and non-interrupting prepared-child execution are now
present (`SubgraphNode`, structural `WorkflowRef`, terminal workflow outcomes,
authoring helpers, and `PreparedSubgraph`). Child graphs execute through their
own scope/lineage and map output back at completion. Interrupt bubbling and
saved/deployed workflow resolution remain later work.
---
@@ -34,11 +34,10 @@ The full native subgraph use of `RuntimeScope` is not implemented yet.
`ForeachBarrierState` still owns scheduling/barrier metadata, but no longer has
to be the primary write store for new concurrent foreach item results.
Direct node commits currently use the explicit root-frame helper
`is_root_lineage_frame(frame)`. That helper still means "root scope plus root
lineage" during migration. The better long-term shape becomes feasible when
native subgraph completion exists: direct commits should be decided by an
explicit scope/lineage commit target, not by root ids.
Direct node commits now use a scope-root decision: the top-level root commits
to `RunState.state`, while a prepared native-subgraph root commits to its child
scope state. Descendant item/branch lineages buffer writes until a barrier or
future gather commits them.
## Problem
@@ -1,6 +1,6 @@
# Native Subgraphs Design
Status: scaffolding implemented; runtime execution planned
Status: prepared-child execution implemented; interrupts/artifact resolution planned
Native subgraphs should make a workflow usable as a workflow step without
collapsing the child run into one opaque Python node call. The current
@@ -8,8 +8,9 @@ collapsing the child run into one opaque Python node call. The current
compatibility wrappers, but they hide the child trace, child frames, and child
interrupt lifecycle from `wf_core`.
This design defines the core runtime shape. The boundary model is implemented;
child execution, interruption, and saved-workflow resolution remain planned.
This design defines the core runtime shape. The boundary model and
non-interrupting prepared-child execution are implemented; interruption and
saved-workflow resolution remain planned.
## Goals
@@ -76,8 +77,11 @@ declare terminal outcomes through `Workflow.outcomes` and `EndNode`.
native boundary, while artifact helpers convert saved/capability workflow
references into core `WorkflowRef` values.
Runtime execution is deliberately not implemented: stepping a `SubgraphNode`
fails explicitly until the next slice adds child scope/frame execution.
Runtime execution now accepts caller-supplied `PreparedSubgraph` dependencies
for local workflow refs. It creates a child scope/lineage, schedules child
frames in the parent run, retains child trace entries, and applies mapped
output only at boundary completion. Saved artifact refs are not loaded by
`wf_core`, and child interrupts fail explicitly until resume routing exists.
`WorkflowRef` should be structural, not a dotted string parser:
@@ -325,11 +329,11 @@ child = parent.subgraph(
```
This copies the compiled child workflow contract into a core `SubgraphNode`,
appends it to the builder, and returns the step for normal routing. It does not
make the child executable yet. The core `workflow` field is structural, but
higher layers still need dependency resolution before saved/deployed workflow
refs can run. The lower-level `subgraph_ref(...)` helper exists for code that
wants only the core step object.
appends it to the builder, and returns the step for normal routing. Runtime
execution requires the child graph and its handlers to be supplied as a
`PreparedSubgraph`; higher layers still need dependency resolution before
saved/deployed workflow refs can run. The lower-level `subgraph_ref(...)`
helper exists for code that wants only the core step object.
Possible API:
@@ -389,18 +393,18 @@ selection out of `wf_core`.
- Artifact conversion helpers bridge saved workflow identities to core
`WorkflowRef` values.
### Slice 1: Non-Interrupting Inline Subgraph Runtime
### Completed Slice 1: Non-Interrupting Prepared Subgraph Runtime
- Resolve local/prepared child `WorkflowRef` dependencies at runtime; do not
load saved artifacts inside `wf_core`.
- Execute child workflow to completion through child frames.
- Give the child an explicit runtime scope/lineage so child state is isolated
- Local/prepared child `WorkflowRef` dependencies resolve through
`PreparedSubgraph`; `wf_core` does not load saved artifacts.
- Child workflows execute through child frames in the parent scheduler.
- Each activation owns a child runtime scope/lineage so child state is isolated
from parent state until boundary completion.
- Preserve child trace in a clearly-owned form.
- Apply child output to parent state through existing output binding code.
- Route the parent step through the child's terminal `RunState.outcome`.
- Tests: child output mapping, child internal trace visibility, parent trace
shape, child runtime failure fails parent.
- Child trace entries remain in the parent run with child frame ids; completion
records the parent `subgraph` trace entry.
- Child output maps to parent state through existing output binding machinery.
- The parent step routes through the child's terminal workflow outcome.
- Child interrupts reject explicitly until structural resume routing exists.
### Slice 2: Interrupt Bubbling and Resume
@@ -450,12 +454,10 @@ selection out of `wf_core`.
## Recommendation
The typed boundary scaffold is complete. Start runtime work with Slice 1 as a
non-interrupting inline/prepared subgraph. It gives us native trace/frame and
scope/lineage semantics without taking on the hardest resume problem
immediately. Do not delete the wrapper-node helpers yet; use them as
compatibility and examples while native subgraphs mature.
The typed boundary scaffold and non-interrupting prepared-child runtime are
complete. Do not delete the wrapper-node helpers yet; use them as compatibility
and examples while native subgraphs mature.
Then implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable but
break at exactly the moment users need persistence and resume.
Implement Slice 2 before exposing saved workflows as broadly reusable child
graphs. Saved workflows without nested interrupt support would look reusable
but break at exactly the moment users need persistence and resume.
+14 -11
View File
@@ -127,30 +127,33 @@ limits and intended adapter seam.
- Foreach supports serial and concurrent execution. Concurrent foreach uses
explicit policy, typed barrier state, lineage-aware patch commits, item error
policy, and quiescent interrupt handling. Remaining gaps are higher-level
graph constructs such as native subgraphs, explicit fork/gather nodes, and
advanced conflict strategies beyond exact-path mergeable reducers.
graph constructs such as explicit fork/gather nodes and advanced conflict
strategies beyond exact-path mergeable reducers.
- Interrupt lifecycle is still node-level and run-state-level. Long-lived
external subscriptions or notification streams need a separate lifecycle
design. Interrupt `request` and `resume` are canonical binding lists; nested
child-workflow resume is still future work.
- Native subgraphs have a core model placeholder, `SubgraphNode`, but runtime
execution is not implemented yet. The placeholder carries a child workflow
reference, declared input/output schemas, binding lists, and declared
outcomes so parent graph structure can validate before execution support
lands.
- Native subgraphs use `SubgraphNode` plus caller-supplied `PreparedSubgraph`
dependencies. A prepared local child executes through a child runtime scope
and lineage; child output commits only through declared boundary bindings and
the parent routes by the child's terminal workflow outcome. Saved/deployed
workflow resolution remains outside core and is not implemented at this
boundary yet.
- Nested subgraph interruption is not first-class yet. The current
`wf_authoring` subgraph helpers wrap a child workflow as an ordinary sync or
async node and validate the child output; they do not preserve a child run
state that can interrupt, bubble to the parent, and later resume inside the
child. See `examples/authoring_workflow_as_node.py` for the current
wrapper-node shape.
child. Native `SubgraphNode` preserves prepared-child execution and trace,
but rejects child interrupts until route-aware resume is implemented. See
`examples/authoring_workflow_as_node.py` for the wrapper-node shape.
- Saved workflow-as-node execution with interrupts requires a core runtime
upgrade: nested run state, child-frame trace preservation, interrupt bubbling
with path metadata, and resume back into the child workflow.
- Frames are no longer only a serial execution stack: the runtime has a ready
queue, `BLOCKED` frame state, lineage isolation, barrier merge semantics, and
pending child results for concurrent foreach. Native subgraphs still need
explicit child workflow/deployment identity and child-scope execution.
pending child results for concurrent foreach. Native prepared subgraphs now
use child-scope execution; saved/deployed child resolution and nested
interruption remain outstanding.
Concurrent foreach is the primary current use case for async concurrent node
handler execution.
- Runtime errors are still ordinary exceptions plus failed run status. A richer
+2
View File
@@ -24,6 +24,7 @@ from .models import (
from .runtime import (
AsyncNodeHandler,
NodeHandler,
PreparedSubgraph,
WorkflowExecutionError,
coerce_node_result,
execute_workflow_async,
@@ -72,6 +73,7 @@ __all__ = [
"SubgraphNode",
"AsyncNodeHandler",
"NodeHandler",
"PreparedSubgraph",
"ExecutionFrame",
"FrameStatus",
"RunState",
+6 -7
View File
@@ -158,12 +158,11 @@ class NodeUse(BaseModel):
class SubgraphNode(BaseModel):
"""Workflow boundary step reserved for native subgraph execution.
"""Workflow boundary step for native prepared-child execution.
This is a contract-bearing placeholder, not the implementation of nested
workflow execution yet. The core can validate the parent graph's bindings
and declared outcomes now; a later runtime slice will resolve ``workflow``
into a child graph, create a child scope/lineage, and commit its result.
The runtime can execute an already-prepared local child graph through a
child scope/lineage and commit only its mapped boundary output. Resolving
saved artifacts and resuming child interrupts remain platform/runtime work.
"""
id: str
@@ -171,8 +170,8 @@ class SubgraphNode(BaseModel):
workflow: WorkflowRef = Field(
description=(
"Reference to the child workflow artifact or registry key. The core "
"does not resolve this reference until native subgraph runtime "
"execution is implemented."
"executes local references only when a PreparedSubgraph dependency "
"is supplied; it does not load saved artifacts."
)
)
desc: str | None = None
+3
View File
@@ -5,6 +5,7 @@ from enum import StrEnum
from typing import Any
from wf_core.models.reducers import ReducerRef
from wf_core.models.workflow_refs import WorkflowRef
from wf_core.paths import StatePath
ROOT_SCOPE_ID = "root"
@@ -54,7 +55,9 @@ class RuntimeScope:
id: str
workflow_name: str
workflow_input: dict[str, Any] = field(default_factory=dict)
committed_state: dict[str, Any] = field(default_factory=dict)
workflow_ref: WorkflowRef | None = None
@dataclass(slots=True)
+2
View File
@@ -13,6 +13,7 @@ from .engine import (
resume_workflow,
resume_workflow_async,
)
from .subgraphs import PreparedSubgraph
from .step import complete_step, step_workflow, step_workflow_async
__all__ = [
@@ -29,4 +30,5 @@ __all__ = [
"resume_workflow_async",
"step_workflow",
"step_workflow_async",
"PreparedSubgraph",
]
+85 -13
View File
@@ -3,17 +3,19 @@ from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from wf_core.errors import WorkflowExecutionError
from wf_core.models.workflow import Workflow
from wf_core.runtime.ops.flow import finalize_run
from wf_core.runtime.ops.merges import ReducerDefinition
from wf_core.runtime.ops.nodes import AsyncNodeHandler, NodeHandler
from wf_core.runtime.ops.runs import create_run_state
from wf_core.runtime.scheduler import resolve_no_ready_frames, select_next_frame
from wf_core.run_state import RunState, RunStatus
from wf_core.run_state import ROOT_SCOPE_ID, RunState, RunStatus
from wf_core.tokens import END
from .preparation import prepare_new_run, prepare_resume
from .step import step_workflow, step_workflow_async
from .subgraphs import PreparedSubgraph, resolve_prepared_subgraph
def execute_workflow(
@@ -22,13 +24,20 @@ def execute_workflow(
registry: Mapping[str, NodeHandler],
*,
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[NodeHandler]] | None = None,
) -> RunState:
"""Create a run and execute a workflow synchronously until it stops."""
run = create_run_state(workflow, workflow_input)
try:
prepare_new_run(workflow, workflow_input, run)
return resume_workflow(workflow, run, registry, reducers=reducers)
return resume_workflow(
workflow,
run,
registry,
reducers=reducers,
subgraphs=subgraphs,
)
except Exception as exc:
run.status = RunStatus.FAILED
run.error = str(exc)
@@ -41,13 +50,20 @@ async def execute_workflow_async(
registry: Mapping[str, AsyncNodeHandler],
*,
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[AsyncNodeHandler]] | None = None,
) -> RunState:
"""Create a run and execute a workflow asynchronously until it stops."""
run = create_run_state(workflow, workflow_input)
try:
prepare_new_run(workflow, workflow_input, run)
return await resume_workflow_async(workflow, run, registry, reducers=reducers)
return await resume_workflow_async(
workflow,
run,
registry,
reducers=reducers,
subgraphs=subgraphs,
)
except Exception as exc:
run.status = RunStatus.FAILED
run.error = str(exc)
@@ -62,6 +78,7 @@ def resume_workflow(
resume_payload: dict[str, Any] | None = None,
resume_outcome: str = "submitted",
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[NodeHandler]] | None = None,
) -> RunState:
"""Resume a synchronous run from its current state."""
index = prepare_resume(
@@ -77,17 +94,22 @@ def resume_workflow(
return run
while True:
if select_next_frame(run) is None:
frame = select_next_frame(run)
if frame is None:
status = resolve_no_ready_frames(run)
if status == RunStatus.COMPLETED:
break
return run
active_workflow, active_registry, active_reducers = _sync_execution_target(
workflow, registry, reducers, run, subgraphs
)
step_workflow(
workflow,
active_workflow,
run,
registry,
index=index,
reducers=reducers,
active_registry,
index=index if frame.scope_id == ROOT_SCOPE_ID else None,
reducers=active_reducers,
subgraphs=subgraphs,
)
if run.status == RunStatus.INTERRUPTED:
return run
@@ -103,6 +125,7 @@ async def resume_workflow_async(
resume_payload: dict[str, Any] | None = None,
resume_outcome: str = "submitted",
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[AsyncNodeHandler]] | None = None,
) -> RunState:
"""Resume an async run from its current state."""
index = prepare_resume(
@@ -118,19 +141,68 @@ async def resume_workflow_async(
return run
while True:
if select_next_frame(run) is None:
frame = select_next_frame(run)
if frame is None:
status = resolve_no_ready_frames(run)
if status == RunStatus.COMPLETED:
break
return run
active_workflow, active_registry, active_reducers = _async_execution_target(
workflow, registry, reducers, run, subgraphs
)
await step_workflow_async(
workflow,
active_workflow,
run,
registry,
index=index,
reducers=reducers,
active_registry,
index=index if frame.scope_id == ROOT_SCOPE_ID else None,
reducers=active_reducers,
subgraphs=subgraphs,
)
if run.status == RunStatus.INTERRUPTED:
return run
return finalize_run(workflow, run)
def _sync_execution_target(
root_workflow: Workflow,
root_registry: Mapping[str, NodeHandler],
root_reducers: Mapping[str, ReducerDefinition] | None,
run: RunState,
subgraphs: Mapping[str, PreparedSubgraph[NodeHandler]] | None,
) -> tuple[Workflow, Mapping[str, NodeHandler], Mapping[str, ReducerDefinition] | None]:
"""Return the workflow dependencies owned by the selected frame scope."""
frame = run.current_frame()
if frame.scope_id == ROOT_SCOPE_ID:
return root_workflow, root_registry, root_reducers
scope = run.scopes.get(frame.scope_id)
if scope is None or scope.workflow_ref is None:
raise WorkflowExecutionError(
f"child frame {frame.id!r} has no prepared workflow scope"
)
child = resolve_prepared_subgraph(scope.workflow_ref, subgraphs)
return child.workflow, child.registry, child.reducers
def _async_execution_target(
root_workflow: Workflow,
root_registry: Mapping[str, AsyncNodeHandler],
root_reducers: Mapping[str, ReducerDefinition] | None,
run: RunState,
subgraphs: Mapping[str, PreparedSubgraph[AsyncNodeHandler]] | None,
) -> tuple[
Workflow,
Mapping[str, AsyncNodeHandler],
Mapping[str, ReducerDefinition] | None,
]:
"""Return async workflow dependencies owned by the selected frame scope."""
frame = run.current_frame()
if frame.scope_id == ROOT_SCOPE_ID:
return root_workflow, root_registry, root_reducers
scope = run.scopes.get(frame.scope_id)
if scope is None or scope.workflow_ref is None:
raise WorkflowExecutionError(
f"child frame {frame.id!r} has no prepared workflow scope"
)
child = resolve_prepared_subgraph(scope.workflow_ref, subgraphs)
return child.workflow, child.registry, child.reducers
+35 -8
View File
@@ -7,10 +7,9 @@ from typing import Any
from wf_core.errors import WorkflowExecutionError
from wf_core.run_state import ExecutionFrame, LineageState, RunState, StateWrite
from wf_core.run_state import ROOT_LINEAGE_ID, ROOT_SCOPE_ID
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
from wf_core.runtime.ops.state import StatePatch
from wf_core.runtime.ops.state import safe_set_nested_value
from wf_core.runtime.ops.state import commit_state_patch, safe_set_nested_value
@dataclass(slots=True)
@@ -79,14 +78,34 @@ def lineage_writes_for_frame(
return pending.patch.writes
def is_root_lineage_frame(frame: ExecutionFrame) -> bool:
"""Return whether a frame currently commits directly to root run state.
def is_scope_root_lineage_frame(run: RunState, frame: ExecutionFrame) -> bool:
"""Return whether writes from this frame commit to its scope state root."""
lineage = run.lineages.get(frame.lineage_id)
return (
lineage is not None
and lineage.scope_id == frame.scope_id
and lineage.parent_id is None
)
This is a migration shortcut, not the final commit policy. Once native
subgraphs can complete, direct commits should be decided by an explicit
scope/lineage commit target rather than only by root ids.
def commit_patch_for_frame(
run: RunState, frame: ExecutionFrame, patch: StatePatch
) -> dict[str, Any]:
"""Commit at a scope root or buffer writes in the frame lineage.
Child workflow root frames own a committed child-state root just like the
top-level root frame owns `RunState.state`. Descendant branch/item frames
remain isolated until an explicit barrier or future gather commits them.
"""
return frame.scope_id == ROOT_SCOPE_ID and frame.lineage_id == ROOT_LINEAGE_ID
if is_scope_root_lineage_frame(run, frame):
return commit_state_patch(scope_state_for_frame(run, frame), patch)
append_lineage_writes(
run,
scope_id=frame.scope_id,
lineage_id=frame.lineage_id,
writes=patch.writes,
)
return {}
def scope_state_for_frame(run: RunState, frame: ExecutionFrame) -> dict[str, Any]:
@@ -97,6 +116,14 @@ def scope_state_for_frame(run: RunState, frame: ExecutionFrame) -> dict[str, Any
return scope.committed_state
def scope_input_for_frame(run: RunState, frame: ExecutionFrame) -> dict[str, Any]:
"""Return the invocation input associated with the frame's workflow scope."""
scope = run.scopes.get(frame.scope_id)
if scope is None:
raise ValueError(f"unknown scope {frame.scope_id!r}")
return scope.workflow_input
def add_lineage(
run: RunState,
*,
+4
View File
@@ -80,6 +80,10 @@ def advance_frame(
frame.activated_incoming_edge = frame.node_id
frame.node_id = next_node_id
if next_node_id == END:
if frame.kind in {"workflow", "subgraph_root"}:
# Legacy terminal routing emits the workflow-level `ok` outcome.
# Explicit EndNode execution stores its declared outcome first.
frame.metadata.setdefault("workflow_outcome", "ok")
frame.status = FrameStatus.COMPLETED
frame.finished_at_node_id = END
wake_parent_for_child_progress(run, frame.id)
+7 -16
View File
@@ -14,18 +14,18 @@ from wf_core.runtime.foreach_state import (
)
from wf_core.runtime.lineage import (
add_lineage,
append_lineage_writes,
is_root_lineage_frame,
commit_patch_for_frame,
lineage_patch,
scope_input_for_frame,
)
from wf_core.runtime.ops.flow import advance_frame, append_step_result_trace
from wf_core.runtime.ops.frames import frame_context_values
from wf_core.runtime.ops.index import WorkflowIndex
from wf_core.runtime.ops.merges import ReducerDefinition
from wf_core.runtime.ops.overlays import state_view_for_frame
from wf_core.runtime.ops.state import (
StatePatch,
build_barrier_patch,
commit_state_patch,
)
from wf_core.runtime.scheduler import (
ForeachIterationMetadata,
@@ -182,8 +182,8 @@ def _resolve_foreach_iterable(
) -> list[object]:
iterable = safe_resolve_path(
str(step.over),
state=run.state,
workflow_input=run.workflow_input,
state=state_view_for_frame(run, frame),
workflow_input=scope_input_for_frame(run, frame),
context=frame_context_values(frame),
)
if not isinstance(iterable, list):
@@ -346,19 +346,10 @@ def _finish_concurrent_foreach(
combined = build_barrier_patch(
workflow,
item_patches,
run.state,
state_view_for_frame(run, frame),
reducers=reducers,
)
if is_root_lineage_frame(frame):
state_changes = commit_state_patch(run.state, combined)
else:
append_lineage_writes(
run,
scope_id=frame.scope_id,
lineage_id=frame.lineage_id,
writes=combined.writes,
)
state_changes = {}
state_changes = commit_patch_for_frame(run, frame, combined)
append_step_result_trace(
run,
frame_id=frame.id,
+4 -2
View File
@@ -3,9 +3,11 @@ from __future__ import annotations
from wf_core.conditions import eval_condition
from wf_core.models.steps import ConditionNode, InterruptNode
from wf_core.run_state import FrameStatus, RunState, RunStatus, StepExecutionResult
from wf_core.runtime.lineage import scope_input_for_frame
from wf_core.runtime.ops.flow import append_trace
from wf_core.runtime.ops.frames import frame_context_values
from wf_core.runtime.ops.interrupts import build_interrupt_request
from wf_core.runtime.ops.overlays import state_view_for_frame
def handle_condition_step(
@@ -15,8 +17,8 @@ def handle_condition_step(
frame = run.current_frame()
predicate = eval_condition(
step.check,
run.state,
run.workflow_input,
state_view_for_frame(run, frame),
scope_input_for_frame(run, frame),
frame.prior_outcome,
)
outcome = "true" if predicate else "false"
+9 -16
View File
@@ -18,12 +18,16 @@ from wf_core.run_state import (
StepExecutionResult,
)
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
from wf_core.runtime.lineage import append_lineage_writes, is_root_lineage_frame
from wf_core.runtime.lineage import (
append_lineage_writes,
commit_patch_for_frame,
scope_input_for_frame,
)
from wf_core.runtime.ops.frames import frame_context_values
from wf_core.runtime.ops.merges import ReducerDefinition
from wf_core.runtime.ops.overlays import state_view_for_frame
from wf_core.runtime.ops.schemas import validate_payload_against_schema
from wf_core.runtime.ops.state import StatePatch, build_output_patch, commit_state_patch
from wf_core.runtime.ops.state import StatePatch, build_output_patch
NodeHandler = Callable[[dict[str, Any], RuntimeContext], NodeResult | dict[str, Any]]
AsyncNodeHandler = Callable[
@@ -62,7 +66,7 @@ def _resolve_node_execution(
value = safe_resolve_path(
str(binding.path),
state=state_view,
workflow_input=run.workflow_input,
workflow_input=scope_input_for_frame(run, frame),
context=context_values,
)
else:
@@ -121,18 +125,7 @@ def _finalize_node_execution(
)
owner = item_frame_owner(frame)
if owner is None:
if is_root_lineage_frame(frame):
state_changes = commit_state_patch(run.state, patch)
else:
# Non-root frames are future subgraph/fork branch execution: writes
# become lineage-local until an explicit boundary/barrier commits.
append_lineage_writes(
run,
scope_id=frame.scope_id,
lineage_id=frame.lineage_id,
writes=patch.writes,
)
state_changes = {}
state_changes = commit_patch_for_frame(run, frame, patch)
else:
parent_frame_id, foreach_node_id, item_index = owner
parent_frame = run.frames[parent_frame_id]
@@ -155,7 +148,7 @@ def _finalize_node_execution(
barrier.save_to_frame(parent_frame, foreach_node_id)
state_changes = {}
else:
state_changes = commit_state_patch(run.state, patch)
state_changes = commit_patch_for_frame(run, parent_frame, patch)
return StepExecutionResult(
outcome=result.outcome,
resolved_input=resolved_input,
+10 -1
View File
@@ -18,12 +18,20 @@ from wf_core.run_state import (
from wf_core.runtime.scheduler import add_frame
def create_run_state(workflow: Workflow, workflow_input: dict[str, object]) -> RunState:
def initial_state(
workflow: Workflow, workflow_input: dict[str, object]
) -> dict[str, object]:
"""Create one scope's committed state from defaults plus workflow input."""
state: dict[str, object] = {}
for field in workflow.state_schema.fields:
if field.default is not None:
set_nested_value(state, list(field.path.parts), deepcopy(field.default))
state.update(dict(workflow_input))
return state
def create_run_state(workflow: Workflow, workflow_input: dict[str, object]) -> RunState:
state = initial_state(workflow, workflow_input)
run = RunState(
workflow_name=workflow.name,
status=RunStatus.PENDING,
@@ -33,6 +41,7 @@ def create_run_state(workflow: Workflow, workflow_input: dict[str, object]) -> R
ROOT_SCOPE_ID: RuntimeScope(
id=ROOT_SCOPE_ID,
workflow_name=workflow.name,
workflow_input=dict(workflow_input),
committed_state=state,
)
},
+33 -8
View File
@@ -38,7 +38,9 @@ from wf_core.runtime.scheduler import (
select_next_frame,
wake_parent_for_child_progress,
)
from wf_core.runtime.subgraphs import PreparedSubgraph, step_subgraph
from wf_core.run_state import ExecutionFrame, FrameStatus, RunState, StepExecutionResult
from wf_core.run_state import ROOT_SCOPE_ID
from wf_core.tokens import END
from .preparation import prepare_step
@@ -85,7 +87,10 @@ def complete_end_step(
) -> RunState:
"""Record an explicit workflow terminal and complete the active frame."""
result = StepExecutionResult(outcome=outcome)
run.outcome = outcome
frame = run.frames[frame_id]
frame.metadata["workflow_outcome"] = outcome
if frame.parent_frame_id is None:
run.outcome = outcome
append_step_result_trace(
run,
frame_id=frame_id,
@@ -96,7 +101,7 @@ def complete_end_step(
)
advance_frame(
run,
run.frames[frame_id],
frame,
outcome=outcome,
next_node_id=END,
)
@@ -110,6 +115,7 @@ def step_workflow(
*,
index: WorkflowIndex | None = None,
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[NodeHandler]] | None = None,
) -> RunState:
"""Execute at most one synchronous workflow step."""
frame = run.current_frame() if run.current_frame_id is not None else None
@@ -149,14 +155,23 @@ def step_workflow(
outcome=step.outcome,
)
elif isinstance(step, InterruptNode):
if frame.kind == "subgraph_root" or frame.scope_id != ROOT_SCOPE_ID:
raise WorkflowExecutionError(
"child interrupts are not supported until native subgraph resume routing exists"
)
return handle_interrupt_step(run, step)
elif isinstance(step, ForeachNode):
return step_foreach(workflow, run, step, index, reducers=reducers)
elif isinstance(step, SubgraphNode):
raise WorkflowExecutionError(
f"subgraph step {step.id!r} references {step.workflow!r}, "
"but native subgraph execution is not implemented yet"
step_result = step_subgraph(
workflow,
run,
step,
subgraphs=subgraphs,
reducers=reducers,
)
if step_result is None:
return run
else:
raise WorkflowExecutionError(
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
@@ -205,6 +220,7 @@ async def step_workflow_async(
*,
index: WorkflowIndex | None = None,
reducers: Mapping[str, ReducerDefinition] | None = None,
subgraphs: Mapping[str, PreparedSubgraph[AsyncNodeHandler]] | None = None,
) -> RunState:
"""Execute at most one async workflow step."""
frame = run.current_frame() if run.current_frame_id is not None else None
@@ -255,14 +271,23 @@ async def step_workflow_async(
outcome=step.outcome,
)
elif isinstance(step, InterruptNode):
if frame.kind == "subgraph_root" or frame.scope_id != ROOT_SCOPE_ID:
raise WorkflowExecutionError(
"child interrupts are not supported until native subgraph resume routing exists"
)
return handle_interrupt_step(run, step)
elif isinstance(step, ForeachNode):
return step_foreach(workflow, run, step, index, reducers=reducers)
elif isinstance(step, SubgraphNode):
raise WorkflowExecutionError(
f"subgraph step {step.id!r} references {step.workflow!r}, "
"but native subgraph execution is not implemented yet"
step_result = step_subgraph(
workflow,
run,
step,
subgraphs=subgraphs,
reducers=reducers,
)
if step_result is None:
return run
else:
raise WorkflowExecutionError(
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
+273
View File
@@ -0,0 +1,273 @@
from __future__ import annotations
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from typing import Any, Generic, TypeVar
from wf_core.conditions import safe_resolve_path
from wf_core.errors import WorkflowExecutionError
from wf_core.local_paths import LocalPathError, set_local_value
from wf_core.models.steps import (
InputBinding,
InputPathBinding,
InputValueBinding,
SubgraphNode,
)
from wf_core.models.workflow import Workflow
from wf_core.models.workflow_refs import WorkflowRef
from wf_core.run_state import (
ExecutionFrame,
FrameStatus,
LineageState,
RunState,
RuntimeScope,
StepExecutionResult,
)
from wf_core.runtime.lineage import commit_patch_for_frame
from wf_core.runtime.ops.frames import frame_context_values
from wf_core.runtime.ops.merges import ReducerDefinition
from wf_core.runtime.ops.overlays import state_view_for_frame
from wf_core.runtime.ops.runs import initial_state
from wf_core.runtime.ops.schemas import validate_payload_against_schema
from wf_core.runtime.ops.state import build_output_patch, project_output
from wf_core.runtime.scheduler import add_frame, block_frame_on_children
HandlerT = TypeVar("HandlerT", bound=Callable[..., object])
_ACTIVATION_KEY = "subgraph_activation"
@dataclass(slots=True, frozen=True)
class PreparedSubgraph(Generic[HandlerT]):
"""Executable local child dependency supplied by the caller.
Core owns child execution semantics but does not load artifacts or resolve
deployment/source bindings. Higher layers must resolve those concerns into
this prepared dependency before a run starts.
"""
workflow: Workflow
registry: Mapping[str, HandlerT]
reducers: Mapping[str, ReducerDefinition] | None = None
@dataclass(slots=True, frozen=True)
class SubgraphActivation:
"""Runtime ownership record for one in-flight subgraph boundary."""
workflow_ref: WorkflowRef
scope_id: str
lineage_id: str
child_frame_id: str
child_input: dict[str, Any]
@classmethod
def from_frame(cls, frame: ExecutionFrame) -> SubgraphActivation | None:
raw = frame.metadata.get(_ACTIVATION_KEY)
if raw is None:
return None
if not isinstance(raw, Mapping):
raise WorkflowExecutionError(
f"malformed subgraph activation for frame {frame.id!r}"
)
try:
return cls(
workflow_ref=WorkflowRef.model_validate(raw["workflow_ref"]),
scope_id=str(raw["scope_id"]),
lineage_id=str(raw["lineage_id"]),
child_frame_id=str(raw["child_frame_id"]),
child_input=dict(raw["child_input"]),
)
except (KeyError, TypeError, ValueError) as exc:
raise WorkflowExecutionError(
f"malformed subgraph activation for frame {frame.id!r}"
) from exc
def save_to_frame(self, frame: ExecutionFrame) -> None:
frame.metadata[_ACTIVATION_KEY] = {
"workflow_ref": self.workflow_ref.model_dump(mode="json"),
"scope_id": self.scope_id,
"lineage_id": self.lineage_id,
"child_frame_id": self.child_frame_id,
"child_input": dict(self.child_input),
}
def resolve_prepared_subgraph(
ref: WorkflowRef,
subgraphs: Mapping[str, PreparedSubgraph[HandlerT]] | None,
) -> PreparedSubgraph[HandlerT]:
"""Resolve a prepared local child; artifact loading is not a core concern."""
if ref.name is None:
raise WorkflowExecutionError(
f"saved child workflow reference {ref.display!r} is not prepared for core execution"
)
prepared = None if subgraphs is None else subgraphs.get(ref.name)
if prepared is None:
raise WorkflowExecutionError(
f"no prepared child workflow registered for {ref.name!r}"
)
return prepared
def resolve_input_bindings(
bindings: Sequence[InputBinding],
*,
state: Mapping[str, Any],
workflow_input: Mapping[str, Any],
context: Mapping[str, Any],
label: str,
) -> dict[str, Any]:
"""Build a local input payload from canonical value/path bindings."""
payload: dict[str, Any] = {}
for binding in bindings:
if isinstance(binding, InputValueBinding):
value = binding.value
elif isinstance(binding, InputPathBinding):
value = safe_resolve_path(
str(binding.path),
state=state,
workflow_input=workflow_input,
context=context,
)
else:
raise WorkflowExecutionError(f"unsupported input binding for {label}")
try:
set_local_value(payload, binding.target, value)
except LocalPathError as exc:
raise WorkflowExecutionError(str(exc)) from exc
return payload
def step_subgraph(
workflow: Workflow,
run: RunState,
step: SubgraphNode,
*,
subgraphs: Mapping[str, PreparedSubgraph[HandlerT]] | None,
reducers: Mapping[str, ReducerDefinition] | None,
) -> StepExecutionResult | None:
"""Start or finish one native child activation.
Returning ``None`` means the parent frame is blocked while child frames run.
Returning a result means child execution completed and the parent boundary
can advance normally through the child's terminal workflow outcome.
"""
frame = run.current_frame()
activation = SubgraphActivation.from_frame(frame)
prepared = resolve_prepared_subgraph(step.workflow, subgraphs)
if activation is None:
_start_subgraph(run, frame, step, prepared)
return None
return _finish_subgraph(workflow, run, frame, step, activation, prepared, reducers)
def _start_subgraph(
run: RunState,
frame: ExecutionFrame,
step: SubgraphNode,
prepared: PreparedSubgraph[HandlerT],
) -> None:
prepared.workflow.validate_structure().raise_for_errors()
parent_scope = run.scopes[frame.scope_id]
child_input = resolve_input_bindings(
step.input,
state=state_view_for_frame(run, frame),
workflow_input=parent_scope.workflow_input,
context=frame_context_values(frame),
label=f"subgraph {step.id!r}",
)
validate_payload_against_schema(
step.input_schema, child_input, f"subgraph input for {step.id}"
)
validate_payload_against_schema(
prepared.workflow.input_schema,
child_input,
f"child workflow input for {step.id}",
)
scope_id = f"{frame.id}:subgraph:{step.id}"
lineage_id = f"{scope_id}:root"
child_frame_id = f"{scope_id}:frame"
if scope_id in run.scopes or lineage_id in run.lineages:
raise WorkflowExecutionError(
f"duplicate subgraph activation identifiers for step {step.id!r}"
)
run.scopes[scope_id] = RuntimeScope(
id=scope_id,
workflow_name=prepared.workflow.name,
workflow_input=dict(child_input),
committed_state=initial_state(prepared.workflow, child_input),
workflow_ref=step.workflow,
)
run.lineages[lineage_id] = LineageState(id=lineage_id, scope_id=scope_id)
add_frame(
run,
ExecutionFrame(
id=child_frame_id,
kind="subgraph_root",
node_id=prepared.workflow.start,
status=FrameStatus.PENDING,
parent_frame_id=frame.id,
scope_id=scope_id,
lineage_id=lineage_id,
),
ready=True,
)
SubgraphActivation(
workflow_ref=step.workflow,
scope_id=scope_id,
lineage_id=lineage_id,
child_frame_id=child_frame_id,
child_input=child_input,
).save_to_frame(frame)
block_frame_on_children(run, frame.id, (child_frame_id,))
def _finish_subgraph(
workflow: Workflow,
run: RunState,
frame: ExecutionFrame,
step: SubgraphNode,
activation: SubgraphActivation,
prepared: PreparedSubgraph[HandlerT],
reducers: Mapping[str, ReducerDefinition] | None,
) -> StepExecutionResult:
child_frame = run.frames.get(activation.child_frame_id)
if child_frame is None or child_frame.status != FrameStatus.COMPLETED:
raise WorkflowExecutionError(
f"subgraph step {step.id!r} resumed before its child completed"
)
child_scope = run.scopes.get(activation.scope_id)
if child_scope is None:
raise WorkflowExecutionError(
f"subgraph step {step.id!r} is missing child scope {activation.scope_id!r}"
)
child_outcome = child_frame.metadata.get("workflow_outcome")
if not isinstance(child_outcome, str):
raise WorkflowExecutionError(
f"subgraph step {step.id!r} child completed without a workflow outcome"
)
child_output = project_output(prepared.workflow, child_scope.committed_state)
validate_payload_against_schema(
prepared.workflow.output_schema,
child_output,
f"child workflow output for {step.id}",
)
validate_payload_against_schema(
step.output_schema, child_output, f"subgraph output for {step.id}"
)
patch = build_output_patch(
workflow,
step.output,
child_output,
state_view_for_frame(run, frame),
reducers=reducers,
missing_field_message="subgraph output did not include required field {field}",
)
state_changes = commit_patch_for_frame(run, frame, patch)
return StepExecutionResult(
outcome=child_outcome,
resolved_input=activation.child_input,
output=child_output,
state_changes=state_changes,
)
+1 -1
View File
@@ -68,7 +68,7 @@ def validate_subgraph_node(
workflow: Workflow,
report: ValidationReport,
) -> None:
"""Validate a subgraph boundary contract before runtime support exists."""
"""Validate a subgraph boundary independently of runtime child resolution."""
_validate_boundary_bindings(
input_bindings=node.input,
output_bindings=node.output,
+205 -9
View File
@@ -1,19 +1,29 @@
from __future__ import annotations
import asyncio
import pytest
from wf_core import (
END,
Edge,
EndNode,
InterruptNode,
NodeDef,
NodeUse,
PreparedSubgraph,
RunState,
SchemaRef,
StateField,
StateSchema,
SubgraphNode,
Workflow,
WorkflowExecutionError,
execute_workflow_async,
execute_workflow,
)
from wf_core.validation.issues import ValidationIssueCode
from wf_core.models.steps import Step
def test_subgraph_step_validates_boundary_bindings_and_outcomes() -> None:
@@ -73,33 +83,219 @@ def test_subgraph_step_rejects_unwired_declared_outcome() -> None:
)
def test_subgraph_step_runtime_fails_explicitly_until_native_execution_exists() -> None:
def test_subgraph_step_requires_prepared_child_dependency() -> None:
workflow = _workflow()
with pytest.raises(WorkflowExecutionError, match="native subgraph execution"):
with pytest.raises(WorkflowExecutionError, match="prepared child workflow"):
execute_workflow(workflow, {"text": "hello"}, {})
def _workflow(*, node: SubgraphNode | None = None) -> Workflow:
subgraph = node or SubgraphNode.model_validate(
def test_subgraph_step_executes_prepared_child_in_isolated_scope() -> None:
workflow = _workflow(
node=_subgraph_node(input_bindings=[{"target": "text", "value": "child-only"}]),
output_schema=_schema({"answer": {"type": "string"}}),
)
child = _child_workflow()
run = execute_workflow(
workflow,
{"text": "hello"},
{},
subgraphs={
"child.workflow": PreparedSubgraph(
workflow=child,
registry={
"answer": lambda payload, _ctx: {
"answer": f"child:{payload['text']}"
}
},
)
},
)
assert run.output["answer"] == "child:child-only"
assert run.state["answer"] == "child:child-only"
assert run.scopes["root"].committed_state["answer"] == "child:child-only"
child_scope = next(
scope for scope in run.scopes.values() if scope.workflow_name == child.name
)
assert child_scope.workflow_input["text"] == "child-only"
assert child_scope.committed_state["answer"] == "child:child-only"
assert run.trace[0].node_id == "answer"
assert run.trace[0].frame_id != "root"
assert run.trace[-1].node_id == "child"
assert run.trace[-1].step_type == "subgraph"
def test_subgraph_step_executes_prepared_async_child() -> None:
async def answer(payload: dict[str, object], _ctx: object) -> dict[str, object]:
return {"answer": f"async:{payload['text']}"}
async def execute() -> RunState:
return await execute_workflow_async(
_workflow(output_schema=_schema({"answer": {"type": "string"}})),
{"text": "hello"},
{},
subgraphs={
"child.workflow": PreparedSubgraph(
workflow=_child_workflow(),
registry={"answer": answer},
)
},
)
run = asyncio.run(execute())
assert run.output["answer"] == "async:hello"
assert run.trace[-1].step_type == "subgraph"
def test_subgraph_step_routes_through_child_terminal_outcome() -> None:
child = _child_workflow(
outcomes=["error"],
terminal=EndNode(id="child_error", type="end", outcome="error"),
edges=[
Edge.model_validate(
{"from": "answer", "outcome": "ok", "to": "child_error"}
)
],
)
subgraph = _subgraph_node(outcomes=["error"])
workflow = _workflow(
node=subgraph,
outcomes=["ok", "error"],
nodes=[subgraph, EndNode(id="parent_error", type="end", outcome="error")],
edges=[
Edge.model_validate(
{"from": "child", "outcome": "error", "to": "parent_error"}
)
],
)
run = execute_workflow(
workflow,
{"text": "hello"},
{},
subgraphs={
"child.workflow": PreparedSubgraph(
workflow=child,
registry={"answer": lambda payload, _ctx: {"answer": payload["text"]}},
)
},
)
assert run.outcome == "error"
assert any(
entry.node_id == "child_error" and entry.outcome == "error"
for entry in run.trace
)
assert any(
entry.node_id == "child" and entry.outcome == "error" for entry in run.trace
)
def test_subgraph_step_rejects_child_interrupt_until_resume_route_exists() -> None:
child = Workflow(
name="child.workflow",
input_schema=_schema({"text": {"type": "string"}}),
state_schema=StateSchema.from_field_map({}),
output_schema=_schema({}),
start="ask",
nodes=[
InterruptNode.model_validate(
{"id": "ask", "type": "interrupt", "kind": "input"}
)
],
edges=[Edge.model_validate({"from": "ask", "outcome": "submitted", "to": END})],
)
with pytest.raises(WorkflowExecutionError, match="child interrupts"):
execute_workflow(
_workflow(),
{"text": "hello"},
{},
subgraphs={"child.workflow": PreparedSubgraph(workflow=child, registry={})},
)
def _workflow(
*,
node: SubgraphNode | None = None,
outcomes: list[str] | None = None,
nodes: list[Step] | None = None,
edges: list[Edge] | None = None,
output_schema: SchemaRef | None = None,
) -> Workflow:
subgraph = node or _subgraph_node()
return Workflow(
name="subgraph_parent",
input_schema=_schema({"text": {"type": "string"}}),
state_schema=StateSchema.from_field_map({"answer": StateField(type="string")}),
output_schema=output_schema or _schema({}),
outcomes=outcomes or ["ok"],
start="child",
nodes=[subgraph] if nodes is None else nodes,
edges=edges
or [Edge.model_validate({"from": "child", "outcome": "ok", "to": END})],
)
def _subgraph_node(
*,
outcomes: list[str] | None = None,
input_bindings: list[dict[str, object]] | None = None,
) -> SubgraphNode:
return SubgraphNode.model_validate(
{
"id": "child",
"type": "subgraph",
"workflow": "child.workflow",
"input_schema": _schema({"text": {"type": "string"}}),
"output_schema": _schema({"answer": {"type": "string"}}),
"input": (
[{"target": "text", "path": "input.text"}]
if input_bindings is None
else input_bindings
),
"output": [{"source": "answer", "target": "state.answer"}],
"outcomes": outcomes or ["ok"],
}
)
def _child_workflow(
*,
outcomes: list[str] | None = None,
terminal: EndNode | None = None,
edges: list[Edge] | None = None,
) -> Workflow:
node = NodeUse.model_validate(
{
"id": "answer",
"type": "node",
"node": "answer",
"input": [{"target": "text", "path": "input.text"}],
"output": [{"source": "answer", "target": "state.answer"}],
}
)
return Workflow(
name="subgraph_parent",
name="child.workflow",
input_schema=_schema({"text": {"type": "string"}}),
state_schema=StateSchema.from_field_map({"answer": StateField(type="string")}),
output_schema=_schema({}),
start="child",
nodes=[subgraph],
edges=[Edge.model_validate({"from": "child", "outcome": "ok", "to": END})],
output_schema=_schema({"answer": {"type": "string"}}),
node_defs=[
NodeDef(
name="answer",
input_schema=_schema({"text": {"type": "string"}}),
output_schema=_schema({"answer": {"type": "string"}}),
outcomes=["ok"],
)
],
outcomes=outcomes or ["ok"],
start="answer",
nodes=[node] if terminal is None else [node, terminal],
edges=edges
or [Edge.model_validate({"from": "answer", "outcome": "ok", "to": END})],
)