in-memory MCP resuming an interrupted deployment

This commit is contained in:
lda
2026-05-26 01:54:04 +07:00 Verified
parent 95532b1cf9
commit 9d11f78111
14 changed files with 585 additions and 91 deletions
+8 -8
View File
@@ -61,8 +61,9 @@ implementation state.
`SubgraphNode` is now the graph-as-node path for prepared children. `SubgraphNode` is now the graph-as-node path for prepared children.
`WorkflowBuilder.prepare_subgraph()` and `WorkflowBuilder.resume()` make the `WorkflowBuilder.prepare_subgraph()` and `WorkflowBuilder.resume()` make the
local runnable/resumable path available without core-runtime plumbing. local runnable/resumable path available without core-runtime plumbing.
Saved interrupting artifacts remain unrunnable through one-shot Saved interrupting artifacts can now pause and resume through
`run_deployment` until the platform exposes persisted resume. `run_deployment`/`resume_run` while the MCP server process stays alive.
Persisted resume remains future work.
- **Concurrent foreach**: implemented in core with explicit scheduling, - **Concurrent foreach**: implemented in core with explicit scheduling,
reducer/merge semantics, item error policy, async handler batching, and reducer/merge semantics, item error policy, async handler batching, and
quiescent interrupt behavior. Remaining work is polish and future reuse of quiescent interrupt behavior. Remaining work is polish and future reuse of
@@ -104,9 +105,8 @@ Frame stress points remaining for native subgraphs and future fork/gather:
## Why This Order ## Why This Order
The MCP workflow authoring path is now usable enough for real testing. The next The MCP workflow authoring path is now usable enough for real testing. The next
bottleneck is runtime/platform correctness: persisted resume for saved bottleneck is runtime/platform correctness: durable resume/run history,
interrupting children, optional per-use-site child deployment overrides, optional per-use-site child deployment overrides, and protocol-native progress
persistent run history, and protocol-native progress reporting. Concurrent reporting. Concurrent foreach, native saved child execution, and process-local
foreach and native saved child execution now supply scheduler/lineage interrupt resume now supply scheduler/lineage precedent. Those remaining pieces
precedent. Those remaining pieces should come before adding more high-level should come before adding more high-level authoring sugar.
authoring sugar.
+5 -3
View File
@@ -142,8 +142,9 @@ limits and intended adapter seam.
and lineage; child output commits only through declared boundary bindings and and lineage; child output commits only through declared boundary bindings and
the parent routes by the child's terminal workflow outcome. Saved/deployed the parent routes by the child's terminal workflow outcome. Saved/deployed
workflow resolution remains outside core; the workflow platform can now workflow resolution remains outside core; the workflow platform can now
supply non-interrupting saved child artifacts as prepared dependencies using supply saved child artifacts as prepared dependencies using one inherited
one inherited deployment binding environment. For local authoring, deployment binding environment, including process-local pause/resume for
child interrupts. For local authoring,
`WorkflowBuilder.prepare_subgraph()` `WorkflowBuilder.prepare_subgraph()`
registers a child builder and `WorkflowBuilder.resume()` continues a paused registers a child builder and `WorkflowBuilder.resume()` continues a paused
prepared-child interrupt without requiring direct core-runtime calls. prepared-child interrupt without requiring direct core-runtime calls.
@@ -162,7 +163,8 @@ limits and intended adapter seam.
queue, `BLOCKED` frame state, lineage isolation, barrier merge semantics, and queue, `BLOCKED` frame state, lineage isolation, barrier merge semantics, and
pending child results for concurrent foreach. Native prepared subgraphs now pending child results for concurrent foreach. Native prepared subgraphs now
use child-scope execution and typed routed child interruption; the platform use child-scope execution and typed routed child interruption; the platform
resolves non-interrupting saved/deployed child artifacts before core starts. resolves saved/deployed child artifacts before core starts and retains paused
deployment runs in memory for resume.
Concurrent foreach is the primary current use case for async concurrent node Concurrent foreach is the primary current use case for async concurrent node
handler execution. handler execution.
- Runtime errors are still ordinary exceptions plus failed run status. A richer - Runtime errors are still ordinary exceptions plus failed run status. A richer
+17 -7
View File
@@ -298,18 +298,28 @@ Inspect the returned diagnostics first. Then use the matching section above:
Do not debug runtime behavior before dependency validation is clean. Do not debug runtime behavior before dependency validation is clean.
## `run_deployment` Refuses An Interrupting Artifact ## `run_deployment` Returns `interrupted`
Interrupting saved artifacts are not fully supported through the current saved The deployment paused at an interrupt node. The response should include:
artifact execution path yet.
Expected diagnostic:
```text ```text
interrupting_artifact_unsupported status: interrupted
outcome: null
run_id: <process-local id>
interrupt: <request payload and metadata>
``` ```
That is a known platform limitation, not a missing deployment binding. Send the requested resume payload to:
```text
wf.workflow.resume_run
```
The `run_id` is intentionally process-local. It is valid only while the current
MCP server process keeps the paused run in memory. If the server restarts,
there is no durable run store yet; rerun the deployment from the beginning.
After resume completes, `status` is `completed` and `outcome` reports the
workflow terminal outcome such as `ok` or `error`.
## A Raw MCP Tool Works But The Workflow Version Is Awkward ## A Raw MCP Tool Works But The Workflow Version Is Awkward
+23 -17
View File
@@ -69,9 +69,13 @@ semantics.
Dynamic projection of saved workflows as individual MCP tools can exist later, Dynamic projection of saved workflows as individual MCP tools can exist later,
but it should be optional. The stable run tool is the reliable base layer. but it should be optional. The stable run tool is the reliable base layer.
Current `run_deployment` calls are synchronous request/response executions. They Current `run_deployment` calls are synchronous request/response executions until
return compact status, output, diagnostics, and `trace_count`; optional ranged the workflow pauses or completes. They return compact execution status,
trace detail is for debugging only. terminal workflow outcome, output, diagnostics, and `trace_count`; optional
ranged trace detail is for debugging
only. If a run pauses at an interrupt, the response includes a process-local
`run_id` and interrupt payload. Use `wf.workflow.resume_run` with that `run_id`
to continue while the same MCP server process is alive.
Non-interrupting saved workflow children can now execute natively through this Non-interrupting saved workflow children can now execute natively through this
deployment surface. A parent deployment resolves its saved descendants by exact deployment surface. A parent deployment resolves its saved descendants by exact
@@ -80,20 +84,21 @@ bindings for the whole child tree. This is intentionally one configured graph
environment; future per-child deployment overrides, if added, must be keyed by environment; future per-child deployment overrides, if added, must be keyed by
the subgraph use site rather than only the child artifact id. the subgraph use site rather than only the child artifact id.
Interrupting saved artifacts remain unrunnable through `run_deployment`. Interrupting saved artifacts can pause and resume through the current
Although core prepared children can interrupt and resume, this one-shot public deployment surface, including interrupts raised inside saved child workflows.
surface does not yet persist a run for a later resume request. This support is in-memory only: server restart, reload that replaces process
state, or another frontend process invalidates the `run_id`.
Future run history should introduce a stable `run_id` only when there is a real Future run history should replace process-local `run_id` values with durable
run store behind it. A `run_id` without persisted state, trace paging, and run records. The likely shape is:
status lookup would be misleading. The likely shape is:
- `run_deployment` starts or completes a run and returns `run_id` - `run_deployment` starts or completes a run and returns `run_id`
- `inspect_run(run_id)` returns status, output, diagnostics, and trace metadata - `inspect_run(run_id)` returns status, output, diagnostics, and trace metadata
- `read_run_trace(run_id, range)` returns bounded trace slices - `read_run_trace(run_id, range)` returns bounded trace slices
Until that exists, clients should treat the current response as the complete Until that exists, clients should treat the current response as the complete
ephemeral run result for this request. ephemeral run result for this request, or as a process-local resume handle when
`status` is `interrupted`.
For long-running workflow execution, prefer MCP-native execution mechanisms For long-running workflow execution, prefer MCP-native execution mechanisms
where available: where available:
@@ -664,12 +669,13 @@ parent-side contract: child workflow reference, declared child input/output
schemas, binding lists, and declared outcomes. When callers resolve a local schemas, binding lists, and declared outcomes. When callers resolve a local
child into `PreparedSubgraph`, core executes it in child scope, preserves its child into `PreparedSubgraph`, core executes it in child scope, preserves its
trace, and can bubble and resume child interrupts without exposing child state trace, and can bubble and resume child interrupts without exposing child state
as parent state. The current `wf_authoring.subgraph_node` and as parent state. Saved child artifacts are resolved by the workflow surface into
the same `PreparedSubgraph` shape before execution. The current
`wf_authoring.subgraph_node` and
`async_subgraph_node` helpers still execute a child workflow as a plain node `async_subgraph_node` helpers still execute a child workflow as a plain node
and validate the child output. The async helper is explicit because hiding and validate the child output. The async helper is explicit because hiding
`asyncio.run()` inside the sync wrapper would break inside already-running `asyncio.run()` inside the sync wrapper would break inside already-running
event loops. Saved workflow-as-node execution still needs platform-level event loops.
artifact/deployment resolution into prepared children before core can run it.
See `examples/authoring_workflow_as_node.py` for the compatibility wrapper-node See `examples/authoring_workflow_as_node.py` for the compatibility wrapper-node
approach and `examples/authoring_native_subgraph.py` for native prepared-child approach and `examples/authoring_native_subgraph.py` for native prepared-child
@@ -678,10 +684,10 @@ native child pause and builder-driven resume. In the wrapper example the
parent trace sees one node call; in the native examples child trace entries parent trace sees one node call; in the native examples child trace entries
remain in the parent run state. remain in the parent run state.
Until that core upgrade exists, artifact tooling must not assume that an Persisted resume is still not implemented. In-memory resume works because the
interrupting saved workflow can safely be used as a child node. Top-level saved server keeps the paused `RunState`; a durable run store will need to snapshot
workflows with interrupt nodes are valid, but nested interrupting workflows the root workflow, prepared child dependencies, deployment bindings, and trace
should be reported as unsupported for composition. metadata before this can survive restart or move across processes.
Blocking dependency failures happen before workflow execution and are not normal Blocking dependency failures happen before workflow execution and are not normal
workflow outcomes. A missing source, disabled source, unresolved binding, or workflow outcomes. A missing source, disabled source, unresolved binding, or
+110 -3
View File
@@ -96,11 +96,101 @@ Important details:
- `steps` are keyed by stable ids so patches do not depend on array positions. - `steps` are keyed by stable ids so patches do not depend on array positions.
- `start` names one step id. - `start` names one step id.
- `routes` map step outcomes to another step id or `__end__`. - `routes` map step outcomes to another step id or `__end__`.
- top-level `outcomes` declares public workflow terminal outcomes; if omitted,
it defaults to `["ok"]`.
- top-level `output` maps final graph values such as `state.result` into the
public workflow output payload. Step-level `output` only writes a node result
into workflow state.
- `capability` may be concrete during exploration, such as - `capability` may be concrete during exploration, such as
`demo.personal.echo_tool`. `demo.personal.echo_tool`.
- When saved with source bindings, concrete refs can be normalized to logical - When saved with source bindings, concrete refs can be normalized to logical
refs such as `demo.echo_tool`. refs such as `demo.echo_tool`.
## Explicit Outputs And Error Outcomes
Use `__end__` as the compact terminal path for the normal `ok` workflow outcome.
For any other public terminal outcome, add an explicit `end` step and route to
it. The end step itself is terminal; do not add an edge out of it.
This complete draft shape:
```json
{
"name": "echo_with_error",
"input_schema": {
"type": "object",
"properties": {
"text": { "type": "string" },
"fail": { "type": "boolean" }
},
"required": ["text"]
},
"state_schema": {
"type": "object",
"properties": {
"raw": {
"type": "object",
"properties": {
"echoed": { "type": "string" }
}
}
}
},
"output_schema": {
"type": "object",
"properties": {
"message": { "type": "string" }
}
},
"outcomes": ["ok", "error"],
"output": [
{
"target": { "root": "local", "parts": ["message"] },
"path": { "root": "state", "parts": ["raw", "echoed"] }
}
],
"start": "call",
"steps": {
"call": {
"use": "demo.echo",
"input": [
{
"target": { "root": "local", "parts": ["text"] },
"path": { "root": "input", "parts": ["text"] }
},
{
"target": { "root": "local", "parts": ["fail"] },
"path": { "root": "input", "parts": ["fail"] }
}
],
"output": [
{
"source": { "root": "local", "parts": ["echoed"] },
"target": { "root": "state", "parts": ["raw", "echoed"] }
}
]
},
"end_error": {
"end": { "outcome": "error" }
}
},
"routes": {
"call": {
"ok": "__end__",
"error": "end_error"
}
}
}
```
Read it as:
- `call.ok` finishes the workflow with outcome `ok`.
- `call.error` executes `end_error`, which finishes the workflow with outcome
`error`.
- both terminal paths project `state.raw.echoed` into public output field
`message`.
## Binding Shape ## Binding Shape
Draft `use` steps use the same canonical binding structs as core `NodeUse`: Draft `use` steps use the same canonical binding structs as core `NodeUse`:
@@ -356,9 +446,26 @@ provided on resume back into workflow state. Older map-shaped `request` and
`resume` values are accepted only as parse compatibility and dump back to the `resume` values are accepted only as parse compatibility and dump back to the
canonical list shape. canonical list shape.
Saved interrupting artifacts are still limited in the current execution Saved interrupting artifacts can pause and resume through deployment runs while
surface. If a deployment reports `interrupting_artifact_unsupported`, that is a the MCP server process stays alive. The run response includes a process-local
known platform limitation rather than a draft bug. `run_id`; pass that to `wf.workflow.resume_run` with the resume payload.
Persisted run storage is still future work, so a server restart invalidates
that in-memory run id.
### `end`
Declares an explicit workflow terminal outcome.
```json
{
"end": {
"outcome": "error"
}
}
```
Use explicit `end` steps for non-`ok` workflow outcomes. The legacy `__end__`
destination remains the shorthand for public workflow outcome `ok`.
### `join` ### `join`
+13
View File
@@ -79,3 +79,16 @@ def register_artifact_tools(server: FastMCP, service: WfMcpService) -> None:
deployment_id=deployment_id, deployment_id=deployment_id,
workflow_input=workflow_input, workflow_input=workflow_input,
) )
@server.tool()
async def resume_workflow_run(
run_id: str,
resume_payload: dict[str, Any],
resume_outcome: str = "submitted",
) -> dict[str, Any]:
"""Resume a process-local interrupted deployment run."""
return await handlers.resume_run(
run_id=run_id,
resume_payload=resume_payload,
resume_outcome=resume_outcome,
)
+85 -15
View File
@@ -18,7 +18,13 @@ from wf_artifacts import (
artifact_catalog_entry, artifact_catalog_entry,
) )
from wf_authoring import NodeReturn, NodeSpec from wf_authoring import NodeReturn, NodeSpec
from wf_core import NodeUse, Workflow, execute_workflow_async from wf_core import (
NodeUse,
RunState,
Workflow,
execute_workflow_async,
resume_workflow_async,
)
from wf_platform import ( from wf_platform import (
CapabilityBuckets, CapabilityBuckets,
@@ -664,6 +670,7 @@ class WfMcpService:
"state_schema": plan.state_schema, "state_schema": plan.state_schema,
"output_schema": plan.output_schema, "output_schema": plan.output_schema,
"output": [binding.model_dump(mode="json") for binding in plan.output], "output": [binding.model_dump(mode="json") for binding in plan.output],
"outcomes": plan.outcomes,
"start": plan.start, "start": plan.start,
"node_defs": [node.model_dump() for node in node_defs.values()], "node_defs": [node.model_dump() for node in node_defs.values()],
"nodes": nodes, "nodes": nodes,
@@ -671,20 +678,19 @@ class WfMcpService:
} }
return Workflow.model_validate(payload) return Workflow.model_validate(payload)
async def run_workflow_from_plan( def _prepare_workflow_runtime(
self, self,
plan: RawWorkflowPlan, plan: RawWorkflowPlan,
workflow_input: dict[str, Any], *,
deployment: WorkflowDeployment | None = None, deployment: WorkflowDeployment | None,
artifact: WorkflowArtifact | None = None, artifact: WorkflowArtifact | None,
): ) -> tuple[Workflow, dict[str, Any], dict[str, Any], dict[str, Any]]:
self._record_event( """Resolve bindings once into the executable pieces core expects.
make_event(
"workflow_run_started", Saved-run resume must rebuild prepared dependencies from the current
workflow_name=plan.name, in-memory service state. Durable resume will need a stricter snapshot,
payload={"input_keys": sorted(workflow_input.keys())}, but this keeps the current platform boundary explicit.
) """
)
plan_node_names = [ plan_node_names = [
node.node for node in plan.nodes if isinstance(node, NodeUse) node.node for node in plan.nodes if isinstance(node, NodeUse)
] ]
@@ -716,11 +722,39 @@ class WfMcpService:
compile_plan=self.compile_plan, compile_plan=self.compile_plan,
) )
workflow = self.compile_plan(plan, dependencies.node_name_bindings) workflow = self.compile_plan(plan, dependencies.node_name_bindings)
return (
workflow,
dependencies.node_registry,
dependencies.reducers,
prepared_subgraphs,
)
async def run_workflow_from_plan(
self,
plan: RawWorkflowPlan,
workflow_input: dict[str, Any],
deployment: WorkflowDeployment | None = None,
artifact: WorkflowArtifact | None = None,
):
self._record_event(
make_event(
"workflow_run_started",
workflow_name=plan.name,
payload={"input_keys": sorted(workflow_input.keys())},
)
)
workflow, registry, reducers, prepared_subgraphs = (
self._prepare_workflow_runtime(
plan,
deployment=deployment,
artifact=artifact,
)
)
run = await execute_workflow_async( run = await execute_workflow_async(
workflow, workflow,
workflow_input, workflow_input,
dependencies.node_registry, registry,
reducers=dependencies.reducers, reducers=reducers,
subgraphs=prepared_subgraphs, subgraphs=prepared_subgraphs,
) )
self._record_event( self._record_event(
@@ -732,6 +766,42 @@ class WfMcpService:
) )
return run return run
async def resume_workflow_from_plan(
self,
plan: RawWorkflowPlan,
run: RunState,
*,
resume_payload: dict[str, Any],
resume_outcome: str = "submitted",
deployment: WorkflowDeployment | None = None,
artifact: WorkflowArtifact | None = None,
) -> RunState:
"""Resume one in-memory run using freshly resolved runtime dependencies."""
workflow, registry, reducers, prepared_subgraphs = (
self._prepare_workflow_runtime(
plan,
deployment=deployment,
artifact=artifact,
)
)
resumed = await resume_workflow_async(
workflow,
run,
registry,
resume_payload=resume_payload,
resume_outcome=resume_outcome,
reducers=reducers,
subgraphs=prepared_subgraphs,
)
self._record_event(
make_event(
"workflow_run_resumed",
workflow_name=plan.name,
payload={"status": resumed.status.value},
)
)
return resumed
def list_events(self) -> list[McpEvent]: def list_events(self) -> list[McpEvent]:
return self.event_bus.list_events() return self.event_bus.list_events()
+7
View File
@@ -49,6 +49,13 @@ class RawWorkflowPlan(BaseModel):
input_schema: dict[str, Any] input_schema: dict[str, Any]
state_schema: dict[str, Any] state_schema: dict[str, Any]
output_schema: dict[str, Any] output_schema: dict[str, Any]
outcomes: list[str] = Field(
default_factory=lambda: ["ok"],
description=(
"Declared public workflow outcomes. Legacy plans without this field "
"default to ok."
),
)
output: list[InputBinding] = Field( output: list[InputBinding] = Field(
default_factory=list, default_factory=list,
description=( description=(
+128 -12
View File
@@ -1,8 +1,9 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Sequence from collections.abc import Sequence
from dataclasses import asdict from dataclasses import asdict, dataclass
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING, Any
from uuid import uuid4
from wf_artifacts import ( from wf_artifacts import (
ArtifactKind, ArtifactKind,
@@ -52,21 +53,39 @@ from .constants import (
from .models import TraceRange from .models import TraceRange
from .refs import parse_workflow_surface_capability_id from .refs import parse_workflow_surface_capability_id
from .saved_subgraphs import ( from .saved_subgraphs import (
interrupting_artifact_diagnostic, direct_wrapper_interrupt_diagnostic,
resolve_saved_subgraph_tree, resolve_saved_subgraph_tree,
validate_saved_subgraph_tree, validate_saved_subgraph_tree,
) )
from .wrapper_hints import wrapper_hints_for_capability from .wrapper_hints import wrapper_hints_for_capability
if TYPE_CHECKING: if TYPE_CHECKING:
from wf_core import RunState
from ..broker.service import WfMcpService from ..broker.service import WfMcpService
@dataclass(slots=True)
class ActiveWorkflowRun:
"""In-memory paused deployment run.
This is intentionally not durable. It only makes interrupt/resume usable
while the current MCP server process is alive; persisted run storage remains
a separate platform concern.
"""
deployment: WorkflowDeployment
artifact: WorkflowArtifact
plan: RawWorkflowPlan
run: "RunState"
class WorkflowSurfaceHandlers: class WorkflowSurfaceHandlers:
"""Reusable implementation behind MCP workflow artifact tools.""" """Reusable implementation behind MCP workflow artifact tools."""
def __init__(self, service: WfMcpService) -> None: def __init__(self, service: WfMcpService) -> None:
self.service = service self.service = service
self._active_runs: dict[str, ActiveWorkflowRun] = {}
async def list_artifacts( async def list_artifacts(
self, self,
@@ -308,7 +327,7 @@ class WorkflowSurfaceHandlers:
deployment_id: str | None, deployment_id: str | None,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Execute a saved wrapper artifact through the workflow runner.""" """Execute a saved wrapper artifact through the workflow runner."""
unsupported = interrupting_artifact_diagnostic(artifact) unsupported = direct_wrapper_interrupt_diagnostic(artifact)
if unsupported is not None: if unsupported is not None:
raise ValueError(unsupported.message) raise ValueError(unsupported.message)
@@ -911,15 +930,6 @@ class WorkflowSurfaceHandlers:
diagnostics=diagnostics, diagnostics=diagnostics,
) )
unsupported = interrupting_artifact_diagnostic(artifact)
if unsupported is not None:
return _run_payload(
deployment=deployment,
artifact=artifact,
status="unsupported",
diagnostics=[unsupported],
)
plan = _raw_plan_from_artifact(artifact) plan = _raw_plan_from_artifact(artifact)
run = await self.service.run_workflow_from_plan( run = await self.service.run_workflow_from_plan(
plan, plan,
@@ -927,10 +937,19 @@ class WorkflowSurfaceHandlers:
deployment=deployment, deployment=deployment,
artifact=artifact, artifact=artifact,
) )
run_id = self._save_active_run(
deployment=deployment,
artifact=artifact,
plan=plan,
run=run,
)
return _run_payload( return _run_payload(
deployment=deployment, deployment=deployment,
artifact=artifact, artifact=artifact,
status=run.status.value, status=run.status.value,
run_id=run_id,
interrupt=_interrupt_payload(run),
outcome=run.outcome,
output=run.output, output=run.output,
trace_count=len(run.trace), trace_count=len(run.trace),
trace=( trace=(
@@ -951,6 +970,82 @@ class WorkflowSurfaceHandlers:
), ),
) )
async def resume_run(
self,
*,
run_id: str,
resume_payload: dict[str, Any],
resume_outcome: str = "submitted",
trace_range: TraceRange | None = None,
) -> dict[str, Any]:
"""Resume one interrupted in-memory deployment run."""
active = self._active_runs[run_id]
run = await self.service.resume_workflow_from_plan(
active.plan,
active.run,
resume_payload=resume_payload,
resume_outcome=resume_outcome,
deployment=active.deployment,
artifact=active.artifact,
)
active.run = run
next_run_id = self._save_active_run(
deployment=active.deployment,
artifact=active.artifact,
plan=active.plan,
run=run,
run_id=run_id,
)
return _run_payload(
deployment=active.deployment,
artifact=active.artifact,
status=run.status.value,
run_id=next_run_id,
interrupt=_interrupt_payload(run),
outcome=run.outcome,
output=run.output,
trace_count=len(run.trace),
trace=(
[
asdict(entry)
for entry in run.trace[
trace_range.start : trace_range.start + trace_range.limit
]
]
if trace_range is not None
else None
),
trace_start=trace_range.start if trace_range is not None else None,
trace_limit=trace_range.limit if trace_range is not None else None,
trace_truncated=(
trace_range is not None
and len(run.trace) > trace_range.start + trace_range.limit
),
)
def _save_active_run(
self,
*,
deployment: WorkflowDeployment,
artifact: WorkflowArtifact,
plan: RawWorkflowPlan,
run: RunState,
run_id: str | None = None,
) -> str | None:
"""Store only interrupted runs; terminal runs leave no resume handle."""
if run.status.value != "interrupted":
if run_id is not None:
self._active_runs.pop(run_id, None)
return None
key = run_id or f"run_{uuid4().hex}"
self._active_runs[key] = ActiveWorkflowRun(
deployment=deployment,
artifact=artifact,
plan=plan,
run=run,
)
return key
def _deployment_validation( def _deployment_validation(
self, self,
deployment_id: str, deployment_id: str,
@@ -1221,6 +1316,8 @@ def _raw_plan_from_artifact(artifact: WorkflowArtifact) -> RawWorkflowPlan:
"input_schema": _plan_field(artifact, "input_schema"), "input_schema": _plan_field(artifact, "input_schema"),
"state_schema": _plan_field(artifact, "state_schema"), "state_schema": _plan_field(artifact, "state_schema"),
"output_schema": _plan_field(artifact, "output_schema"), "output_schema": _plan_field(artifact, "output_schema"),
"outcomes": artifact.plan.get("outcomes", ["ok"]),
"output": artifact.plan.get("output", []),
"start": _plan_field(artifact, "start"), "start": _plan_field(artifact, "start"),
"nodes": _plan_field(artifact, "nodes"), "nodes": _plan_field(artifact, "nodes"),
"edges": _plan_field(artifact, "edges"), "edges": _plan_field(artifact, "edges"),
@@ -1248,6 +1345,9 @@ def _run_payload(
deployment: WorkflowDeployment, deployment: WorkflowDeployment,
artifact: WorkflowArtifact, artifact: WorkflowArtifact,
status: str, status: str,
run_id: str | None = None,
interrupt: dict[str, Any] | None = None,
outcome: str | None = None,
diagnostics: list[DependencyDiagnostic] | None = None, diagnostics: list[DependencyDiagnostic] | None = None,
output: dict[str, Any] | None = None, output: dict[str, Any] | None = None,
trace_count: int = 0, trace_count: int = 0,
@@ -1261,6 +1361,9 @@ def _run_payload(
"artifact_id": artifact.id, "artifact_id": artifact.id,
"artifact_version": artifact.version, "artifact_version": artifact.version,
"status": status, "status": status,
"run_id": run_id,
"interrupt": interrupt,
"outcome": outcome,
"output": output, "output": output,
"diagnostics": [ "diagnostics": [
diagnostic.model_dump(mode="json") for diagnostic in diagnostics or [] diagnostic.model_dump(mode="json") for diagnostic in diagnostics or []
@@ -1277,6 +1380,19 @@ def _run_payload(
return payload return payload
def _interrupt_payload(run: RunState) -> dict[str, Any] | None:
"""Return a JSON-safe interrupt payload for the current run, if paused."""
if run.interrupt is None:
return None
payload = asdict(run.interrupt)
route = payload.get("route")
if isinstance(route, dict) and "workflow_ref" in route:
workflow_ref = route["workflow_ref"]
if hasattr(workflow_ref, "model_dump"):
route["workflow_ref"] = workflow_ref.model_dump(mode="json")
return payload
def _deployment_summary(deployment: WorkflowDeployment) -> dict[str, Any]: def _deployment_summary(deployment: WorkflowDeployment) -> dict[str, Any]:
"""Return compact deployment metadata for progressive list responses.""" """Return compact deployment metadata for progressive list responses."""
return { return {
+10 -12
View File
@@ -83,9 +83,6 @@ def validate_saved_subgraph_tree(
sources=sources, sources=sources,
) )
) )
interrupt_diagnostic = interrupting_artifact_diagnostic(child)
if interrupt_diagnostic is not None:
diagnostics.append(interrupt_diagnostic)
return diagnostics return diagnostics
@@ -125,24 +122,25 @@ def prepare_saved_subgraphs(
return prepared return prepared
def interrupting_artifact_diagnostic( def direct_wrapper_interrupt_diagnostic(
artifact: WorkflowArtifact, artifact: WorkflowArtifact,
) -> DependencyDiagnostic | None: ) -> DependencyDiagnostic | None:
"""Reject saved interrupt workflows until the platform exposes resume.""" """Reject direct wrapper calls that cannot return a resumable run handle.
Deployment execution supports interrupt/resume through an in-memory
`run_id`; `call_capability` remains a single-call authoring probe.
"""
if not any(isinstance(node, InterruptNode) for node in _artifact_steps(artifact)): if not any(isinstance(node, InterruptNode) for node in _artifact_steps(artifact)):
return None return None
return DependencyDiagnostic( return DependencyDiagnostic(
severity=DiagnosticSeverity.ERROR, severity=DiagnosticSeverity.ERROR,
code="interrupting_artifact_unsupported", code="interrupting_wrapper_call_unsupported",
logical_ref=f"workflow.{artifact.id}.v{artifact.version}", logical_ref=f"workflow.{artifact.id}.v{artifact.version}",
message=( message=(
"Running saved workflow artifacts with interrupt nodes is unsupported " "Direct wrapper calls cannot pause for interrupt input; run the "
"until nested run-state resume is implemented." "artifact through a deployment to receive a resumable run_id."
),
repair_hint=(
"Run this workflow as a top-level core workflow or remove interrupt "
"nodes before saving it as a runnable deployment."
), ),
repair_hint="Save a deployment and call wf.workflow.run_deployment instead.",
) )
+31 -2
View File
@@ -603,8 +603,8 @@ def register_workflow_tools(server: FastMCP[Any], service: WfMcpService) -> None
name="wf.workflow.run_deployment", name="wf.workflow.run_deployment",
title="Run Workflow Deployment", title="Run Workflow Deployment",
description=( description=(
"Run deployment_id with workflow_input and return status, output, " "Run deployment_id with workflow_input and return status, terminal "
"diagnostics, and trace_count. Debug traces can include resolved " "outcome, output, diagnostics, and trace_count. Debug traces can include resolved "
"inputs and state changes; pass trace_range only when needed." "inputs and state changes; pass trace_range only when needed."
), ),
) )
@@ -627,3 +627,32 @@ def register_workflow_tools(server: FastMCP[Any], service: WfMcpService) -> None
workflow_input=workflow_input, workflow_input=workflow_input,
trace_range=trace_range, trace_range=trace_range,
) )
@server.tool(
name="wf.workflow.resume_run",
title="Resume Workflow Run",
description=(
"Resume an interrupted in-memory deployment run returned by "
"run_deployment. Run IDs are process-local and are not durable."
),
)
async def resume_run(
run_id: str,
resume_payload: dict[str, Any],
resume_outcome: str = "submitted",
trace_range: Annotated[
TraceRange | None,
Field(
description=(
"Debug traces range to return after resume. Omit for the "
"normal compact response."
)
),
] = None,
) -> dict[str, Any]:
return await handlers.resume_run(
run_id=run_id,
resume_payload=resume_payload,
resume_outcome=resume_outcome,
trace_range=trace_range,
)
+91 -1
View File
@@ -5,7 +5,15 @@ from pydantic import ValidationError
from wf_artifacts.drafts import WorkflowDraft from wf_artifacts.drafts import WorkflowDraft
from wf_artifacts.drafts.api import compile_workflow_draft, validate_workflow_draft from wf_artifacts.drafts.api import compile_workflow_draft, validate_workflow_draft
from wf_artifacts.drafts.adapter import build_workflow_from_draft from wf_artifacts.drafts.adapter import build_workflow_from_draft
from wf_core import ConditionNode, EndNode, ForeachNode, NodeUse from wf_core import (
ConditionNode,
EndNode,
ForeachNode,
NodeDef,
NodeUse,
SchemaRef,
execute_workflow,
)
from wf_core.models.steps import InputValueBinding from wf_core.models.steps import InputValueBinding
@@ -107,6 +115,88 @@ def test_adapter_lowers_root_workflow_output_bindings() -> None:
assert dumped["output"][0]["path"] == {"root": "state", "parts": ["raw", "echoed"]} assert dumped["output"][0]["path"] == {"root": "state", "parts": ["raw", "echoed"]}
def test_adapter_golden_draft_executes_ok_and_error_outcomes() -> None:
draft = WorkflowDraft.model_validate(
{
"name": "golden_echo",
"input_schema": {
"type": "object",
"properties": {
"text": {"type": "string"},
"fail": {"type": "boolean"},
},
"required": ["text"],
},
"state_schema": {
"type": "object",
"properties": {"raw": {"type": "object"}},
},
"output_schema": {
"type": "object",
"properties": {"message": {"type": "string"}},
},
"outcomes": ["ok", "error"],
"output": [{"target": "message", "path": "state.raw.echoed"}],
"start": "call",
"steps": {
"call": {
"use": "demo.echo",
"input": [
{"target": "text", "path": "input.text"},
{"target": "fail", "path": "input.fail"},
],
"output": [{"source": "echoed", "target": "state.raw.echoed"}],
},
"end_error": {"end": {"outcome": "error"}},
},
"routes": {
"call": {"ok": "__end__", "error": "end_error"},
},
}
)
workflow = build_workflow_from_draft(draft)
workflow = workflow.model_copy(
update={
"node_defs": [
NodeDef(
name="demo.echo",
input_schema=SchemaRef.model_validate(draft.input_schema),
output_schema=SchemaRef.model_validate(
{
"type": "object",
"properties": {"echoed": {"type": "string"}},
}
),
outcomes=["ok", "error"],
)
]
}
)
def echo(payload: dict[str, object], _ctx: object) -> dict[str, object]:
if payload.get("fail") is True:
return {"outcome": "error", "output": {"echoed": "failed"}}
return {"outcome": "ok", "output": {"echoed": str(payload["text"])}}
ok = execute_workflow(
workflow,
{"text": "hello", "fail": False},
{"demo.echo": echo},
)
error = execute_workflow(
workflow,
{"text": "hello", "fail": True},
{"demo.echo": echo},
)
assert ok.status == "completed"
assert ok.outcome == "ok"
assert ok.output["message"] == "hello"
assert error.status == "completed"
assert error.outcome == "error"
assert error.output["message"] == "failed"
def test_adapter_lowers_static_inputs_for_constant_like_steps() -> None: def test_adapter_lowers_static_inputs_for_constant_like_steps() -> None:
draft = WorkflowDraft.model_validate( draft = WorkflowDraft.model_validate(
{ {
+26 -6
View File
@@ -415,7 +415,7 @@ def test_broker_run_deployment_returns_unrunnable_for_dependency_errors() -> Non
assert payload["diagnostics"][0]["code"] == "source_missing" assert payload["diagnostics"][0]["code"] == "source_missing"
def test_broker_run_deployment_rejects_interrupting_artifacts() -> None: def test_broker_run_deployment_pauses_and_resumes_interrupting_artifacts() -> None:
artifact_store = FileWorkflowArtifactStore( artifact_store = FileWorkflowArtifactStore(
local_temp_root() / "broker_run_interrupt_artifacts" local_temp_root() / "broker_run_interrupt_artifacts"
) )
@@ -445,9 +445,25 @@ def test_broker_run_deployment_rejects_interrupting_artifacts() -> None:
) )
payload = cast(dict[str, Any], cast(object, structured)) payload = cast(dict[str, Any], cast(object, structured))
assert payload["status"] == "unsupported" assert payload["status"] == "interrupted"
assert payload["output"] is None assert payload["output"] == {}
assert payload["diagnostics"][0]["code"] == "interrupting_artifact_unsupported" assert isinstance(payload["run_id"], str)
assert payload["interrupt"]["payload"]["message"] == "send?"
_content, structured = asyncio.run(
server.call_tool(
"resume_workflow_run",
{
"run_id": payload["run_id"],
"resume_payload": {},
},
)
)
resumed = cast(dict[str, Any], cast(object, structured))
assert resumed["status"] == "completed"
assert resumed["outcome"] == "submitted"
assert resumed["run_id"] is None
def test_build_service_from_config_uses_store_root_for_artifacts() -> None: def test_build_service_from_config_uses_store_root_for_artifacts() -> None:
@@ -556,6 +572,7 @@ def _interrupt_artifact() -> WorkflowArtifact:
}, },
"state_schema": {"fields": {}}, "state_schema": {"fields": {}},
"output_schema": {"type": "object", "properties": {}}, "output_schema": {"type": "object", "properties": {}},
"outcomes": ["submitted"],
"start": "approval", "start": "approval",
"nodes": [ "nodes": [
{ {
@@ -565,8 +582,11 @@ def _interrupt_artifact() -> WorkflowArtifact:
"request": [input_binding("input.message", "message")], "request": [input_binding("input.message", "message")],
"resume": [], "resume": [],
"outcomes": ["submitted"], "outcomes": ["submitted"],
} },
{"id": "end_submitted", "type": "end", "outcome": "submitted"},
],
"edges": [
{"from": "approval", "outcome": "submitted", "to": "end_submitted"}
], ],
"edges": [{"from": "approval", "outcome": "submitted", "to": "__end__"}],
}, },
) )
+31 -5
View File
@@ -94,18 +94,44 @@ def test_saved_child_missing_parent_binding_is_unrunnable() -> None:
assert result["diagnostics"][0]["logical_ref"] == "demo.echo_tool" assert result["diagnostics"][0]["logical_ref"] == "demo.echo_tool"
def test_interrupting_saved_child_remains_unrunnable_on_deployment_surface() -> None: def test_interrupting_saved_child_pauses_and_resumes_through_deployment_surface() -> (
None
):
store = FileWorkflowArtifactStore(local_temp_root() / "saved_subgraph_interrupt") store = FileWorkflowArtifactStore(local_temp_root() / "saved_subgraph_interrupt")
store.save_artifact(_parent_artifact()) store.save_artifact(_parent_artifact())
store.save_artifact(_interrupting_child_artifact()) store.save_artifact(_interrupting_child_artifact())
store.save_deployment(_deployment()) store.save_deployment(_deployment())
handlers = _handlers(store) handlers = _handlers(store)
result = asyncio.run(handlers.validate_deployment(deployment_id="parent.personal")) validation = asyncio.run(
handlers.validate_deployment(deployment_id="parent.personal")
)
assert result["status"] == "unrunnable" assert validation["status"] == "runnable"
assert result["diagnostics"][0]["code"] == "interrupting_artifact_unsupported" assert validation["diagnostics"] == []
assert result["diagnostics"][0]["logical_ref"] == "workflow.child.v1"
paused = asyncio.run(
handlers.run_deployment(
deployment_id="parent.personal",
workflow_input={"text": "hello"},
)
)
assert paused["status"] == "interrupted"
assert isinstance(paused["run_id"], str)
assert paused["interrupt"]["node_id"] == "child_step"
assert paused["interrupt"]["payload"]["question"] == "hello"
resumed = asyncio.run(
handlers.resume_run(
run_id=paused["run_id"],
resume_payload={"answer": "world"},
)
)
assert resumed["status"] == "completed"
assert resumed["outcome"] == "ok"
assert resumed["output"]["echoed"] == "world"
def test_missing_saved_child_is_unrunnable_on_deployment_surface() -> None: def test_missing_saved_child_is_unrunnable_on_deployment_surface() -> None: