in-memory MCP resuming an interrupted deployment
This commit is contained in:
@@ -61,8 +61,9 @@ implementation state.
|
||||
`SubgraphNode` is now the graph-as-node path for prepared children.
|
||||
`WorkflowBuilder.prepare_subgraph()` and `WorkflowBuilder.resume()` make the
|
||||
local runnable/resumable path available without core-runtime plumbing.
|
||||
Saved interrupting artifacts remain unrunnable through one-shot
|
||||
`run_deployment` until the platform exposes persisted resume.
|
||||
Saved interrupting artifacts can now pause and resume through
|
||||
`run_deployment`/`resume_run` while the MCP server process stays alive.
|
||||
Persisted resume remains future work.
|
||||
- **Concurrent foreach**: implemented in core with explicit scheduling,
|
||||
reducer/merge semantics, item error policy, async handler batching, and
|
||||
quiescent interrupt behavior. Remaining work is polish and future reuse of
|
||||
@@ -104,9 +105,8 @@ Frame stress points remaining for native subgraphs and future fork/gather:
|
||||
## Why This Order
|
||||
|
||||
The MCP workflow authoring path is now usable enough for real testing. The next
|
||||
bottleneck is runtime/platform correctness: persisted resume for saved
|
||||
interrupting children, optional per-use-site child deployment overrides,
|
||||
persistent run history, and protocol-native progress reporting. Concurrent
|
||||
foreach and native saved child execution now supply scheduler/lineage
|
||||
precedent. Those remaining pieces should come before adding more high-level
|
||||
authoring sugar.
|
||||
bottleneck is runtime/platform correctness: durable resume/run history,
|
||||
optional per-use-site child deployment overrides, and protocol-native progress
|
||||
reporting. Concurrent foreach, native saved child execution, and process-local
|
||||
interrupt resume now supply scheduler/lineage precedent. Those remaining pieces
|
||||
should come before adding more high-level authoring sugar.
|
||||
|
||||
@@ -142,8 +142,9 @@ limits and intended adapter seam.
|
||||
and lineage; child output commits only through declared boundary bindings and
|
||||
the parent routes by the child's terminal workflow outcome. Saved/deployed
|
||||
workflow resolution remains outside core; the workflow platform can now
|
||||
supply non-interrupting saved child artifacts as prepared dependencies using
|
||||
one inherited deployment binding environment. For local authoring,
|
||||
supply saved child artifacts as prepared dependencies using one inherited
|
||||
deployment binding environment, including process-local pause/resume for
|
||||
child interrupts. For local authoring,
|
||||
`WorkflowBuilder.prepare_subgraph()`
|
||||
registers a child builder and `WorkflowBuilder.resume()` continues a paused
|
||||
prepared-child interrupt without requiring direct core-runtime calls.
|
||||
@@ -162,7 +163,8 @@ limits and intended adapter seam.
|
||||
queue, `BLOCKED` frame state, lineage isolation, barrier merge semantics, and
|
||||
pending child results for concurrent foreach. Native prepared subgraphs now
|
||||
use child-scope execution and typed routed child interruption; the platform
|
||||
resolves non-interrupting saved/deployed child artifacts before core starts.
|
||||
resolves saved/deployed child artifacts before core starts and retains paused
|
||||
deployment runs in memory for resume.
|
||||
Concurrent foreach is the primary current use case for async concurrent node
|
||||
handler execution.
|
||||
- Runtime errors are still ordinary exceptions plus failed run status. A richer
|
||||
|
||||
@@ -298,18 +298,28 @@ Inspect the returned diagnostics first. Then use the matching section above:
|
||||
|
||||
Do not debug runtime behavior before dependency validation is clean.
|
||||
|
||||
## `run_deployment` Refuses An Interrupting Artifact
|
||||
## `run_deployment` Returns `interrupted`
|
||||
|
||||
Interrupting saved artifacts are not fully supported through the current saved
|
||||
artifact execution path yet.
|
||||
|
||||
Expected diagnostic:
|
||||
The deployment paused at an interrupt node. The response should include:
|
||||
|
||||
```text
|
||||
interrupting_artifact_unsupported
|
||||
status: interrupted
|
||||
outcome: null
|
||||
run_id: <process-local id>
|
||||
interrupt: <request payload and metadata>
|
||||
```
|
||||
|
||||
That is a known platform limitation, not a missing deployment binding.
|
||||
Send the requested resume payload to:
|
||||
|
||||
```text
|
||||
wf.workflow.resume_run
|
||||
```
|
||||
|
||||
The `run_id` is intentionally process-local. It is valid only while the current
|
||||
MCP server process keeps the paused run in memory. If the server restarts,
|
||||
there is no durable run store yet; rerun the deployment from the beginning.
|
||||
After resume completes, `status` is `completed` and `outcome` reports the
|
||||
workflow terminal outcome such as `ok` or `error`.
|
||||
|
||||
## A Raw MCP Tool Works But The Workflow Version Is Awkward
|
||||
|
||||
|
||||
+23
-17
@@ -69,9 +69,13 @@ semantics.
|
||||
Dynamic projection of saved workflows as individual MCP tools can exist later,
|
||||
but it should be optional. The stable run tool is the reliable base layer.
|
||||
|
||||
Current `run_deployment` calls are synchronous request/response executions. They
|
||||
return compact status, output, diagnostics, and `trace_count`; optional ranged
|
||||
trace detail is for debugging only.
|
||||
Current `run_deployment` calls are synchronous request/response executions until
|
||||
the workflow pauses or completes. They return compact execution status,
|
||||
terminal workflow outcome, output, diagnostics, and `trace_count`; optional
|
||||
ranged trace detail is for debugging
|
||||
only. If a run pauses at an interrupt, the response includes a process-local
|
||||
`run_id` and interrupt payload. Use `wf.workflow.resume_run` with that `run_id`
|
||||
to continue while the same MCP server process is alive.
|
||||
|
||||
Non-interrupting saved workflow children can now execute natively through this
|
||||
deployment surface. A parent deployment resolves its saved descendants by exact
|
||||
@@ -80,20 +84,21 @@ bindings for the whole child tree. This is intentionally one configured graph
|
||||
environment; future per-child deployment overrides, if added, must be keyed by
|
||||
the subgraph use site rather than only the child artifact id.
|
||||
|
||||
Interrupting saved artifacts remain unrunnable through `run_deployment`.
|
||||
Although core prepared children can interrupt and resume, this one-shot public
|
||||
surface does not yet persist a run for a later resume request.
|
||||
Interrupting saved artifacts can pause and resume through the current
|
||||
deployment surface, including interrupts raised inside saved child workflows.
|
||||
This support is in-memory only: server restart, reload that replaces process
|
||||
state, or another frontend process invalidates the `run_id`.
|
||||
|
||||
Future run history should introduce a stable `run_id` only when there is a real
|
||||
run store behind it. A `run_id` without persisted state, trace paging, and
|
||||
status lookup would be misleading. The likely shape is:
|
||||
Future run history should replace process-local `run_id` values with durable
|
||||
run records. The likely shape is:
|
||||
|
||||
- `run_deployment` starts or completes a run and returns `run_id`
|
||||
- `inspect_run(run_id)` returns status, output, diagnostics, and trace metadata
|
||||
- `read_run_trace(run_id, range)` returns bounded trace slices
|
||||
|
||||
Until that exists, clients should treat the current response as the complete
|
||||
ephemeral run result for this request.
|
||||
ephemeral run result for this request, or as a process-local resume handle when
|
||||
`status` is `interrupted`.
|
||||
|
||||
For long-running workflow execution, prefer MCP-native execution mechanisms
|
||||
where available:
|
||||
@@ -664,12 +669,13 @@ parent-side contract: child workflow reference, declared child input/output
|
||||
schemas, binding lists, and declared outcomes. When callers resolve a local
|
||||
child into `PreparedSubgraph`, core executes it in child scope, preserves its
|
||||
trace, and can bubble and resume child interrupts without exposing child state
|
||||
as parent state. The current `wf_authoring.subgraph_node` and
|
||||
as parent state. Saved child artifacts are resolved by the workflow surface into
|
||||
the same `PreparedSubgraph` shape before execution. The current
|
||||
`wf_authoring.subgraph_node` and
|
||||
`async_subgraph_node` helpers still execute a child workflow as a plain node
|
||||
and validate the child output. The async helper is explicit because hiding
|
||||
`asyncio.run()` inside the sync wrapper would break inside already-running
|
||||
event loops. Saved workflow-as-node execution still needs platform-level
|
||||
artifact/deployment resolution into prepared children before core can run it.
|
||||
event loops.
|
||||
|
||||
See `examples/authoring_workflow_as_node.py` for the compatibility wrapper-node
|
||||
approach and `examples/authoring_native_subgraph.py` for native prepared-child
|
||||
@@ -678,10 +684,10 @@ native child pause and builder-driven resume. In the wrapper example the
|
||||
parent trace sees one node call; in the native examples child trace entries
|
||||
remain in the parent run state.
|
||||
|
||||
Until that core upgrade exists, artifact tooling must not assume that an
|
||||
interrupting saved workflow can safely be used as a child node. Top-level saved
|
||||
workflows with interrupt nodes are valid, but nested interrupting workflows
|
||||
should be reported as unsupported for composition.
|
||||
Persisted resume is still not implemented. In-memory resume works because the
|
||||
server keeps the paused `RunState`; a durable run store will need to snapshot
|
||||
the root workflow, prepared child dependencies, deployment bindings, and trace
|
||||
metadata before this can survive restart or move across processes.
|
||||
|
||||
Blocking dependency failures happen before workflow execution and are not normal
|
||||
workflow outcomes. A missing source, disabled source, unresolved binding, or
|
||||
|
||||
+110
-3
@@ -96,11 +96,101 @@ Important details:
|
||||
- `steps` are keyed by stable ids so patches do not depend on array positions.
|
||||
- `start` names one step id.
|
||||
- `routes` map step outcomes to another step id or `__end__`.
|
||||
- top-level `outcomes` declares public workflow terminal outcomes; if omitted,
|
||||
it defaults to `["ok"]`.
|
||||
- top-level `output` maps final graph values such as `state.result` into the
|
||||
public workflow output payload. Step-level `output` only writes a node result
|
||||
into workflow state.
|
||||
- `capability` may be concrete during exploration, such as
|
||||
`demo.personal.echo_tool`.
|
||||
- When saved with source bindings, concrete refs can be normalized to logical
|
||||
refs such as `demo.echo_tool`.
|
||||
|
||||
## Explicit Outputs And Error Outcomes
|
||||
|
||||
Use `__end__` as the compact terminal path for the normal `ok` workflow outcome.
|
||||
For any other public terminal outcome, add an explicit `end` step and route to
|
||||
it. The end step itself is terminal; do not add an edge out of it.
|
||||
|
||||
This complete draft shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "echo_with_error",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"text": { "type": "string" },
|
||||
"fail": { "type": "boolean" }
|
||||
},
|
||||
"required": ["text"]
|
||||
},
|
||||
"state_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"raw": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"echoed": { "type": "string" }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"output_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"message": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"outcomes": ["ok", "error"],
|
||||
"output": [
|
||||
{
|
||||
"target": { "root": "local", "parts": ["message"] },
|
||||
"path": { "root": "state", "parts": ["raw", "echoed"] }
|
||||
}
|
||||
],
|
||||
"start": "call",
|
||||
"steps": {
|
||||
"call": {
|
||||
"use": "demo.echo",
|
||||
"input": [
|
||||
{
|
||||
"target": { "root": "local", "parts": ["text"] },
|
||||
"path": { "root": "input", "parts": ["text"] }
|
||||
},
|
||||
{
|
||||
"target": { "root": "local", "parts": ["fail"] },
|
||||
"path": { "root": "input", "parts": ["fail"] }
|
||||
}
|
||||
],
|
||||
"output": [
|
||||
{
|
||||
"source": { "root": "local", "parts": ["echoed"] },
|
||||
"target": { "root": "state", "parts": ["raw", "echoed"] }
|
||||
}
|
||||
]
|
||||
},
|
||||
"end_error": {
|
||||
"end": { "outcome": "error" }
|
||||
}
|
||||
},
|
||||
"routes": {
|
||||
"call": {
|
||||
"ok": "__end__",
|
||||
"error": "end_error"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Read it as:
|
||||
|
||||
- `call.ok` finishes the workflow with outcome `ok`.
|
||||
- `call.error` executes `end_error`, which finishes the workflow with outcome
|
||||
`error`.
|
||||
- both terminal paths project `state.raw.echoed` into public output field
|
||||
`message`.
|
||||
|
||||
## Binding Shape
|
||||
|
||||
Draft `use` steps use the same canonical binding structs as core `NodeUse`:
|
||||
@@ -356,9 +446,26 @@ provided on resume back into workflow state. Older map-shaped `request` and
|
||||
`resume` values are accepted only as parse compatibility and dump back to the
|
||||
canonical list shape.
|
||||
|
||||
Saved interrupting artifacts are still limited in the current execution
|
||||
surface. If a deployment reports `interrupting_artifact_unsupported`, that is a
|
||||
known platform limitation rather than a draft bug.
|
||||
Saved interrupting artifacts can pause and resume through deployment runs while
|
||||
the MCP server process stays alive. The run response includes a process-local
|
||||
`run_id`; pass that to `wf.workflow.resume_run` with the resume payload.
|
||||
Persisted run storage is still future work, so a server restart invalidates
|
||||
that in-memory run id.
|
||||
|
||||
### `end`
|
||||
|
||||
Declares an explicit workflow terminal outcome.
|
||||
|
||||
```json
|
||||
{
|
||||
"end": {
|
||||
"outcome": "error"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use explicit `end` steps for non-`ok` workflow outcomes. The legacy `__end__`
|
||||
destination remains the shorthand for public workflow outcome `ok`.
|
||||
|
||||
### `join`
|
||||
|
||||
|
||||
@@ -79,3 +79,16 @@ def register_artifact_tools(server: FastMCP, service: WfMcpService) -> None:
|
||||
deployment_id=deployment_id,
|
||||
workflow_input=workflow_input,
|
||||
)
|
||||
|
||||
@server.tool()
|
||||
async def resume_workflow_run(
|
||||
run_id: str,
|
||||
resume_payload: dict[str, Any],
|
||||
resume_outcome: str = "submitted",
|
||||
) -> dict[str, Any]:
|
||||
"""Resume a process-local interrupted deployment run."""
|
||||
return await handlers.resume_run(
|
||||
run_id=run_id,
|
||||
resume_payload=resume_payload,
|
||||
resume_outcome=resume_outcome,
|
||||
)
|
||||
|
||||
@@ -18,7 +18,13 @@ from wf_artifacts import (
|
||||
artifact_catalog_entry,
|
||||
)
|
||||
from wf_authoring import NodeReturn, NodeSpec
|
||||
from wf_core import NodeUse, Workflow, execute_workflow_async
|
||||
from wf_core import (
|
||||
NodeUse,
|
||||
RunState,
|
||||
Workflow,
|
||||
execute_workflow_async,
|
||||
resume_workflow_async,
|
||||
)
|
||||
|
||||
from wf_platform import (
|
||||
CapabilityBuckets,
|
||||
@@ -664,6 +670,7 @@ class WfMcpService:
|
||||
"state_schema": plan.state_schema,
|
||||
"output_schema": plan.output_schema,
|
||||
"output": [binding.model_dump(mode="json") for binding in plan.output],
|
||||
"outcomes": plan.outcomes,
|
||||
"start": plan.start,
|
||||
"node_defs": [node.model_dump() for node in node_defs.values()],
|
||||
"nodes": nodes,
|
||||
@@ -671,20 +678,19 @@ class WfMcpService:
|
||||
}
|
||||
return Workflow.model_validate(payload)
|
||||
|
||||
async def run_workflow_from_plan(
|
||||
def _prepare_workflow_runtime(
|
||||
self,
|
||||
plan: RawWorkflowPlan,
|
||||
workflow_input: dict[str, Any],
|
||||
deployment: WorkflowDeployment | None = None,
|
||||
artifact: WorkflowArtifact | None = None,
|
||||
):
|
||||
self._record_event(
|
||||
make_event(
|
||||
"workflow_run_started",
|
||||
workflow_name=plan.name,
|
||||
payload={"input_keys": sorted(workflow_input.keys())},
|
||||
)
|
||||
)
|
||||
*,
|
||||
deployment: WorkflowDeployment | None,
|
||||
artifact: WorkflowArtifact | None,
|
||||
) -> tuple[Workflow, dict[str, Any], dict[str, Any], dict[str, Any]]:
|
||||
"""Resolve bindings once into the executable pieces core expects.
|
||||
|
||||
Saved-run resume must rebuild prepared dependencies from the current
|
||||
in-memory service state. Durable resume will need a stricter snapshot,
|
||||
but this keeps the current platform boundary explicit.
|
||||
"""
|
||||
plan_node_names = [
|
||||
node.node for node in plan.nodes if isinstance(node, NodeUse)
|
||||
]
|
||||
@@ -716,11 +722,39 @@ class WfMcpService:
|
||||
compile_plan=self.compile_plan,
|
||||
)
|
||||
workflow = self.compile_plan(plan, dependencies.node_name_bindings)
|
||||
return (
|
||||
workflow,
|
||||
dependencies.node_registry,
|
||||
dependencies.reducers,
|
||||
prepared_subgraphs,
|
||||
)
|
||||
|
||||
async def run_workflow_from_plan(
|
||||
self,
|
||||
plan: RawWorkflowPlan,
|
||||
workflow_input: dict[str, Any],
|
||||
deployment: WorkflowDeployment | None = None,
|
||||
artifact: WorkflowArtifact | None = None,
|
||||
):
|
||||
self._record_event(
|
||||
make_event(
|
||||
"workflow_run_started",
|
||||
workflow_name=plan.name,
|
||||
payload={"input_keys": sorted(workflow_input.keys())},
|
||||
)
|
||||
)
|
||||
workflow, registry, reducers, prepared_subgraphs = (
|
||||
self._prepare_workflow_runtime(
|
||||
plan,
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
)
|
||||
)
|
||||
run = await execute_workflow_async(
|
||||
workflow,
|
||||
workflow_input,
|
||||
dependencies.node_registry,
|
||||
reducers=dependencies.reducers,
|
||||
registry,
|
||||
reducers=reducers,
|
||||
subgraphs=prepared_subgraphs,
|
||||
)
|
||||
self._record_event(
|
||||
@@ -732,6 +766,42 @@ class WfMcpService:
|
||||
)
|
||||
return run
|
||||
|
||||
async def resume_workflow_from_plan(
|
||||
self,
|
||||
plan: RawWorkflowPlan,
|
||||
run: RunState,
|
||||
*,
|
||||
resume_payload: dict[str, Any],
|
||||
resume_outcome: str = "submitted",
|
||||
deployment: WorkflowDeployment | None = None,
|
||||
artifact: WorkflowArtifact | None = None,
|
||||
) -> RunState:
|
||||
"""Resume one in-memory run using freshly resolved runtime dependencies."""
|
||||
workflow, registry, reducers, prepared_subgraphs = (
|
||||
self._prepare_workflow_runtime(
|
||||
plan,
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
)
|
||||
)
|
||||
resumed = await resume_workflow_async(
|
||||
workflow,
|
||||
run,
|
||||
registry,
|
||||
resume_payload=resume_payload,
|
||||
resume_outcome=resume_outcome,
|
||||
reducers=reducers,
|
||||
subgraphs=prepared_subgraphs,
|
||||
)
|
||||
self._record_event(
|
||||
make_event(
|
||||
"workflow_run_resumed",
|
||||
workflow_name=plan.name,
|
||||
payload={"status": resumed.status.value},
|
||||
)
|
||||
)
|
||||
return resumed
|
||||
|
||||
def list_events(self) -> list[McpEvent]:
|
||||
return self.event_bus.list_events()
|
||||
|
||||
|
||||
@@ -49,6 +49,13 @@ class RawWorkflowPlan(BaseModel):
|
||||
input_schema: dict[str, Any]
|
||||
state_schema: dict[str, Any]
|
||||
output_schema: dict[str, Any]
|
||||
outcomes: list[str] = Field(
|
||||
default_factory=lambda: ["ok"],
|
||||
description=(
|
||||
"Declared public workflow outcomes. Legacy plans without this field "
|
||||
"default to ok."
|
||||
),
|
||||
)
|
||||
output: list[InputBinding] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import asdict
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from uuid import uuid4
|
||||
|
||||
from wf_artifacts import (
|
||||
ArtifactKind,
|
||||
@@ -52,21 +53,39 @@ from .constants import (
|
||||
from .models import TraceRange
|
||||
from .refs import parse_workflow_surface_capability_id
|
||||
from .saved_subgraphs import (
|
||||
interrupting_artifact_diagnostic,
|
||||
direct_wrapper_interrupt_diagnostic,
|
||||
resolve_saved_subgraph_tree,
|
||||
validate_saved_subgraph_tree,
|
||||
)
|
||||
from .wrapper_hints import wrapper_hints_for_capability
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from wf_core import RunState
|
||||
|
||||
from ..broker.service import WfMcpService
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ActiveWorkflowRun:
|
||||
"""In-memory paused deployment run.
|
||||
|
||||
This is intentionally not durable. It only makes interrupt/resume usable
|
||||
while the current MCP server process is alive; persisted run storage remains
|
||||
a separate platform concern.
|
||||
"""
|
||||
|
||||
deployment: WorkflowDeployment
|
||||
artifact: WorkflowArtifact
|
||||
plan: RawWorkflowPlan
|
||||
run: "RunState"
|
||||
|
||||
|
||||
class WorkflowSurfaceHandlers:
|
||||
"""Reusable implementation behind MCP workflow artifact tools."""
|
||||
|
||||
def __init__(self, service: WfMcpService) -> None:
|
||||
self.service = service
|
||||
self._active_runs: dict[str, ActiveWorkflowRun] = {}
|
||||
|
||||
async def list_artifacts(
|
||||
self,
|
||||
@@ -308,7 +327,7 @@ class WorkflowSurfaceHandlers:
|
||||
deployment_id: str | None,
|
||||
) -> dict[str, Any]:
|
||||
"""Execute a saved wrapper artifact through the workflow runner."""
|
||||
unsupported = interrupting_artifact_diagnostic(artifact)
|
||||
unsupported = direct_wrapper_interrupt_diagnostic(artifact)
|
||||
if unsupported is not None:
|
||||
raise ValueError(unsupported.message)
|
||||
|
||||
@@ -911,15 +930,6 @@ class WorkflowSurfaceHandlers:
|
||||
diagnostics=diagnostics,
|
||||
)
|
||||
|
||||
unsupported = interrupting_artifact_diagnostic(artifact)
|
||||
if unsupported is not None:
|
||||
return _run_payload(
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
status="unsupported",
|
||||
diagnostics=[unsupported],
|
||||
)
|
||||
|
||||
plan = _raw_plan_from_artifact(artifact)
|
||||
run = await self.service.run_workflow_from_plan(
|
||||
plan,
|
||||
@@ -927,10 +937,19 @@ class WorkflowSurfaceHandlers:
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
)
|
||||
run_id = self._save_active_run(
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
plan=plan,
|
||||
run=run,
|
||||
)
|
||||
return _run_payload(
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
status=run.status.value,
|
||||
run_id=run_id,
|
||||
interrupt=_interrupt_payload(run),
|
||||
outcome=run.outcome,
|
||||
output=run.output,
|
||||
trace_count=len(run.trace),
|
||||
trace=(
|
||||
@@ -951,6 +970,82 @@ class WorkflowSurfaceHandlers:
|
||||
),
|
||||
)
|
||||
|
||||
async def resume_run(
|
||||
self,
|
||||
*,
|
||||
run_id: str,
|
||||
resume_payload: dict[str, Any],
|
||||
resume_outcome: str = "submitted",
|
||||
trace_range: TraceRange | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Resume one interrupted in-memory deployment run."""
|
||||
active = self._active_runs[run_id]
|
||||
run = await self.service.resume_workflow_from_plan(
|
||||
active.plan,
|
||||
active.run,
|
||||
resume_payload=resume_payload,
|
||||
resume_outcome=resume_outcome,
|
||||
deployment=active.deployment,
|
||||
artifact=active.artifact,
|
||||
)
|
||||
active.run = run
|
||||
next_run_id = self._save_active_run(
|
||||
deployment=active.deployment,
|
||||
artifact=active.artifact,
|
||||
plan=active.plan,
|
||||
run=run,
|
||||
run_id=run_id,
|
||||
)
|
||||
return _run_payload(
|
||||
deployment=active.deployment,
|
||||
artifact=active.artifact,
|
||||
status=run.status.value,
|
||||
run_id=next_run_id,
|
||||
interrupt=_interrupt_payload(run),
|
||||
outcome=run.outcome,
|
||||
output=run.output,
|
||||
trace_count=len(run.trace),
|
||||
trace=(
|
||||
[
|
||||
asdict(entry)
|
||||
for entry in run.trace[
|
||||
trace_range.start : trace_range.start + trace_range.limit
|
||||
]
|
||||
]
|
||||
if trace_range is not None
|
||||
else None
|
||||
),
|
||||
trace_start=trace_range.start if trace_range is not None else None,
|
||||
trace_limit=trace_range.limit if trace_range is not None else None,
|
||||
trace_truncated=(
|
||||
trace_range is not None
|
||||
and len(run.trace) > trace_range.start + trace_range.limit
|
||||
),
|
||||
)
|
||||
|
||||
def _save_active_run(
|
||||
self,
|
||||
*,
|
||||
deployment: WorkflowDeployment,
|
||||
artifact: WorkflowArtifact,
|
||||
plan: RawWorkflowPlan,
|
||||
run: RunState,
|
||||
run_id: str | None = None,
|
||||
) -> str | None:
|
||||
"""Store only interrupted runs; terminal runs leave no resume handle."""
|
||||
if run.status.value != "interrupted":
|
||||
if run_id is not None:
|
||||
self._active_runs.pop(run_id, None)
|
||||
return None
|
||||
key = run_id or f"run_{uuid4().hex}"
|
||||
self._active_runs[key] = ActiveWorkflowRun(
|
||||
deployment=deployment,
|
||||
artifact=artifact,
|
||||
plan=plan,
|
||||
run=run,
|
||||
)
|
||||
return key
|
||||
|
||||
def _deployment_validation(
|
||||
self,
|
||||
deployment_id: str,
|
||||
@@ -1221,6 +1316,8 @@ def _raw_plan_from_artifact(artifact: WorkflowArtifact) -> RawWorkflowPlan:
|
||||
"input_schema": _plan_field(artifact, "input_schema"),
|
||||
"state_schema": _plan_field(artifact, "state_schema"),
|
||||
"output_schema": _plan_field(artifact, "output_schema"),
|
||||
"outcomes": artifact.plan.get("outcomes", ["ok"]),
|
||||
"output": artifact.plan.get("output", []),
|
||||
"start": _plan_field(artifact, "start"),
|
||||
"nodes": _plan_field(artifact, "nodes"),
|
||||
"edges": _plan_field(artifact, "edges"),
|
||||
@@ -1248,6 +1345,9 @@ def _run_payload(
|
||||
deployment: WorkflowDeployment,
|
||||
artifact: WorkflowArtifact,
|
||||
status: str,
|
||||
run_id: str | None = None,
|
||||
interrupt: dict[str, Any] | None = None,
|
||||
outcome: str | None = None,
|
||||
diagnostics: list[DependencyDiagnostic] | None = None,
|
||||
output: dict[str, Any] | None = None,
|
||||
trace_count: int = 0,
|
||||
@@ -1261,6 +1361,9 @@ def _run_payload(
|
||||
"artifact_id": artifact.id,
|
||||
"artifact_version": artifact.version,
|
||||
"status": status,
|
||||
"run_id": run_id,
|
||||
"interrupt": interrupt,
|
||||
"outcome": outcome,
|
||||
"output": output,
|
||||
"diagnostics": [
|
||||
diagnostic.model_dump(mode="json") for diagnostic in diagnostics or []
|
||||
@@ -1277,6 +1380,19 @@ def _run_payload(
|
||||
return payload
|
||||
|
||||
|
||||
def _interrupt_payload(run: RunState) -> dict[str, Any] | None:
|
||||
"""Return a JSON-safe interrupt payload for the current run, if paused."""
|
||||
if run.interrupt is None:
|
||||
return None
|
||||
payload = asdict(run.interrupt)
|
||||
route = payload.get("route")
|
||||
if isinstance(route, dict) and "workflow_ref" in route:
|
||||
workflow_ref = route["workflow_ref"]
|
||||
if hasattr(workflow_ref, "model_dump"):
|
||||
route["workflow_ref"] = workflow_ref.model_dump(mode="json")
|
||||
return payload
|
||||
|
||||
|
||||
def _deployment_summary(deployment: WorkflowDeployment) -> dict[str, Any]:
|
||||
"""Return compact deployment metadata for progressive list responses."""
|
||||
return {
|
||||
|
||||
@@ -83,9 +83,6 @@ def validate_saved_subgraph_tree(
|
||||
sources=sources,
|
||||
)
|
||||
)
|
||||
interrupt_diagnostic = interrupting_artifact_diagnostic(child)
|
||||
if interrupt_diagnostic is not None:
|
||||
diagnostics.append(interrupt_diagnostic)
|
||||
return diagnostics
|
||||
|
||||
|
||||
@@ -125,24 +122,25 @@ def prepare_saved_subgraphs(
|
||||
return prepared
|
||||
|
||||
|
||||
def interrupting_artifact_diagnostic(
|
||||
def direct_wrapper_interrupt_diagnostic(
|
||||
artifact: WorkflowArtifact,
|
||||
) -> DependencyDiagnostic | None:
|
||||
"""Reject saved interrupt workflows until the platform exposes resume."""
|
||||
"""Reject direct wrapper calls that cannot return a resumable run handle.
|
||||
|
||||
Deployment execution supports interrupt/resume through an in-memory
|
||||
`run_id`; `call_capability` remains a single-call authoring probe.
|
||||
"""
|
||||
if not any(isinstance(node, InterruptNode) for node in _artifact_steps(artifact)):
|
||||
return None
|
||||
return DependencyDiagnostic(
|
||||
severity=DiagnosticSeverity.ERROR,
|
||||
code="interrupting_artifact_unsupported",
|
||||
code="interrupting_wrapper_call_unsupported",
|
||||
logical_ref=f"workflow.{artifact.id}.v{artifact.version}",
|
||||
message=(
|
||||
"Running saved workflow artifacts with interrupt nodes is unsupported "
|
||||
"until nested run-state resume is implemented."
|
||||
),
|
||||
repair_hint=(
|
||||
"Run this workflow as a top-level core workflow or remove interrupt "
|
||||
"nodes before saving it as a runnable deployment."
|
||||
"Direct wrapper calls cannot pause for interrupt input; run the "
|
||||
"artifact through a deployment to receive a resumable run_id."
|
||||
),
|
||||
repair_hint="Save a deployment and call wf.workflow.run_deployment instead.",
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -603,8 +603,8 @@ def register_workflow_tools(server: FastMCP[Any], service: WfMcpService) -> None
|
||||
name="wf.workflow.run_deployment",
|
||||
title="Run Workflow Deployment",
|
||||
description=(
|
||||
"Run deployment_id with workflow_input and return status, output, "
|
||||
"diagnostics, and trace_count. Debug traces can include resolved "
|
||||
"Run deployment_id with workflow_input and return status, terminal "
|
||||
"outcome, output, diagnostics, and trace_count. Debug traces can include resolved "
|
||||
"inputs and state changes; pass trace_range only when needed."
|
||||
),
|
||||
)
|
||||
@@ -627,3 +627,32 @@ def register_workflow_tools(server: FastMCP[Any], service: WfMcpService) -> None
|
||||
workflow_input=workflow_input,
|
||||
trace_range=trace_range,
|
||||
)
|
||||
|
||||
@server.tool(
|
||||
name="wf.workflow.resume_run",
|
||||
title="Resume Workflow Run",
|
||||
description=(
|
||||
"Resume an interrupted in-memory deployment run returned by "
|
||||
"run_deployment. Run IDs are process-local and are not durable."
|
||||
),
|
||||
)
|
||||
async def resume_run(
|
||||
run_id: str,
|
||||
resume_payload: dict[str, Any],
|
||||
resume_outcome: str = "submitted",
|
||||
trace_range: Annotated[
|
||||
TraceRange | None,
|
||||
Field(
|
||||
description=(
|
||||
"Debug traces range to return after resume. Omit for the "
|
||||
"normal compact response."
|
||||
)
|
||||
),
|
||||
] = None,
|
||||
) -> dict[str, Any]:
|
||||
return await handlers.resume_run(
|
||||
run_id=run_id,
|
||||
resume_payload=resume_payload,
|
||||
resume_outcome=resume_outcome,
|
||||
trace_range=trace_range,
|
||||
)
|
||||
|
||||
@@ -5,7 +5,15 @@ from pydantic import ValidationError
|
||||
from wf_artifacts.drafts import WorkflowDraft
|
||||
from wf_artifacts.drafts.api import compile_workflow_draft, validate_workflow_draft
|
||||
from wf_artifacts.drafts.adapter import build_workflow_from_draft
|
||||
from wf_core import ConditionNode, EndNode, ForeachNode, NodeUse
|
||||
from wf_core import (
|
||||
ConditionNode,
|
||||
EndNode,
|
||||
ForeachNode,
|
||||
NodeDef,
|
||||
NodeUse,
|
||||
SchemaRef,
|
||||
execute_workflow,
|
||||
)
|
||||
from wf_core.models.steps import InputValueBinding
|
||||
|
||||
|
||||
@@ -107,6 +115,88 @@ def test_adapter_lowers_root_workflow_output_bindings() -> None:
|
||||
assert dumped["output"][0]["path"] == {"root": "state", "parts": ["raw", "echoed"]}
|
||||
|
||||
|
||||
def test_adapter_golden_draft_executes_ok_and_error_outcomes() -> None:
|
||||
draft = WorkflowDraft.model_validate(
|
||||
{
|
||||
"name": "golden_echo",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"text": {"type": "string"},
|
||||
"fail": {"type": "boolean"},
|
||||
},
|
||||
"required": ["text"],
|
||||
},
|
||||
"state_schema": {
|
||||
"type": "object",
|
||||
"properties": {"raw": {"type": "object"}},
|
||||
},
|
||||
"output_schema": {
|
||||
"type": "object",
|
||||
"properties": {"message": {"type": "string"}},
|
||||
},
|
||||
"outcomes": ["ok", "error"],
|
||||
"output": [{"target": "message", "path": "state.raw.echoed"}],
|
||||
"start": "call",
|
||||
"steps": {
|
||||
"call": {
|
||||
"use": "demo.echo",
|
||||
"input": [
|
||||
{"target": "text", "path": "input.text"},
|
||||
{"target": "fail", "path": "input.fail"},
|
||||
],
|
||||
"output": [{"source": "echoed", "target": "state.raw.echoed"}],
|
||||
},
|
||||
"end_error": {"end": {"outcome": "error"}},
|
||||
},
|
||||
"routes": {
|
||||
"call": {"ok": "__end__", "error": "end_error"},
|
||||
},
|
||||
}
|
||||
)
|
||||
workflow = build_workflow_from_draft(draft)
|
||||
workflow = workflow.model_copy(
|
||||
update={
|
||||
"node_defs": [
|
||||
NodeDef(
|
||||
name="demo.echo",
|
||||
input_schema=SchemaRef.model_validate(draft.input_schema),
|
||||
output_schema=SchemaRef.model_validate(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"echoed": {"type": "string"}},
|
||||
}
|
||||
),
|
||||
outcomes=["ok", "error"],
|
||||
)
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
def echo(payload: dict[str, object], _ctx: object) -> dict[str, object]:
|
||||
if payload.get("fail") is True:
|
||||
return {"outcome": "error", "output": {"echoed": "failed"}}
|
||||
return {"outcome": "ok", "output": {"echoed": str(payload["text"])}}
|
||||
|
||||
ok = execute_workflow(
|
||||
workflow,
|
||||
{"text": "hello", "fail": False},
|
||||
{"demo.echo": echo},
|
||||
)
|
||||
error = execute_workflow(
|
||||
workflow,
|
||||
{"text": "hello", "fail": True},
|
||||
{"demo.echo": echo},
|
||||
)
|
||||
|
||||
assert ok.status == "completed"
|
||||
assert ok.outcome == "ok"
|
||||
assert ok.output["message"] == "hello"
|
||||
assert error.status == "completed"
|
||||
assert error.outcome == "error"
|
||||
assert error.output["message"] == "failed"
|
||||
|
||||
|
||||
def test_adapter_lowers_static_inputs_for_constant_like_steps() -> None:
|
||||
draft = WorkflowDraft.model_validate(
|
||||
{
|
||||
|
||||
@@ -415,7 +415,7 @@ def test_broker_run_deployment_returns_unrunnable_for_dependency_errors() -> Non
|
||||
assert payload["diagnostics"][0]["code"] == "source_missing"
|
||||
|
||||
|
||||
def test_broker_run_deployment_rejects_interrupting_artifacts() -> None:
|
||||
def test_broker_run_deployment_pauses_and_resumes_interrupting_artifacts() -> None:
|
||||
artifact_store = FileWorkflowArtifactStore(
|
||||
local_temp_root() / "broker_run_interrupt_artifacts"
|
||||
)
|
||||
@@ -445,9 +445,25 @@ def test_broker_run_deployment_rejects_interrupting_artifacts() -> None:
|
||||
)
|
||||
payload = cast(dict[str, Any], cast(object, structured))
|
||||
|
||||
assert payload["status"] == "unsupported"
|
||||
assert payload["output"] is None
|
||||
assert payload["diagnostics"][0]["code"] == "interrupting_artifact_unsupported"
|
||||
assert payload["status"] == "interrupted"
|
||||
assert payload["output"] == {}
|
||||
assert isinstance(payload["run_id"], str)
|
||||
assert payload["interrupt"]["payload"]["message"] == "send?"
|
||||
|
||||
_content, structured = asyncio.run(
|
||||
server.call_tool(
|
||||
"resume_workflow_run",
|
||||
{
|
||||
"run_id": payload["run_id"],
|
||||
"resume_payload": {},
|
||||
},
|
||||
)
|
||||
)
|
||||
resumed = cast(dict[str, Any], cast(object, structured))
|
||||
|
||||
assert resumed["status"] == "completed"
|
||||
assert resumed["outcome"] == "submitted"
|
||||
assert resumed["run_id"] is None
|
||||
|
||||
|
||||
def test_build_service_from_config_uses_store_root_for_artifacts() -> None:
|
||||
@@ -556,6 +572,7 @@ def _interrupt_artifact() -> WorkflowArtifact:
|
||||
},
|
||||
"state_schema": {"fields": {}},
|
||||
"output_schema": {"type": "object", "properties": {}},
|
||||
"outcomes": ["submitted"],
|
||||
"start": "approval",
|
||||
"nodes": [
|
||||
{
|
||||
@@ -565,8 +582,11 @@ def _interrupt_artifact() -> WorkflowArtifact:
|
||||
"request": [input_binding("input.message", "message")],
|
||||
"resume": [],
|
||||
"outcomes": ["submitted"],
|
||||
}
|
||||
},
|
||||
{"id": "end_submitted", "type": "end", "outcome": "submitted"},
|
||||
],
|
||||
"edges": [
|
||||
{"from": "approval", "outcome": "submitted", "to": "end_submitted"}
|
||||
],
|
||||
"edges": [{"from": "approval", "outcome": "submitted", "to": "__end__"}],
|
||||
},
|
||||
)
|
||||
|
||||
@@ -94,18 +94,44 @@ def test_saved_child_missing_parent_binding_is_unrunnable() -> None:
|
||||
assert result["diagnostics"][0]["logical_ref"] == "demo.echo_tool"
|
||||
|
||||
|
||||
def test_interrupting_saved_child_remains_unrunnable_on_deployment_surface() -> None:
|
||||
def test_interrupting_saved_child_pauses_and_resumes_through_deployment_surface() -> (
|
||||
None
|
||||
):
|
||||
store = FileWorkflowArtifactStore(local_temp_root() / "saved_subgraph_interrupt")
|
||||
store.save_artifact(_parent_artifact())
|
||||
store.save_artifact(_interrupting_child_artifact())
|
||||
store.save_deployment(_deployment())
|
||||
handlers = _handlers(store)
|
||||
|
||||
result = asyncio.run(handlers.validate_deployment(deployment_id="parent.personal"))
|
||||
validation = asyncio.run(
|
||||
handlers.validate_deployment(deployment_id="parent.personal")
|
||||
)
|
||||
|
||||
assert result["status"] == "unrunnable"
|
||||
assert result["diagnostics"][0]["code"] == "interrupting_artifact_unsupported"
|
||||
assert result["diagnostics"][0]["logical_ref"] == "workflow.child.v1"
|
||||
assert validation["status"] == "runnable"
|
||||
assert validation["diagnostics"] == []
|
||||
|
||||
paused = asyncio.run(
|
||||
handlers.run_deployment(
|
||||
deployment_id="parent.personal",
|
||||
workflow_input={"text": "hello"},
|
||||
)
|
||||
)
|
||||
|
||||
assert paused["status"] == "interrupted"
|
||||
assert isinstance(paused["run_id"], str)
|
||||
assert paused["interrupt"]["node_id"] == "child_step"
|
||||
assert paused["interrupt"]["payload"]["question"] == "hello"
|
||||
|
||||
resumed = asyncio.run(
|
||||
handlers.resume_run(
|
||||
run_id=paused["run_id"],
|
||||
resume_payload={"answer": "world"},
|
||||
)
|
||||
)
|
||||
|
||||
assert resumed["status"] == "completed"
|
||||
assert resumed["outcome"] == "ok"
|
||||
assert resumed["output"]["echoed"] == "world"
|
||||
|
||||
|
||||
def test_missing_saved_child_is_unrunnable_on_deployment_surface() -> None:
|
||||
|
||||
Reference in New Issue
Block a user