concurrent foreach work
This commit is contained in:
@@ -95,6 +95,12 @@ Patch creation and commit must extract/reuse the existing node output
|
|||||||
validation, output binding, and reducer logic. Concurrent foreach must not
|
validation, output binding, and reducer logic. Concurrent foreach must not
|
||||||
create a second write system.
|
create a second write system.
|
||||||
|
|
||||||
|
Current sync V1 implements the barrier commit path only for `loop -> one node ->
|
||||||
|
END` item bodies. The runtime includes an explicit no-op overlay seam
|
||||||
|
(`state_view_for_frame`) so the next slice can add lineage-local reads without
|
||||||
|
rewiring node execution. Until that seam becomes real, multi-step concurrent
|
||||||
|
item bodies are rejected instead of reading stale parent state.
|
||||||
|
|
||||||
## Merge and Reducer Rules
|
## Merge and Reducer Rules
|
||||||
|
|
||||||
At a barrier, missing reducer means default replace only for single-writer
|
At a barrier, missing reducer means default replace only for single-writer
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
# Concurrent Foreach Phase 4 Roadmap
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Split concurrent foreach execution into safe, independently testable runtime slices.
|
||||||
|
|
||||||
|
**Architecture:** `foreach(mode="concurrent")` is the workflow mode. Sync runtime should interleave admitted item frames one node call at a time; async runtime may later run admitted async node handlers simultaneously. All item writes must flow through `StatePatch` and commit at a foreach barrier, not directly from child frames into shared state.
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.14, dataclasses, Pydantic v2, pytest, `wf_core.runtime.scheduler`, `wf_core.runtime.foreach_state`, `wf_core.runtime.ops.state`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
Already implemented:
|
||||||
|
|
||||||
|
- `ForeachNode.mode` accepts canonical `"concurrent"`.
|
||||||
|
- `ForeachConcurrentPolicy` exists with `max_active`, `max_outstanding`, and `interrupt="quiesce"`.
|
||||||
|
- Legacy `mode="parallel"` / `parallel={...}` parse into canonical concurrent shape.
|
||||||
|
- `ForeachItemErrorPolicy` exists with `fail`, `skip`, and `collect`.
|
||||||
|
- `completed_with_errors` is derived for `skip` and `collect`.
|
||||||
|
- `collect_to` is validated as a declared array state field.
|
||||||
|
- `StatePatch`, `build_output_patch(...)`, and `commit_state_patch(...)` exist.
|
||||||
|
- `ForeachBarrierState`, `PendingItemResult`, and `ItemErrorRecord` exist.
|
||||||
|
- Serial foreach progress now uses `ForeachBarrierState`.
|
||||||
|
- Runtime still rejects `foreach(mode="concurrent")`.
|
||||||
|
|
||||||
|
## Non-Goals For Phase 4
|
||||||
|
|
||||||
|
- Do not implement Fork/Gather graph nodes.
|
||||||
|
- Do not turn `JoinNode` into a real barrier.
|
||||||
|
- Do not run sync node handlers in threads or processes.
|
||||||
|
- Do not add OpenTelemetry.
|
||||||
|
- Do not add platform-level source/tool semaphores.
|
||||||
|
- Do not add persistent run storage.
|
||||||
|
|
||||||
|
## Slice 1: Sync Concurrent Foreach, Fail-Only
|
||||||
|
|
||||||
|
Implement first because it proves the scheduler and frame admission model without async task orchestration or handled item failures.
|
||||||
|
|
||||||
|
Scope:
|
||||||
|
|
||||||
|
- `foreach(mode="concurrent", item_error.action="fail")` runs in sync runtime.
|
||||||
|
- Parent foreach admits up to `concurrent.max_active` item frames.
|
||||||
|
- Scheduler interleaves item frames one step at a time.
|
||||||
|
- Child output writes are buffered as per-item `StatePatch` objects.
|
||||||
|
- Barrier commits all successful item patches only when every item succeeds.
|
||||||
|
- Any item runtime failure fails the whole run.
|
||||||
|
- `skip` and `collect` remain runtime-unsupported for concurrent mode.
|
||||||
|
|
||||||
|
Plan:
|
||||||
|
|
||||||
|
- See [`2026-05-22-concurrent-foreach-v1-sync-fail-only.md`](2026-05-22-concurrent-foreach-v1-sync-fail-only.md).
|
||||||
|
|
||||||
|
## Slice 2: Barrier Commit Conflict Semantics
|
||||||
|
|
||||||
|
Implement after Slice 1 if V1 keeps conflict behavior too conservative.
|
||||||
|
|
||||||
|
Scope:
|
||||||
|
|
||||||
|
- Detect sibling lineage writes to the same state path.
|
||||||
|
- If exactly one lineage writes a destination path, default replace is allowed.
|
||||||
|
- If multiple sibling lineages write the same destination path, a declared reducer is required.
|
||||||
|
- Ancestor/descendant writes across sibling lineages are conflicts unless an explicit future merge strategy covers them.
|
||||||
|
- Commit order is item index order, never completion order.
|
||||||
|
|
||||||
|
Files likely touched:
|
||||||
|
|
||||||
|
- `src/wf_core/runtime/foreach_state.py`
|
||||||
|
- `src/wf_core/runtime/ops/state.py`
|
||||||
|
- `src/wf_core/runtime/ops/foreach.py`
|
||||||
|
- `tests/core/test_concurrent_foreach.py`
|
||||||
|
|
||||||
|
Key tests:
|
||||||
|
|
||||||
|
- `test_concurrent_foreach_rejects_sibling_writes_without_reducer`
|
||||||
|
- `test_concurrent_foreach_applies_reducer_in_item_index_order`
|
||||||
|
- `test_concurrent_foreach_rejects_ancestor_descendant_write_conflict`
|
||||||
|
|
||||||
|
## Slice 3: Item Error Policies
|
||||||
|
|
||||||
|
Implement after barrier success commits are correct.
|
||||||
|
|
||||||
|
Scope:
|
||||||
|
|
||||||
|
- `item_error.action="skip"` continues after item runtime failures.
|
||||||
|
- `item_error.action="collect"` continues and writes structured errors to `collect_to`.
|
||||||
|
- Both emit `completed_with_errors` if at least one item failed.
|
||||||
|
- `collect` writes an empty list and emits `done` if all items succeed.
|
||||||
|
- Failed item frames remain `FAILED`; parent foreach decides whether the failure is handled.
|
||||||
|
|
||||||
|
Files likely touched:
|
||||||
|
|
||||||
|
- `src/wf_core/runtime/foreach_state.py`
|
||||||
|
- `src/wf_core/runtime/ops/foreach.py`
|
||||||
|
- `tests/core/test_concurrent_foreach_errors.py`
|
||||||
|
|
||||||
|
Key tests:
|
||||||
|
|
||||||
|
- `test_concurrent_foreach_skip_emits_completed_with_errors`
|
||||||
|
- `test_concurrent_foreach_collect_writes_ordered_error_records`
|
||||||
|
- `test_concurrent_foreach_collect_writes_empty_list_on_clean_success`
|
||||||
|
|
||||||
|
## Slice 4: Async Concurrent Foreach
|
||||||
|
|
||||||
|
Implement only after sync semantics are stable.
|
||||||
|
|
||||||
|
Scope:
|
||||||
|
|
||||||
|
- Async runtime may have multiple async node handler calls in flight.
|
||||||
|
- `concurrent.max_active` caps admitted/running item work.
|
||||||
|
- Sync handlers are still called normally; no thread/process executor.
|
||||||
|
- Trace remains append-only chronological execution history.
|
||||||
|
- Barrier commit order remains item index order.
|
||||||
|
|
||||||
|
Files likely touched:
|
||||||
|
|
||||||
|
- `src/wf_core/runtime/engine.py`
|
||||||
|
- `src/wf_core/runtime/step.py`
|
||||||
|
- `src/wf_core/runtime/ops/nodes.py`
|
||||||
|
- `src/wf_core/runtime/ops/foreach.py`
|
||||||
|
- `tests/core/test_concurrent_foreach_async.py`
|
||||||
|
|
||||||
|
Key tests:
|
||||||
|
|
||||||
|
- `test_async_concurrent_foreach_respects_max_active`
|
||||||
|
- `test_async_concurrent_foreach_commits_in_item_index_order`
|
||||||
|
|
||||||
|
## Slice 5: Interrupt Quiescence
|
||||||
|
|
||||||
|
Implement after async execution exists.
|
||||||
|
|
||||||
|
Scope:
|
||||||
|
|
||||||
|
- If any concurrent item interrupts, the whole run pauses.
|
||||||
|
- No new item frames are admitted after the interrupt.
|
||||||
|
- Already-started async node calls drain to pending results.
|
||||||
|
- The caller gets control only at a quiescent point.
|
||||||
|
- Pending results do not commit until resume/commit policy allows it.
|
||||||
|
|
||||||
|
Files likely touched:
|
||||||
|
|
||||||
|
- `src/wf_core/runtime/engine.py`
|
||||||
|
- `src/wf_core/runtime/preparation.py`
|
||||||
|
- `src/wf_core/runtime/ops/interrupts.py`
|
||||||
|
- `src/wf_core/runtime/ops/foreach.py`
|
||||||
|
- `tests/core/test_concurrent_foreach_interrupts.py`
|
||||||
|
|
||||||
|
Key tests:
|
||||||
|
|
||||||
|
- `test_concurrent_foreach_interrupt_returns_after_quiescence`
|
||||||
|
- `test_resume_prioritizes_interrupted_item_frame_before_siblings`
|
||||||
|
|
||||||
|
## Execution Order
|
||||||
|
|
||||||
|
1. Sync concurrent foreach, fail-only.
|
||||||
|
2. Barrier conflict semantics, if not fully covered by slice 1.
|
||||||
|
3. `skip` / `collect` item error policies.
|
||||||
|
4. Async concurrent foreach.
|
||||||
|
5. Interrupt quiescence.
|
||||||
|
|
||||||
|
## Self-Review
|
||||||
|
|
||||||
|
- Spec coverage: the roadmap covers scheduler admission, barrier commits, reducer conflicts, handled item failures, async handler execution, and interrupt quiescence.
|
||||||
|
- Placeholder scan: each slice has scope, likely files, and named tests; concrete code lives in the slice-specific plan.
|
||||||
|
- Type consistency: the roadmap uses canonical `concurrent`, `ForeachConcurrentPolicy`, `ForeachBarrierState`, `PendingItemResult`, and `StatePatch`.
|
||||||
@@ -22,6 +22,10 @@
|
|||||||
- Phase 4 is not implemented: `foreach(mode="concurrent")` still validates as a
|
- Phase 4 is not implemented: `foreach(mode="concurrent")` still validates as a
|
||||||
model shape but runtime execution rejects it until concurrent scheduling,
|
model shape but runtime execution rejects it until concurrent scheduling,
|
||||||
barrier commits, and item failure handling are implemented.
|
barrier commits, and item failure handling are implemented.
|
||||||
|
- Phase 4 is expanded into a dedicated roadmap:
|
||||||
|
[`2026-05-22-concurrent-foreach-phase4-roadmap.md`](2026-05-22-concurrent-foreach-phase4-roadmap.md).
|
||||||
|
Start with
|
||||||
|
[`2026-05-22-concurrent-foreach-v1-sync-fail-only.md`](2026-05-22-concurrent-foreach-v1-sync-fail-only.md).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,7 @@ from typing import Any, Literal
|
|||||||
from wf_core.errors import WorkflowExecutionError
|
from wf_core.errors import WorkflowExecutionError
|
||||||
from wf_core.run_state import ExecutionFrame
|
from wf_core.run_state import ExecutionFrame
|
||||||
from wf_core.runtime.ops.state import StatePatch
|
from wf_core.runtime.ops.state import StatePatch
|
||||||
|
from wf_core.runtime.scheduler import ForeachIterationMetadata
|
||||||
|
|
||||||
_BARRIER_METADATA_KEY = "foreach_barriers"
|
_BARRIER_METADATA_KEY = "foreach_barriers"
|
||||||
|
|
||||||
@@ -120,6 +121,7 @@ class ForeachBarrierState:
|
|||||||
"""Resumable state owned by one foreach parent frame."""
|
"""Resumable state owned by one foreach parent frame."""
|
||||||
|
|
||||||
next_index: int = 0
|
next_index: int = 0
|
||||||
|
mode: Literal["serial", "concurrent"] = "serial"
|
||||||
active_frame_ids: tuple[str, ...] = ()
|
active_frame_ids: tuple[str, ...] = ()
|
||||||
outstanding_frame_ids: tuple[str, ...] = ()
|
outstanding_frame_ids: tuple[str, ...] = ()
|
||||||
pending_results: dict[int, PendingItemResult] = field(default_factory=dict)
|
pending_results: dict[int, PendingItemResult] = field(default_factory=dict)
|
||||||
@@ -156,11 +158,14 @@ class ForeachBarrierState:
|
|||||||
if not isinstance(raw, dict):
|
if not isinstance(raw, dict):
|
||||||
raise WorkflowExecutionError("malformed foreach barrier state")
|
raise WorkflowExecutionError("malformed foreach barrier state")
|
||||||
next_index = raw.get("next_index")
|
next_index = raw.get("next_index")
|
||||||
|
mode = raw.get("mode", "serial")
|
||||||
active_frame_ids = _string_tuple(raw.get("active_frame_ids", ()))
|
active_frame_ids = _string_tuple(raw.get("active_frame_ids", ()))
|
||||||
outstanding_frame_ids = _string_tuple(raw.get("outstanding_frame_ids", ()))
|
outstanding_frame_ids = _string_tuple(raw.get("outstanding_frame_ids", ()))
|
||||||
pending_results = raw.get("pending_results", {})
|
pending_results = raw.get("pending_results", {})
|
||||||
if not isinstance(next_index, int):
|
if not isinstance(next_index, int):
|
||||||
raise WorkflowExecutionError("malformed foreach barrier next_index")
|
raise WorkflowExecutionError("malformed foreach barrier next_index")
|
||||||
|
if mode not in {"serial", "concurrent"}:
|
||||||
|
raise WorkflowExecutionError("malformed foreach barrier mode")
|
||||||
if not isinstance(pending_results, dict):
|
if not isinstance(pending_results, dict):
|
||||||
raise WorkflowExecutionError("malformed foreach barrier pending results")
|
raise WorkflowExecutionError("malformed foreach barrier pending results")
|
||||||
parsed_results: dict[int, PendingItemResult] = {}
|
parsed_results: dict[int, PendingItemResult] = {}
|
||||||
@@ -174,6 +179,7 @@ class ForeachBarrierState:
|
|||||||
parsed_results[index] = PendingItemResult.from_metadata(raw_result)
|
parsed_results[index] = PendingItemResult.from_metadata(raw_result)
|
||||||
return cls(
|
return cls(
|
||||||
next_index=next_index,
|
next_index=next_index,
|
||||||
|
mode=mode,
|
||||||
active_frame_ids=active_frame_ids,
|
active_frame_ids=active_frame_ids,
|
||||||
outstanding_frame_ids=outstanding_frame_ids,
|
outstanding_frame_ids=outstanding_frame_ids,
|
||||||
pending_results=parsed_results,
|
pending_results=parsed_results,
|
||||||
@@ -196,6 +202,7 @@ class ForeachBarrierState:
|
|||||||
def to_metadata(self) -> dict[str, Any]:
|
def to_metadata(self) -> dict[str, Any]:
|
||||||
return {
|
return {
|
||||||
"next_index": self.next_index,
|
"next_index": self.next_index,
|
||||||
|
"mode": self.mode,
|
||||||
"active_frame_ids": list(self.active_frame_ids),
|
"active_frame_ids": list(self.active_frame_ids),
|
||||||
"outstanding_frame_ids": list(self.outstanding_frame_ids),
|
"outstanding_frame_ids": list(self.outstanding_frame_ids),
|
||||||
"pending_results": {
|
"pending_results": {
|
||||||
@@ -204,6 +211,45 @@ class ForeachBarrierState:
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def start_child(self, frame_id: str) -> None:
|
||||||
|
"""Record one admitted child frame as active and outstanding."""
|
||||||
|
if frame_id in self.active_frame_ids or frame_id in self.outstanding_frame_ids:
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
f"foreach child frame {frame_id!r} already active"
|
||||||
|
)
|
||||||
|
self.active_frame_ids = (*self.active_frame_ids, frame_id)
|
||||||
|
self.outstanding_frame_ids = (*self.outstanding_frame_ids, frame_id)
|
||||||
|
|
||||||
|
def finish_child(self, frame_id: str) -> None:
|
||||||
|
"""Record one child frame as no longer active or outstanding."""
|
||||||
|
self.active_frame_ids = tuple(
|
||||||
|
item for item in self.active_frame_ids if item != frame_id
|
||||||
|
)
|
||||||
|
self.outstanding_frame_ids = tuple(
|
||||||
|
item for item in self.outstanding_frame_ids if item != frame_id
|
||||||
|
)
|
||||||
|
|
||||||
|
def add_success_patch(
|
||||||
|
self, *, index: int, frame_id: str, patch: StatePatch
|
||||||
|
) -> None:
|
||||||
|
"""Buffer one successful item patch by item index."""
|
||||||
|
self.pending_results[index] = PendingItemResult(
|
||||||
|
index=index,
|
||||||
|
frame_id=frame_id,
|
||||||
|
status="succeeded",
|
||||||
|
patch=patch,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def item_frame_owner(frame: ExecutionFrame) -> tuple[str, str, int] | None:
|
||||||
|
"""Return parent frame id, foreach node id, and item index for item frames."""
|
||||||
|
if frame.kind != "foreach_iteration" or frame.parent_frame_id is None:
|
||||||
|
return None
|
||||||
|
metadata = ForeachIterationMetadata.from_frame(frame)
|
||||||
|
if metadata is None:
|
||||||
|
return None
|
||||||
|
return frame.parent_frame_id, metadata.foreach_node_id, metadata.loop_index
|
||||||
|
|
||||||
|
|
||||||
def _string_tuple(raw: object) -> tuple[str, ...]:
|
def _string_tuple(raw: object) -> tuple[str, ...]:
|
||||||
if isinstance(raw, tuple) and all(isinstance(item, str) for item in raw):
|
if isinstance(raw, tuple) and all(isinstance(item, str) for item in raw):
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ from wf_core.runtime.ops.schemas import validate_payload_against_schema
|
|||||||
from wf_core.runtime.ops.state import project_output
|
from wf_core.runtime.ops.state import project_output
|
||||||
from wf_core.runtime.scheduler import (
|
from wf_core.runtime.scheduler import (
|
||||||
mark_frame_pending,
|
mark_frame_pending,
|
||||||
wake_parent_if_children_complete,
|
wake_parent_for_child_progress,
|
||||||
)
|
)
|
||||||
from wf_core.tokens import END
|
from wf_core.tokens import END
|
||||||
|
|
||||||
@@ -81,7 +81,7 @@ def advance_frame(
|
|||||||
if next_node_id == END:
|
if next_node_id == END:
|
||||||
frame.status = FrameStatus.COMPLETED
|
frame.status = FrameStatus.COMPLETED
|
||||||
frame.finished_at_node_id = END
|
frame.finished_at_node_id = END
|
||||||
wake_parent_if_children_complete(run, frame.id)
|
wake_parent_for_child_progress(run, frame.id)
|
||||||
else:
|
else:
|
||||||
frame.finished_at_node_id = None
|
frame.finished_at_node_id = None
|
||||||
mark_frame_pending(run, frame.id)
|
mark_frame_pending(run, frame.id)
|
||||||
|
|||||||
@@ -1,19 +1,24 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
|
||||||
from wf_core.conditions import safe_resolve_path
|
from wf_core.conditions import safe_resolve_path
|
||||||
from wf_core.errors import WorkflowExecutionError
|
from wf_core.errors import WorkflowExecutionError
|
||||||
from wf_core.models.steps import ForeachNode
|
from wf_core.models.steps import ForeachNode, NodeUse
|
||||||
from wf_core.models.workflow import Workflow
|
from wf_core.models.workflow import Workflow
|
||||||
from wf_core.run_state import ExecutionFrame, FrameStatus, RunState, StepExecutionResult
|
from wf_core.run_state import ExecutionFrame, FrameStatus, RunState, StepExecutionResult
|
||||||
from wf_core.runtime.foreach_state import ForeachBarrierState
|
from wf_core.runtime.foreach_state import ForeachBarrierState
|
||||||
from wf_core.runtime.ops.flow import advance_frame, append_step_result_trace
|
from wf_core.runtime.ops.flow import advance_frame, append_step_result_trace
|
||||||
from wf_core.runtime.ops.frames import frame_context_values
|
from wf_core.runtime.ops.frames import frame_context_values
|
||||||
from wf_core.runtime.ops.index import WorkflowIndex
|
from wf_core.runtime.ops.index import WorkflowIndex
|
||||||
|
from wf_core.runtime.ops.merges import ReducerDefinition
|
||||||
|
from wf_core.runtime.ops.state import build_barrier_patch, commit_state_patch
|
||||||
from wf_core.runtime.scheduler import (
|
from wf_core.runtime.scheduler import (
|
||||||
ForeachIterationMetadata,
|
ForeachIterationMetadata,
|
||||||
add_frame,
|
add_frame,
|
||||||
block_frame_on_children,
|
block_frame_on_children,
|
||||||
)
|
)
|
||||||
|
from wf_core.tokens import END
|
||||||
|
|
||||||
|
|
||||||
def step_foreach(
|
def step_foreach(
|
||||||
@@ -21,25 +26,32 @@ def step_foreach(
|
|||||||
run: RunState,
|
run: RunState,
|
||||||
step: ForeachNode,
|
step: ForeachNode,
|
||||||
index: WorkflowIndex,
|
index: WorkflowIndex,
|
||||||
|
*,
|
||||||
|
reducers: Mapping[str, ReducerDefinition] | None = None,
|
||||||
|
) -> RunState:
|
||||||
|
if step.mode == "serial":
|
||||||
|
return _step_foreach_serial(workflow, run, step, index)
|
||||||
|
return _step_foreach_concurrent(
|
||||||
|
workflow,
|
||||||
|
run,
|
||||||
|
step,
|
||||||
|
index,
|
||||||
|
reducers=reducers,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _step_foreach_serial(
|
||||||
|
workflow: Workflow,
|
||||||
|
run: RunState,
|
||||||
|
step: ForeachNode,
|
||||||
|
index: WorkflowIndex,
|
||||||
) -> RunState:
|
) -> RunState:
|
||||||
if step.mode != "serial":
|
if step.mode != "serial":
|
||||||
raise WorkflowExecutionError(
|
raise WorkflowExecutionError("serial foreach helper received non-serial mode")
|
||||||
"concurrent foreach execution is not implemented yet"
|
|
||||||
)
|
|
||||||
|
|
||||||
frame = run.current_frame()
|
frame = run.current_frame()
|
||||||
barrier = ForeachBarrierState.from_frame(frame, step.id) or ForeachBarrierState()
|
barrier = ForeachBarrierState.from_frame(frame, step.id) or ForeachBarrierState()
|
||||||
|
iterable = _resolve_foreach_iterable(run, frame, step)
|
||||||
iterable = safe_resolve_path(
|
|
||||||
str(step.over),
|
|
||||||
state=run.state,
|
|
||||||
workflow_input=run.workflow_input,
|
|
||||||
context=frame_context_values(frame),
|
|
||||||
)
|
|
||||||
if not isinstance(iterable, list):
|
|
||||||
raise WorkflowExecutionError(
|
|
||||||
f"foreach source {str(step.over)!r} must resolve to a list"
|
|
||||||
)
|
|
||||||
|
|
||||||
loop_index = barrier.next_index
|
loop_index = barrier.next_index
|
||||||
if loop_index >= len(iterable):
|
if loop_index >= len(iterable):
|
||||||
@@ -62,17 +74,10 @@ def step_foreach(
|
|||||||
return run
|
return run
|
||||||
|
|
||||||
loop_start = index.next_node_id(frame.node_id, "loop")
|
loop_start = index.next_node_id(frame.node_id, "loop")
|
||||||
|
|
||||||
item = iterable[loop_index]
|
item = iterable[loop_index]
|
||||||
barrier.next_index = loop_index + 1
|
barrier.next_index = loop_index + 1
|
||||||
barrier.save_to_frame(frame, step.id)
|
barrier.save_to_frame(frame, step.id)
|
||||||
child_id = f"{frame.id}:{step.id}:{loop_index}"
|
child_id = f"{frame.id}:{step.id}:{loop_index}"
|
||||||
child_metadata = ForeachIterationMetadata(
|
|
||||||
foreach_node_id=step.id,
|
|
||||||
loop_index=loop_index,
|
|
||||||
loop_item=item,
|
|
||||||
loop_alias=step.as_,
|
|
||||||
)
|
|
||||||
add_frame(
|
add_frame(
|
||||||
run,
|
run,
|
||||||
ExecutionFrame(
|
ExecutionFrame(
|
||||||
@@ -81,7 +86,12 @@ def step_foreach(
|
|||||||
node_id=loop_start,
|
node_id=loop_start,
|
||||||
status=FrameStatus.PENDING,
|
status=FrameStatus.PENDING,
|
||||||
parent_frame_id=frame.id,
|
parent_frame_id=frame.id,
|
||||||
metadata=child_metadata.to_metadata(),
|
metadata=ForeachIterationMetadata(
|
||||||
|
foreach_node_id=step.id,
|
||||||
|
loop_index=loop_index,
|
||||||
|
loop_item=item,
|
||||||
|
loop_alias=step.as_,
|
||||||
|
).to_metadata(),
|
||||||
),
|
),
|
||||||
ready=True,
|
ready=True,
|
||||||
)
|
)
|
||||||
@@ -101,3 +111,210 @@ def step_foreach(
|
|||||||
)
|
)
|
||||||
run.sync_from_current_frame()
|
run.sync_from_current_frame()
|
||||||
return run
|
return run
|
||||||
|
|
||||||
|
|
||||||
|
def _step_foreach_concurrent(
|
||||||
|
workflow: Workflow,
|
||||||
|
run: RunState,
|
||||||
|
step: ForeachNode,
|
||||||
|
index: WorkflowIndex,
|
||||||
|
*,
|
||||||
|
reducers: Mapping[str, ReducerDefinition] | None = None,
|
||||||
|
) -> RunState:
|
||||||
|
if step.item_error.action != "fail":
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
"concurrent foreach v1 only supports item_error.action='fail'"
|
||||||
|
)
|
||||||
|
if step.concurrent is None:
|
||||||
|
raise WorkflowExecutionError("concurrent foreach requires concurrent policy")
|
||||||
|
_validate_single_node_loop_body(index, step)
|
||||||
|
|
||||||
|
frame = run.current_frame()
|
||||||
|
barrier = ForeachBarrierState.from_frame(frame, step.id)
|
||||||
|
if barrier is None:
|
||||||
|
barrier = ForeachBarrierState(mode="concurrent")
|
||||||
|
elif barrier.mode != "concurrent":
|
||||||
|
raise WorkflowExecutionError("malformed concurrent foreach barrier mode")
|
||||||
|
|
||||||
|
_finish_completed_children(run, barrier)
|
||||||
|
iterable = _resolve_foreach_iterable(run, frame, step)
|
||||||
|
_admit_concurrent_children(
|
||||||
|
run=run,
|
||||||
|
frame=frame,
|
||||||
|
step=step,
|
||||||
|
index=index,
|
||||||
|
barrier=barrier,
|
||||||
|
iterable=iterable,
|
||||||
|
)
|
||||||
|
|
||||||
|
if barrier.next_index >= len(iterable) and not barrier.outstanding_frame_ids:
|
||||||
|
return _finish_concurrent_foreach(
|
||||||
|
workflow=workflow,
|
||||||
|
run=run,
|
||||||
|
frame=frame,
|
||||||
|
step=step,
|
||||||
|
index=index,
|
||||||
|
barrier=barrier,
|
||||||
|
reducers=reducers,
|
||||||
|
)
|
||||||
|
|
||||||
|
barrier.save_to_frame(frame, step.id)
|
||||||
|
block_frame_on_children(run, frame.id, barrier.outstanding_frame_ids)
|
||||||
|
run.sync_from_current_frame()
|
||||||
|
return run
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_foreach_iterable(
|
||||||
|
run: RunState,
|
||||||
|
frame: ExecutionFrame,
|
||||||
|
step: ForeachNode,
|
||||||
|
) -> list[object]:
|
||||||
|
iterable = safe_resolve_path(
|
||||||
|
str(step.over),
|
||||||
|
state=run.state,
|
||||||
|
workflow_input=run.workflow_input,
|
||||||
|
context=frame_context_values(frame),
|
||||||
|
)
|
||||||
|
if not isinstance(iterable, list):
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
f"foreach source {str(step.over)!r} must resolve to a list"
|
||||||
|
)
|
||||||
|
return iterable
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_single_node_loop_body(index: WorkflowIndex, step: ForeachNode) -> None:
|
||||||
|
"""Reject multi-step concurrent item bodies until item overlays are real.
|
||||||
|
|
||||||
|
The current slice has a no-op item-state overlay seam. Without a real overlay,
|
||||||
|
multi-node item bodies would read stale parent state after earlier item-local
|
||||||
|
writes, so V1 only allows loop -> one node -> END.
|
||||||
|
"""
|
||||||
|
loop_start = index.next_node_id(step.id, "loop")
|
||||||
|
loop_step = index.nodes_by_id.get(loop_start)
|
||||||
|
if not isinstance(loop_step, NodeUse):
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
"concurrent foreach v1 only supports loop bodies with one node"
|
||||||
|
)
|
||||||
|
node_def = index.node_defs[loop_step.node]
|
||||||
|
for outcome in node_def.outcomes:
|
||||||
|
if index.next_node_id(loop_step.id, outcome) != END:
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
"concurrent foreach v1 only supports loop bodies with one node"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _finish_completed_children(run: RunState, barrier: ForeachBarrierState) -> None:
|
||||||
|
for child_id in tuple(barrier.outstanding_frame_ids):
|
||||||
|
child = run.frames[child_id]
|
||||||
|
if child.status == FrameStatus.COMPLETED:
|
||||||
|
barrier.finish_child(child_id)
|
||||||
|
elif child.status == FrameStatus.FAILED:
|
||||||
|
message = child.metadata.get("error", "unknown item failure")
|
||||||
|
raise WorkflowExecutionError(
|
||||||
|
f"concurrent foreach item frame {child_id!r} failed: {message}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _admit_concurrent_children(
|
||||||
|
*,
|
||||||
|
run: RunState,
|
||||||
|
frame: ExecutionFrame,
|
||||||
|
step: ForeachNode,
|
||||||
|
index: WorkflowIndex,
|
||||||
|
barrier: ForeachBarrierState,
|
||||||
|
iterable: list[object],
|
||||||
|
) -> int:
|
||||||
|
if step.concurrent is None:
|
||||||
|
raise WorkflowExecutionError("concurrent foreach requires concurrent policy")
|
||||||
|
|
||||||
|
admitted = 0
|
||||||
|
loop_start = index.next_node_id(frame.node_id, "loop")
|
||||||
|
while (
|
||||||
|
barrier.next_index < len(iterable)
|
||||||
|
and len(barrier.active_frame_ids) < step.concurrent.max_active
|
||||||
|
and len(barrier.outstanding_frame_ids) < step.concurrent.max_outstanding
|
||||||
|
):
|
||||||
|
loop_index = barrier.next_index
|
||||||
|
item = iterable[loop_index]
|
||||||
|
child_id = f"{frame.id}:{step.id}:{loop_index}"
|
||||||
|
active_count = len(barrier.active_frame_ids)
|
||||||
|
barrier.next_index = loop_index + 1
|
||||||
|
barrier.start_child(child_id)
|
||||||
|
add_frame(
|
||||||
|
run,
|
||||||
|
ExecutionFrame(
|
||||||
|
id=child_id,
|
||||||
|
kind="foreach_iteration",
|
||||||
|
node_id=loop_start,
|
||||||
|
status=FrameStatus.PENDING,
|
||||||
|
parent_frame_id=frame.id,
|
||||||
|
metadata=ForeachIterationMetadata(
|
||||||
|
foreach_node_id=step.id,
|
||||||
|
loop_index=loop_index,
|
||||||
|
loop_item=item,
|
||||||
|
loop_alias=step.as_,
|
||||||
|
).to_metadata(),
|
||||||
|
),
|
||||||
|
ready=True,
|
||||||
|
)
|
||||||
|
append_step_result_trace(
|
||||||
|
run,
|
||||||
|
frame_id=frame.id,
|
||||||
|
node_id=frame.node_id,
|
||||||
|
step_type=step.type,
|
||||||
|
next_node_id=loop_start,
|
||||||
|
result=StepExecutionResult(
|
||||||
|
outcome="loop",
|
||||||
|
resolved_input={
|
||||||
|
"item": item,
|
||||||
|
"index": loop_index,
|
||||||
|
"active_count": active_count,
|
||||||
|
},
|
||||||
|
output={},
|
||||||
|
state_changes={},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
admitted += 1
|
||||||
|
return admitted
|
||||||
|
|
||||||
|
|
||||||
|
def _finish_concurrent_foreach(
|
||||||
|
*,
|
||||||
|
workflow: Workflow,
|
||||||
|
run: RunState,
|
||||||
|
frame: ExecutionFrame,
|
||||||
|
step: ForeachNode,
|
||||||
|
index: WorkflowIndex,
|
||||||
|
barrier: ForeachBarrierState,
|
||||||
|
reducers: Mapping[str, ReducerDefinition] | None = None,
|
||||||
|
) -> RunState:
|
||||||
|
next_node_id = index.next_node_id(frame.node_id, "done")
|
||||||
|
combined = build_barrier_patch(
|
||||||
|
workflow,
|
||||||
|
[
|
||||||
|
barrier.pending_results[item_index].patch
|
||||||
|
for item_index in sorted(barrier.pending_results)
|
||||||
|
],
|
||||||
|
run.state,
|
||||||
|
reducers=reducers,
|
||||||
|
)
|
||||||
|
state_changes = commit_state_patch(run.state, combined)
|
||||||
|
append_step_result_trace(
|
||||||
|
run,
|
||||||
|
frame_id=frame.id,
|
||||||
|
node_id=frame.node_id,
|
||||||
|
step_type=step.type,
|
||||||
|
next_node_id=next_node_id,
|
||||||
|
result=StepExecutionResult(
|
||||||
|
outcome="done",
|
||||||
|
resolved_input={
|
||||||
|
"count": barrier.next_index,
|
||||||
|
"index": barrier.next_index,
|
||||||
|
"committed_items": len(barrier.pending_results),
|
||||||
|
},
|
||||||
|
output={},
|
||||||
|
state_changes=state_changes,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
advance_frame(run, frame, outcome="done", next_node_id=next_node_id)
|
||||||
|
return run
|
||||||
|
|||||||
@@ -11,10 +11,12 @@ from wf_core.models.schemas import NodeDef
|
|||||||
from wf_core.models.steps import InputPathBinding, InputValueBinding, NodeUse
|
from wf_core.models.steps import InputPathBinding, InputValueBinding, NodeUse
|
||||||
from wf_core.models.workflow import Workflow
|
from wf_core.models.workflow import Workflow
|
||||||
from wf_core.run_state import RunState, RuntimeContext, StepExecutionResult
|
from wf_core.run_state import RunState, RuntimeContext, StepExecutionResult
|
||||||
|
from wf_core.runtime.foreach_state import ForeachBarrierState, item_frame_owner
|
||||||
from wf_core.runtime.ops.frames import frame_context_values
|
from wf_core.runtime.ops.frames import frame_context_values
|
||||||
from wf_core.runtime.ops.merges import ReducerDefinition
|
from wf_core.runtime.ops.merges import ReducerDefinition
|
||||||
|
from wf_core.runtime.ops.overlays import state_view_for_frame
|
||||||
from wf_core.runtime.ops.schemas import validate_payload_against_schema
|
from wf_core.runtime.ops.schemas import validate_payload_against_schema
|
||||||
from wf_core.runtime.ops.state import apply_output_bindings
|
from wf_core.runtime.ops.state import build_output_patch, commit_state_patch
|
||||||
|
|
||||||
NodeHandler = Callable[[dict[str, Any], RuntimeContext], NodeResult | dict[str, Any]]
|
NodeHandler = Callable[[dict[str, Any], RuntimeContext], NodeResult | dict[str, Any]]
|
||||||
AsyncNodeHandler = Callable[
|
AsyncNodeHandler = Callable[
|
||||||
@@ -32,6 +34,7 @@ def _resolve_node_execution(
|
|||||||
) -> tuple[dict[str, Any], RuntimeContext]:
|
) -> tuple[dict[str, Any], RuntimeContext]:
|
||||||
frame = run.current_frame()
|
frame = run.current_frame()
|
||||||
context_values = frame_context_values(frame)
|
context_values = frame_context_values(frame)
|
||||||
|
state_view = state_view_for_frame(run, frame)
|
||||||
resolved_input: dict[str, Any] = {}
|
resolved_input: dict[str, Any] = {}
|
||||||
for binding in node.input:
|
for binding in node.input:
|
||||||
if isinstance(binding, InputValueBinding):
|
if isinstance(binding, InputValueBinding):
|
||||||
@@ -39,7 +42,7 @@ def _resolve_node_execution(
|
|||||||
elif isinstance(binding, InputPathBinding):
|
elif isinstance(binding, InputPathBinding):
|
||||||
value = safe_resolve_path(
|
value = safe_resolve_path(
|
||||||
str(binding.path),
|
str(binding.path),
|
||||||
state=run.state,
|
state=state_view,
|
||||||
workflow_input=run.workflow_input,
|
workflow_input=run.workflow_input,
|
||||||
context=context_values,
|
context=context_values,
|
||||||
)
|
)
|
||||||
@@ -85,13 +88,30 @@ def _finalize_node_execution(
|
|||||||
validate_payload_against_schema(
|
validate_payload_against_schema(
|
||||||
node_def.output_schema, result.output, f"node output for {node.id}"
|
node_def.output_schema, result.output, f"node output for {node.id}"
|
||||||
)
|
)
|
||||||
state_changes = apply_output_bindings(
|
patch = build_output_patch(
|
||||||
workflow,
|
workflow,
|
||||||
node.output,
|
node.output,
|
||||||
result.output,
|
result.output,
|
||||||
run.state,
|
run.state,
|
||||||
reducers=reducers,
|
reducers=reducers,
|
||||||
)
|
)
|
||||||
|
owner = item_frame_owner(run.current_frame())
|
||||||
|
if owner is None:
|
||||||
|
state_changes = commit_state_patch(run.state, patch)
|
||||||
|
else:
|
||||||
|
parent_frame_id, foreach_node_id, item_index = owner
|
||||||
|
parent_frame = run.frames[parent_frame_id]
|
||||||
|
barrier = ForeachBarrierState.from_frame(parent_frame, foreach_node_id)
|
||||||
|
if barrier is not None and barrier.mode == "concurrent":
|
||||||
|
barrier.add_success_patch(
|
||||||
|
index=item_index,
|
||||||
|
frame_id=run.current_frame().id,
|
||||||
|
patch=patch,
|
||||||
|
)
|
||||||
|
barrier.save_to_frame(parent_frame, foreach_node_id)
|
||||||
|
state_changes = {}
|
||||||
|
else:
|
||||||
|
state_changes = commit_state_patch(run.state, patch)
|
||||||
return StepExecutionResult(
|
return StepExecutionResult(
|
||||||
outcome=result.outcome,
|
outcome=result.outcome,
|
||||||
resolved_input=resolved_input,
|
resolved_input=resolved_input,
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from wf_core.run_state import ExecutionFrame, RunState
|
||||||
|
|
||||||
|
|
||||||
|
def state_view_for_frame(run: RunState, frame: ExecutionFrame) -> dict[str, Any]:
|
||||||
|
"""Return the state view visible to one execution frame.
|
||||||
|
|
||||||
|
This is intentionally a no-op seam for concurrent foreach V1. The first
|
||||||
|
sync-concurrent slice only supports single-node item bodies, so item frames
|
||||||
|
do not need to read their own prior buffered writes yet. The overlay slice
|
||||||
|
should replace this with parent-state plus item-local staged writes.
|
||||||
|
"""
|
||||||
|
return run.state
|
||||||
@@ -145,6 +145,46 @@ def commit_state_patch(state: dict[str, Any], patch: StatePatch) -> dict[str, An
|
|||||||
return dict(patch.changes)
|
return dict(patch.changes)
|
||||||
|
|
||||||
|
|
||||||
|
def build_barrier_patch(
|
||||||
|
workflow: Workflow,
|
||||||
|
item_patches: Sequence[StatePatch],
|
||||||
|
state: dict[str, Any],
|
||||||
|
*,
|
||||||
|
reducers: Mapping[str, ReducerDefinition] | None = None,
|
||||||
|
) -> StatePatch:
|
||||||
|
"""Build one committed barrier patch by replaying item writes in order.
|
||||||
|
|
||||||
|
Item patches are built against the parent-visible state. Their prepared
|
||||||
|
writes cannot be blindly merged because reducers must see the value produced
|
||||||
|
by earlier item patches. The barrier therefore replays trace-facing incoming
|
||||||
|
changes against a single staged state in deterministic item order.
|
||||||
|
"""
|
||||||
|
state_fields = workflow.state_schema.field_index()
|
||||||
|
staged_state = deepcopy(state)
|
||||||
|
prepared_patch: dict[StatePath, tuple[list[str], Any]] = {}
|
||||||
|
committed_changes: dict[str, Any] = {}
|
||||||
|
for item_patch in item_patches:
|
||||||
|
for destination, incoming_value in item_patch.changes.items():
|
||||||
|
destination_path = StatePath.parse(destination)
|
||||||
|
key_path, merged_value = prepare_state_value(
|
||||||
|
workflow,
|
||||||
|
staged_state,
|
||||||
|
destination_path,
|
||||||
|
incoming_value,
|
||||||
|
reducers=reducers,
|
||||||
|
state_fields=state_fields,
|
||||||
|
)
|
||||||
|
safe_set_nested_value(staged_state, key_path, merged_value)
|
||||||
|
prepared_patch[destination_path] = (key_path, merged_value)
|
||||||
|
committed_changes[destination] = merged_value
|
||||||
|
validate_staged_state_patch(staged_state, prepared_patch, state_fields)
|
||||||
|
return StatePatch(
|
||||||
|
changes=committed_changes,
|
||||||
|
_prepared_writes=prepared_patch,
|
||||||
|
_staged_state=staged_state,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def apply_mapped_state(
|
def apply_mapped_state(
|
||||||
workflow: Workflow,
|
workflow: Workflow,
|
||||||
source_data: dict[str, Any],
|
source_data: dict[str, Any],
|
||||||
|
|||||||
@@ -177,6 +177,21 @@ def wake_parent_if_children_complete(run: RunState, child_frame_id: str) -> None
|
|||||||
wake_frame(run, parent_id)
|
wake_frame(run, parent_id)
|
||||||
|
|
||||||
|
|
||||||
|
def wake_parent_for_child_progress(run: RunState, child_frame_id: str) -> None:
|
||||||
|
"""Wake a blocked parent after one child finishes so it can refill slots."""
|
||||||
|
child = _frame(run, child_frame_id)
|
||||||
|
parent_id = child.parent_frame_id
|
||||||
|
if parent_id is None:
|
||||||
|
return
|
||||||
|
parent = _frame(run, parent_id)
|
||||||
|
if parent.status != FrameStatus.BLOCKED:
|
||||||
|
return
|
||||||
|
block = BlockedOnChildren.from_frame(parent)
|
||||||
|
if block is None or child_frame_id not in block.child_frame_ids:
|
||||||
|
return
|
||||||
|
wake_frame(run, parent_id)
|
||||||
|
|
||||||
|
|
||||||
def resolve_no_ready_frames(run: RunState) -> RunStatus:
|
def resolve_no_ready_frames(run: RunState) -> RunStatus:
|
||||||
"""Classify an empty ready queue into terminal, paused, or deadlocked state."""
|
"""Classify an empty ready queue into terminal, paused, or deadlocked state."""
|
||||||
if run.status == RunStatus.INTERRUPTED:
|
if run.status == RunStatus.INTERRUPTED:
|
||||||
|
|||||||
@@ -99,7 +99,7 @@ def step_workflow(
|
|||||||
elif isinstance(step, InterruptNode):
|
elif isinstance(step, InterruptNode):
|
||||||
return handle_interrupt_step(run, step)
|
return handle_interrupt_step(run, step)
|
||||||
elif isinstance(step, ForeachNode):
|
elif isinstance(step, ForeachNode):
|
||||||
return step_foreach(workflow, run, step, index)
|
return step_foreach(workflow, run, step, index, reducers=reducers)
|
||||||
else:
|
else:
|
||||||
raise WorkflowExecutionError(
|
raise WorkflowExecutionError(
|
||||||
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
|
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
|
||||||
@@ -152,7 +152,7 @@ async def step_workflow_async(
|
|||||||
elif isinstance(step, InterruptNode):
|
elif isinstance(step, InterruptNode):
|
||||||
return handle_interrupt_step(run, step)
|
return handle_interrupt_step(run, step)
|
||||||
elif isinstance(step, ForeachNode):
|
elif isinstance(step, ForeachNode):
|
||||||
return step_foreach(workflow, run, step, index)
|
return step_foreach(workflow, run, step, index, reducers=reducers)
|
||||||
else:
|
else:
|
||||||
raise WorkflowExecutionError(
|
raise WorkflowExecutionError(
|
||||||
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
|
f"unsupported step type {getattr(step, 'type', type(step).__name__)!r}"
|
||||||
|
|||||||
@@ -0,0 +1,279 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from wf_core import (
|
||||||
|
END,
|
||||||
|
Edge,
|
||||||
|
ForeachNode,
|
||||||
|
NodeDef,
|
||||||
|
NodeUse,
|
||||||
|
ReducerRef,
|
||||||
|
SchemaRef,
|
||||||
|
StateField,
|
||||||
|
StateSchema,
|
||||||
|
Workflow,
|
||||||
|
WorkflowExecutionError,
|
||||||
|
execute_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_concurrent_foreach_interleaves_items_and_commits_at_barrier() -> None:
|
||||||
|
workflow = _workflow(
|
||||||
|
state_schema=StateSchema.from_field_map(
|
||||||
|
{
|
||||||
|
"items": StateField(type="array"),
|
||||||
|
"seen": StateField(
|
||||||
|
type="array",
|
||||||
|
reducer=ReducerRef(name="wf.std.append"),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
foreach=ForeachNode.model_validate(
|
||||||
|
{
|
||||||
|
"id": "each",
|
||||||
|
"type": "foreach",
|
||||||
|
"over": "state.items",
|
||||||
|
"as": "item",
|
||||||
|
"mode": "concurrent",
|
||||||
|
"concurrent": {"max_active": 2, "max_outstanding": 2},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
run = execute_workflow(
|
||||||
|
workflow,
|
||||||
|
{"items": ["a", "b", "c"]},
|
||||||
|
{"record": lambda payload, _ctx: {"outcome": "ok", "output": payload}},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert run.output["seen"] == ["a", "b", "c"]
|
||||||
|
assert run.state["seen"] == ["a", "b", "c"]
|
||||||
|
foreach_entries = [entry for entry in run.trace if entry.step_type == "foreach"]
|
||||||
|
assert foreach_entries[-1].outcome == "done"
|
||||||
|
assert foreach_entries[-1].state_changes["state.seen"] == ["a", "b", "c"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_concurrent_foreach_respects_max_active_by_refill_trace() -> None:
|
||||||
|
workflow = _workflow(
|
||||||
|
state_schema=StateSchema.from_field_map(
|
||||||
|
{
|
||||||
|
"items": StateField(type="array"),
|
||||||
|
"seen": StateField(
|
||||||
|
type="array",
|
||||||
|
reducer=ReducerRef(name="wf.std.append"),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
foreach=ForeachNode.model_validate(
|
||||||
|
{
|
||||||
|
"id": "each",
|
||||||
|
"type": "foreach",
|
||||||
|
"over": "state.items",
|
||||||
|
"as": "item",
|
||||||
|
"mode": "concurrent",
|
||||||
|
"concurrent": {"max_active": 2, "max_outstanding": 2},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
run = execute_workflow(
|
||||||
|
workflow,
|
||||||
|
{"items": ["a", "b", "c", "d"]},
|
||||||
|
{"record": lambda payload, _ctx: {"outcome": "ok", "output": payload}},
|
||||||
|
)
|
||||||
|
|
||||||
|
loop_entries = [
|
||||||
|
entry
|
||||||
|
for entry in run.trace
|
||||||
|
if entry.step_type == "foreach" and entry.outcome == "loop"
|
||||||
|
]
|
||||||
|
assert loop_entries[0].resolved_input["active_count"] == 0
|
||||||
|
assert loop_entries[1].resolved_input["active_count"] == 1
|
||||||
|
assert all(entry.resolved_input["active_count"] < 2 for entry in loop_entries)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_concurrent_foreach_rejects_non_fail_item_policy_for_now() -> None:
|
||||||
|
workflow = _workflow(
|
||||||
|
state_schema=StateSchema.from_field_map(
|
||||||
|
{
|
||||||
|
"items": StateField(type="array"),
|
||||||
|
"seen": StateField(
|
||||||
|
type="array",
|
||||||
|
reducer=ReducerRef(name="wf.std.append"),
|
||||||
|
),
|
||||||
|
"errors": StateField(type="array"),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
foreach=ForeachNode.model_validate(
|
||||||
|
{
|
||||||
|
"id": "each",
|
||||||
|
"type": "foreach",
|
||||||
|
"over": "state.items",
|
||||||
|
"as": "item",
|
||||||
|
"mode": "concurrent",
|
||||||
|
"concurrent": {"max_active": 2, "max_outstanding": 2},
|
||||||
|
"item_error": {"action": "collect", "collect_to": "state.errors"},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
include_completed_with_errors=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
WorkflowExecutionError,
|
||||||
|
match="only supports item_error.action='fail'",
|
||||||
|
):
|
||||||
|
execute_workflow(
|
||||||
|
workflow,
|
||||||
|
{"items": ["a"]},
|
||||||
|
{"record": lambda payload, _ctx: {"outcome": "ok", "output": payload}},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_concurrent_foreach_fails_run_on_item_runtime_error() -> None:
|
||||||
|
workflow = _workflow(
|
||||||
|
state_schema=StateSchema.from_field_map(
|
||||||
|
{
|
||||||
|
"items": StateField(type="array"),
|
||||||
|
"seen": StateField(
|
||||||
|
type="array",
|
||||||
|
reducer=ReducerRef(name="wf.std.append"),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
foreach=ForeachNode.model_validate(
|
||||||
|
{
|
||||||
|
"id": "each",
|
||||||
|
"type": "foreach",
|
||||||
|
"over": "state.items",
|
||||||
|
"as": "item",
|
||||||
|
"mode": "concurrent",
|
||||||
|
"concurrent": {"max_active": 2, "max_outstanding": 2},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def fail_on_b(payload: dict[str, Any], _ctx: object) -> dict[str, Any]:
|
||||||
|
if payload["value"] == "b":
|
||||||
|
raise ValueError("bad item")
|
||||||
|
return {"outcome": "ok", "output": payload}
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="bad item"):
|
||||||
|
execute_workflow(workflow, {"items": ["a", "b", "c"]}, {"record": fail_on_b})
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_concurrent_foreach_rejects_multi_step_item_body_for_now() -> None:
|
||||||
|
workflow = _workflow(
|
||||||
|
state_schema=StateSchema.from_field_map(
|
||||||
|
{
|
||||||
|
"items": StateField(type="array"),
|
||||||
|
"seen": StateField(
|
||||||
|
type="array",
|
||||||
|
reducer=ReducerRef(name="wf.std.append"),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
),
|
||||||
|
foreach=ForeachNode.model_validate(
|
||||||
|
{
|
||||||
|
"id": "each",
|
||||||
|
"type": "foreach",
|
||||||
|
"over": "state.items",
|
||||||
|
"as": "item",
|
||||||
|
"mode": "concurrent",
|
||||||
|
"concurrent": {"max_active": 2, "max_outstanding": 2},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
workflow.nodes.append(
|
||||||
|
NodeUse.model_validate(
|
||||||
|
{
|
||||||
|
"id": "after_record",
|
||||||
|
"type": "node",
|
||||||
|
"node": "record",
|
||||||
|
"input": [{"target": "value", "path": "state.seen"}],
|
||||||
|
"output": [{"source": "seen", "target": "state.seen"}],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
workflow.edges = [
|
||||||
|
Edge.model_validate({"from": "each", "outcome": "loop", "to": "record"}),
|
||||||
|
Edge.model_validate({"from": "record", "outcome": "ok", "to": "after_record"}),
|
||||||
|
Edge.model_validate({"from": "after_record", "outcome": "ok", "to": END}),
|
||||||
|
Edge.model_validate({"from": "each", "outcome": "done", "to": END}),
|
||||||
|
]
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
WorkflowExecutionError,
|
||||||
|
match="only supports loop bodies with one node",
|
||||||
|
):
|
||||||
|
execute_workflow(
|
||||||
|
workflow,
|
||||||
|
{"items": ["a"]},
|
||||||
|
{"record": lambda payload, _ctx: {"outcome": "ok", "output": payload}},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _workflow(
|
||||||
|
*,
|
||||||
|
state_schema: StateSchema,
|
||||||
|
foreach: ForeachNode,
|
||||||
|
include_completed_with_errors: bool = False,
|
||||||
|
) -> Workflow:
|
||||||
|
edges = [
|
||||||
|
Edge.model_validate({"from": "each", "outcome": "loop", "to": "record"}),
|
||||||
|
Edge.model_validate({"from": "record", "outcome": "ok", "to": END}),
|
||||||
|
Edge.model_validate({"from": "each", "outcome": "done", "to": END}),
|
||||||
|
]
|
||||||
|
if include_completed_with_errors:
|
||||||
|
edges.append(
|
||||||
|
Edge.model_validate(
|
||||||
|
{"from": "each", "outcome": "completed_with_errors", "to": END}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return Workflow(
|
||||||
|
name="concurrent_foreach_v1",
|
||||||
|
input_schema=SchemaRef(
|
||||||
|
type="object",
|
||||||
|
properties={"items": {"type": "array"}},
|
||||||
|
),
|
||||||
|
state_schema=state_schema,
|
||||||
|
output_schema=SchemaRef(
|
||||||
|
type="object",
|
||||||
|
properties={"seen": {"type": "array"}},
|
||||||
|
),
|
||||||
|
node_defs=[
|
||||||
|
NodeDef(
|
||||||
|
name="record",
|
||||||
|
input_schema=SchemaRef(
|
||||||
|
type="object",
|
||||||
|
properties={"value": {}, "seen": {}},
|
||||||
|
required=["value", "seen"],
|
||||||
|
),
|
||||||
|
output_schema=SchemaRef(
|
||||||
|
type="object",
|
||||||
|
properties={"value": {}, "seen": {}},
|
||||||
|
required=["seen"],
|
||||||
|
),
|
||||||
|
outcomes=["ok"],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
start="each",
|
||||||
|
nodes=[
|
||||||
|
foreach,
|
||||||
|
NodeUse.model_validate(
|
||||||
|
{
|
||||||
|
"id": "record",
|
||||||
|
"type": "node",
|
||||||
|
"node": "record",
|
||||||
|
"input": [
|
||||||
|
{"target": "value", "path": "context.item"},
|
||||||
|
{"target": "seen", "path": "context.item"},
|
||||||
|
],
|
||||||
|
"output": [{"source": "seen", "target": "state.seen"}],
|
||||||
|
}
|
||||||
|
),
|
||||||
|
],
|
||||||
|
edges=edges,
|
||||||
|
)
|
||||||
@@ -66,6 +66,25 @@ def test_foreach_barrier_state_rejects_malformed_metadata() -> None:
|
|||||||
ForeachBarrierState.from_frame(frame, "each")
|
ForeachBarrierState.from_frame(frame, "each")
|
||||||
|
|
||||||
|
|
||||||
|
def test_foreach_barrier_tracks_active_and_outstanding_children() -> None:
|
||||||
|
barrier = ForeachBarrierState()
|
||||||
|
|
||||||
|
barrier.start_child("child-0")
|
||||||
|
barrier.start_child("child-1")
|
||||||
|
barrier.finish_child("child-0")
|
||||||
|
|
||||||
|
assert barrier.active_frame_ids == ("child-1",)
|
||||||
|
assert barrier.outstanding_frame_ids == ("child-1",)
|
||||||
|
|
||||||
|
|
||||||
|
def test_foreach_barrier_rejects_duplicate_child_start() -> None:
|
||||||
|
barrier = ForeachBarrierState()
|
||||||
|
barrier.start_child("child-0")
|
||||||
|
|
||||||
|
with pytest.raises(WorkflowExecutionError, match="already active"):
|
||||||
|
barrier.start_child("child-0")
|
||||||
|
|
||||||
|
|
||||||
def test_item_error_record_rejects_negative_index() -> None:
|
def test_item_error_record_rejects_negative_index() -> None:
|
||||||
with pytest.raises(WorkflowExecutionError, match="index"):
|
with pytest.raises(WorkflowExecutionError, match="index"):
|
||||||
ItemErrorRecord.from_metadata(
|
ItemErrorRecord.from_metadata(
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ from wf_core.runtime.scheduler import (
|
|||||||
select_next_frame,
|
select_next_frame,
|
||||||
resolve_no_ready_frames,
|
resolve_no_ready_frames,
|
||||||
wake_frame,
|
wake_frame,
|
||||||
|
wake_parent_for_child_progress,
|
||||||
wake_parent_if_children_complete,
|
wake_parent_if_children_complete,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -160,6 +161,27 @@ def test_child_completion_wakes_blocked_parent() -> None:
|
|||||||
assert run.ready_frame_ids == ["parent"]
|
assert run.ready_frame_ids == ["parent"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_wake_parent_when_child_finishes_for_refill() -> None:
|
||||||
|
run = _run()
|
||||||
|
add_frame(run, ExecutionFrame(id="parent", kind="root", node_id="foreach"))
|
||||||
|
add_frame(
|
||||||
|
run,
|
||||||
|
ExecutionFrame(
|
||||||
|
id="child",
|
||||||
|
kind="foreach_iteration",
|
||||||
|
node_id="__end__",
|
||||||
|
parent_frame_id="parent",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
block_frame_on_children(run, "parent", ("child", "other"))
|
||||||
|
run.frames["child"].status = FrameStatus.COMPLETED
|
||||||
|
|
||||||
|
wake_parent_for_child_progress(run, "child")
|
||||||
|
|
||||||
|
assert run.frames["parent"].status == FrameStatus.PENDING
|
||||||
|
assert run.ready_frame_ids == ["parent"]
|
||||||
|
|
||||||
|
|
||||||
def test_resume_wakes_interrupted_frame_at_front() -> None:
|
def test_resume_wakes_interrupted_frame_at_front() -> None:
|
||||||
run = _run()
|
run = _run()
|
||||||
add_frame(run, ExecutionFrame(id="waiting", kind="root", node_id="ask"))
|
add_frame(run, ExecutionFrame(id="waiting", kind="root", node_id="ask"))
|
||||||
|
|||||||
Reference in New Issue
Block a user