This commit is contained in:
lda
2026-05-22 22:37:10 +07:00 Verified
parent 9fe9c1452b
commit 1c5cf15815
13 changed files with 89 additions and 41 deletions
@@ -333,6 +333,8 @@ Add:
def validate_barrier_writes(
item_patches: Sequence[StatePatch],
state_fields: Mapping[StatePath, StateFieldDecl],
*,
reducers: Mapping[str, ReducerDefinition] | None = None,
) -> None:
"""Reject ambiguous sibling writes before replaying a foreach barrier.
@@ -369,7 +371,7 @@ In `build_barrier_patch(...)`, after:
add:
```python
validate_barrier_writes(item_patches, state_fields)
validate_barrier_writes(item_patches, state_fields, reducers=reducers)
```
- [ ] **Step 7: Verify focused unit tests**
@@ -135,7 +135,11 @@ def test_sync_concurrent_foreach_item_reads_own_buffered_write() -> None:
assert run.state["seen"] == ["scratch:a", "scratch:b", "scratch:c"]
```
Important detail: update the existing `record` `NodeUse` in `_workflow(...)` or inside this test so it writes `scratch` to `state.scratch` for this workflow. If `_workflow(...)` is too fixed for that, create a small dedicated helper for this test instead of making `_workflow(...)` harder to read.
Important detail: do not mutate the generic `_workflow(...)` helper for this test.
Create a dedicated helper such as `_multi_step_overlay_workflow()` whose `record`
`NodeUse` writes `scratch` to `state.scratch`. The test is specifically about an
item-local write followed by an item-local read, so the workflow shape should be
self-contained and obvious.
- [ ] **Step 2: Add a sibling isolation test**
@@ -498,7 +502,8 @@ def test_sync_concurrent_foreach_allows_multi_step_item_body_with_overlay() -> N
# Assert the workflow completes and output contains all expected values.
```
Prefer not duplicating the full workflow; extract a helper:
Prefer not duplicating the full workflow; reuse the dedicated helper introduced
for the overlay read/write tests:
```python
def _multi_step_overlay_workflow() -> Workflow:
@@ -4,7 +4,7 @@
**Goal:** Implement concurrent foreach incrementally without breaking serial workflows or duplicating state-write logic.
**Architecture:** The work is split into four independently shippable layers: policy models, state patch extraction, barrier runtime state, and concurrent execution. Each layer preserves current serial behavior and adds tests before implementation. `foreach(mode="concurrent")` remains unsupported until the final layer. Sync runtime should support deterministic interleaving once the mode is enabled; async runtime can additionally run admitted async node handlers simultaneously.
**Architecture:** The work was split into four independently shippable layers: policy models, state patch extraction, barrier runtime state, and concurrent execution. Each layer preserves current serial behavior and adds tests before implementation. Sync runtime supports deterministic interleaving for concurrent foreach; async runtime can additionally run admitted async node handlers simultaneously.
**Tech Stack:** Python 3.14, Pydantic v2, dataclasses, pytest, basedpyright, ruff, existing `wf_core` scheduler/runtime modules.
@@ -19,10 +19,10 @@
- Phase 3 is implemented: `wf_core.runtime.foreach_state` owns typed barrier
metadata and serial foreach progress now uses that metadata instead of ad hoc
`foreach_progress`.
- Phase 4 is not implemented: `foreach(mode="concurrent")` still validates as a
model shape but runtime execution rejects it until concurrent scheduling,
barrier commits, and item failure handling are implemented.
- Phase 4 is expanded into a dedicated roadmap:
- Phase 4 is implemented: `foreach(mode="concurrent")` supports sync
interleaving, async item-node batching, barrier commits, item error policies,
and quiescent interrupt handling.
- Phase 4 details live in the dedicated roadmap:
[`2026-05-22-concurrent-foreach-phase4-roadmap.md`](2026-05-22-concurrent-foreach-phase4-roadmap.md).
Start with
[`2026-05-22-concurrent-foreach-v1-sync-fail-only.md`](2026-05-22-concurrent-foreach-v1-sync-fail-only.md).
@@ -466,7 +466,8 @@ async def test_async_runtime_accepts_concurrent_foreach() -> None:
...
```
Expected before implementation: both tests fail because runtime still rejects concurrent mode.
Historical expectation before Phase 4: both tests failed because runtime rejected
concurrent mode. Current implementation status: these tests should pass.
- [ ] **Step 2: Add capacity tests**
@@ -575,10 +576,10 @@ Ship these as separate commits/PRs:
3. Phase 3: barrier metadata with serial behavior unchanged
4. Phase 4: concurrent execution
Do not start Phase 4 until Phase 2 and Phase 3 are stable. Concurrent foreach depends on patch extraction and resumable barrier state.
Phase 4 depended on Phase 2 and Phase 3 because concurrent foreach needs patch extraction and resumable barrier state. Keep future fork/gather or native subgraph work layered on top of those runtime primitives instead of replacing them.
## Self-Review
- Spec coverage: ADR 0002 decisions are represented across the four phases.
- Intentional gaps: explicit Fork/Gather, lineage-token graph nodes, OpenTelemetry, platform source/tool caps, and full run persistence are not included.
- Risk control: phases 1-3 preserve serial behavior and keep `mode="concurrent"` unsupported until phase 4.
- Risk control: phases 1-3 preserved serial behavior until phase 4 enabled `mode="concurrent"`.
+5 -4
View File
@@ -118,10 +118,11 @@ limits and intended adapter seam.
state patch commits. The remaining mapping design notes for future reducer
metadata are documented in
[`core_state_mapping_and_merge.md`](core_state_mapping_and_merge.md).
- Foreach is still serial-only. The scheduler foundation exists, but concurrent
foreach still needs explicit policy, implicit barrier state, lineage-aware
patch commits, and quiescent interrupt handling. `ForeachNode.over` is typed
as a `GraphSourcePath`, but execution is still serial.
- Foreach supports serial and concurrent execution. Concurrent foreach uses
explicit policy, typed barrier state, lineage-aware patch commits, item error
policy, and quiescent interrupt handling. Remaining gaps are higher-level
graph constructs such as native subgraphs, explicit fork/gather nodes, and
advanced conflict strategies beyond exact-path mergeable reducers.
- Interrupt lifecycle is still node-level and run-state-level. Long-lived
external subscriptions or notification streams need a separate lifecycle
design. Interrupt `request` and `resume` are canonical binding lists; nested