reducer policy

This commit is contained in:
lda
2026-05-22 20:54:03 +07:00 Verified
parent 70f3b19bc5
commit adbad56c80
13 changed files with 166 additions and 31 deletions
@@ -104,17 +104,19 @@ the foreach barrier commits.
## Merge and Reducer Rules ## Merge and Reducer Rules
At a barrier, missing reducer means default replace only for single-writer At a barrier, missing reducer means default replace only for single-writer
paths. Multiple sibling lineages writing the same state path require an explicit paths. Multiple sibling lineages writing the same state path require a reducer
reducer. Ancestor/descendant overlapping writes across lineages are conflicts whose metadata declares it `mergeable`. `replace` is an explicit reducer, but it
unless an explicit merge strategy covers them. is `exclusive`, so it is rejected for same-path sibling writes. Ancestor/
descendant overlapping writes across lineages are conflicts unless an explicit
future merge strategy covers them.
Reducers apply incrementally in deterministic lineage order. For foreach, that Reducers apply incrementally in deterministic lineage order. For foreach, that
means item index order. means item index order.
Current barrier validation enforces this policy for sibling foreach item Current barrier validation enforces this policy for sibling foreach item
lineages. Same-path sibling writes require an explicit non-`replace` reducer on lineages. Same-path sibling writes require a `mergeable` reducer on the exact
the exact destination state path. Ancestor/descendant sibling writes are destination state path. Ancestor/descendant sibling writes are rejected until a
rejected until a future explicit deep merge policy exists. future explicit deep merge policy exists.
## Interrupt and Failure Quiescence ## Interrupt and Failure Quiescence
+10
View File
@@ -288,6 +288,16 @@ to `dict.update` or `operator.or_`. It is not a recursive deep merge.
If recursive merge is ever needed, it should be explicit rather than hidden If recursive merge is ever needed, it should be explicit rather than hidden
inside `merge_object`. inside `merge_object`.
Reducers also declare a sibling-write policy for concurrent barriers:
- `exclusive`: valid for ordinary writes, rejected when multiple sibling
lineages write the same state path at one barrier.
- `mergeable`: allowed for same-path sibling writes; the barrier still replays
writes in deterministic item-index order.
`wf.std.replace` is `exclusive`. The other built-in reducers are currently
`mergeable`.
## Future Reducers ## Future Reducers
Reducers are a capability family, similar to reusable node specs: Reducers are a capability family, similar to reusable node specs:
@@ -4,7 +4,7 @@
**Goal:** Enforce deterministic and explicit write semantics when concurrent foreach sibling item lineages commit at the barrier. **Goal:** Enforce deterministic and explicit write semantics when concurrent foreach sibling item lineages commit at the barrier.
**Architecture:** Keep per-node patch building unchanged. Add a barrier-only validation step before replaying item patches: inspect all item patch destination paths, reject ambiguous sibling writes, and allow multi-writer paths only when the exact declared state path has an explicit non-replace reducer. Barrier replay still happens in item-index order through existing reducer logic. **Architecture:** Keep per-node patch building unchanged. Add a barrier-only validation step before replaying item patches: inspect all item patch destination paths, reject ambiguous sibling writes, and allow multi-writer paths only when the exact declared state path has a reducer whose metadata is `mergeable`. Barrier replay still happens in item-index order through existing reducer logic.
**Tech Stack:** Python 3.14, dataclasses, Pydantic v2 models, pytest, `StatePath`, `StateSchema.field_index()`, `StatePatch`, `ForeachBarrierState`, and existing reducer definitions. **Tech Stack:** Python 3.14, dataclasses, Pydantic v2 models, pytest, `StatePath`, `StateSchema.field_index()`, `StatePatch`, `ForeachBarrierState`, and existing reducer definitions.
@@ -17,14 +17,14 @@ This slice owns sibling write policy at a foreach barrier.
Allowed: Allowed:
- A destination path written by exactly one item lineage uses normal state rules. - A destination path written by exactly one item lineage uses normal state rules.
- A destination path written by multiple item lineages is allowed only if that exact declared state path has an explicit reducer other than `wf.std.replace`. - A destination path written by multiple item lineages is allowed only if that exact declared state path has a `mergeable` reducer.
- Multi-writer reducer replay is deterministic item-index order. - Multi-writer reducer replay is deterministic item-index order.
- Multiple nodes inside the same item lineage may write multiple paths; that is item-local overlay behavior from Slice 2. - Multiple nodes inside the same item lineage may write multiple paths; that is item-local overlay behavior from Slice 2.
Rejected: Rejected:
- Multiple sibling item lineages writing the same destination path with missing reducer/default replace. - Multiple sibling item lineages writing the same destination path with missing reducer/default replace.
- Multiple sibling item lineages writing the same destination path with explicit `wf.std.replace`. - Multiple sibling item lineages writing the same destination path with an `exclusive` reducer such as `wf.std.replace`.
- Sibling item lineages writing ancestor/descendant paths such as `state.person` and `state.person.name`. - Sibling item lineages writing ancestor/descendant paths such as `state.person` and `state.person.name`.
Important distinction: Important distinction:
@@ -508,9 +508,9 @@ In `docs/adr/0002-concurrent-foreach-policy-and-barrier-commits.md`, under
```markdown ```markdown
Current barrier validation enforces this policy for sibling foreach item Current barrier validation enforces this policy for sibling foreach item
lineages. Same-path sibling writes require an explicit non-`replace` reducer on lineages. Same-path sibling writes require a `mergeable` reducer on the exact
the exact destination state path. Ancestor/descendant sibling writes are destination state path. Ancestor/descendant sibling writes are rejected until a
rejected until a future explicit deep merge policy exists. future explicit deep merge policy exists.
``` ```
- [ ] **Step 2: Update roadmap Slice 3** - [ ] **Step 2: Update roadmap Slice 3**
@@ -28,7 +28,7 @@ Already implemented:
commits. commits.
- Multi-step concurrent item bodies are supported for fail-only item policy. - Multi-step concurrent item bodies are supported for fail-only item policy.
- Barrier write validation rejects ambiguous sibling writes: same-path sibling - Barrier write validation rejects ambiguous sibling writes: same-path sibling
writes require an explicit non-`replace` reducer, and ancestor/descendant writes require a `mergeable` reducer, and ancestor/descendant
sibling writes are rejected. sibling writes are rejected.
## Non-Goals For Phase 4 ## Non-Goals For Phase 4
@@ -88,7 +88,8 @@ Scope:
- Detect sibling lineage writes to the same state path. - Detect sibling lineage writes to the same state path.
- If exactly one lineage writes a destination path, default replace is allowed. - If exactly one lineage writes a destination path, default replace is allowed.
- If multiple sibling lineages write the same destination path, a declared reducer is required. - If multiple sibling lineages write the same destination path, a declared
`mergeable` reducer is required.
- Ancestor/descendant writes across sibling lineages are conflicts unless an explicit future merge strategy covers them. - Ancestor/descendant writes across sibling lineages are conflicts unless an explicit future merge strategy covers them.
- Commit order is item index order, never completion order. - Commit order is item index order, never completion order.
+7 -1
View File
@@ -6,7 +6,7 @@ from typing import Any, TypeVar, cast, overload
from pydantic import BaseModel from pydantic import BaseModel
from wf_core import ReducerSpec from wf_core import ReducerSpec, SiblingWritePolicy
from wf_core.runtime.ops.merges import ReducerDefinition from wf_core.runtime.ops.merges import ReducerDefinition
from .callables import ConfigReducerCallable, ConfigT, PlainReducerCallable from .callables import ConfigReducerCallable, ConfigT, PlainReducerCallable
@@ -28,6 +28,7 @@ def reducer(
*, *,
name: str | None = None, name: str | None = None,
description: str | None = None, description: str | None = None,
sibling_write_policy: SiblingWritePolicy = SiblingWritePolicy.MERGEABLE,
) -> AuthoredReducer: ... ) -> AuthoredReducer: ...
@@ -36,6 +37,7 @@ def reducer(
*, *,
name: str, name: str,
description: str | None = None, description: str | None = None,
sibling_write_policy: SiblingWritePolicy = SiblingWritePolicy.MERGEABLE,
) -> Callable[[Callable[..., Any]], AuthoredReducer]: ... ) -> Callable[[Callable[..., Any]], AuthoredReducer]: ...
@@ -45,6 +47,7 @@ def reducer(
name: str, name: str,
config_model: type[ConfigT], config_model: type[ConfigT],
description: str | None = None, description: str | None = None,
sibling_write_policy: SiblingWritePolicy = SiblingWritePolicy.MERGEABLE,
) -> Callable[[Callable[..., Any]], AuthoredReducer]: ... ) -> Callable[[Callable[..., Any]], AuthoredReducer]: ...
@@ -55,6 +58,7 @@ def reducer(
name: str | None = None, name: str | None = None,
config_model: type[BaseModel] | None = None, config_model: type[BaseModel] | None = None,
description: str | None = None, description: str | None = None,
sibling_write_policy: SiblingWritePolicy = SiblingWritePolicy.MERGEABLE,
) -> AuthoredReducer | Callable[[Callable[..., Any]], AuthoredReducer]: ) -> AuthoredReducer | Callable[[Callable[..., Any]], AuthoredReducer]:
"""Wrap a Python reducer function as a runtime reducer definition.""" """Wrap a Python reducer function as a runtime reducer definition."""
@@ -67,6 +71,7 @@ def reducer(
spec=ReducerSpec( spec=ReducerSpec(
name=reducer_name, name=reducer_name,
description=_clean_doc(reducer_description), description=_clean_doc(reducer_description),
sibling_write_policy=sibling_write_policy,
), ),
fn=raw, fn=raw,
) )
@@ -92,6 +97,7 @@ def reducer(
name=reducer_name, name=reducer_name,
description=_clean_doc(reducer_description), description=_clean_doc(reducer_description),
config_schema=config_model.model_json_schema(), config_schema=config_model.model_json_schema(),
sibling_write_policy=sibling_write_policy,
), ),
fn=runtime_fn, fn=runtime_fn,
accepts_config=True, accepts_config=True,
+2
View File
@@ -12,6 +12,7 @@ from .models import (
ReducerRef, ReducerRef,
ReducerSpec, ReducerSpec,
SchemaRef, SchemaRef,
SiblingWritePolicy,
StateField, StateField,
StateSchema, StateSchema,
Workflow, Workflow,
@@ -60,6 +61,7 @@ __all__ = [
"ReducerRef", "ReducerRef",
"ReducerSpec", "ReducerSpec",
"SchemaRef", "SchemaRef",
"SiblingWritePolicy",
"StateField", "StateField",
"StateSchema", "StateSchema",
"AsyncNodeHandler", "AsyncNodeHandler",
+2 -1
View File
@@ -9,7 +9,7 @@ from wf_core.models.conditions import (
VariadicCondition, VariadicCondition,
) )
from wf_core.models.results import NodeResult from wf_core.models.results import NodeResult
from wf_core.models.reducers import ReducerRef, ReducerSpec from wf_core.models.reducers import ReducerRef, ReducerSpec, SiblingWritePolicy
from wf_core.models.schemas import NodeDef, SchemaRef, StateField, StateSchema from wf_core.models.schemas import NodeDef, SchemaRef, StateField, StateSchema
from wf_core.models.steps import ( from wf_core.models.steps import (
ConditionNode, ConditionNode,
@@ -44,6 +44,7 @@ __all__ = [
"Operand", "Operand",
"PathOperand", "PathOperand",
"SchemaRef", "SchemaRef",
"SiblingWritePolicy",
"StateField", "StateField",
"StateSchema", "StateSchema",
"Step", "Step",
+16
View File
@@ -1,6 +1,7 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Mapping from collections.abc import Mapping
from enum import StrEnum
from typing import Any from typing import Any
from pydantic import BaseModel, ConfigDict, Field, model_validator from pydantic import BaseModel, ConfigDict, Field, model_validator
@@ -64,6 +65,13 @@ class ReducerRef(BaseModel):
return str(self.ref) return str(self.ref)
class SiblingWritePolicy(StrEnum):
"""Whether a reducer is safe for sibling foreach lineages at a barrier."""
EXCLUSIVE = "exclusive"
MERGEABLE = "mergeable"
class ReducerSpec(BaseModel): class ReducerSpec(BaseModel):
"""Inspectable metadata for one named pure state reducer.""" """Inspectable metadata for one named pure state reducer."""
@@ -78,3 +86,11 @@ class ReducerSpec(BaseModel):
"additionalProperties": False, "additionalProperties": False,
} }
) )
sibling_write_policy: SiblingWritePolicy = Field(
default=SiblingWritePolicy.MERGEABLE,
description=(
"Whether sibling foreach item lineages may write this state path "
"at one barrier. Exclusive reducers are valid for ordinary writes "
"but ambiguous for sibling barrier commits."
),
)
+30 -6
View File
@@ -5,7 +5,7 @@ from dataclasses import dataclass
from typing import Any, cast from typing import Any, cast
from wf_core.errors import WorkflowExecutionError from wf_core.errors import WorkflowExecutionError
from wf_core.models.reducers import ReducerRef, ReducerSpec from wf_core.models.reducers import ReducerRef, ReducerSpec, SiblingWritePolicy
from wf_core.runtime.ops.schemas import validate_payload_against_schema from wf_core.runtime.ops.schemas import validate_payload_against_schema
PlainReducer = Callable[[Any, Any], Any] PlainReducer = Callable[[Any, Any], Any]
@@ -116,6 +116,7 @@ DEFAULT_REDUCER_DEFINITIONS: Mapping[str, ReducerDefinition] = {
spec=ReducerSpec( spec=ReducerSpec(
name="wf.std.replace", name="wf.std.replace",
description="Replace the current state value with the incoming value.", description="Replace the current state value with the incoming value.",
sibling_write_policy=SiblingWritePolicy.EXCLUSIVE,
), ),
fn=replace_reducer, fn=replace_reducer,
), ),
@@ -157,6 +158,33 @@ DEFAULT_REDUCER_DEFINITIONS: Mapping[str, ReducerDefinition] = {
} }
def get_reducer_definition(
reducer: ReducerRef,
reducers: Mapping[str, ReducerDefinition] | None = None,
) -> ReducerDefinition:
"""Resolve a reducer from injected definitions plus built-ins."""
definition = None if reducers is None else reducers.get(reducer.name)
if definition is None:
definition = DEFAULT_REDUCER_DEFINITIONS.get(reducer.name)
if definition is None:
raise WorkflowExecutionError(f"unknown reducer {reducer.name!r}")
return definition
def reducer_allows_sibling_writes(
reducer: ReducerRef,
reducers: Mapping[str, ReducerDefinition] | None = None,
) -> bool:
"""Return whether a reducer can merge sibling foreach item writes.
Reducers remain pure value functions. Barrier conflict rules live here as
metadata so `replace` can be exclusive without teaching the reducer about
foreach, frames, or item ordering.
"""
definition = get_reducer_definition(reducer, reducers)
return definition.spec.sibling_write_policy is SiblingWritePolicy.MERGEABLE
def apply_reducer( def apply_reducer(
*, *,
reducer: ReducerRef, reducer: ReducerRef,
@@ -171,11 +199,7 @@ def apply_reducer(
tests and local packages can provide custom reducers without re-registering tests and local packages can provide custom reducers without re-registering
every `wf.std.*` reducer. every `wf.std.*` reducer.
""" """
definition = None if reducers is None else reducers.get(reducer.name) definition = get_reducer_definition(reducer, reducers)
if definition is None:
definition = DEFAULT_REDUCER_DEFINITIONS.get(reducer.name)
if definition is None:
raise WorkflowExecutionError(f"unknown reducer {reducer.name!r}")
return definition.apply( return definition.apply(
reducer=reducer, reducer=reducer,
current_value=current_value, current_value=current_value,
+13 -6
View File
@@ -19,7 +19,11 @@ from wf_core.paths import (
set_nested_value, set_nested_value,
split_graph_path, split_graph_path,
) )
from wf_core.runtime.ops.merges import ReducerDefinition, apply_reducer from wf_core.runtime.ops.merges import (
ReducerDefinition,
apply_reducer,
reducer_allows_sibling_writes,
)
from wf_core.runtime.ops.schemas import validate_payload_against_schema from wf_core.runtime.ops.schemas import validate_payload_against_schema
_MISSING = object() _MISSING = object()
@@ -175,7 +179,7 @@ def build_barrier_patch(
values would hide what actually landed in `RunState.state`. values would hide what actually landed in `RunState.state`.
""" """
state_fields = workflow.state_schema.field_index() state_fields = workflow.state_schema.field_index()
validate_barrier_writes(item_patches, state_fields) validate_barrier_writes(item_patches, state_fields, reducers=reducers)
staged_state = deepcopy(state) staged_state = deepcopy(state)
prepared_patch: dict[StatePath, tuple[list[str], Any]] = {} prepared_patch: dict[StatePath, tuple[list[str], Any]] = {}
committed_changes: dict[str, Any] = {} committed_changes: dict[str, Any] = {}
@@ -204,6 +208,8 @@ def build_barrier_patch(
def validate_barrier_writes( def validate_barrier_writes(
item_patches: Sequence[StatePatch], item_patches: Sequence[StatePatch],
state_fields: Mapping[StatePath, StateFieldDecl], state_fields: Mapping[StatePath, StateFieldDecl],
*,
reducers: Mapping[str, ReducerDefinition] | None = None,
) -> None: ) -> None:
"""Reject ambiguous sibling writes before replaying a foreach barrier. """Reject ambiguous sibling writes before replaying a foreach barrier.
@@ -216,11 +222,11 @@ def validate_barrier_writes(
if left.item_index == right.item_index: if left.item_index == right.item_index:
continue continue
if left.path == right.path: if left.path == right.path:
if _has_explicit_non_replace_reducer(left.path, state_fields): if _allows_sibling_writes(left.path, state_fields, reducers):
continue continue
raise WorkflowExecutionError( raise WorkflowExecutionError(
"multiple sibling writes to " "multiple sibling writes to "
f"{left.source_key!r} require an explicit reducer" f"{left.source_key!r} require a mergeable reducer"
) )
if _state_paths_overlap(left.path, right.path): if _state_paths_overlap(left.path, right.path):
raise WorkflowExecutionError( raise WorkflowExecutionError(
@@ -244,14 +250,15 @@ def _barrier_writes(item_patches: Sequence[StatePatch]) -> list[_BarrierWrite]:
return writes return writes
def _has_explicit_non_replace_reducer( def _allows_sibling_writes(
path: StatePath, path: StatePath,
state_fields: Mapping[StatePath, StateFieldDecl], state_fields: Mapping[StatePath, StateFieldDecl],
reducers: Mapping[str, ReducerDefinition] | None,
) -> bool: ) -> bool:
field = state_fields.get(path) field = state_fields.get(path)
if field is None or field.reducer is None: if field is None or field.reducer is None:
return False return False
return field.reducer.name != "wf.std.replace" return reducer_allows_sibling_writes(field.reducer, reducers)
def _state_paths_overlap(left: StatePath, right: StatePath) -> bool: def _state_paths_overlap(left: StatePath, right: StatePath) -> bool:
+61 -2
View File
@@ -8,7 +8,9 @@ from wf_core import (
NodeDef, NodeDef,
NodeUse, NodeUse,
ReducerRef, ReducerRef,
ReducerSpec,
SchemaRef, SchemaRef,
SiblingWritePolicy,
StateField, StateField,
StateSchema, StateSchema,
Workflow, Workflow,
@@ -16,6 +18,7 @@ from wf_core import (
) )
from wf_core.models.steps import OutputBinding from wf_core.models.steps import OutputBinding
from wf_core.runtime.engine import resume_workflow from wf_core.runtime.engine import resume_workflow
from wf_core.runtime.ops.merges import ReducerDefinition
from wf_core.runtime.ops.runs import create_run_state from wf_core.runtime.ops.runs import create_run_state
from wf_core.runtime.ops.state import ( from wf_core.runtime.ops.state import (
StatePatch, StatePatch,
@@ -215,7 +218,7 @@ def test_build_and_commit_patch_matches_apply_output_bindings() -> None:
def test_barrier_rejects_sibling_same_path_writes_without_reducer() -> None: def test_barrier_rejects_sibling_same_path_writes_without_reducer() -> None:
workflow = _workflow(fields={"value": StateField(type="string")}) workflow = _workflow(fields={"value": StateField(type="string")})
with pytest.raises(WorkflowExecutionError, match="explicit reducer"): with pytest.raises(WorkflowExecutionError, match="mergeable reducer"):
build_barrier_patch( build_barrier_patch(
workflow, workflow,
[ [
@@ -236,7 +239,7 @@ def test_barrier_rejects_sibling_same_path_writes_with_explicit_replace() -> Non
} }
) )
with pytest.raises(WorkflowExecutionError, match="explicit reducer"): with pytest.raises(WorkflowExecutionError, match="mergeable reducer"):
build_barrier_patch( build_barrier_patch(
workflow, workflow,
[ [
@@ -269,6 +272,62 @@ def test_barrier_allows_sibling_same_path_writes_with_non_replace_reducer() -> N
assert patch.changes["state.seen"] == ["a", "b"] assert patch.changes["state.seen"] == ["a", "b"]
def test_barrier_uses_reducer_policy_instead_of_reducer_name() -> None:
workflow = _workflow(
fields={
"value": StateField(
type="integer",
reducer=ReducerRef(name="test.keep_latest"),
)
}
)
reducer = ReducerDefinition(
spec=ReducerSpec(name="test.keep_latest"),
fn=lambda _current, incoming: incoming,
)
patch = build_barrier_patch(
workflow,
[
StatePatch(changes={"state.value": 1}),
StatePatch(changes={"state.value": 2}),
],
{},
reducers={"test.keep_latest": reducer},
)
assert patch.changes["state.value"] == 2
def test_barrier_rejects_custom_exclusive_reducer() -> None:
workflow = _workflow(
fields={
"value": StateField(
type="integer",
reducer=ReducerRef(name="test.last"),
)
}
)
reducer = ReducerDefinition(
spec=ReducerSpec(
name="test.last",
sibling_write_policy=SiblingWritePolicy.EXCLUSIVE,
),
fn=lambda _current, incoming: incoming,
)
with pytest.raises(WorkflowExecutionError, match="mergeable reducer"):
build_barrier_patch(
workflow,
[
StatePatch(changes={"state.value": 1}),
StatePatch(changes={"state.value": 2}),
],
{},
reducers={"test.last": reducer},
)
def test_barrier_rejects_sibling_ancestor_descendant_writes() -> None: def test_barrier_rejects_sibling_ancestor_descendant_writes() -> None:
workflow = _workflow_from_state_schema( workflow = _workflow_from_state_schema(
StateSchema.model_validate( StateSchema.model_validate(
+1 -1
View File
@@ -210,7 +210,7 @@ def test_sync_concurrent_foreach_sibling_overlays_do_not_leak() -> None:
def test_sync_concurrent_foreach_rejects_sibling_replace_writes() -> None: def test_sync_concurrent_foreach_rejects_sibling_replace_writes() -> None:
workflow = _same_path_replace_workflow() workflow = _same_path_replace_workflow()
with pytest.raises(WorkflowExecutionError, match="explicit reducer"): with pytest.raises(WorkflowExecutionError, match="mergeable reducer"):
execute_workflow( execute_workflow(
workflow, workflow,
{"items": ["a", "b"]}, {"items": ["a", "b"]},
+7
View File
@@ -6,6 +6,7 @@ from wf_core import (
ReducerRef, ReducerRef,
ReducerSpec, ReducerSpec,
SchemaRef, SchemaRef,
SiblingWritePolicy,
StateField, StateField,
StateSchema, StateSchema,
Workflow, Workflow,
@@ -468,6 +469,12 @@ def test_reducer_definition_can_wrap_plain_two_arg_callable() -> None:
assert result == 5 assert result == 5
def test_reducer_spec_defaults_to_mergeable_sibling_write_policy() -> None:
spec = ReducerSpec(name="test.reducer")
assert spec.sibling_write_policy is SiblingWritePolicy.MERGEABLE
def test_reducer_definition_can_wrap_config_aware_callable() -> None: def test_reducer_definition_can_wrap_config_aware_callable() -> None:
definition = ReducerDefinition( definition = ReducerDefinition(
spec=ReducerSpec( spec=ReducerSpec(