lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs of dipping my hand on wf_core

Browse Source
This commit is contained in:
lda
2026-05-17 15:08:02 +07:00 Verified
parent 173a72fa82
commit 37914449a1
5 changed files with 581 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/core_state_mapping_and_merge.md
+253
View File
@@ -0,0 +1,253 @@
# Core State Mapping and Merge Semantics
This document records the intended direction for `wf_core` mapping and state
updates before the next round of core implementation work.
## Why This Exists
The current core model is intentionally explicit:
- graph data moves between steps through workflow state
- nodes declare their own input and output contracts
- graph use-sites map between graph data and node-local data
- merge behavior belongs to workflow state, not to node implementations
That model is still right. The next pressure comes from structured tools and
future nested execution:
- real tools often accept deep payloads such as `user.name` and `job.title`
- real tools often return deep payloads such as `job.wage`
- wrappers should not need extra runtime nodes for pure boundary wiring
- future native subgraphs and parallel foreach both need a stronger commit model
- nested state reducers can become a differentiator over systems that only merge
whole top-level values
## Canonical Mapping Rule
The graph-facing side of a map is a graph path. The node-facing side is a
node-local path.
```text
in_map:
graph source path -> node-local input path
out_map:
node-local output path -> graph state destination path
```
Examples:
```python
in_map = {
"state.person.name": "user.name",
"state.digital.email": "user.email",
"state.job.title": "job.title",
}
out_map = {
"job.wage": "state.job.wage",
"job.years": "state.experience.years",
"user.age": "state.person.age",
}
```
Whole-object mapping remains valid:
```python
in_map = {"state.person": "user"}
out_map = {"user": "state.person"}
```
The change from the current implementation is only on the node-local sides:
today they are top-level fields; in the target model they may be nested paths.
## Explicitness Rules
### Node-local writes must not overlap
`in_map` constructs node input payloads. Its destination node-local paths must
be pairwise non-overlapping.
Valid:
```python
{
"state.person.name": "user.name",
"state.person.email": "user.email",
}
```
Invalid:
```python
{
"state.person": "user",
"state.person.name": "user.name",
}
```
The invalid form would require implicit object patch precedence. Authors must
choose either whole-object mapping or explicit child mapping.
### State writes must not overlap in one commit
`out_map` mutates workflow state. Its destination graph paths must be pairwise
non-overlapping inside one logical commit.
Invalid:
```python
{
"user": "state.person",
"user.name": "state.person.name",
}
```
The target model rejects these writes before mutating state.
### Read overlap is allowed
Overlap is forbidden on write targets, not read sources. It is valid to read
both a whole object and one child into separate destinations when no constructed
target overlaps.
## Required Mapped Paths
Mapped paths are assertions by the workflow author.
- a missing graph source path in `in_map` is a runtime error
- a missing node-local output path in `out_map` is a runtime error
- optional/default behavior must be modeled explicitly later, not inferred from
a missing path
## State Patch Commit
A successful step should produce a logical state patch before state is mutated:
1. resolve all mapped output paths
2. ensure every required output path exists
3. ensure destination state paths do not overlap
4. prepare the complete write set
5. commit the write set according to state merge rules
This preserves the existing “no partial state commit before success” rule and
creates a reusable boundary for:
- ordinary node completion
- serial foreach iteration completion
- future parallel foreach result combination
- future subgraph completion
## State Declarations and Merge Rules
The current implementation attaches merge behavior to declared top-level state
fields. The target model should later allow nested declared state paths while
keeping the internal representation flat:
```python
fields = {
"person.name": StateField(type="string", merge_strategy="replace"),
"person.tags": StateField(type="array", merge_strategy="append"),
"profile": StateField(type="object", merge_strategy="merge_object"),
}
```
Presentation layers may rebuild a tree for humans. Core should keep the simpler
path-keyed representation.
### Exact-path ownership
Merge behavior belongs only to the exact declared state path being written.
- no ancestor inheritance
- no descendant declarations altering parent writes
- writing `state.profile` uses only the rule for `profile`
- writing `state.profile.avatar` uses only the rule for `profile.avatar`
If a path is undeclared, it defaults to `replace`, as undeclared state does
today.
### Built-in strategies
Existing built-ins remain distinct:
- `replace`
- `append`
- `merge_object`
`merge_object` means shallow object merge at the exact destination path, similar
to `dict.update` or `operator.or_`. It is not a recursive deep merge.
If recursive merge is ever needed, it should be explicit rather than hidden
inside `merge_object`.
## Future Reducers
Custom reducers should become a future capability family, similar to reusable
node specs:
- named
- source-owned
- inspectable
- dependency-trackable
State fields should reference reducers declaratively. Workflow artifacts should
not embed arbitrary Python callables.
Reducers should be pure:
```text
current_value, incoming_value -> merged_value
```
They should not receive node ids, frame ids, loop indexes, timestamps, or other
runtime context. If behavior depends on workflow context, that is business logic
and belongs in nodes or graph structure.
Examples a future reducer library could support:
- `max`
- `set_union`
- `modulo_add` with configuration such as modulus `10`
## Implementation Phases
### Phase 1: Nested node-local mappings
- allow nested node-local paths on the destination side of `in_map`
- allow nested node-local paths on the source side of `out_map`
- reject overlapping write targets
- commit node output through a validated state patch
- validate top-level node-local roots statically; validate deeper shape when the
existing node schema makes that practical
This phase immediately helps wrappers and structured tools while preserving
current state merge behavior.
### Phase 2: Nested declared state paths
- allow exact nested state path declarations
- resolve merge strategies by exact destination path only
- keep undeclared paths as `replace`
- keep `merge_object` shallow
### Phase 3: Reducer capabilities
- design source-owned reducer specs
- add reducer dependency references to state metadata
- resolve pure reducers through runtime/deployment registries
### Phase 4: Core features that depend on this foundation
- native subgraphs should reuse the same mapping and patch semantics at graph
boundaries
- parallel foreach should combine validated patches and reject conflicts unless
exact-path merge rules permit combination
## Non-Goals for the First Implementation
- direct node-to-node data wires outside workflow state
- implicit object patching from overlapping mapped paths
- recursive deep merge hidden inside `merge_object`
- arbitrary Python reducer callables stored in workflow models
- shipping native subgraphs or parallel foreach as part of nested mapping work
docs/schema_validation.md
+1
View File
@@ -23,6 +23,7 @@ It still does not solve:
external JSON Schema dialect
- typed Python object creation from arbitrary JSON Schema
- workflow state merge behavior
- deep node-local map-path validation beyond statically knowable schema roots
- better domain-specific error payloads beyond `WorkflowExecutionError`
This means schema fields are mostly contracts for authoring, planning,
docs/scratchpad.md
+4 -4
View File
@@ -394,13 +394,13 @@ Practical notes:
`in_map`:
- source is graph `input`, `state`, or future `context` path
- destination is a declared node input field
- destination is a node-local input path
- missing required source path is a graph/runtime error
- optional destination fields may be omitted
`out_map`:
- source is a declared node output field
- source is a node-local output path
- destination is a graph state path
- mapping to undeclared state keys is allowed, but loses typed merge behavior
- mapping a required output field that does not exist at runtime is node failure
@@ -452,8 +452,8 @@ Before execution, validator should be able to check:
- every edge destination exists or is `__end__`
- every edge outcome is declared by its source node type
- every reachable declared outcome is wired
- every `in_map` destination exists in node input schema
- every `out_map` source exists in node output schema
- every `in_map` destination has a valid declared node-input root
- every `out_map` source has a valid declared node-output root
- every condition node uses valid operators and operand shapes
At runtime, executor should still check:
docs/superpowers/plans/2026-05-17-nested-node-local-mappings.md
+319
View File
@@ -0,0 +1,319 @@
# Nested Node-Local Mappings Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let workflow maps address nested node-local input/output paths while preserving explicit state-mediated data flow and preparing state writes for future reducer libraries.
**Architecture:** Keep graph-facing paths unchanged. Add a small node-local path helper layer, validate non-overlapping write targets, construct nested node inputs from `in_map`, resolve nested node outputs from `out_map`, and refactor state writes into a prepared patch commit. Extract built-in merge-rule application behind a focused reducer-like module so future named reducer libraries can replace the dispatch seam without changing patch semantics.
**Tech Stack:** Python, Pydantic, pytest, existing `wf_core` path/state runtime.
---
## File Structure
- Modify `src/wf_core/paths.py`
- keep graph-path helpers
- add reusable path-overlap utility if it belongs at the generic path layer
- Create `src/wf_core/local_paths.py`
- node-local dotted path parsing
- nested get/set for node-local payloads
- overlap checks for write targets
- Create `src/wf_core/runtime/ops/merges.py`
- built-in exact-path merge-rule implementations
- future seam for named reducer registry
- Modify `src/wf_core/runtime/ops/state.py`
- replace per-write mutation loop with prepared patch commit
- delegate merge-rule application to `merges.py`
- Modify `src/wf_core/runtime/ops/nodes.py`
- build nested node inputs from `in_map`
- resolve nested node outputs into state patch writes
- Modify `src/wf_core/validation/steps.py`
- validate node-local top-level roots
- reject overlapping write targets
- Add/modify tests under `tests/core/`
- nested `in_map`
- nested `out_map`
- whole-object mapping still works
- overlapping write targets rejected
- missing nested output path fails
- merge dispatch remains behaviorally unchanged
---
### Task 1: Pin Node-Local Path Behavior
**Files:**
- Modify: `tests/core/test_validation.py`
- Modify: `tests/core/test_runtime.py`
- [ ] **Step 1: Add failing validation tests**
Cover:
```python
def test_validation_allows_nested_node_local_paths() -> None:
...
def test_validation_rejects_overlapping_node_input_destinations() -> None:
...
def test_validation_rejects_overlapping_state_write_destinations() -> None:
...
```
Expected rules:
- `state.person.name -> user.name` is valid when `user` exists in the node input schema
- `user -> state.person` and `user.name -> state.person.name` in one `out_map` is invalid because destination state paths overlap
- `state.person -> user` plus `state.person.name -> user.name` in one `in_map` is invalid because destination node-local paths overlap
- [ ] **Step 2: Add failing runtime tests**
Cover:
```python
def test_runtime_builds_nested_node_input_from_in_map() -> None:
...
def test_runtime_reads_nested_node_output_from_out_map() -> None:
...
def test_runtime_missing_nested_node_output_path_fails() -> None:
...
```
- [ ] **Step 3: Run focused tests and confirm failure**
Run:
```bash
uv run --with pytest pytest tests/core/test_validation.py tests/core/test_runtime.py -q
```
Expected: FAIL because node-local map sides are top-level-only today.
### Task 2: Add Node-Local Path Helpers
**Files:**
- Create: `src/wf_core/local_paths.py`
- Modify: `src/wf_core/validation/steps.py`
- [ ] **Step 1: Add minimal helper API**
Implement:
```python
def split_local_path(path: str) -> list[str]: ...
def get_local_value(payload: Mapping[str, Any], path: str) -> Any: ...
def set_local_value(payload: dict[str, Any], path: str, value: Any) -> None: ...
def paths_overlap(left: str, right: str) -> bool: ...
def has_overlapping_paths(paths: Iterable[str]) -> bool: ...
```
Rules:
- dotted local paths only
- no empty segments
- overlap means same path or ancestor/descendant path
- [ ] **Step 2: Update validation**
Use node-local path roots for schema checks:
```python
input_root = split_local_path(destination_path)[0]
output_root = split_local_path(source_path)[0]
```
Reject:
- overlapping `in_map` destination local paths
- overlapping `out_map` destination state paths
- [ ] **Step 3: Run focused validation tests**
Run:
```bash
uv run --with pytest pytest tests/core/test_validation.py -q
```
Expected: PASS for validation-specific cases.
### Task 3: Execute Nested Local Mappings
**Files:**
- Modify: `src/wf_core/runtime/ops/nodes.py`
- Modify: `src/wf_core/runtime/ops/state.py`
- [ ] **Step 1: Build nested node inputs**
Replace flat input construction with:
```python
resolved_input: dict[str, Any] = {}
for source_path, destination_path in node.in_map.items():
value = safe_resolve_path(...)
set_local_value(resolved_input, destination_path, value)
```
- [ ] **Step 2: Resolve nested node outputs**
When preparing mapped output writes, use `get_local_value()` for each `out_map`
source path instead of indexing only top-level output keys.
- [ ] **Step 3: Preserve missing-path failures**
Raise `WorkflowExecutionError` when a mapped nested output path is missing.
- [ ] **Step 4: Run focused runtime tests**
Run:
```bash
uv run --with pytest pytest tests/core/test_runtime.py -q
```
Expected: PASS for nested mapping behavior.
### Task 4: Introduce Prepared Patch Commits and Extract Merge Dispatch
**Files:**
- Create: `src/wf_core/runtime/ops/merges.py`
- Modify: `src/wf_core/runtime/ops/state.py`
- Modify: `tests/core/test_state_ops.py`
- [ ] **Step 1: Add failing patch-level tests**
Cover:
```python
def test_state_patch_rejects_overlapping_destinations_before_mutation() -> None:
...
def test_builtin_merge_rules_preserve_existing_behavior() -> None:
...
```
- [ ] **Step 2: Extract built-in merge implementations**
Move the current strategy body out of `write_state_value()` into focused helpers:
```python
def apply_builtin_merge(
*,
strategy: str,
current_value: Any,
incoming_value: Any,
destination_path: str,
) -> Any: ...
```
Keep current semantics:
- `replace`
- `append`
- shallow `merge_object`
Add a docstring that this is the future seam for source-owned named reducers,
not custom reducer support yet.
- [ ] **Step 3: Prepare full write sets before mutation**
Refactor output mapping so it:
1. resolves all mapped output values
2. validates destination overlap
3. prepares the patch
4. applies merge behavior
No state changes should occur before all mapped output paths are known-good.
- [ ] **Step 4: Run state/runtime tests**
Run:
```bash
uv run --with pytest pytest tests/core/test_state_ops.py tests/core/test_runtime.py -q
```
Expected: PASS.
### Task 5: Keep Authoring and Docs Aligned
**Files:**
- Modify: `src/wf_authoring/builder/mapping.py`
- Modify: `docs/core_state_mapping_and_merge.md` if implementation details differ
- Modify: `docs/scratchpad.md` only if wording drift appears
- Modify/Add: authoring tests as needed
- [ ] **Step 1: Confirm authoring auto-maps remain top-level**
Automatic maps should stay conservative unless there is an explicit reason to
infer nested paths. The new feature is for explicit maps first.
- [ ] **Step 2: Add one authoring regression**
Prove that a builder can compile a workflow using explicit nested local map
paths without extra helper nodes.
- [ ] **Step 3: Update docs only for implementation drift**
The design doc already states the target behavior. Keep docs in sync with final
names and module boundaries, but do not broaden scope into nested state
declarations yet.
### Task 6: Verify the Whole Project
**Files:**
- No additional files.
- [ ] **Step 1: Run focused suites**
```bash
uv run --with pytest pytest tests/core tests/authoring -q
```
- [ ] **Step 2: Run the full suite**
```bash
uv run --with pytest pytest -q
```
- [ ] **Step 3: Run type checking**
```bash
uv run basedpyright --level error
```
Expected:
- tests pass
- any remaining basedpyright failures are called out explicitly if they come
from existing generated/build/doc-fixture noise rather than this work
---
## Deliberate Non-Goals
- nested declared state merge metadata
- reducer capability registry
- deep merge behavior
- native subgraphs
- parallel foreach
- automatic inference of nested maps from schemas
## Follow-On Plans
After this lands:
1. nested declared state paths with exact-path merge lookup
2. reducer capability model / registry seam
3. native subgraph design on the same map + patch boundary
4. async-only parallel foreach using patch combination rules
docs/wf_core_architecture.md
+4
View File
@@ -71,6 +71,10 @@ limits and intended adapter seam.
## What This Cleanup Does Not Solve Yet
- Node-local mapping paths are still top-level only in the current runtime.
The intended direction for nested node-local paths, patch commits, and future
nested merge metadata is documented in
[`core_state_mapping_and_merge.md`](core_state_mapping_and_merge.md).
- Foreach is still serial-only. Parallel foreach needs an explicit scheduling
model, not just `asyncio.gather`.
- Interrupt lifecycle is still node-level and run-state-level. Long-lived