lda-wf/docs/authoring_sketch.md

# Authoring Layer Sketch

This document sketches the next layer on top of `wf_core`.

The goal is to make workflow authoring pleasant for:

- Python developers using `@node`
- builder-style graph construction
- client LLMs that consume a node catalog and emit graph structure

This layer should compile down to the existing core model without changing
runtime semantics.

## Principles

1. `wf_core` remains the execution model.
2. Authoring APIs compile to `wf_core.Workflow`.
3. `NodeDef` should usually be derived, not written manually.
4. The LLM should usually choose from a node catalog, not invent raw node defs.
5. Generics belong at the authoring boundary, not in the runtime core.

## Main Objects

### `NodeSpec[InputT, OutputT]`

The durable product of `@node`.

Responsibilities:

- hold typed Python callable metadata
- expose input and output model types
- expose node outcomes
- generate a core `NodeDef`
- register a runtime handler
- remain directly callable in Python

This should be the single central wrapper object. Avoid creating multiple
parallel wrapper types with overlapping meanings.

### `WorkflowBuilder`

Builder API for human Python authors.

Responsibilities:

- collect node uses
- collect control-flow nodes
- collect edges
- derive unique `NodeDef`s from referenced `NodeSpec`s
- compile to a core `Workflow`

Typical entry points:

- `use(node_spec, id=..., input=[...], output=[...])`
- `condition(...)`
- `foreach(...)`
- `interrupt(...)`
- `connect(...)`
- `compile()`

`input` and `output` use the same canonical binding-list shape as core
`NodeUse`:

```json
{
      "input": [
        {
          "target": "text",
          "path": "input.text"
        }
      ],
      "output": [
        {
          "source": "echoed",
          "target": "state.echoed"
        }
      ]
}
```

`in_map`, `input_values`, and `out_map` are deprecated compatibility sugar for
Python authors. Client LLMs and MCP/JSON callers should use canonical binding
lists so structural paths live inside structs, not as unhashable map keys.

### `NodeCatalog`

LLM-facing registry of available nodes.

Responsibilities:

- expose name, docs, schemas, and outcomes
- provide a normalized machine-readable view for MCP consumers
- allow the client LLM to build `NodeUse`s against known nodes

The client LLM should usually receive a node catalog and emit graph structure
that references those known nodes. It should not usually generate new raw
`NodeDef`s.

## Flow

### Python authoring flow

1. declare `InputModel` and `OutputModel`
2. decorate a function with `@node(...)`
3. receive a `NodeSpec`
4. add `NodeSpec`s to a `WorkflowBuilder`
5. compile builder to core `Workflow`
6. build a registry from the same `NodeSpec`s
7. run with existing runtime

### LLM graph authoring flow

1. MCP exposes a `NodeCatalog`
2. client LLM selects nodes from the catalog
3. client LLM emits graph structure:
   - node uses
   - canonical input/output bindings
   - conditions
   - foreach nodes
   - interrupt nodes
   - edges
4. server compiles or validates that structure into core `Workflow`
5. runtime executes core `Workflow`

## Docs and schema descriptions

Input and output models should prefer `pydantic.BaseModel`.

Recommended sources of documentation:

- class docstring: model-level description
- `Field(description=...)`: strongest field-level description
- attribute docstrings with `ConfigDict(use_attribute_docstrings=True)`: good authoring UX

The authoring layer should normalize these into schema descriptions so MCP can
surface them to client LLMs.

## Async stance

Do not hide async behind `.result()`.

Preferred design:

- `NodeSpec` knows whether a callable is sync or async
- sync runtime accepts sync handlers
- future async runtime accepts async handlers

If sync runtime encounters an async node, fail clearly rather than faking a sync
bridge.

## Async runtime seam

The intended async work should be additive, not a rewrite.

Recommended shape:

- keep current sync runtime as the stable baseline
- add async siblings instead of mutating the sync path into a mixed mode
- avoid hidden sync-to-async or async-to-sync bridges in core execution

Likely async entry points:

- `execute_workflow_async(...)`
- `resume_workflow_async(...)`
- `step_workflow_async(...)`
- `execute_node_use_async(...)`

Likely registry split:

- sync registry: `dict[str, SyncNodeHandler]`
- async registry: `dict[str, AsyncNodeHandler]`

`NodeSpec` should support both export paths:

- `to_registry_handler()` for sync callables only
- `to_async_registry_handler()` for sync or async callables
- `build_registry(...)` for sync specs
- `build_async_registry(...)` for mixed or async specs

This keeps the rules simple:

- sync runtime executes sync handlers only
- async runtime can execute both sync and async handlers
- async runtime is the natural home for future MCP-backed tool nodes

## MCP proxy layer

MCP integration should sit above `wf_core`, not inside it.

Recommended layering:

1. MCP client or proxy code discovers tools
2. each MCP tool is wrapped as a `NodeSpec`
3. wrapped specs enter the same `NodeCatalog` as handwritten nodes
4. the client LLM builds graphs against one unified catalog
5. workflows compile to the existing core `Workflow`
6. runtime executes registry handlers without caring whether the backing tool is local Python or MCP

This means MCP tools should look like ordinary nodes at the authoring boundary:

- declared input model
- declared output model
- declared outcomes
- description/docs for LLM consumption
- sync or async execution capability

The proxy/MCP layer should be responsible for:

- tool discovery
- schema translation
- auth/session concerns
- wrapping tool calls into `NodeSpec`s

The core runtime should remain responsible only for:

- mapping input/state/context into node payloads
- validating payloads
- routing outcomes
- writing mapped output into workflow state
- interrupts, frames, trace, and foreach semantics

## Type validation stance

Current core validation is intentionally shallow. Today it mainly enforces:

- object payloads are dict-like
- required keys exist

It does not yet fully enforce:

- scalar field types
- nested object structure
- array item types
- enums/literals

Near-term recommendation:

- keep `SchemaRef` as the portable contract/export shape
- keep using `pydantic.BaseModel` as the strongest validation layer for authored nodes
- gradually strengthen core schema validation where it pays off

This is especially useful for MCP wrapping, because the proxy layer can often
normalize a tool contract into Pydantic models before the workflow runtime sees
it.

## Future extension

### `Workflow -> NodeSpec`

Today there are two different authoring paths:

- `subgraph_node` / `async_subgraph_node` wrap a compiled workflow as a normal
  `NodeSpec`. The parent sees one node call and child frames/interrupts are not
  native parent state.
- `subgraph_ref` builds a native `SubgraphNode` contract from a compiled
  `Workflow`: child input schema, output schema, and workflow outcomes are
  copied into the boundary. The workflow reference is structural: local
  workflows use `{"name": "child"}`, saved artifacts can use
  `{"artifact_id": "child", "version": 1}`. Runtime execution still raises
  until native subgraph scopes are implemented.
- `WorkflowBuilder.subgraph(...)` is the builder-facing version of
  `subgraph_ref`: it appends the native boundary step and returns it as a
  `StepRef` for `connect()` / `set_entry_point()`.

In the future, a compiled workflow or subgraph can also be exposed as a reusable
`NodeSpec`, likely by treating workflow input and output schemas as the node's
input and output schemas.

That should be an authoring-layer transformation, not a core runtime rewrite.