docs

by mimo opencode

docs/add/architecture.md

@@ -0,0 +1,541 @@
+---
+title: |
+  lda.chat\
+  \Large wf: Architecture \& Design
+author: "draft"
+date: "2026-05-30"
+documentclass: report
+papersize: a4
+fontsize: 12pt
+geometry:
+  - top=30mm
+  - bottom=30mm
+  - left=32mm
+  - right=32mm
+mainfont: "Libertinus Serif"
+sansfont: "Libertinus Sans"
+monofont: "Libertinus Mono"
+mathfont: "Libertinus Math"
+header-includes:
+  - \usepackage{graphicx}
+  - \usepackage{booktabs}
+  - \usepackage{hyperref}
+  - \usepackage{fancyhdr}
+  - \pagestyle{fancy}
+  - \fancyhead[L]{\small\leftmark}
+  - \fancyhead[R]{\small lda.chat}
+  - \fancyfoot[C]{\thepage}
+  - \setlength{\parskip}{0.6em}
+  - \setlength{\parindent}{0pt}
+  - \setkeys{Gin}{width=\linewidth,height=0.55\textheight,keepaspectratio}
+  - \renewcommand{\arraystretch}{1.3}
+  - '\newcommand{\status}[1]{\hfill\textsc{\small #1}}'
+diagram:
+  engine:
+    mermaid:
+      theme: neutral
+      outputFormat: svg
+---
+
+# Introduction
+
+`lda.chat` is an AI agent that turns natural language requests into executable
+workspace workflows. The LLM plans a directed graph of typed steps. A
+deterministic executor runs it. Registered node handlers do the real
+work---whether those handlers call MCP tools, run local Python functions,
+execute saved subworkflows, or compose authoring-layer operations.
+
+The core architectural split:
+
+> **The LLM plans, but does not execute. The executor executes, but does not
+> think. Nodes do work. Edges are dumb routing.**
+
+**Thesis scope** is narrow: prove the planner/executor/tool split end to end
+with a few working MCP tools, LLM-generated workflow JSON, validated execution,
+and structured results. The broader product vision---tool registries, scheduling,
+multi-user---is out of scope.
+
+# System Architecture \status{implemented}
+
+## Package Structure
+
+The system is organized into three main packages with strict dependency
+boundaries.
+
+```mermaid
+graph TD
+    subgraph Core["wf_core --- Execution Kernel"]
+        direction TB
+        Models["models"]
+        Runtime["runtime"]
+        Scheduler["scheduler"]
+        Ops["runtime.ops"]
+        Validation["validation"]
+        Paths["paths"]
+        RunState["run_state"]
+    end
+
+    subgraph Authoring["wf_authoring --- DSL"]
+        direction TB
+        Builder["WorkflowBuilder"]
+        NodeDec["@node"]
+        DSL["dsl/"]
+        Reducers["reducers/"]
+    end
+
+    subgraph MCP["wf_mcp --- MCP Platform"]
+        direction TB
+        Broker["broker/"]
+        Proxy["proxy/"]
+        SDK["sdk/"]
+        WorkflowSurf["workflow_surface/"]
+        WfConvert["workflow/"]
+    end
+
+    Authoring --> Core
+    MCP --> Core
+    MCP --> Authoring
+```
+
+**Dependency rules:**
+
+- `wf_core` must not import `wf_authoring` or `wf_mcp`.
+- `wf_mcp.sdk` should not import `wf_core` or `wf_authoring`.
+- `wf_mcp.proxy` should not import `wf_mcp.workflow`.
+- `wf_mcp.workflow` is the only layer that converts MCP tools into node specs.
+
+## Data Flow
+
+```mermaid
+flowchart LR
+    User["User request"] --> LLM["LLM Planner"]
+    LLM --> WJ["Workflow JSON"]
+    WJ --> EX["Deterministic Executor"]
+    EX --> NH["Node Handlers"]
+    NH --> State["Shared Workflow State"]
+```
+
+# Workflow Model \status{implemented}
+
+A workflow is a directed graph with explicit schemas and explicit control flow.
+
+## Top-Level Structure
+
+```mermaid
+classDiagram
+    class Workflow {
+        +str name
+        +SchemaRef input_schema
+        +StateSchema state_schema
+        +SchemaRef output_schema
+        +list~InputBinding~ output
+        +list~NodeDef~ node_defs
+        +list~str~ outcomes
+        +str start
+        +list~Step~ nodes
+        +list~Edge~ edges
+    }
+
+    class NodeDef {
+        +str name
+        +SchemaRef input_schema
+        +SchemaRef output_schema
+        +list~str~ outcomes
+    }
+
+    class Edge {
+        +str from
+        +str outcome
+        +str to
+    }
+
+    Workflow --> NodeDef
+    Workflow --> Edge
+```
+
+## Schema Boundaries
+
+Three schemas define distinct contracts:
+
+- **`input_schema`** validates run input once. Input is stored independently and
+  remains readable throughout execution via `input.*` paths.
+- **`state_schema`** defines typed workflow memory with per-field merge
+  reducers. Nodes read/write state via `state.*` paths.
+- **`output_schema`** declares the output contract. Output is projected via
+  explicit `workflow.output` bindings that can read from `state.*`, `input.*`,
+  or `context.*` paths. A legacy fallback derives output from same-name top-level
+  state keys when bindings are omitted.
+
+Input, state, and context are **separate read roots**. The runtime resolves
+bindings against all three independently; input is not merged into state.
+
+## Step Types
+
+Steps form a discriminated union on the `type` field:
+
+```mermaid
+graph
+    Step["Step (type)"]
+    Step --> NU["NodeUse"]
+    Step --> SG["SubgraphNode"]
+    Step --> CN["ConditionNode"]
+    Step --> FE["ForeachNode"]
+    Step --> JN["JoinNode"]
+    Step --> EN["EndNode"]
+    Step --> IN["InterruptNode"]
+```
+
+**NodeUse** binds a reusable `NodeDef` with explicit input/output path
+bindings. **ConditionNode** evaluates structured JSON expressions (not code
+strings). **ForeachNode** iterates serial or concurrent. **InterruptNode**
+pauses for typed external input. **EndNode** sets non-`ok` workflow outcomes.
+
+## Path Bindings
+
+Bindings connect graph state to node-local payloads using structural paths:
+
+```json
+{
+  "input": [
+    { "target": { "root": "local", "parts": ["text"] },
+      "path":   { "root": "input", "parts": ["text"] } }
+  ],
+  "output": [
+    { "source": { "root": "local", "parts": ["echoed"] },
+      "target": { "root": "state", "parts": ["echoed"] } }
+  ]
+}
+```
+
+Three path kinds: **LocalPath** (node payload), **GraphSourcePath**
+(read from `input`/`state`/`context`), **StatePath** (write to state).
+
+## State Reducers
+
+State fields declare merge behavior. This matters for parallel and foreach
+execution where multiple writers target the same path.
+
+| Reducer | Policy | Use case |
+|:--------|:-------|:---------|
+| `wf.std.replace` | exclusive | Default scalars |
+| `wf.std.append` | mergeable | Foreach accumulation |
+| `wf.std.merge_object` | mergeable | Parallel object updates |
+| `wf.std.add` | mergeable | Counters |
+| `wf.std.set_union` | mergeable | Tag/ID collection |
+
+# Runtime Execution \status{implemented}
+
+## Execution Flow
+
+```mermaid
+flowchart
+    A["execute_workflow()"] --> B["prepare_new_run()"]
+    B --> C["create_run_state()"]
+    C --> D["resume_workflow()"]
+    D --> E{"select_next_frame()"}
+    E -->|"frame found"| F["step_workflow()"]
+    F --> G{"interrupted?"}
+    G -->|Yes| H["return RunState"]
+    G -->|No| E
+    E -->|"no frames"| I{"classify terminal"}
+    I -->|"completed"| J["finalize_run()"]
+    I -->|"failed"| K["return RunState"]
+    J --> M["return RunState"]
+```
+
+The executor loops: select a frame from the ready queue, dispatch one step,
+re-enqueue or terminate. It never thinks. It only routes.
+
+## Step Dispatch
+
+```mermaid
+flowchart LR
+    SW["step_workflow()"] -->|"NodeUse"| NE["execute node"]
+    SW -->|"Condition"| CE["evaluate"]
+    SW -->|"Foreach"| FE["foreach step"]
+    SW -->|"Join"| JO["done"]
+    SW -->|"End"| EN["complete"]
+    SW -->|"Interrupt"| IR["pause run"]
+    SW -->|"Subgraph"| SG["child scope"]
+```
+
+Node handlers are registered callables---anything from a thin MCP tool wrapper
+to a full Python function. The executor does not distinguish handler origin; it
+resolves by name from the registry, invokes, validates, and commits.
+
+## Node Execution
+
+```mermaid
+sequenceDiagram
+    participant E as Executor
+    participant H as Handler
+    participant S as State
+
+    E->>E: resolve input bindings
+    E->>H: handler(payload, context)
+    H-->>E: NodeResult{outcome, output}
+    E->>E: validate outcome + output
+    E->>S: commit state patch (reducer-aware)
+    E->>E: record trace, advance frame
+```
+
+Two channels stay separate: **outcome** controls routing, **output** carries
+typed business data.
+
+## Scheduler
+
+The scheduler uses an explicit FIFO **ready queue** of frame IDs.
+
+```mermaid
+stateDiagram-v2
+    [*] --> PENDING
+    PENDING --> RUNNING: scheduler selects
+    RUNNING --> PENDING: re-enqueue
+    RUNNING --> COMPLETED: reached end
+    RUNNING --> FAILED: unhandled error
+    RUNNING --> INTERRUPTED: interrupt node
+    RUNNING --> BLOCKED: waiting on child
+    BLOCKED --> PENDING: child completes
+    INTERRUPTED --> PENDING: resume delivered
+```
+
+When the ready queue is empty, the scheduler classifies the run as completed,
+interrupted, failed, or deadlocked.
+
+# Foreach \status{implemented}
+
+## Serial
+
+Each iteration creates a child frame, blocks the parent, and runs to
+completion before the next item starts.
+
+## Concurrent
+
+Multiple item frames run simultaneously. Each gets its own **lineage**---a
+write overlay that isolates sibling state. At the barrier, patches commit in
+item-index order through declared reducers.
+
+```mermaid
+flowchart TD
+    PF["foreach parent"] --> C1["item 0"]
+    PF --> C2["item 1"]
+    PF --> C3["item 2"]
+    C1 & C2 & C3 --> BUF["buffer patches"]
+    BUF --> BARRIER["barrier: all done?"]
+    BARRIER --> COMMIT["commit in index order"]
+    COMMIT --> DONE["done / completed_with_errors"]
+```
+
+Capacity is bounded by `ForeachConcurrentPolicy`: `max_active` (default 4)
+limits ready/running item frames; `max_outstanding` (default 20) limits
+active + blocked. Item error policies: **fail** (stop), **skip** (continue,
+no state), **collect** (continue, write structured error records).
+
+# Interrupts \status{implemented}
+
+Interrupts are graph-native and typed, not arbitrary line-level pauses.
+
+```mermaid
+sequenceDiagram
+    participant N as Node
+    participant I as InterruptNode
+    participant R as Runtime
+    participant E as Caller
+
+    N->>I: outcome "needs_input"
+    I->>R: InterruptRequest{kind, payload}
+    R->>E: surface request
+    E->>R: resume payload + outcome
+    R->>I: map resume into state
+    I->>I: continue routing
+```
+
+Pause points stay visible in the graph. Payloads are validated by `kind`.
+Traces stay clean.
+
+# Subgraph Composition \status{implemented}
+
+A workflow can be used as a node, giving **graph composition** with one
+consistent contract.
+
+```mermaid
+flowchart LR
+    subgraph Parent["Parent"]
+        P1["A"] --> SG["SubgraphNode"]
+        SG --> P2["C"]
+    end
+    subgraph Child["Child (prepared)"]
+        C1["X"] --> C2["Y"]
+    end
+    SG --> C1
+    C2 --> SG
+```
+
+The parent supplies input bindings. The child runs in its own scope with its
+own lineage. Output bindings project child state back to the parent.
+
+**Interrupt bubbling (v1):** When a child subgraph hits an interrupt node, the
+runtime constructs an `InterruptRoute` that identifies the child frame, scope,
+and workflow ref. The parent subgraph frame stays `BLOCKED`; the entire run
+returns `INTERRUPTED`. Resume restores the child scope and continues inside it.
+This works for prepared local children. Artifact/deployment resolution for
+nested saved children remains outside core; the platform resolves saved
+descendants before execution starts.
+
+# Authoring Layer \status{implemented}
+
+## WorkflowBuilder
+
+Fluent Python DSL for constructing workflows without raw dicts:
+
+```mermaid
+flowchart LR
+    WB["WorkflowBuilder"] -->|"use(spec)"| NU["NodeUse"]
+    WB -->|"condition()"| CN["Condition"]
+    WB -->|"foreach()"| FE["Foreach"]
+    WB -->|"interrupt()"| IN["Interrupt"]
+    WB -->|"compile()"| WF["Workflow"]
+```
+
+Common operations: `use(spec)`, `connect(from, outcome, to)`,
+`branch(from, branches)`, `when(cond, then, otherwise)`,
+`match(value, cases)`, `foreach(over, mode)`.
+
+## @node Decorator
+
+Converts a typed Python function into a reusable `NodeSpec`:
+
+```python
+@node(name="summarize", outcomes=["ok"])
+def summarize_doc(input: SummarizeInput, ctx: RuntimeContext) -> SummarizeOutput:
+    """Summarize a single document."""
+    return SummarizeOutput(summary=input.text[:500])
+```
+
+The decorator infers input/output models from type annotations, detects async,
+and generates the `NodeDef` with schema refs.
+
+# MCP Platform \status{implemented}
+
+## Architecture
+
+```mermaid
+flowchart TB
+    subgraph Server["Unified MCP Server"]
+        Admin["wf_admin.*"]
+        Wf["wf.workflow.*"]
+        Proxy["connection.tool_name"]
+    end
+    subgraph Broker["WfMcpService"]
+        Cat["catalog"]
+        Disc["discovery"]
+        Art["artifacts"]
+        Drf["drafts"]
+    end
+    subgraph ProxyLayer["ProxyRuntime"]
+        Mount["mount registry"]
+        FMP["proxy per connection"]
+    end
+    Server --> Broker
+    Server --> ProxyLayer
+    ProxyLayer --> Upstream["Upstream MCP Servers"]
+```
+
+Each upstream connection gets a namespaced `FastMCPProxy` mount
+(`connection_id.tool_name`). Hot reload is best-effort; FastMCP lacks a
+complete unmount lifecycle.
+
+## Tool Conversion
+
+```mermaid
+flowchart LR
+    DT["DiscoveredTool"] --> WDT["wrap_discovered_tool()"]
+    WDT --> NS["NodeSpec"]
+    NS --> CS["CapabilitySource"]
+```
+
+Every MCP tool gets `outcomes=("ok", "error")` by default. The wrapper
+preserves the original JSON Schema as a contract and generates an async
+handler that calls the upstream tool.
+
+# Artifact Lifecycle \status{implemented}
+
+## Draft to Deployment
+
+```mermaid
+flowchart LR
+    D["Draft<br/>(mutable)"] -->|"compile"| A["Artifact<br/>(immutable)"]
+    A -->|"bind sources"| DP["Deployment"]
+    DP -->|"execute"| R["Run"]
+```
+
+**Draft workspaces** support iterative LLM authoring with optimistic
+concurrency (revision-checked patches). **Artifacts** are versioned snapshots
+with dependency contract hashes. **Deployments** bind logical source aliases
+to concrete MCP connections.
+
+## Durable Runs \status{v1 boundaries}
+
+Every `run_deployment` call persists a stopped run record (file-backed JSON)
+with a stable `run_id`. Interrupted runs can be resumed via `resume_run` after
+process restart, provided the same `FileRunStore` root is accessible.
+
+**V1 limits:**
+
+- Checkpointing happens at stopped boundaries only (interrupted, completed,
+  failed). No per-node or mid-call crash recovery.
+- Resume revalidates the pinned dependency environment. If a required source is
+  missing or disabled, resume returns `resume_readiness=blocked` without
+  consuming the payload.
+- During live execution, source failures produce a `failed` run, not a
+  `blocked` one. The `blocked` gate applies only at resume time.
+
+Two deployments can point the same artifact at different accounts:
+
+```
+summarize_docs.personal -> context7.personal
+summarize_docs.work     -> context7.work
+```
+
+# Validation \status{implemented}
+
+## Structural (Pre-Execution)
+
+```mermaid
+flowchart LR
+    VW["validate_workflow()"] --> A["node def uniqueness"]
+    VW --> B["node id uniqueness"]
+    VW --> C["edge source/dest existence"]
+    VW --> D["outcome declarations"]
+    VW --> E["binding path validity"]
+    VW --> F["reachable outcome wiring"]
+```
+
+Reports multiple issues via `ValidationReport` instead of failing at the
+first.
+
+## Runtime
+
+At runtime: outcome is declared, output matches schema, merge conflicts are
+caught, final output validates against `output_schema`.
+
+# Open Questions
+
+- Output derivation: direct projection vs explicit final output mapping?
+- Trace model for raw extra node output?
+- Isolated pure-Python execution: node type, tool backend, or separate API?
+- Retry policy vs non-idempotent side-effecting tools?
+
+# Stack
+
+| Layer | Technology |
+|:------|:-----------|
+| Language | Python 3.14+ |
+| Validation | Pydantic v2 |
+| Tool protocol | MCP |
+| MCP framework | FastMCP 3.2.4+ |
+| Storage | SQLite/PostgreSQL (planned) |
+
+---
+
+*Incomplete. See `docs/README.md` for the full documentation index.*

stuff/.gitignore

@@ -0,0 +1,4 @@
+*
+!diagram.patch
+!pandoc-diagram.ps1
+!.gitignore

stuff/diagram.patch

@@ -0,0 +1,114 @@
+--- stuff\diagram-og.lua	2026-05-30 03:52:48.703616700 +0700
++++ stuff\diagram.lua	2026-05-30 06:45:13.472927700 +0700
+@@ -6,6 +6,14 @@
+ -- The filter uses the Figure AST element, which was added in pandoc 3.
+ PANDOC_VERSION:must_be_at_least '3.0'
+ 
++warn = warn or function (...)
++  local parts = {}
++  for i = 1, select('#', ...) do
++    parts[#parts+1] = tostring(select(i, ...))
++  end
++  io.stderr:write(table.concat(parts, '') .. '\n')
++end
++
+ local version = pandoc.types.Version '1.2.0'
+ 
+ -- Report Lua warnings to stderr if the `warn` function is not plugged into
+@@ -148,7 +156,7 @@
+ local mermaid = {
+   line_comment_start = '%%',
+   mime_types = mime_types_set{'pdf', 'png', 'svg'},
+-  compile = function (self, code)
++  compile = function (self, code, user_opts)
+     local mime_type = self.mime_type or 'image/svg+xml'
+     local file_extension = extension_for_mimetype[mime_type]
+     return with_temporary_directory("diagram", function (tmpdir)
+@@ -156,11 +164,30 @@
+         local infile = 'diagram.mmd'
+         local outfile = 'diagram.' .. file_extension
+         write_file(infile, code)
+-        pipe(
+-          self.execpath or 'mmdc',
+-          {"--pdfFit", "--input", infile, "--output", outfile},
+-          ''
+-        )
++
++        local args = {"--pdfFit", "--input", infile, "--output", outfile}
++
++        -- Map YAML options to mmdc flags
++        if user_opts then
++          local opt_map = {
++            theme           = '--theme',
++            backgroundColor = '--backgroundColor',
++            width           = '--width',
++            height          = '--height',
++            scale           = '--scale',
++            configFile      = '--configFile',
++            cssFile         = '--cssFile',
++            outputFormat    = '--outputFormat',
++          }
++          for key, flag in pairs(opt_map) do
++            if user_opts[key] then
++              table.insert(args, flag)
++              table.insert(args, stringify(user_opts[key]))
++            end
++          end
++        end
++
++        pipe(self.execpath or 'mmdc', args, '')
+         return read_file(outfile), mime_type
+       end)
+     end)
+@@ -223,11 +250,11 @@
+           ''
+         )
+         if not success then
+-          warn(string.format(
+-                 "The call\n%s\nfailed with error code %s. Output:\n%s",
+-                 result.command,
+-                 result.error_code,
+-                 result.output
++          io.stderr:write(string.format(
++                 "The call\n%s\nfailed with error code %s. Output:\n%s\n",
++                 tostring(result and result.command),
++                 tostring(result and result.error_code),
++                 tostring(result and result.output)
+           ))
+         end
+         return read_file(pdf_file), 'application/pdf'
+@@ -385,7 +412,7 @@
+ 
+   -- Sanity check
+   if not engine then
+-    warn(PANDOC_SCRIPT_FILE, ": No such engine '", name, "'.")
++    io.stderr:write(PANDOC_SCRIPT_FILE .. ": No such engine '" .. name .. "'.\n")
+     return nil
+   elseif engopts == false then
+     -- engine is disabled
+@@ -402,7 +429,7 @@
+     engopts['mime-type'] or engopts['mime-types']
+   )
+   if not mime_type then
+-    warn(PANDOC_SCRIPT_FILE, ": Cannot use ", name, " with ", format.name)
++    io.stderr:write(PANDOC_SCRIPT_FILE .. ": Cannot use " .. name .. " with " .. format.name .. "\n")
+     return nil
+   end
+ 
+@@ -600,13 +627,13 @@
+       -- Bail if an error occurred; imgdata contains the error message
+       -- when that happens.
+       if not success then
+-        warn(PANDOC_SCRIPT_FILE, ': ', tostring(imgdata))
++        io.stderr:write(PANDOC_SCRIPT_FILE .. ': ' .. tostring(imgdata) .. '\n')
+         return nil
+       elseif not imgdata then
+-        warn(PANDOC_SCRIPT_FILE, ': Diagram engine returned no image data.')
++        io.stderr:write(PANDOC_SCRIPT_FILE .. ': Diagram engine returned no image data.\n')
+         return nil
+       elseif not imgtype then
+-        warn(PANDOC_SCRIPT_FILE, ': Diagram engine did not return a MIME type.')
++        io.stderr:write(PANDOC_SCRIPT_FILE .. ': Diagram engine did not return a MIME type.\n')
+         return nil
+       end
+ 

stuff/pandoc-diagram.ps1

@@ -0,0 +1,19 @@
+<#
+.SYNOPSIS
+    pandoc wrapper with diagram.lua filter, mermaid, and tikz/xelatex support.
+.DESCRIPTION
+    Sets MERMAID_BIN, TIKZ_BIN, and passes --lua-filter diagram.lua to pandoc.
+    All other arguments are forwarded to pandoc.
+.EXAMPLE
+    .\pandoc-diagram.ps1 -- input.md -o output.pdf
+    .\pandoc-diagram.ps1 -- input.md -o output.html --embed-resources
+    .\pandoc-diagram.ps1 -- input.md -o output.pdf --pdf-engine=xelatex
+#>
+function whereis($cmd) { Get-Command $cmd -CommandType Application | select-object -ExpandProperty Path -First 1 }
+
+$filter = Join-Path $PSScriptRoot 'diagram.lua'
+
+$env:MERMAID_BIN = whereis mmdc
+$env:TIKZ_BIN = 'xelatex'
+
+& pandoc --lua-filter $filter --pdf-engine=xelatex @args