docs: streamline project entrypoint

2026-06-08 23:05:14 +07:00
parent 7043866259
commit 5b81380f2a
3 changed files with 70 additions and 545 deletions
@@ -12,6 +12,8 @@ implementation plans are kept for context, not as active instructions.
  core type-shape cleanup.
 - [`wf_core_architecture.md`](wf_core_architecture.md): kernel package
  boundaries, runtime flow, validation flow, and known runtime gaps.
 - [`wf_api_architecture.md`](wf_api_architecture.md): workflow application API,
  server/transport boundaries, and source package split.
 - [`wf_mcp_architecture.md`](wf_mcp_architecture.md): MCP package boundaries,
  dependency rules, reload/proxy behavior, and extraction seams.
@@ -1,6 +1,8 @@
 # Project Map
-This repository has three main packages plus examples and tests.
+This repository has workflow kernel, API/server, transport, source, CLI, examples,
 and tests packages. The older MCP package still exists, but new durable client
 paths should go through `wf_server` plus transport/source packages.
 ## Packages
@@ -8,7 +10,12 @@ This repository has three main packages plus examples and tests.
 | --- | --- | --- |
 | `wf_core` | Deterministic workflow kernel: models, validation, runtime, run state, traces, interrupts, foreach, and path/state operations. | Runtime users, `wf_authoring`, workflow adapters. |
 | `wf_authoring` | Ergonomic workflow construction: `@node`, `NodeSpec`, builder DSL, conditions, path helpers, reusable ops, subgraph nodes. | Humans, tests, future LLM workflow builders. |
-| `wf_mcp` | MCP-specific integration: SDK adapters, upstream MCP runtime, proxy/debug compatibility, legacy broker entrypoints, and wrappers for discovered MCP tools. | MCP source/provider code, MCP transport work, and compatibility callers. Durable workflow server paths should prefer `wf_server` plus transport packages. |
+| `wf_api` | Workflow application surface over core/artifacts/platform: capabilities, drafts, artifacts, deployments, runs, and source/admin surfaces. | `wf_cli`, `wf_server`, JSON-RPC clients, future transports. |
 | `wf_server` | Durable server composition boundary around `WorkflowApi` plus optional admin/source-registry surfaces. | Transport packages and server startup code. |
 | `wf_transport_rpc_http` | JSON-RPC-over-HTTP app/client and `wf-rpc-server` CLI. | Remote `wf` clients and local server smoke tests. |
 | `wf_sources_mcp` | MCP-as-upstream-source implementation: ids, registry DTOs, auth/catalog stores, discovery, SDK client/facade, runtime pool, wrappers. | `wf_server`, broker glue, MCP source tests. |
 | `wf_mcp` | MCP frontend/compatibility package: legacy `wf-mcp` entrypoints, broker glue, proxy/admin tools, and shims while extraction continues. | Compatibility callers and MCP transport work. |
 | `wf_cli` | Command-line frontend over local or remote workflow APIs. | Humans, scripts, agent skills. |
 ## Important Entry Points
@@ -22,7 +29,13 @@ This repository has three main packages plus examples and tests.
 - `wf_authoring.node`: typed Python function to `NodeSpec`.
 - [`docs/wf_authoring_control_flow.md`](wf_authoring_control_flow.md): when to
  use `branch`, `handle`, `match`, `when`, and `choose`.
- `wf_mcp`: MCP-specific facade and compatibility package.
+- `wf_api.WorkflowApi`: process-local workflow application facade.
 - `wf_server.WorkflowServer`: durable workflow server composition object.
 - `wf_transport_rpc_http.RpcWorkflowApiClient`: JSON-RPC client implementing
  the workflow/admin surfaces over HTTP.
 - `wf_sources_mcp.McpRuntimePool`: persistent MCP source runtime for stateful
  upstream tools/resources/prompts.
 - `wf_mcp`: MCP-specific frontend and compatibility package.
 - `wf-mcp`: legacy/special-purpose MCP script from `pyproject.toml`.
 - `wf-rpc-server`: preferred durable workflow server script for CLI/API clients.
 - `wf_mcp.broker.WfMcpService.get_catalog()`: backend MCP catalog snapshots.
@@ -70,8 +83,9 @@ need local environment configuration.
 - Add new graph/model syntax in `wf_core.models`, then validate it in
  `wf_core.validation`.
 - Add author convenience helpers in `wf_authoring`, not `wf_core`.
- Add upstream MCP source/provider behavior in `wf_mcp` concern packages until a
+- Add upstream MCP source/provider behavior in `wf_sources_mcp`.
-  dedicated `wf_sources_mcp` split exists.
+- Keep `wf_mcp` changes limited to MCP frontend/broker/proxy compatibility
  unless the work is explicitly retiring old callers.
 - Add durable workflow server behavior in `wf_server` or transport packages, not
  the legacy `wf-mcp` entrypoint.
 - Add broker-local workflow utilities as `WfMcpService` spec sources, not as
@@ -1,550 +1,59 @@
-# lda.chat - Running Design Notes
+# lda.chat workflow platform
-## Related docs
+This repository is building a workflow runtime for AI-assisted digital work.
 Agents discover capabilities, author typed workflows, validate them, deploy
 them, and run them through a deterministic executor. The LLM plans; the runtime
 executes.
- [docs/README.md](docs/README.md): documentation index; start here for current vs historical docs.
+## Quick Start
 - [docs/authoring_sketch.md](docs/authoring_sketch.md): `wf_authoring` direction, including `@node`, `NodeSpec`, builder ergonomics, async registry seams, and subgraph-as-node.
 - [docs/project_map.md](docs/project_map.md): current package map, entrypoints, examples, tests, and verification commands.
 - [docs/current_roadmap.md](docs/current_roadmap.md): short active next-work list after the core type-shape cleanup.
 - [docs/wf_core_architecture.md](docs/wf_core_architecture.md): current `wf_core` package boundaries, runtime flow, validation flow, and remaining cleanup seams.
 - [docs/schema_validation.md](docs/schema_validation.md): current payload schema validation limits and intended validation seam.
 - [docs/wf_mcp_plan.md](docs/wf_mcp_plan.md): `wf_mcp` direction as a namespaced MCP capability broker plus workflow build/run layer.
 - [docs/wf_mcp_architecture.md](docs/wf_mcp_architecture.md): current `wf_mcp` package boundaries and dependency rules.
 - [docs/historical/scratchpad.md](docs/historical/scratchpad.md): historical design notes that fed the current model.
-## What is this
+```powershell
 uv run wf --help
 uv run wf --local cap list --format ids
 uv run wf --local cap inspect wf.std.constant
 uv run wf --local cap call wf.std.constant --input '{"value":"hello"}'
 uv run pytest -q
 ```
-`lda.chat` is an AI agent that turns natural language requests into executable
+Run the durable JSON-RPC server:
 workspace workflows. The LLM plans. The executor runs. Tools do the real work.
-Target users are knowledge workers doing repetitive digital tasks like
+```powershell
-collecting, summarizing, reporting, and monitoring without writing code.
+uv run wf-rpc-server --config wf.config.json --host 127.0.0.1 --port 8765
 uv run wf --url http://127.0.0.1:8765/rpc cap list --format ids
 ```
---
+## Current Shape
 ## Thesis scope vs product
 **Thesis:** prove the architecture end to end.
 - a few working MCP tools
 - LLM plans a workflow from natural language
 - executor validates and runs it
 - results come back in structured form
 **Real product:** everything around it.
 - broad MCP bridge layer
 - per-tool auth handling
 - tool registry / marketplace
 - scheduling, monitoring, multi-user support
 The thesis should demonstrate the core split cleanly, not solve the whole platform.
 ---
 ## Core architecture
 ```text
-User request
+wf_cli
-  -> LLM planner
+  -> wf_transport_rpc_http or local target
-    -> Workflow JSON
+  -> wf_server
-      -> Deterministic executor
+  -> wf_api
-        -> MCP tools / subworkflows
+  -> wf_core / wf_artifacts / wf_sources_*
          -> Shared workflow state
 ```
-Principles:
+- `wf_core`: deterministic graph/runtime kernel.
-
+- `wf_authoring`: Python authoring helpers and `NodeSpec` creation.
- the LLM plans, but does not execute
+- `wf_api`: workflow application surface shared by local and remote frontends.
- the executor executes, but does not think
+- `wf_server`: durable server composition boundary.
- nodes do work
+- `wf_transport_rpc_http`: JSON-RPC-over-HTTP client/server transport.
- edges are dumb routing
+- `wf_sources_mcp`: upstream MCP source implementation and persistent runtime.
-
+- `wf_mcp`: MCP frontend, legacy entrypoints, proxy/admin glue, and compatibility
---
+  shims during extraction.
-
+- `wf_cli`: command-line frontend over local or remote workflow APIs.
-## Workflow model
+
-
+## Start Reading
-A workflow is a directed graph with explicit schemas and explicit control flow.
+
-
+- [docs/README.md](docs/README.md): documentation index.
-```text
+- [docs/wf_cli.md](docs/wf_cli.md): CLI lifecycle, remote targets, and common
-Workflow
+  diagnostics.
-  .name
+- [docs/wf_api_architecture.md](docs/wf_api_architecture.md): current API,
-  .input_schema
+  server, transport, and source package boundaries.
-  .state_schema
+- [docs/project_map.md](docs/project_map.md): package map, entrypoints, examples,
-  .output_schema
+  and verification commands.
-  .node_defs[]
+- [docs/current_roadmap.md](docs/current_roadmap.md): active next work.
-  .start
+
-  .nodes[]
+Historical design notes and completed implementation plans live under
-  .edges[]
+[docs/historical/](docs/historical/). The previous long top-level running-design
-```
+note is intentionally no longer the project front door; current model details
-
+are split across the docs index above.
 ### Schema boundaries
 - `input_schema` validates the run input once
 - input is copied into mutable workflow state at run start
 - `state_schema` defines typed workflow memory and merge behavior
 - `output_schema` is derived from final state at the end
 - internal state may contain values that never appear in output
 This separation matters because a workflow can later be reused as a node.
 ### Reusable node definition
 Nodes are defined independently from any one graph.
 ```text
 NodeDef
  .name
  .input_schema
  .output_schema
  .outcomes[]         // ex: ["ok"], ["approved", "rejected"]
  .retry?
  .timeout_seconds?
 ```
 Even a simple node still declares its outcomes explicitly.
 ### Node use inside a graph
 Graph use-sites bind reusable nodes into graph state.
 ```text
 NodeUse
  .id
  .type: "node"
  .node               // reference to NodeDef
  .desc?
  .input[]            // graph/value -> node-local input bindings
  .output[]           // node-local output -> graph state bindings
  .retry?             // optional override
  .timeout_seconds?   // optional override
 ```
 The node does not get ambient access to all workflow state. It receives:
 - the payload built by explicit `input` bindings
 - runtime context
 This keeps nodes reusable instead of graph-coupled.
 Path bindings use structural paths in saved JSON:
 ```json
 {
  "input": [
    {
      "target": { "root": "local", "parts": ["text"] },
      "path": { "root": "input", "parts": ["text"] }
    }
  ],
  "output": [
    {
      "source": { "root": "local", "parts": ["echoed"] },
      "target": { "root": "state", "parts": ["echoed"] }
    }
  ]
 }
 ```
 Legacy `in_map`, `input_values`, and `out_map` shapes remain parse-only
 compatibility inputs. New authored or saved plans should write `input` and
 `output`.
 ### Control-flow nodes
 #### **ConditionNode**
 ```text
 .id
 .type: "condition"
 .check
 ```
 Conditions are structured JSON, not freeform code strings.
 #### **ForeachNode**
 ```text
 .id
 .type: "foreach"
 .over
 .as
 .mode               // serial | parallel
 .on_item_error      // fail | collect | skip
 ```
 #### **JoinNode**
 ```text
 .id
 .type: "join"
 ```
 #### **InterruptNode**
 ```text
 .id
 .type: "interrupt"
 .kind               // approval | text_input | choice | tool_auth | ...
 .request[]          // graph/value -> public interrupt payload bindings
 .resume[]           // resume payload -> graph state bindings
 .outcomes[]         // ex: submitted, cancelled
 ```
 Interrupt nodes are explicit graph nodes, not arbitrary line-level pauses inside Python code.
 This keeps pause points visible, typed, traceable, and easier to validate.
 ### Edges
 Routing is outcome-based.
 ```text
 Edge
  .from
  .outcome
  .to
 ```
 - source nodes declare which outcomes are possible
 - edges map those outcomes to next nodes
 - terminal routing can go to builtin `__end__`, which is compatibility
  shorthand for workflow outcome `ok`
 - explicit `EndNode` steps set non-`ok` workflow outcomes such as `error` or
  `needs_input`
 Reaching an undeclared or unwired outcome is runtime failure.
 ---
 ## Node result contract
 At the engine boundary, every node result should look uniform:
 ```text
 NodeResult
  .outcome
  .output
  .meta?
  .extra?
 ```
 Two channels stay separate:
 - `outcome` is for control flow
 - `output` is for typed business data
 This lets richer nodes branch directly, while dumb MCP tools can still be adapted.
 If a tool only returns data, a later `ConditionNode` can decide routing.
 Extra undeclared returned keys:
 - do not affect typed execution
 - are preserved in trace
 - are not visible to later nodes unless a future raw-trace access feature is added
 ---
 ## Condition model
 The condition language should stay small and structured.
 Supported v1 operators:
 - `and`
 - `or`
 - `not`
 - `eq`
 - `ne`
 - `gt`
 - `lt`
 - `exists`
 Operands are paths or literals, such as:
 - `state.summary_count`
 - `context.retry_count`
 - string, number, bool, null literals
 This is less work than inventing a mini-language and generally easier for LLMs to emit reliably.
 ---
 ## State and merge semantics
 State fields carry metadata, especially merge behavior.
 ```text
 StateField
  .path
  .schema            // JSON Schema with reducer extension metadata
  .reducer           // wf.std.replace | wf.std.append | ...
 ```
 Rules:
 - merge behavior belongs to the state field, not the node
 - declared fields get typed merge semantics
 - undeclared keys behave like plain map entries
 - concurrent writes to the same untyped key should fail
 - concurrent writes to the same typed key should fail unless that field's merge strategy allows it
 This keeps parallel and foreach behavior predictable.
 ---
 ## Execution semantics
 Executor steps:
 1. Validate workflow structure
 2. Validate workflow input
 3. Copy input into state
 4. Start at the declared `start` node
 5. Resolve node input snapshot from canonical `input` bindings
 6. Execute node with mapped input plus runtime context
 7. Validate typed node output
 8. Commit mapped output into state
 9. Route by returned outcome
 10. Stop when routing reaches `__end__` or an explicit `EndNode`
 11. Derive and validate final output from state
 Commit rules:
 - no partial commit before success
 - validation failure is node failure
 - execution failure is not a normal business outcome
 Runtime failure means things like:
 - timeout
 - transport error
 - MCP failure
 - invalid node output
 Expected business branching should use declared outcomes instead.
 ### Trace
 Each run should produce a structured trace.
 At minimum, each trace entry should capture:
 - node id
 - step type
 - resolved input snapshot
 - returned outcome
 - mapped output
 - state changes committed
 - next routed node
 This keeps debugging honest as loops, retries, and richer tool adapters are added.
 ### Run State
 Execution should return a structured run object, not just loose output maps.
 At minimum, run state should track:
 - workflow name
 - run status
 - original workflow input
 - current mutable state
 - final output when available
 - current node id
 - prior outcome
 - activated incoming edge
 - accumulated trace
 - terminal error if execution failed
 This gives the engine a clean path toward checkpointing, interrupt, and resume later.
 Run state should also carry execution frames.
 - v1 may only have a root workflow frame
 - future `foreach` and subgraph execution should attach work to child frames
 - interrupts should belong to a frame, not just to the run globally
 Two useful execution entry points fall out of this:
 - `step_workflow(...)` for one-node advancement
 - `resume_workflow(...)` for continuing from an existing run state
 That keeps the main execution loop small and makes future interrupt/re-invoke semantics easier to model.
 ### Interrupts
 Interrupts should be graph-native and typed.
 Preferred model:
 1. A normal business node returns an outcome such as `needs_input`
 2. The graph routes to an `InterruptNode`
 3. The interrupt node produces a typed interrupt request
 4. The runtime marks the run interrupted and surfaces the request externally
 5. External code supplies a typed resume payload
 6. The interrupt node maps resume payload back into state
 7. The graph continues via a declared interrupt outcome such as `submitted` or `cancelled`
 This is deliberately stricter than line-level dynamic interrupts inside arbitrary node code.
 Why:
 - pause points stay visible in the graph
 - interrupt payloads can be validated by `kind`
 - traces stay clean
 - child graph interruption is easier to surface to parent graphs
 - `foreach` and nested execution are less likely to become cursed
 #### Interrupt request shape
 At runtime, an interrupt should become a structured request attached to run state.
 ```text
 InterruptRequest
  .id
  .node_id
  .kind
  .payload
  .resumable
 ```
 Notes:
 - `kind` chooses the interrupt contract
 - `payload` is still JSON-like, but should be typed by `kind`
 - v1 should allow only one active interrupt per run at a time
 - the whole run pauses in v1, even if the interrupt originated inside a child graph or future foreach frame
 #### Resume semantics
 Resume should not jump back into the middle of arbitrary Python code.
 Instead:
 - resume data is delivered to the interrupt node
 - the interrupt node maps resume fields back into state through explicit `resume` bindings
 - graph routing continues normally from that node
 This means the primary interrupt design is node-based, not line-based.
 #### Child graph behavior
 If a child graph interrupts:
 - the parent run is considered interrupted in v1
 - the parent only needs to know that the child graph node interrupted
 - the child interrupt payload should still be surfaced to the external caller/UI
 #### Foreach compatibility
 Interrupt design must be compatible with future foreach support.
 Bad designs to avoid:
 - one anonymous global interrupt payload with no origin
 - treating interrupt like a normal business outcome only
 - resuming from arbitrary instruction pointers
 Better design:
 - interrupt belongs to a specific execution frame / node
 - v1 may still pause the whole run
 - future foreach support should record which iteration/frame produced the interrupt
 ### Retry
 Default retry behavior should be strict.
 - retry can be declared on the reusable node definition
 - graph use-site can override it
 - retries should reuse the original resolved input snapshot by default
 - failfast is the default engine behavior
 ### Foreach
 `foreach` should be explicit, not hidden.
 - execution mode is graph-declared: `serial` or `parallel`
 - item failure policy is graph-declared: `fail`, `collect`, or `skip`
 - accumulation follows target state field merge strategy
 ### Join
 Join is barrier-only.
 - it does not do custom merge logic
 - it waits only for branches actually activated in this run
 - it should not wait for statically possible but never-taken branches
 ---
 ## Validation
 Before execution, the validator should be able to check:
 - workflow has a valid `start`
 - node def names are unique
 - node ids are unique
 - every referenced node definition exists
 - every edge source exists
 - every edge destination exists or is `__end__`
 - every edge outcome is declared by its source node
 - each source + outcome pair is wired at most once
 - every reachable outcome is wired
 - every `input` binding target is a valid node-local path
 - every `output` binding source exists in node output schema where schema detail is known
 - every condition path references valid `input`, `state`, or `context`
 - every `foreach.over` path references valid `input` or `state`
 At runtime, the executor should still check:
 - returned outcome is declared
 - typed output matches schema
 - merge conflicts
 - required mapped outputs are present
 - final output validates
 ---
 ## Registry and composition
 A saved workflow can be registered as a reusable node.
 That gives:
 - workflow as function
 - graph composition
 - one consistent node contract whether the implementation is MCP-backed or workflow-backed
 ---
 ## Thesis demo direction
 A good thesis demo is narrow but real, for example:
 - use an external MCP such as Google Drive
 - have the planner generate a workflow
 - validate it against schema
 - execute the workflow once end to end
 - return a structured result or report
 That proves the planner/executor/tool split without pretending to solve general automation.
 ---
 ## Open questions
 - how strict should output derivation be: direct projection vs explicit final output mapping?
 - what is the best trace model for raw extra node output?
 - how should isolated pure-Python execution fit later: node type, tool backend, or separate functional API?
 - how should retry policy interact with side-effecting tools that are not safely idempotent?
 ---
 ## Stack
 - Backend: Python
 - Validation and schema: Pydantic
 - Tool execution: MCP
 - Storage: SQLite or PostgreSQL
 - Future isolated compute: restricted Python sandbox, possibly remote