docs: clarify agent and source architecture guides

AGENTS.md

@@ -1,80 +1,62 @@
-# Agent guide
+# Agent Guide
 
-## pitfalls / guide
+## Runtime And Syntax
 
-### tech stack
+- Python baseline is 3.14 (`requires-python = ">=3.14"`).
+- Python 3.14 syntax is allowed. Do not rewrite valid new syntax just because it
+  looks unusual.
+- Example: Parentheses-Free Exceptions (PEP 758).
 
-Python baseline is 3.14 (`requires-python = ">=3.14"`). Python 3.14 syntax is allowed; do not "fix" valid new syntax just because it looks unusual.
+## Code Organization
 
-Example new syntax:
+- Prefer focused packages/modules over large flat files when adding new areas.
+- Preserve current package boundaries. If unsure, check `docs/project_map.md`
+  and `docs/source_architecture.md`.
+- Put related files in folders from the start when the area is likely to grow.
+  Example: prefer `src/pack/foo/bar.py` over adding many unrelated
+  `src/pack/foo_bar.py` files.
 
-- Parentheses-Free Exceptions (PEP 758) <!-- coderabbit! -->
+## Tests
 
-### extra fields
+- Prefer pytest `tmp_path` for test-local filesystem state.
+- Avoid fixed paths under `local_temp_root()` unless the test explicitly needs
+  cross-process persistence and cleans up after itself.
+- Prefer `async def test_x()` with pytest-asyncio over
+  `def test_x(): async def scenario(): ...; asyncio.run(scenario())`.
+- Prefer field-level assertions like `actual["field"] == expected["field"]`
+  over whole-object equality unless extra fields are intentionally forbidden.
+- Scope test runs. This repo is large; broad test commands can be slow.
 
-prefer asserts actual['field'] == expected['field'] over assert actual == expected unless we know better (eg. no extra fields allowed)
+## Verification Commands
-
-### tests
-
-Prefer pytest `tmp_path` for test-local filesystem state. Avoid fixed paths under `local_temp_root()` for tests that create durable files unless the test explicitly cleans or needs cross-process persistence; stale files there can change later test runs. (almost never the case btw)
-
-Now that pytest-asyncio is installed, prefer `async def test_x()`
-instead of `def test_x(): async def scenario(): ...; asyncio.run(scenario())`
-
-### mgmt
-
-More packages please. we spent a while cleaning flatten packages/modules; putting files of similar interests in folders and sub-folders.
-example: some of tests/ and some packages. (simple example: src/pack/foo_bar.py -> src/pack/foo/bar.py)
-
-Lets just do that from the start this time, ok?
-
-### Docs mgmt 
-
-read docs/AGENTS.md
-
-more later
-
-## Test suite
 
 ```bash
-uv run /* --env-file .env */ pytest -q
+uv run pytest -q
-uv run ruff check; uv run ruff format
+uv run ruff check
-uv run basedpyright # --level error # to cut spam if typeCheckingMode = "recommended", but its "basic" now
+uv run ruff format
-## maybe uvx ty
+uv run basedpyright --level error
 ```
 
-or so i think.
+Use `uv run --env-file .env pytest -q` when live MCP-backed tests need local
+environment configuration.
 
-### project is getting big
+## Docs
 
-scope your calls lads. else timeouts. not good for rapid testings
+- Before editing docs, read `docs/AGENTS.md`.
+- `docs/current_roadmap.md` is the live roadmap.
+- If docs mention a partial implementation, add a short comment or docstring at
+  the code seam too. Future agents see code before they see old plans.
 
-## code
+## Comments And Docstrings
 
-### docstrings/comment
+- Add comments/docstrings around weird or non-obvious logic.
+- Add docstrings explaining compound return types that otherwise say little,
+  for example `tuple[list[str], Any]`.
+- Polish common helper docs if you keep using the helper.
 
-- add docstrings or comments around weird or non-obvious logic.
+## Skills And Tools
-  - Add docstrings explaining compound return types that otherwise say nothing (e.g. `tuple[list[str], Any]`)
-- Polish the thing (at least its docs) if you keep using it (helper fn, common class)
 
-### partial impls
+- Skills can be useful but can overstate urgency. Use judgment.
-
+- Serena is useful for symbol discovery and rename-like navigation. Prefer
-If code has a partial implementation and docs mention the limitation, add a
+  built-in edit tools for ordinary file edits.
-short comment or docstring at the code seam too. Future agents see code before
+- If an MCP tool is irrelevant to the project or clutters available tools,
-they see old plans.
+  mention it so it can be disabled.
-
-## skills
-
-im looking at you superpowers
-
-skills screaming at you IMPORTANT CRITICAL bs. Use your best judgements. maybe they are critical idk you tell me
-
-### mcp tools
-
-<!-- looking at you opencode/mimo -->
-`serena-agent` is likely set up. You want to use it for symbol discovery (akin to the `outline` tab in vscode)
-(maybe for symbol renames as well). It is a strong tool!
-
-if you reach for it to do general file editing, built-in tools may be better
-
-If you notice an MCP tool that seems irrelevant to the project or is cluttering your available tools, mention it so we can disable it.

docs/README.md

@@ -14,6 +14,8 @@ implementation plans are kept for context, not as active instructions.
   boundaries, runtime flow, validation flow, and known runtime gaps.
 - [`wf_api_architecture.md`](wf_api_architecture.md): workflow application API,
   server/transport boundaries, and source package split.
+- [`source_architecture.md`](source_architecture.md): source provider package
+  map for built-in, MCP, Python, and future source families.
 - [`wf_mcp_architecture.md`](wf_mcp_architecture.md): MCP package boundaries,
   dependency rules, reload/proxy behavior, and extraction seams.
 

docs/project_map.md

@@ -4,6 +4,9 @@ This repository has workflow kernel, API/server, transport, source, CLI, example
 and tests packages. The older MCP package still exists, but new durable client
 paths should go through `wf_server` plus transport/source packages.
 
+For the source-provider-specific map, see
+[`source_architecture.md`](source_architecture.md).
+
 ## Packages
 
 | Package | Purpose | Usual callers |

docs/source_architecture.md

@@ -0,0 +1,149 @@
+# Source Architecture
+
+This map explains how workflow capability sources fit together. Use it when
+deciding where to add a new source family or when untangling `wf_sources_mcp`
+from the older `wf_mcp` compatibility package.
+
+## Short Version
+
+```text
+wf_authoring
+  creates NodeSpec, reducers, and authored workflows
+
+wf_platform
+  defines neutral source DTOs:
+    CapabilitySource
+    CapabilityBuckets
+    SourceVisibility
+    SourcePermissions
+
+wf_api
+  consumes source DTOs through WorkflowSpecProvider
+  owns built-in local sources:
+    wf.std
+    wf.recipes
+
+wf_sources_mcp
+  implements MCP as an upstream source provider
+
+wf_sources_python
+  implements trusted in-process Python source loading
+
+wf_server
+  composes configured providers into WorkflowServer
+
+wf_transport_*
+  exposes WorkflowServer over protocols
+```
+
+The workflow API should not care whether a capability came from MCP, Python,
+OpenAPI, or a built-in source. It should see `CapabilitySource` and executable
+`NodeSpec` objects.
+
+## Package Responsibilities
+
+| Package | Owns | Does not own |
+| --- | --- | --- |
+| `wf_authoring` | `@node`, `NodeSpec`, builder DSL, reducers, reusable authored ops. | Server config, source loading, MCP sessions. |
+| `wf_platform` | Neutral source DTOs and source visibility/permission metadata. | Provider-specific loading or execution. |
+| `wf_api` | Application operations over capabilities, drafts, artifacts, deployments, and runs. Built-in `wf.std` / `wf.recipes` sources. | MCP/Python/OpenAPI source-specific behavior. |
+| `wf_sources_mcp` | MCP source ids, connection models, auth/catalog stores, discovery, SDK facade, persistent runtime pool, converters, wrappers. | MCP frontend/proxy compatibility, durable server composition. |
+| `wf_sources_python` | Trusted Python module registry loading and projection to `CapabilitySource`. | Authoring primitives, registry mutation/apply, sandboxing. |
+| `wf_server` | `WorkflowServer` composition from config/store/source providers. | JSON-RPC method definitions, MCP protocol frontend. |
+| `wf_transport_rpc_http` | JSON-RPC HTTP app/client around an existing `WorkflowServer`. | Server startup policy, source-provider composition. |
+| `wf_mcp` | Legacy/special-purpose MCP frontend, broker glue, proxy, compatibility shims. | New durable product behavior unless explicitly retiring old callers. |
+
+## Data Flow
+
+```text
+wf_config.server.sources[]
+  -> wf_server.config selects source providers
+  -> provider loads live source inventory
+  -> CapabilitySource map
+  -> WorkflowSpecProvider
+  -> WorkflowApi / WorkflowServer
+  -> transport or CLI
+```
+
+For MCP, the provider also owns stateful upstream sessions:
+
+```text
+McpSourceConnection
+  -> McpRuntimePool
+  -> McpSourceClient
+  -> MCP ClientSession
+```
+
+For Python, the provider is simpler:
+
+```text
+PythonSourceConfig(module, registry)
+  -> import module
+  -> load NodeSpec registry
+  -> qualify specs under source id
+  -> CapabilitySource(kind="python")
+```
+
+## Built-Ins Versus Configured Sources
+
+`wf.std` and `wf.recipes` are built-in local sources owned by `wf_api.local_sources`.
+They are always platform-versioned content.
+
+Configured sources are explicit server/operator choices:
+
+- `kind: "mcp"`: upstream MCP server capabilities.
+- `kind: "python"`: trusted in-process project capabilities.
+- future `kind: "openapi"`: HTTP/OpenAPI operations.
+
+Do not move `wf.std` or `wf.recipes` into `wf_sources_python`. They are not
+operator-configured project sources.
+
+## `wf_sources_mcp` Internal Layers
+
+`wf_sources_mcp` is clearer if read from bottom to top:
+
+```text
+ids / transports / connections
+  source identity and MCP connection description
+
+auth / storage
+  source auth records and catalog cache files
+
+client
+  one live MCP ClientSession facade
+
+runtime
+  persistent session pool for stateful upstream operations
+
+sdk
+  one-shot adapter, operation protocols, and MCP-to-workflow converters
+
+catalog / discovery / tool_wrappers
+  turn MCP tools/resources/prompts into workflow-facing source inventories
+```
+
+`wf_mcp` may still re-export or adapt some of this while old callers exist. New
+durable server/source work should prefer `wf_sources_mcp` directly.
+
+## Adding A New Source Family
+
+Start with a provider package:
+
+```text
+src/wf_sources_<kind>/
+  __init__.py
+  loader.py or provider.py
+  tests/
+```
+
+Then add:
+
+1. A `wf_config` discriminated-union source model.
+2. A provider loader that returns `CapabilitySource`.
+3. `wf_server.config` composition for that source kind.
+4. Tests proving `cap list`, `cap call`, and one workflow run path.
+5. Docs stating which parts are static, mutable, reloadable, or deferred.
+
+Do not add source-family branches inside `wf_api` run execution. Source-specific
+logic belongs in the provider package or server composition layer.
+