lda-wf/docs/source_architecture.md

Source Architecture

This map explains how workflow capability sources fit together. Use it when deciding where to add a new source family or when untangling wf_sources_mcp from the older wf_mcp compatibility package.

Short Version

wf_authoring
  creates NodeSpec, reducers, and authored workflows

wf_platform
  defines neutral source DTOs:
    CapabilitySource
    CapabilityBuckets
    SourceVisibility
    SourcePermissions

wf_api
  consumes source DTOs through WorkflowSpecProvider
  owns built-in local sources:
    wf.std
    wf.recipes

wf_sources_mcp
  implements MCP as an upstream source provider

wf_sources_python
  implements trusted in-process Python source loading

wf_server
  composes configured providers into WorkflowServer

wf_transport_*
  exposes WorkflowServer over protocols

The workflow API should not care whether a capability came from MCP, Python, OpenAPI, or a built-in source. It should see CapabilitySource and executable NodeSpec objects.

Package Responsibilities

Package	Owns	Does not own
wf_authoring	@node, NodeSpec, builder DSL, reducers, reusable authored ops.	Server config, source loading, MCP sessions.
wf_platform	Neutral source DTOs and source visibility/permission metadata.	Provider-specific loading or execution.
wf_api	Application operations over capabilities, drafts, artifacts, deployments, and runs. Built-in wf.std / wf.recipes sources.	MCP/Python/OpenAPI source-specific behavior.
wf_sources_mcp	MCP source ids, connection models, auth/catalog stores, discovery, SDK facade, persistent runtime pool, converters, wrappers.	MCP frontend/proxy compatibility, durable server composition.
wf_sources_python	Trusted Python module registry loading and projection to CapabilitySource.	Authoring primitives, registry mutation/apply, sandboxing.
wf_server	WorkflowServer composition from config/store/source providers.	JSON-RPC method definitions, MCP protocol frontend.
wf_transport_rpc_http	JSON-RPC HTTP app/client around an existing WorkflowServer.	Server startup policy, source-provider composition.
wf_client	Async Python facade over a narrow client port: capability discovery, local authoring, artifact/deployment snapshots, and durable runs.	Draft workspace authoring, server composition, transport registration.
wf_mcp	Legacy/special-purpose MCP frontend, broker glue, proxy, compatibility shims.	New durable product behavior unless explicitly retiring old callers.

Data Flow

wf_config.server.sources[]
  -> wf_server.config selects source providers
  -> provider loads live source inventory
  -> CapabilitySource map
  -> WorkflowSpecProvider
  -> WorkflowApi / WorkflowServer
  -> transport or CLI

Python applications consume the same server through wf_client.App; the client's local builder and immutable snapshots sit above the transport:

wf_client.App
  -> WorkflowClientPort
  -> RpcWorkflowApiClient
  -> JSON-RPC HTTP
  -> WorkflowServer / WorkflowApi

The first shared provider seam is intentionally static:

class WorkflowSourceProvider(Protocol):
    def load_sources(self) -> Mapping[str, CapabilitySource]: ...

This covers source families that can project configured inventory into workflow-facing CapabilitySource objects. Provider-specific runtime pools, admin/apply hooks, auth, catalog caches, and live health checks stay outside this narrow seam until a source family needs them.

Capability, Tool, Resource, And Prompt

Use these terms precisely:

• A source is the owner and namespace. Examples: everything.default, wf.std, wf.source, or a future OpenAPI/Python source.
• A tool is provider-native. For MCP, it is an MCP tool discovered from an upstream server.
• A workflow capability is workflow-native. It is the NodeSpec shape a graph can call with typed input, typed output, outcomes, validation, and trace behavior. Tools can be projected into workflow capabilities, but the two are not identical concepts.
• A resource is source-owned addressable content. The URI is not globally meaningful by itself; it must be interpreted with the source that owns it.
• A prompt is source-owned prompt/template inventory. Listing prompts is inventory; rendering a prompt is an upstream operation and may be stateful.

Saved workflow data should prefer logical source references over concrete source ids. For example, a resource ref should store:

{"logical_source": "drive", "uri": "gdrive://file/abc"}

The deployment binding decides whether drive means drive.personal, drive.work, or another concrete source. Platform sources such as wf.std and wf.source are special because their logical source id is also their concrete source id, so they do not require deployment bindings. Legacy explicit self-bindings such as wf.std=wf.std are accepted as no-op compatibility, but new deployments should omit them. Validation rejects non-self platform bindings such as wf.std=custom.std as stale or misleading configuration.

Runtime dereference is explicit. Passing a resource ref by value does not fetch content. A helper capability such as wf.source.read_resource receives the ref, uses runtime/platform context to resolve the logical source, and applies bounded output policy before returning text into workflow state/output.

Prompt rendering uses the same source ownership rules, but it is not yet exposed as a workflow helper source. Keep prompt support at inventory/inspection level until there is a concrete graph use case, argument schema contract, and bounded output policy equivalent to wf.source.read_resource.

For MCP, the provider also owns stateful upstream sessions:

McpSourceConnection
  -> McpRuntimePool
  -> McpSourceClient
  -> MCP ClientSession

For Python, the provider is simpler:

PythonSourceConfig(path, module, registry)
  -> PythonSourceProvider
  -> import module
  -> load NodeSpec registry
  -> qualify specs under source id
  -> CapabilitySource(kind="python")

For a concrete operator flow, see the Python source runbook. It covers writing ops.py, configuring path/module/registry, validating the config, calling a capability, and running a saved workflow deployment.

Built-Ins Versus Configured Sources

wf.std and wf.recipes are built-in local sources owned by wf_api.local_sources. They are always platform-versioned content.

Configured sources are explicit server/operator choices:

• kind: "mcp": upstream MCP server capabilities.
• kind: "python": trusted in-process project capabilities.
• future kind: "openapi": HTTP/OpenAPI operations.

Do not move wf.std or wf.recipes into wf_sources_python. They are not operator-configured project sources.

Generated draft workflows may still use built-in helper sources such as wf.std. Platform sources do not need deployment bindings, so deployment examples should bind configured sources only. If validation reports a platform source binding, remove it unless it is an old self-binding kept only for compatibility.

wf_sources_mcp Internal Layers

wf_sources_mcp is clearer if read from bottom to top:

ids / transports / connections
  source identity and MCP connection description

auth / storage
  source auth records and catalog cache files

client
  one live MCP ClientSession facade

runtime
  persistent session pool for stateful upstream operations

sdk
  one-shot adapter, operation protocols, and MCP-to-workflow converters

catalog / discovery / tool_wrappers
  turn MCP tools/resources/prompts into workflow-facing source inventories

wf_mcp may still re-export or adapt some of this while old callers exist. New durable server/source work should prefer wf_sources_mcp directly.

Adding A New Source Family

Start with a provider package:

src/wf_sources_<kind>/
  __init__.py
  loader.py or provider.py
  tests/

Then add:

1. A wf_config discriminated-union source model.
2. A provider loader that returns CapabilitySource.
3. wf_server.config composition for that source kind.
4. Tests proving cap list, cap call, and one workflow run path.
5. Docs stating which parts are static, mutable, reloadable, or deferred.

Do not add source-family branches inside wf_api run execution. Source-specific logic belongs in the provider package or server composition layer.

Python client boundary

wf_client is a consumer-facing layer, not another server API. App.from_http_jsonrpc() creates a lazy HTTP transport; the first capability/artifact/deployment/run operation performs I/O. EditableWorkflow keeps graph construction local and uses the port only for remote validation and artifact persistence. The server remains responsible for durable artifact, deployment, and run stores and for resolving concrete source bindings.

The client intentionally omits draft workspace operations. Drafts are a server-side authoring/admin surface, and a server composition must opt in to registering their JSON-RPC methods. This keeps a normal Python application focused on the stable artifact -> deployment -> run lifecycle while retaining the underlying draft implementation for explicit server users.