242 lines
9.2 KiB
Markdown
242 lines
9.2 KiB
Markdown
# Source Architecture
|
|
|
|
This map explains how workflow capability sources fit together. Use it when
|
|
deciding where to add a new source family or when untangling `wf_sources_mcp`
|
|
from the older `wf_mcp` compatibility package.
|
|
|
|
## Short Version
|
|
|
|
```text
|
|
wf_authoring
|
|
creates NodeSpec, reducers, and authored workflows
|
|
|
|
wf_platform
|
|
defines neutral source DTOs:
|
|
CapabilitySource
|
|
CapabilityBuckets
|
|
SourceVisibility
|
|
SourcePermissions
|
|
|
|
wf_api
|
|
consumes source DTOs through WorkflowSpecProvider
|
|
owns built-in local sources:
|
|
wf.std
|
|
wf.recipes
|
|
|
|
wf_sources_mcp
|
|
implements MCP as an upstream source provider
|
|
|
|
wf_sources_python
|
|
implements trusted in-process Python source loading
|
|
|
|
wf_server
|
|
composes configured providers into WorkflowServer
|
|
|
|
wf_transport_*
|
|
exposes WorkflowServer over protocols
|
|
```
|
|
|
|
The workflow API should not care whether a capability came from MCP, Python,
|
|
OpenAPI, or a built-in source. It should see `CapabilitySource` and executable
|
|
`NodeSpec` objects.
|
|
|
|
## Package Responsibilities
|
|
|
|
| Package | Owns | Does not own |
|
|
| --- | --- | --- |
|
|
| `wf_authoring` | `@node`, `NodeSpec`, builder DSL, reducers, reusable authored ops. | Server config, source loading, MCP sessions. |
|
|
| `wf_platform` | Neutral source DTOs and source visibility/permission metadata. | Provider-specific loading or execution. |
|
|
| `wf_api` | Application operations over capabilities, drafts, artifacts, deployments, and runs. Built-in `wf.std` / `wf.recipes` sources. | MCP/Python/OpenAPI source-specific behavior. |
|
|
| `wf_sources_mcp` | MCP source ids, connection models, auth/catalog stores, discovery, SDK facade, persistent runtime pool, converters, wrappers. | MCP frontend/proxy compatibility, durable server composition. |
|
|
| `wf_sources_python` | Trusted Python module registry loading and projection to `CapabilitySource`. | Authoring primitives, registry mutation/apply, sandboxing. |
|
|
| `wf_server` | `WorkflowServer` composition from config/store/source providers. | JSON-RPC method definitions, MCP protocol frontend. |
|
|
| `wf_transport_rpc_http` | JSON-RPC HTTP app/client around an existing `WorkflowServer`. | Server startup policy, source-provider composition. |
|
|
| `wf_client` | Async Python facade over a narrow client port: capability discovery, local authoring, artifact/deployment snapshots, and durable runs. | Draft workspace authoring, server composition, transport registration. |
|
|
| `wf_mcp` | Legacy/special-purpose MCP frontend, broker glue, proxy, compatibility shims. | New durable product behavior unless explicitly retiring old callers. |
|
|
|
|
## Data Flow
|
|
|
|
```text
|
|
wf_config.server.sources[]
|
|
-> wf_server.config selects source providers
|
|
-> provider loads live source inventory
|
|
-> CapabilitySource map
|
|
-> WorkflowSpecProvider
|
|
-> WorkflowApi / WorkflowServer
|
|
-> transport or CLI
|
|
```
|
|
|
|
Python applications consume the same server through `wf_client.App`; the
|
|
client's local builder and immutable snapshots sit above the transport:
|
|
|
|
```text
|
|
wf_client.App
|
|
-> WorkflowClientPort
|
|
-> RpcWorkflowApiClient
|
|
-> JSON-RPC HTTP
|
|
-> WorkflowServer / WorkflowApi
|
|
```
|
|
|
|
The first shared provider seam is intentionally static:
|
|
|
|
```python
|
|
class WorkflowSourceProvider(Protocol):
|
|
def load_sources(self) -> Mapping[str, CapabilitySource]: ...
|
|
```
|
|
|
|
This covers source families that can project configured inventory into
|
|
workflow-facing `CapabilitySource` objects. Provider-specific runtime pools,
|
|
admin/apply hooks, auth, catalog caches, and live health checks stay outside
|
|
this narrow seam until a source family needs them.
|
|
|
|
## Capability, Tool, Resource, And Prompt
|
|
|
|
Use these terms precisely:
|
|
|
|
- A **source** is the owner and namespace. Examples: `everything.default`,
|
|
`wf.std`, `wf.source`, or a future OpenAPI/Python source.
|
|
- A **tool** is provider-native. For MCP, it is an MCP tool discovered from an
|
|
upstream server.
|
|
- A **workflow capability** is workflow-native. It is the `NodeSpec` shape a
|
|
graph can call with typed input, typed output, outcomes, validation, and trace
|
|
behavior. Tools can be projected into workflow capabilities, but the two are
|
|
not identical concepts.
|
|
- A **resource** is source-owned addressable content. The URI is not globally
|
|
meaningful by itself; it must be interpreted with the source that owns it.
|
|
- A **prompt** is source-owned prompt/template inventory. Listing prompts is
|
|
inventory; rendering a prompt is an upstream operation and may be stateful.
|
|
|
|
Saved workflow data should prefer logical source references over concrete
|
|
source ids. For example, a resource ref should store:
|
|
|
|
```json
|
|
{"logical_source": "drive", "uri": "gdrive://file/abc"}
|
|
```
|
|
|
|
The deployment binding decides whether `drive` means `drive.personal`,
|
|
`drive.work`, or another concrete source. Platform sources such as `wf.std` and
|
|
`wf.source` are special because their logical source id is also their concrete
|
|
source id, so they do not require deployment bindings. Legacy explicit
|
|
self-bindings such as `wf.std=wf.std` are accepted as no-op compatibility, but
|
|
new deployments should omit them. Validation rejects non-self platform bindings
|
|
such as `wf.std=custom.std` as stale or misleading configuration.
|
|
|
|
Runtime dereference is explicit. Passing a resource ref by value does not fetch
|
|
content. A helper capability such as `wf.source.read_resource` receives the ref,
|
|
uses runtime/platform context to resolve the logical source, and applies bounded
|
|
output policy before returning text into workflow state/output.
|
|
|
|
Prompt rendering uses the same source ownership rules, but it is not yet exposed
|
|
as a workflow helper source. Keep prompt support at inventory/inspection level
|
|
until there is a concrete graph use case, argument schema contract, and bounded
|
|
output policy equivalent to `wf.source.read_resource`.
|
|
|
|
For MCP, the provider also owns stateful upstream sessions:
|
|
|
|
```text
|
|
McpSourceConnection
|
|
-> McpRuntimePool
|
|
-> McpSourceClient
|
|
-> MCP ClientSession
|
|
```
|
|
|
|
For Python, the provider is simpler:
|
|
|
|
```text
|
|
PythonSourceConfig(path, module, registry)
|
|
-> PythonSourceProvider
|
|
-> import module
|
|
-> load NodeSpec registry
|
|
-> qualify specs under source id
|
|
-> CapabilitySource(kind="python")
|
|
```
|
|
|
|
For a concrete operator flow, see the
|
|
[`Python source runbook`](runbooks/python-source.md). It covers writing `ops.py`,
|
|
configuring `path`/`module`/`registry`, validating the config, calling a
|
|
capability, and running a saved workflow deployment.
|
|
|
|
## Built-Ins Versus Configured Sources
|
|
|
|
`wf.std` and `wf.recipes` are built-in local sources owned by `wf_api.local_sources`.
|
|
They are always platform-versioned content.
|
|
|
|
Configured sources are explicit server/operator choices:
|
|
|
|
- `kind: "mcp"`: upstream MCP server capabilities.
|
|
- `kind: "python"`: trusted in-process project capabilities.
|
|
- future `kind: "openapi"`: HTTP/OpenAPI operations.
|
|
|
|
Do not move `wf.std` or `wf.recipes` into `wf_sources_python`. They are not
|
|
operator-configured project sources.
|
|
|
|
Generated draft workflows may still use built-in helper sources such as
|
|
`wf.std`. Platform sources do not need deployment bindings, so deployment
|
|
examples should bind configured sources only. If validation reports a platform
|
|
source binding, remove it unless it is an old self-binding kept only for
|
|
compatibility.
|
|
|
|
## `wf_sources_mcp` Internal Layers
|
|
|
|
`wf_sources_mcp` is clearer if read from bottom to top:
|
|
|
|
```text
|
|
ids / transports / connections
|
|
source identity and MCP connection description
|
|
|
|
auth / storage
|
|
source auth records and catalog cache files
|
|
|
|
client
|
|
one live MCP ClientSession facade
|
|
|
|
runtime
|
|
persistent session pool for stateful upstream operations
|
|
|
|
sdk
|
|
one-shot adapter, operation protocols, and MCP-to-workflow converters
|
|
|
|
catalog / discovery / tool_wrappers
|
|
turn MCP tools/resources/prompts into workflow-facing source inventories
|
|
```
|
|
|
|
`wf_mcp` may still re-export or adapt some of this while old callers exist. New
|
|
durable server/source work should prefer `wf_sources_mcp` directly.
|
|
|
|
## Adding A New Source Family
|
|
|
|
Start with a provider package:
|
|
|
|
```text
|
|
src/wf_sources_<kind>/
|
|
__init__.py
|
|
loader.py or provider.py
|
|
tests/
|
|
```
|
|
|
|
Then add:
|
|
|
|
1. A `wf_config` discriminated-union source model.
|
|
2. A provider loader that returns `CapabilitySource`.
|
|
3. `wf_server.config` composition for that source kind.
|
|
4. Tests proving `cap list`, `cap call`, and one workflow run path.
|
|
5. Docs stating which parts are static, mutable, reloadable, or deferred.
|
|
|
|
Do not add source-family branches inside `wf_api` run execution. Source-specific
|
|
logic belongs in the provider package or server composition layer.
|
|
|
|
## Python client boundary
|
|
|
|
`wf_client` is a consumer-facing layer, not another server API.
|
|
`App.from_http_jsonrpc()` creates a lazy HTTP transport; the first
|
|
capability/artifact/deployment/run operation performs I/O. `EditableWorkflow`
|
|
keeps graph construction local and uses the port only for remote validation and
|
|
artifact persistence. The server remains responsible for durable artifact,
|
|
deployment, and run stores and for resolving concrete source bindings.
|
|
|
|
The client intentionally omits draft workspace operations. Drafts are a
|
|
server-side authoring/admin surface, and a server composition must opt in to
|
|
registering their JSON-RPC methods. This keeps a normal Python application
|
|
focused on the stable artifact -> deployment -> run lifecycle while retaining
|
|
the underlying draft implementation for explicit server users.
|