741 lines
37 KiB
Markdown
741 lines
37 KiB
Markdown
# WF MCP Runtime & Source Provider Map
|
|
|
|
Date: 2026-06-07
|
|
|
|
> **Historical note (2026-06-08):** This research snapshot predates the
|
|
> `wf_sources_mcp` extraction slices completed on 2026-06-08. Several blockers
|
|
> called out below have since moved: source ID helpers live in
|
|
> `wf_sources_mcp.ids`, MCP runtime/SDK/client code lives in `wf_sources_mcp`,
|
|
> and broker DTO construction helpers live in `wf_mcp.source_registry`.
|
|
|
|
## Executive Summary
|
|
|
|
`wf_mcp` is a monolith containing four distinct responsibilities that should become separate packages:
|
|
|
|
1. **MCP source provider runtime** (transport opening, session pooling, auth) -- belongs in `wf_sources_mcp`
|
|
2. **MCP frontend transport** (FastMCP server, proxy mounts, MCP tool registration) -- belongs in a future `wf_transport_mcp`
|
|
3. **Broker service layer** (connection catalog, discovery, source catalog, workflow runtime) -- stays in `wf_mcp` or becomes `wf_server`
|
|
4. **Compatibility shims** (re-export modules, `WfMcpService` facade) -- remains in `wf_mcp` until callers migrate
|
|
|
|
The critical blocker is `ConnectionConfig` (defined at `src/wf_mcp/broker/models.py:37`). This dataclass is consumed by `wf_sources_mcp.sdk.protocols`, `wf_sources_mcp.auth`, `wf_sources_mcp.source_registry`, `wf_sources_mcp.storage.store`, the runtime factory, the adapter, the pool, and every broker service. Until `ConnectionConfig` moves to a neutral package or is replaced by a typed protocol, extraction is blocked.
|
|
|
|
The next safe slice is **Slice 0: Define a `SourceConnection` protocol in `wf_sources_mcp`** so runtime/session code stops depending on the broker DTO. After that, the transport-opening logic in `runtime/factory.py` and `sdk/adapter.py` can merge into `wf_sources_mcp.sdk` without dragging the broker layer along.
|
|
|
|
---
|
|
|
|
## Package Responsibility Map
|
|
|
|
### `wf_sources_mcp` (source provider core)
|
|
|
|
Owns: MCP source provider identity, auth, catalog entries, storage, and the SDK adapter/executor protocols.
|
|
|
|
| Module | Responsibility | Status |
|
|
|--------|---------------|--------|
|
|
| `auth.py` | MCP auth record, env/header extraction, diagnostic helpers | Canonical. Has TYPE_CHECKING dep on `wf_mcp.broker.models.ConnectionConfig`. |
|
|
| `sdk/protocols.py` | `BackendAdapter`, `ToolExecutor`, `ToolCallResult` | Canonical. Has TYPE_CHECKING dep on `ConnectionConfig`. |
|
|
| `sdk/converters.py` | MCP tool/resource/prompt conversion | Canonical. No `wf_mcp` deps. |
|
|
| `catalog/models.py` | `CatalogSnapshot`, `dump_catalog_snapshot` | Canonical. No `wf_mcp` deps. |
|
|
| `catalog/entries.py` | `CatalogNodeEntry`, `CatalogResourceEntry`, `CatalogPromptEntry`, `DiscoveredTool`, etc. | Canonical. No `wf_mcp` deps. |
|
|
| `source_registry.py` | `McpSourceRegistryEntry`, `SourceRegistryFile`, conversion helpers | Canonical. Imports `parse_connection_id` from `wf_mcp.connections` and `RESERVED_CONNECTION_IDS` from `wf_mcp.shared.names`. TYPE_CHECKING dep on `ConnectionConfig`. |
|
|
| `storage/store.py` | `AuthStore`, `CatalogStore`, `FileAuthStore`, `FileCatalogStore` | Canonical. `FileCatalogStore._connection_path` imports `parse_connection_id` from `wf_mcp.connections` at runtime. |
|
|
|
|
### Future `wf_transport_mcp` (MCP frontend transport)
|
|
|
|
Owns: FastMCP server creation, proxy mounting, MCP tool/resource/prompt registration, admin tool registration, workflow surface tool registration.
|
|
|
|
| Module | Responsibility | Status |
|
|
|--------|---------------|--------|
|
|
| `proxy/runtime.py` | `ProxyRuntime` -- mount upstream MCP connections into FastMCP | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, `EventBus`, `BrokerConfigManager`. |
|
|
| `proxy/mounts.py` | `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy` | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, FastMCP. |
|
|
| `proxy/tools.py` | Proxy tool listing/filtering | `wf_mcp` internal. |
|
|
| `proxy/safe_names.py` | `SafeToolNames` transform | `wf_mcp` internal. FastMCP transform. |
|
|
| `proxy/admin.py` | Proxy admin tools | `wf_mcp` internal. |
|
|
| `server/core.py` | `create_server`, `run_server`, `create_server_client` | `wf_mcp` entrypoint. Wires broker service + proxy + workflow surface + admin surface. |
|
|
| `admin_surface/tools.py` | `register_service_admin_tools` | `wf_mcp` internal. |
|
|
| `admin_surface/handlers/*.py` | Admin tool handlers | `wf_mcp` internal. |
|
|
| `workflow_surface/tools.py` | `register_workflow_tools` | `wf_mcp` internal. |
|
|
| `workflow_surface/*.py` | Workflow tool models, handlers, lifecycle | `wf_mcp` internal. |
|
|
| `shared/names.py` | `ProxyNamespace`, `ADMIN_NAMESPACE`, `RESERVED_CONNECTION_IDS`, namespace helpers | `wf_mcp` internal. FastMCP-specific. |
|
|
| `cli.py` | CLI entrypoint | `wf_mcp` entrypoint. |
|
|
|
|
### `wf_server` / `wf_api` (server core)
|
|
|
|
Owns: Workflow API, operation context, artifact management, deployment, run lifecycle.
|
|
|
|
Already extracted. These packages consume `wf_sources_mcp` types and `wf_mcp.broker.service.WfMcpService` through `wf_api.WorkflowApi` and `wf_api.WorkflowRuntimeAdapter`.
|
|
|
|
### Legacy `wf_mcp` (broker + compatibility shims)
|
|
|
|
Owns: Connection registry, broker config, broker service coordination, events, and all compatibility re-export shims.
|
|
|
|
| Module | Responsibility | Status |
|
|
|--------|---------------|--------|
|
|
| `broker/models.py` | `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots`, `SourceConfigOwnership` | **The core blocker.** All other packages depend on this. |
|
|
| `broker/service/core.py` | `WfMcpService` -- compatibility coordinator | Facade. Delegates to focused services. |
|
|
| `broker/service/connection_service.py` | `ConnectionService` | Focused service. Depends on `ConnectionConfig`, `SourceCatalogService`. |
|
|
| `broker/service/source_catalog.py` | `SourceCatalogService` | Focused service. Depends on `ConnectionConfig`, `McpEvent`, `NodeSpec`. |
|
|
| `broker/service/upstream_transport.py` | `UpstreamTransportService` | Focused service. Depends on `ConnectionConfig`, `BackendAdapter`, `ToolExecutor`. |
|
|
| `broker/service/workflow_runtime.py` | `WorkflowRuntimeService` | Focused service. Depends on `SourceCatalogService`. |
|
|
| `broker/service/content_access.py` | `ContentAccessService` | Focused service. |
|
|
| `broker/service/events.py` | `BrokerEventRecorder` | Focused service. Depends on `McpEvent`. |
|
|
| `broker/service/adapters.py` | `require_adapter` | Helper. |
|
|
| `broker/discovery.py` | `discover_connection_capabilities`, `specs_from_discovered_tools` | Broker logic. |
|
|
| `broker/catalog.py` | `snapshot_from_specs`, `CombinedCatalog` | Broker catalog projection. |
|
|
| `broker/config.py` | `build_service_from_config`, `load_broker_config`, `broker_config_from_workflow_config` | Config construction. |
|
|
| `connections.py` | `ConnectionRegistry`, `parse_connection_id`, `qualify_node_name` | Broker connection registry. |
|
|
| `auth.py` | Re-export shim from `wf_sources_mcp.auth` | Compatibility. |
|
|
| `models.py` | Re-export shim aggregating broker models | Compatibility. |
|
|
| `source_registry.py` | Re-export shim from `wf_sources_mcp.source_registry` | Compatibility. |
|
|
| `capabilities.py` | Re-export shim from `wf_sources_mcp.catalog.entries` | Compatibility. |
|
|
| `runtime/protocols.py` | Re-export shim from `wf_sources_mcp.sdk` | Compatibility. |
|
|
| `events/bus.py` | `EventBus`, `InMemoryEventSink` | Broker-local event fanout. |
|
|
| `events/models.py` | `McpEvent`, `make_event` | Broker event model. |
|
|
|
|
---
|
|
|
|
## Current Dependency Blockers
|
|
|
|
### Blocker 1: `ConnectionConfig` origin
|
|
|
|
**File:** `src/wf_mcp/broker/models.py:37-44`
|
|
|
|
```python
|
|
@dataclass(slots=True)
|
|
class ConnectionConfig:
|
|
id: str
|
|
server: str
|
|
account: str
|
|
enabled: bool = True
|
|
metadata: dict[str, Any] = field(default_factory=dict)
|
|
source_config_ownership: SourceConfigOwnership = "locked"
|
|
```
|
|
|
|
This DTO is imported by:
|
|
|
|
| Consumer | Import path | Import type |
|
|
|----------|------------|-------------|
|
|
| `wf_sources_mcp.auth` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
|
|
| `wf_sources_mcp.sdk.protocols` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
|
|
| `wf_sources_mcp.source_registry` | `wf_mcp.models.ConnectionConfig` | `TYPE_CHECKING` |
|
|
| `wf_sources_mcp.storage.store` | (none directly, but `FileCatalogStore._connection_path` calls `parse_connection_id` from `wf_mcp.connections`) | runtime |
|
|
| `wf_mcp.runtime.factory` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.sdk.adapter` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.runtime.pool` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.runtime.session` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.workflow.wrappers` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.broker.discovery` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.broker.service.*` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.proxy.mounts` | `wf_mcp.models.ConnectionConfig` | runtime |
|
|
| `wf_mcp.proxy.runtime` | (indirectly via `BrokerConfig`) | runtime |
|
|
|
|
**Impact:** Until `ConnectionConfig` moves to a neutral package (or `wf_sources_mcp` defines its own protocol), the runtime code in `wf_sources_mcp` cannot be independent of `wf_mcp`.
|
|
|
|
### Blocker 2: `parse_connection_id` origin
|
|
|
|
**File:** `src/wf_mcp/connections.py:11-25`
|
|
|
|
Used by:
|
|
|
|
- `wf_sources_mcp.source_registry` (runtime import, line 32)
|
|
- `wf_sources_mcp.storage.store` (runtime import inside `_connection_path`, line 132)
|
|
- `wf_mcp.connections` (canonical home)
|
|
|
|
**Impact:** `wf_sources_mcp` has a runtime import dependency on `wf_mcp` for connection ID validation. This function should move to a neutral package or `wf_sources_mcp`.
|
|
|
|
### Blocker 3: `RESERVED_CONNECTION_IDS` origin
|
|
|
|
**File:** `src/wf_mcp/shared/names.py:25`
|
|
|
|
```python
|
|
RESERVED_CONNECTION_IDS = frozenset({ADMIN_NAMESPACE, "wf.mcp"})
|
|
```
|
|
|
|
Used by:
|
|
|
|
- `wf_sources_mcp.source_registry` (runtime import, line 33)
|
|
- `wf_mcp.broker.service.connection_service` (runtime import, line 14)
|
|
|
|
**Impact:** `wf_sources_mcp` imports from `wf_mcp.shared.names` which transitively imports FastMCP transforms (line 8-15 of `shared/names.py`). This creates an unwanted dependency chain.
|
|
|
|
### Blocker 4: `McpSdkAdapter` transport-opening duplication
|
|
|
|
**Files:**
|
|
|
|
- `src/wf_mcp/runtime/factory.py:43-90` (`_create_with_stack`)
|
|
- `src/wf_mcp/sdk/adapter.py:34-75` (`_session`)
|
|
|
|
Both methods contain nearly identical logic for:
|
|
|
|
- Reading `connection.metadata["transport"]` to select stdio vs streamable HTTP
|
|
- Creating `StdioServerParameters` and calling `stdio_client`
|
|
- Creating `httpx.AsyncClient` and calling `streamable_http_client`
|
|
- Creating `ClientSession` and calling `session.initialize()`
|
|
- Applying auth via `mcp_auth_env` / `mcp_auth_headers`
|
|
|
|
The difference: factory.py owns the session long-term (persistent actor pattern), while adapter.py opens/closes per call (one-shot pattern).
|
|
|
|
**Impact:** This duplication means both files must be updated together when transport handling changes. They should share a transport-opening helper.
|
|
|
|
### Blocker 5: `BrokerConfig` in proxy/runtime
|
|
|
|
**File:** `src/wf_mcp/proxy/runtime.py:73-85`
|
|
|
|
`ProxyRuntime.__init__` takes `BrokerConfig` directly and reads `config.connections`. The proxy layer is a frontend transport concern that should not depend on the broker config DTO.
|
|
|
|
**Impact:** Proxy runtime cannot move to `wf_transport_mcp` until it consumes a transport-neutral config shape.
|
|
|
|
### Blocker 6: `McpEvent` coupling
|
|
|
|
**File:** `src/wf_mcp/events/models.py`
|
|
|
|
`McpEvent` is used by:
|
|
|
|
- `UpstreamTransportService` (event_sink)
|
|
- `SourceCatalogService` (emit_event)
|
|
- `BrokerEventRecorder` (event_bus)
|
|
- `broker/catalog.py` (indirectly through event callbacks)
|
|
- `workflow/wrappers.py` (emit_event callback)
|
|
|
|
**Impact:** `McpEvent` is broker-specific. If source provider code needs to emit events, it should use a protocol or a neutral event type. Currently `wf_sources_mcp` does not import `McpEvent` directly, which is good.
|
|
|
|
---
|
|
|
|
## Detailed Findings by Worker Scope
|
|
|
|
### Worker 1: Runtime/Session Findings
|
|
|
|
#### Where is MCP session opening duplicated?
|
|
|
|
The transport-opening logic appears in two places:
|
|
|
|
1. **`runtime/factory.py:43-90`** (`PersistentSessionFactory._create_with_stack`):
|
|
- Opens stdio or streamable HTTP transport
|
|
- Creates `ClientSession`
|
|
- Calls `session.initialize()`
|
|
- Returns session owned by `AsyncExitStack`
|
|
- Used for persistent long-lived sessions (actor pattern)
|
|
|
|
2. **`sdk/adapter.py:34-75`** (`McpSdkAdapter._session`):
|
|
- Opens stdio or streamable HTTP transport
|
|
- Creates `ClientSession`
|
|
- Calls `session.initialize()`
|
|
- Yields session in async context manager
|
|
- Used for one-shot per-call sessions (discovery, admin operations)
|
|
|
|
Both import from the same MCP SDK modules:
|
|
|
|
- `mcp.client.stdio.StdioServerParameters`, `stdio_client`
|
|
- `mcp.client.streamable_http.streamable_http_client`
|
|
- `mcp.client.session.ClientSession`
|
|
- `wf_sources_mcp.auth.mcp_auth_env`, `mcp_auth_headers`
|
|
|
|
#### Which client operations are supported by one-shot adapter but not persistent runtime?
|
|
|
|
The `BackendAdapter` protocol (`wf_sources_mcp.sdk.protocols:26-88`) defines:
|
|
|
|
- `list_tools`, `list_resources`, `list_prompts`
|
|
- `get_connection_metadata`
|
|
- `read_resource`, `get_prompt`
|
|
- `invoke_method`, `send_notification`
|
|
- `call_tool`
|
|
|
|
The `PersistentMcpSession` (`runtime/session.py:19-49`) only exposes:
|
|
|
|
- `call_tool`
|
|
- `close`
|
|
|
|
The `McpRuntimePool` (`runtime/pool.py:43-96`) only exposes:
|
|
|
|
- `get_session` (returns `PersistentMcpSession`)
|
|
- `call_tool`
|
|
- `close_connection`, `close_all`
|
|
|
|
**Missing from persistent runtime:** `list_tools`, `list_resources`, `list_prompts`, `get_connection_metadata`, `read_resource`, `get_prompt`, `invoke_method`, `send_notification`.
|
|
|
|
This is intentional -- persistent sessions exist for workflow execution (tool calls only). Discovery and admin operations use one-shot adapters. However, this means the persistent runtime cannot replace the adapter for all operations.
|
|
|
|
#### What common `McpClientSession` / `McpClientSessionFactory` interface should exist?
|
|
|
|
Both factory.py and adapter.py share:
|
|
|
|
1. Transport selection logic (stdio vs streamable HTTP)
|
|
2. Auth application (env vars for stdio, headers for HTTP)
|
|
3. Session creation and initialization
|
|
|
|
A shared `open_mcp_session` helper should:
|
|
|
|
- Accept a connection descriptor (transport type, command/url, env/headers) and optional auth
|
|
- Return an initialized `ClientSession` (or yield it)
|
|
- Be used by both `PersistentSessionFactory._create_with_stack` and `McpSdkAdapter._session`
|
|
|
|
The connection descriptor should NOT be `ConnectionConfig` directly -- it should be a transport-specific DTO that `ConnectionConfig` can convert to.
|
|
|
|
#### What blocks moving this code to `wf_sources_mcp`?
|
|
|
|
1. `ConnectionConfig` dependency (Blocker 1)
|
|
2. `McpEvent` event callbacks in `PersistentSessionFactory` (the `_SessionOwner._run` method does not emit events, but the pool and factory are used by `broker/config.py` which wires events)
|
|
3. The `PersistentMcpSession` dataclass holds `connection: ConnectionConfig` and `auth: AuthRecord` -- the `AuthRecord` is already in `wf_sources_mcp`, but `ConnectionConfig` is not
|
|
|
|
**Recommendation:** Define a `SourceConnection` protocol in `wf_sources_mcp.sdk.protocols` that captures the transport fields `ConnectionConfig` exposes to runtime code. The factory/pool/session code should consume this protocol, not the concrete DTO.
|
|
|
|
### Worker 2: Broker/Upstream Findings
|
|
|
|
#### What is truly MCP-source-provider logic vs broker/catalog projection logic?
|
|
|
|
**MCP-source-provider logic** (belongs in `wf_sources_mcp`):
|
|
|
|
- Transport opening (stdio, streamable HTTP) -- currently in `factory.py` and `adapter.py`
|
|
- Auth application (env vars, headers) -- already in `wf_sources_mcp.auth`
|
|
- `BackendAdapter` protocol and `ToolExecutor` protocol -- already in `wf_sources_mcp.sdk`
|
|
- `ToolCallResult` dataclass -- already in `wf_sources_mcp.sdk`
|
|
- `DiscoveredTool`, `DiscoveredResource`, `DiscoveredPrompt` -- already in `wf_sources_mcp.catalog`
|
|
- `CatalogSnapshot` and catalog entries -- already in `wf_sources_mcp.catalog`
|
|
- Auth/catalog storage -- already in `wf_sources_mcp.storage`
|
|
- Source registry models -- already in `wf_sources_mcp.source_registry`
|
|
|
|
**Broker/catalog projection logic** (stays in `wf_mcp`):
|
|
|
|
- `UpstreamTransportService` -- wraps `BackendAdapter` with auth loading, event recording, catalog refresh orchestration
|
|
- `SourceCatalogService` -- manages `CapabilitySource` registrations, catalog hydration, source inventory
|
|
- `discover_connection_capabilities` -- orchestrates adapter calls and wraps results
|
|
- `specs_from_discovered_tools` -- wraps discovered tools into `NodeSpec` with event emission
|
|
- `snapshot_from_specs` -- builds `CatalogSnapshot` from `NodeSpec` dict
|
|
- `CombinedCatalog` -- aggregates snapshots across connections
|
|
- `ConnectionService` -- connection registry lifecycle
|
|
- `BrokerEventRecorder` -- broker event fanout
|
|
- `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots` -- broker config DTOs
|
|
|
|
#### What should move to `wf_sources_mcp`?
|
|
|
|
1. **Transport-opening helper** (`open_mcp_session` or similar) -- extract from `factory.py` and `adapter.py`
|
|
2. **`parse_connection_id`** and **`RESERVED_CONNECTION_IDS`** -- move from `wf_mcp.connections` and `wf_mcp.shared.names`
|
|
3. **`SourceConnection` protocol** -- new, replacing `ConnectionConfig` in runtime code
|
|
|
|
#### What should stay in broker compatibility wiring?
|
|
|
|
1. `UpstreamTransportService` -- it orchestrates broker-specific concerns (events, catalog store, adapter registry)
|
|
2. `SourceCatalogService` -- it manages `CapabilitySource` which is `wf_platform`-level
|
|
3. `ConnectionService` -- it manages `ConnectionRegistry` which is broker-specific
|
|
4. `WfMcpService` -- the compatibility facade
|
|
5. All `broker/config.py` construction logic
|
|
|
|
#### Where are events/catalog/source_catalog dependencies preventing extraction?
|
|
|
|
- `UpstreamTransportService.refresh_connection_catalog` (lines 203-275) takes `source_catalog: SourceCatalogService` and `record_catalog_change_events` callback. This is broker orchestration, not source provider logic.
|
|
- `SourceCatalogService.register_specs` takes `record_catalog_change_events` callback. This is broker event wiring.
|
|
- `SourceCatalogService.spec_from_snapshot_entry` (lines 257-298) rebuilds executable `NodeSpec` from stored snapshots, routing calls through `tool_executor_for()`. This is broker hydration, not source provider logic.
|
|
|
|
None of these prevent `wf_sources_mcp` from owning transport opening -- they just cannot move with it.
|
|
|
|
### Worker 3: Config/Source/Auth Findings
|
|
|
|
#### What connection/auth/source DTOs are still too coupled to `wf_mcp`?
|
|
|
|
1. **`ConnectionConfig`** (`wf_mcp.broker.models:37-44`) -- the primary blocker. Every runtime and broker module depends on it.
|
|
|
|
2. **`BrokerConfig`** (`wf_mcp.broker.models:47-55`) -- holds `store_root`, `connections`, `store_roots`. Used by proxy/runtime, server/core, broker/config. This is broker-specific and should stay.
|
|
|
|
3. **`BrokerStoreRoots`** (`wf_mcp.broker.models:11-33`) -- filesystem roots for stores. Broker-specific.
|
|
|
|
4. **`SourceConfigOwnership`** (`wf_mcp.broker.models:7`) -- `Literal["locked", "seed"]`. Also defined in `wf_config.models:73`. The broker version should be the canonical one since it controls runtime behavior.
|
|
|
|
5. **`AuthRecord`** (`wf_sources_mcp.auth:22-25`) -- already canonical in `wf_sources_mcp`. The `wf_mcp.auth` module is a re-export shim.
|
|
|
|
#### What neutral or MCP-source-specific config object should runtime/session code consume instead of `ConnectionConfig.metadata`?
|
|
|
|
The runtime code (`factory.py`, `adapter.py`) reads these fields from `ConnectionConfig.metadata`:
|
|
|
|
| Field | Used by | Purpose |
|
|
|-------|---------|---------|
|
|
| `transport` | `factory.py:49`, `adapter.py:40` | Transport type selector (`"stdio"` or `"streamable_http"`) |
|
|
| `command` | `factory.py:56`, `adapter.py:42` | Stdio command |
|
|
| `args` | `factory.py:57`, `adapter.py:43` | Stdio arguments |
|
|
| `env` | `factory.py:51`, `adapter.py:44` | Stdio environment variables |
|
|
| `cwd` | `factory.py:59`, `adapter.py:45` | Stdio working directory |
|
|
| `url` | `factory.py:71`, `adapter.py:62` | HTTP transport URL |
|
|
|
|
A `SourceTransport` union type already exists in `wf_sources_mcp.source_registry:53-69`:
|
|
|
|
```python
|
|
class StdioSourceTransport(SourceRegistryBaseModel):
|
|
kind: Literal["stdio"] = "stdio"
|
|
command: str
|
|
args: tuple[str, ...] = ()
|
|
env: dict[str, str] = Field(default_factory=dict)
|
|
|
|
|
|
class HttpSourceTransport(SourceRegistryBaseModel):
|
|
kind: Literal["http"] = "http"
|
|
url: AnyHttpUrl
|
|
headers: dict[str, str] = Field(default_factory=dict)
|
|
```
|
|
|
|
The runtime code should consume a `SourceConnection` protocol that exposes:
|
|
|
|
- `id: str`
|
|
- `transport: StdioSourceTransport | HttpSourceTransport` (or a discriminated union)
|
|
- No `metadata: dict[str, Any]` bag
|
|
|
|
This would eliminate the `connection.metadata.get("transport", "stdio")` pattern scattered across `factory.py` and `adapter.py`.
|
|
|
|
#### What temporary dependencies remain and how should they be removed?
|
|
|
|
| Dependency | From | To | How to remove |
|
|
|-----------|------|-----|---------------|
|
|
| `parse_connection_id` | `wf_sources_mcp.source_registry:32` | `wf_mcp.connections` | Move `parse_connection_id` to `wf_sources_mcp` or a shared `wf_id` package. |
|
|
| `RESERVED_CONNECTION_IDS` | `wf_sources_mcp.source_registry:33` | `wf_mcp.shared.names` | Move constant to `wf_sources_mcp.source_registry` or shared package. Remove `wf_mcp.shared.names` import that transitively pulls FastMCP. |
|
|
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.auth:18` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
|
|
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.sdk.protocols:16` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
|
|
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.source_registry:36` | `wf_mcp.models` | Replace with `SourceConnection` protocol or keep as converter-only. |
|
|
| `parse_connection_id` runtime | `wf_sources_mcp.storage.store:132` | `wf_mcp.connections` | Move function to `wf_sources_mcp`. |
|
|
|
|
### Worker 4: Frontend/Compat Findings
|
|
|
|
#### What is MCP frontend transport vs old compatibility facade?
|
|
|
|
**MCP frontend transport** (FastMCP server + proxy + tool registration):
|
|
|
|
- `server/core.py` -- creates `FastMCP` server, wires everything
|
|
- `proxy/runtime.py` -- `ProxyRuntime` mounts upstream connections as FastMCP proxies
|
|
- `proxy/mounts.py` -- `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy`
|
|
- `proxy/tools.py` -- proxy tool listing/filtering
|
|
- `proxy/safe_names.py` -- `SafeToolNames` transform for strict clients
|
|
- `admin_surface/tools.py` -- registers `wf.admin.*` tools on the server
|
|
- `workflow_surface/tools.py` -- registers `wf.workflow.*` tools on the server
|
|
- `shared/names.py` -- `ProxyNamespace`, `ADMIN_NAMESPACE`, namespace helpers
|
|
|
|
**Old compatibility facade** (re-export shims and `WfMcpService`):
|
|
|
|
- `auth.py` -- re-exports from `wf_sources_mcp.auth`
|
|
- `models.py` -- re-exports broker models
|
|
- `capabilities.py` -- re-exports from `wf_sources_mcp.catalog.entries`
|
|
- `source_registry.py` -- re-exports from `wf_sources_mcp.source_registry`
|
|
- `runtime/protocols.py` -- re-exports from `wf_sources_mcp.sdk`
|
|
- `broker/service/core.py` -- `WfMcpService` facade
|
|
|
|
#### What should eventually become `wf_transport_mcp`?
|
|
|
|
1. `server/core.py` -- `create_server`, `run_server`
|
|
2. `proxy/runtime.py` -- `ProxyRuntime`
|
|
3. `proxy/mounts.py` -- proxy mount logic
|
|
4. `proxy/tools.py` -- proxy tool helpers
|
|
5. `proxy/safe_names.py` -- `SafeToolNames`
|
|
6. `proxy/admin.py` -- proxy admin tools
|
|
7. `admin_surface/tools.py` -- admin tool registration
|
|
8. `admin_surface/handlers/*.py` -- admin tool handlers
|
|
9. `workflow_surface/tools.py` -- workflow tool registration
|
|
10. `workflow_surface/models.py` -- workflow tool models
|
|
11. `shared/names.py` -- namespace helpers (minus `RESERVED_CONNECTION_IDS`)
|
|
12. `cli.py` -- CLI entrypoint
|
|
|
|
#### What should remain as legacy `wf_mcp` entrypoints/shims?
|
|
|
|
1. `auth.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.auth`)
|
|
2. `models.py` -- re-export shim (keep until all callers import from `wf_mcp.broker.models` directly)
|
|
3. `capabilities.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.catalog`)
|
|
4. `source_registry.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.source_registry`)
|
|
5. `runtime/protocols.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.sdk`)
|
|
6. `runtime/__init__.py` -- re-export `McpRuntimePool`, `PersistentMcpSession`, etc. (keep for backward compat)
|
|
|
|
#### What should not be touched during upstream-source extraction?
|
|
|
|
1. `workflow_surface/*` -- workflow tools are MCP-frontend, not source-provider
|
|
2. `admin_surface/*` -- admin tools are MCP-frontend, not source-provider
|
|
3. `proxy/*` -- proxy mounting is MCP-frontend transport
|
|
4. `server/core.py` -- server creation is MCP-frontend
|
|
5. `cli.py` -- CLI entrypoint
|
|
6. `events/*` -- broker-local event system
|
|
7. `broker/service/workflow_runtime.py` -- workflow execution coordination
|
|
8. `broker/service/content_access.py` -- resource/prompt access
|
|
9. `workflow/wrappers.py` -- NodeSpec wrapping (uses `ToolExecutor` protocol, which is correct)
|
|
|
|
---
|
|
|
|
## Recommended Next Slices
|
|
|
|
Ordered smallest-safe-first.
|
|
|
|
### Slice 0: Define `SourceConnection` protocol in `wf_sources_mcp`
|
|
|
|
**Goal:** Break the TYPE_CHECKING dependency from `wf_sources_mcp.sdk.protocols` and `wf_sources_mcp.auth` on `wf_mcp.broker.models.ConnectionConfig` by defining a transport-level protocol that captures what runtime code actually needs.
|
|
|
|
**Files likely touched:**
|
|
|
|
- `src/wf_sources_mcp/sdk/protocols.py` -- add `SourceConnection` protocol, update `BackendAdapter` and `ToolExecutor` signatures
|
|
- `src/wf_sources_mcp/auth.py` -- update `auth_ref_for_connection` and `connection_auth_diagnostic` to accept protocol
|
|
- `src/wf_sources_mcp/source_registry.py` -- update TYPE_CHECKING import
|
|
- `src/wf_mcp/broker/models.py` -- make `ConnectionConfig` implement the protocol (no structural change needed, just verify compatibility)
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Protocol conformance test: `ConnectionConfig` satisfies `SourceConnection`
|
|
- Verify `BackendAdapter` and `ToolExecutor` protocols still typecheck with `ConnectionConfig`
|
|
|
|
**What must NOT change:**
|
|
|
|
- `ConnectionConfig` fields/shape
|
|
- `BackendAdapter` method signatures (only the type of `connection` parameter changes)
|
|
- Any broker service code
|
|
- Any proxy/server code
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- `SourceConnection` is a new protocol, not a replacement. `ConnectionConfig` already satisfies it structurally.
|
|
- Existing `TYPE_CHECKING` imports in `wf_sources_mcp` become `SourceConnection` protocol imports.
|
|
- If any runtime code needs fields beyond the protocol (e.g., `source_config_ownership`), those stay on `ConnectionConfig` and are accessed through a cast or separate parameter.
|
|
|
|
---
|
|
|
|
### Slice 1: Move `parse_connection_id` and `RESERVED_CONNECTION_IDS` to `wf_sources_mcp`
|
|
|
|
**Goal:** Remove the runtime import dependency from `wf_sources_mcp` on `wf_mcp.connections` and `wf_mcp.shared.names`.
|
|
|
|
**Files likely touched:**
|
|
|
|
- `src/wf_sources_mcp/source_registry.py` -- replace imports, move validation logic
|
|
- `src/wf_sources_mcp/storage/store.py` -- replace `parse_connection_id` import
|
|
- `src/wf_mcp/connections.py` -- make `parse_connection_id` a re-export shim
|
|
- `src/wf_mcp/shared/names.py` -- make `RESERVED_CONNECTION_IDS` a re-export shim (or keep in `source_registry`)
|
|
- `src/wf_mcp/broker/service/connection_service.py` -- update import path
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Existing tests for `parse_connection_id` should still pass
|
|
- Verify `FileCatalogStore._connection_path` still validates connection IDs
|
|
|
|
**What must NOT change:**
|
|
|
|
- Validation logic (same regex, same error messages)
|
|
- `RESERVED_CONNECTION_IDS` values
|
|
- Any broker service behavior
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- Move `CONNECTION_ID_PATTERN`, `parse_connection_id` to `wf_sources_mcp.source_registry` or a new `wf_sources_mcp.validation` module.
|
|
- `wf_mcp.connections` becomes a re-export shim: `from wf_sources_mcp.source_registry import parse_connection_id`
|
|
- `RESERVED_CONNECTION_IDS` moves to `wf_sources_mcp.source_registry` (it's already imported there).
|
|
- `wf_mcp.shared.names` keeps its own copy or re-exports.
|
|
- `wf_mcp.shared.names` can remove the FastMCP import from the top-level module (move `ProxyNamespace` and FastMCP-specific code to a separate submodule if needed).
|
|
|
|
---
|
|
|
|
### Slice 2: Extract `open_mcp_session` transport helper to `wf_sources_mcp`
|
|
|
|
**Goal:** Eliminate the transport-opening duplication between `runtime/factory.py` and `sdk/adapter.py` by extracting a shared helper.
|
|
|
|
**Files likely touched:**
|
|
|
|
- New: `src/wf_sources_mcp/sdk/transport.py` -- `open_mcp_session` async context manager
|
|
- `src/wf_mcp/runtime/factory.py` -- use `open_mcp_session` in `_create_with_stack`
|
|
- `src/wf_mcp/sdk/adapter.py` -- use `open_mcp_session` in `_session`
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Unit test for `open_mcp_session` with mock transport
|
|
- Verify `PersistentSessionFactory` still creates persistent sessions correctly
|
|
- Verify `McpSdkAdapter` still creates one-shot sessions correctly
|
|
- Test error handling (unsupported transport, auth failures)
|
|
|
|
**What must NOT change:**
|
|
|
|
- `PersistentMcpSession` API
|
|
- `McpRuntimePool` behavior
|
|
- `McpSdkAdapter` method signatures
|
|
- Any broker service code
|
|
- Event emission patterns
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- `open_mcp_session` accepts a `SourceConnection` (from Slice 0) and `AuthRecord | None`.
|
|
- Returns an async context manager yielding `ClientSession`.
|
|
- Both `factory.py` and `adapter.py` call this helper instead of duplicating transport logic.
|
|
- The factory wraps it in `AsyncExitStack` for persistent ownership; the adapter uses it directly as a context manager.
|
|
|
|
---
|
|
|
|
### Slice 3: Move `PersistentSessionFactory`, `PersistentMcpSession`, `McpRuntimePool` to `wf_sources_mcp`
|
|
|
|
**Goal:** Move the persistent MCP runtime into the source provider package where it belongs.
|
|
|
|
**Files likely touched:**
|
|
|
|
- New: `src/wf_sources_mcp/runtime/__init__.py` -- package init
|
|
- New: `src/wf_sources_mcp/runtime/factory.py` -- moved from `wf_mcp/runtime/factory.py`
|
|
- New: `src/wf_sources_mcp/runtime/session.py` -- moved from `wf_mcp/runtime/session.py`
|
|
- New: `src/wf_sources_mcp/runtime/pool.py` -- moved from `wf_mcp/runtime/pool.py`
|
|
- `src/wf_mcp/runtime/__init__.py` -- becomes re-export shim
|
|
- `src/wf_mcp/runtime/factory.py` -- becomes re-export shim
|
|
- `src/wf_mcp/runtime/session.py` -- becomes re-export shim
|
|
- `src/wf_mcp/runtime/pool.py` -- becomes re-export shim
|
|
- `src/wf_mcp/broker/config.py` -- update import path
|
|
- `src/wf_mcp/broker/service/core.py` -- update import path (if any)
|
|
|
|
**Tests likely needed:**
|
|
|
|
- All existing `test_stateful_runtime.py` tests must pass unchanged
|
|
- Verify `CrashingSessionFactory` subclass still works
|
|
- Verify `McpRuntimePool` fingerprint logic still works
|
|
|
|
**What must NOT change:**
|
|
|
|
- `PersistentMcpSession` API (connection, auth, call_tool, close)
|
|
- `McpRuntimePool` API (get_session, call_tool, close_connection, close_all)
|
|
- `PersistentSessionFactory.create` signature
|
|
- `connection_runtime_fingerprint` function
|
|
- Any event emission or broker orchestration
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- `wf_mcp.runtime` becomes a re-export shim: `from wf_sources_mcp.runtime import ...`
|
|
- `wf_mcp.runtime.protocols.ToolExecutor` already re-exports from `wf_sources_mcp.sdk`
|
|
- Tests import from `wf_mcp.runtime` still work via shims
|
|
- `broker/config.py` can update to import from `wf_sources_mcp.runtime` directly
|
|
|
|
---
|
|
|
|
### Slice 4: Move `McpSdkAdapter` to `wf_sources_mcp.sdk`
|
|
|
|
**Goal:** Consolidate the one-shot MCP adapter into the source provider SDK.
|
|
|
|
**Files likely touched:**
|
|
|
|
- New: `src/wf_sources_mcp/sdk/adapter.py` -- moved from `wf_mcp/sdk/adapter.py`
|
|
- `src/wf_mcp/sdk/adapter.py` -- becomes re-export shim
|
|
- `src/wf_mcp/sdk/__init__.py` -- update re-exports
|
|
- `src/wf_mcp/broker/config.py` -- update import path
|
|
- `src/wf_mcp/server/core.py` -- update import path
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Existing `test_sdk_adapter.py` tests must pass
|
|
- Verify `McpSdkAdapter` still implements `BackendAdapter`
|
|
|
|
**What must NOT change:**
|
|
|
|
- `McpSdkAdapter` method signatures
|
|
- `BackendAdapter` protocol
|
|
- Any broker service code
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- `wf_mcp.sdk.adapter.McpSdkAdapter` re-exports from `wf_sources_mcp.sdk.adapter`
|
|
- `wf_mcp.sdk.__init__` keeps exporting `McpSdkAdapter`
|
|
|
|
---
|
|
|
|
### Slice 5: Introduce `SourceConnection` dataclass in `wf_sources_mcp` (optional, future)
|
|
|
|
**Goal:** Replace the `metadata: dict[str, Any]` bag in `ConnectionConfig` with a typed transport DTO for source-provider code.
|
|
|
|
**Files likely touched:**
|
|
|
|
- `src/wf_sources_mcp/source_registry.py` -- add `SourceConnection` dataclass
|
|
- `src/wf_mcp/runtime/factory.py` -- accept `SourceConnection` instead of `ConnectionConfig`
|
|
- `src/wf_mcp/sdk/adapter.py` -- accept `SourceConnection` instead of `ConnectionConfig`
|
|
- `src/wf_mcp/runtime/pool.py` -- accept `SourceConnection` in fingerprint
|
|
- Conversion helpers in `wf_sources_mcp.source_registry`
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Conversion test: `ConnectionConfig` -> `SourceConnection`
|
|
- Round-trip test for fingerprint stability
|
|
- Verify `McpRuntimePool` fingerprint changes correctly
|
|
|
|
**What must NOT change:**
|
|
|
|
- `ConnectionConfig` shape (it's still the broker DTO)
|
|
- Broker service behavior
|
|
- Proxy/server behavior
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- `SourceConnection` is a new typed DTO in `wf_sources_mcp`.
|
|
- `ConnectionConfig` gains a `to_source_connection() -> SourceConnection` method or a standalone converter.
|
|
- Runtime code (`factory.py`, `adapter.py`, `pool.py`) accepts `SourceConnection`.
|
|
- `McpRuntimePool` fingerprint uses `SourceConnection` fields.
|
|
- This slice is optional if Slices 0-4 are sufficient.
|
|
|
|
---
|
|
|
|
### Slice 6: Clean up `wf_mcp` re-export shims (final)
|
|
|
|
**Goal:** Remove all compatibility re-export shims from `wf_mcp` once all callers import from canonical packages.
|
|
|
|
**Files likely touched:**
|
|
|
|
- `src/wf_mcp/auth.py` -- delete or leave empty
|
|
- `src/wf_mcp/capabilities.py` -- delete or leave empty
|
|
- `src/wf_mcp/source_registry.py` -- delete or leave empty
|
|
- `src/wf_mcp/runtime/protocols.py` -- delete or leave empty
|
|
- `src/wf_mcp/runtime/__init__.py` -- simplify
|
|
- `src/wf_mcp/models.py` -- simplify
|
|
- `src/wf_mcp/__init__.py` -- simplify
|
|
|
|
**Tests likely needed:**
|
|
|
|
- Verify all existing imports still work (or update them)
|
|
- `test_compat_imports.py` should pass or be updated
|
|
|
|
**What must NOT change:**
|
|
|
|
- Any runtime behavior
|
|
- Any broker service behavior
|
|
|
|
**Migration/shim strategy:**
|
|
|
|
- This is the final cleanup after all other slices are complete.
|
|
- Search for all `from wf_mcp.auth import` etc. and update to canonical paths.
|
|
- Leave shims in place for one release cycle, then remove.
|
|
|
|
---
|
|
|
|
## Should `runtime/factory.py` Move Now?
|
|
|
|
**No.** A typed client/session seam must come first.
|
|
|
|
Reasons:
|
|
|
|
1. `PersistentSessionFactory._create_with_stack` reads `connection.metadata["transport"]`, `connection.metadata["command"]`, etc. These are `dict[str, Any]` bag accesses that should be replaced by typed protocol access.
|
|
2. `PersistentMcpSession` holds `connection: ConnectionConfig` directly. If factory moves to `wf_sources_mcp`, it drags `ConnectionConfig` (and therefore `wf_mcp.broker.models`) into the source provider package at runtime.
|
|
3. The `PersistentSessionFactory.create` method returns `PersistentMcpSession` which stores `connection: ConnectionConfig`. Without the protocol, this creates a circular import.
|
|
|
|
**Correct order:**
|
|
|
|
1. Slice 0: Define `SourceConnection` protocol
|
|
2. Slice 1: Move `parse_connection_id` / `RESERVED_CONNECTION_IDS`
|
|
3. Slice 2: Extract `open_mcp_session` helper
|
|
4. Slice 3: Move runtime code to `wf_sources_mcp`
|
|
5. Slice 4: Move adapter to `wf_sources_mcp.sdk`
|
|
6. Slice 5: (Optional) Typed `SourceConnection` dataclass
|
|
7. Slice 6: Clean up shims
|
|
|
|
---
|
|
|
|
## Test Coverage Summary
|
|
|
|
Existing tests relevant to the extraction:
|
|
|
|
| Test file | What it covers | Extraction impact |
|
|
|-----------|---------------|-------------------|
|
|
| `tests/wf_mcp/test_stateful_runtime.py` | `McpRuntimePool`, `PersistentMcpSession`, `PersistentSessionFactory`, `CrashingSessionFactory` | Must pass unchanged through Slices 0-4 |
|
|
| `tests/wf_mcp/test_sdk_adapter.py` | `McpSdkAdapter` one-shot operations | Must pass unchanged through Slices 0-4 |
|
|
| `tests/wf_mcp/test_compat_imports.py` | Re-export shim compatibility | Must pass through Slice 6 |
|
|
| `tests/wf_mcp/test_workflow_wrappers.py` | `wrap_discovered_tool` with `ToolExecutor` | Must pass unchanged |
|
|
| `tests/wf_mcp/test_store.py` | `FileCatalogStore._connection_path` uses `parse_connection_id` | Must pass after Slice 1 |
|
|
| `tests/wf_mcp/service/test_connection_service.py` | `ConnectionService` | Unaffected |
|
|
| `tests/wf_mcp/service/test_source_registry_admin.py` | Source registry admin tools | Unaffected |
|
|
| `tests/wf_mcp/service/test_events.py` | `BrokerEventRecorder` | Unaffected |
|
|
| `tests/wf_mcp/service/test_workflow_runtime.py` | `WorkflowRuntimeService` | Unaffected |
|
|
|
|
---
|
|
|
|
## Risk Assessment
|
|
|
|
| Risk | Severity | Mitigation |
|
|
|------|----------|------------|
|
|
| Circular import if `wf_sources_mcp` imports `ConnectionConfig` at runtime | High | Use TYPE_CHECKING + protocol pattern (Slice 0) |
|
|
| Breaking re-export shims during move | Medium | Keep shims in place, update canonical imports first |
|
|
| `McpRuntimePool` fingerprint behavioral change | Medium | Preserve exact same fingerprint computation after move |
|
|
| `FileCatalogStore` breaking after `parse_connection_id` move | Low | Move function, keep re-export, run `test_store.py` |
|
|
| FastMCP transitive import in `shared/names.py` | Low | Move `RESERVED_CONNECTION_IDS` first, then clean `shared/names.py` |
|