docs: plan mcp source connection seam

This commit is contained in:
lda
2026-06-07 03:45:51 +07:00 Verified
parent 9960862fc6
commit f79741eb2a
3 changed files with 1628 additions and 4 deletions
+8 -4
View File
@@ -217,10 +217,14 @@ implementation state.
V1 apply reconciles registry state into the current server connection/source
graph; it does not auto-apply mutations, mutate config files, or remount
MCP proxy providers.
- Completed: persisted resume across server rebuild is covered through the
MCP-backed JSON-RPC path. A neutral-config `WorkflowServer` can start an
interrupting run, be rebuilt from the same filesystem stores, inspect the
interrupted run, and resume it to completion through `RpcWorkflowApiClient`.
- Completed: persisted resume across server rebuild is covered through the
MCP-backed JSON-RPC path. A neutral-config `WorkflowServer` can start an
interrupting run, be rebuilt from the same filesystem stores, inspect the
interrupted run, and resume it to completion through `RpcWorkflowApiClient`.
- Planned: MCP upstream source runtime cleanup now starts with a typed
`McpSourceConnection` seam in `wf_sources_mcp`, not by moving
`runtime/factory.py` as-is. The active plan is
[2026-06-07 MCP source connection seam](./superpowers/plans/2026-06-07-mcp-source-connection-seam.md).
- Auth/source secrets boundary: keep registry desired state separate from
upstream credentials, and surface missing auth as validation diagnostics.
The contract is now specified in
@@ -0,0 +1,934 @@
# MCP Source Connection Seam Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Introduce a typed MCP source connection seam in `wf_sources_mcp` so runtime/session code can stop depending on `wf_mcp.broker.models.ConnectionConfig.metadata`.
**Architecture:** This is the preparatory slice before moving MCP runtime/session code. Move low-level source ID validation and MCP transport models into focused `wf_sources_mcp` modules, then add `McpSourceConnection` plus explicit converters from the legacy broker DTO and source registry entries. Do not move `runtime/factory.py`, `runtime/session.py`, `runtime/pool.py`, or `McpSdkAdapter` in this plan.
**Tech Stack:** Python 3.14, dataclasses, Pydantic v2, pytest, ruff, basedpyright.
---
## Why This Slice Exists
Persistent MCP connections are the real product need. Stdio MCP startup and HTTP session initialization are expensive and failure-prone, so runtime code needs one clear object that means:
> one configured upstream MCP source account that can be opened, authenticated, catalogued, called, and eventually reused.
Today that object is implicitly `ConnectionConfig` plus `metadata: dict[str, Any]`. That keeps old code working, but it makes runtime/session extraction unsafe. This slice creates the typed seam first:
```text
ConnectionConfig -> McpSourceConnection -> shared MCP session opener -> persistent runtime
```
After this slice, later runtime moves become mechanical instead of dragging broker config bags into `wf_sources_mcp`.
---
## File Structure
Create:
- `src/wf_sources_mcp/ids.py`
- Owns source/connection ID validation for MCP upstream sources.
- Exports `CONNECTION_ID_PATTERN`, `RESERVED_CONNECTION_IDS`, `parse_connection_id`.
- `src/wf_sources_mcp/transports.py`
- Owns `StdioSourceTransport`, `HttpSourceTransport`, `SourceTransport`.
- Replaces transport model definitions currently embedded in `source_registry.py`.
- `src/wf_sources_mcp/connections.py`
- Owns `McpSourceConnection`.
- Owns conversion helpers from legacy broker `ConnectionConfig` and registry entries.
- `tests/wf_sources_mcp/test_connections.py`
- Tests ID validation, transport parsing, legacy conversion, registry conversion, and package exports.
Modify:
- `src/wf_sources_mcp/source_registry.py`
- Import ID helpers from `wf_sources_mcp.ids`.
- Import transport models from `wf_sources_mcp.transports`.
- Keep existing public exports for compatibility.
- `src/wf_sources_mcp/auth.py`
- Make auth helpers accept `McpSourceConnection` / source-like objects instead of directly depending on `ConnectionConfig`.
- `src/wf_sources_mcp/sdk/protocols.py`
- Introduce a source-connection protocol/type alias for backend protocols.
- Remove the `TYPE_CHECKING` dependency on `wf_mcp.broker.models.ConnectionConfig`.
- `src/wf_sources_mcp/__init__.py`
- Export new seam types lazily if needed to avoid circular imports.
- `src/wf_mcp/connections.py`
- Re-export `CONNECTION_ID_PATTERN` and `parse_connection_id` from `wf_sources_mcp.ids`.
- Keep `ConnectionRegistry` and `qualify_node_name` behavior unchanged.
- `src/wf_mcp/shared/names.py`
- Import/re-export `RESERVED_CONNECTION_IDS` from `wf_sources_mcp.ids`.
- Keep FastMCP namespace helpers unchanged.
- `tests/wf_mcp/test_compat_imports.py`
- Add identity checks for moved ID/transport exports where appropriate.
- `docs/current_roadmap.md`
- Add a status note for the typed MCP source connection seam.
- `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
- Add this slice to the MCP source provider package direction list.
---
## Non-Goals
- Do not move `src/wf_mcp/runtime/factory.py`.
- Do not move `src/wf_mcp/runtime/session.py`.
- Do not move `src/wf_mcp/runtime/pool.py`.
- Do not move `src/wf_mcp/sdk/adapter.py`.
- Do not change `ConnectionConfig` field shape.
- Do not change on-disk registry/auth/catalog JSON shapes.
- Do not change MCP proxy/frontend behavior.
---
## Task 1: Move ID Validation Into `wf_sources_mcp.ids`
**Files:**
- Create: `src/wf_sources_mcp/ids.py`
- Modify: `src/wf_mcp/connections.py`
- Modify: `src/wf_mcp/shared/names.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Write failing ID tests**
Create `tests/wf_sources_mcp/test_connections.py` with these tests first:
```python
import pytest
from wf_sources_mcp.ids import (
CONNECTION_ID_PATTERN,
RESERVED_CONNECTION_IDS,
parse_connection_id,
)
def test_parse_connection_id_splits_provider_and_account() -> None:
assert parse_connection_id("github.work") == ("github", "work")
@pytest.mark.parametrize(
"source_id",
["github", ".github.work", "github.", "github/work", "github work"],
)
def test_parse_connection_id_rejects_unsafe_or_unqualified_ids(source_id: str) -> None:
with pytest.raises(ValueError):
parse_connection_id(source_id)
def test_reserved_connection_ids_are_source_provider_constants() -> None:
assert "wf.admin" in RESERVED_CONNECTION_IDS
assert "wf.mcp" in RESERVED_CONNECTION_IDS
assert CONNECTION_ID_PATTERN.startswith("^")
```
- [ ] **Step 2: Run the failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.ids` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.ids`**
Create `src/wf_sources_mcp/ids.py`:
```python
from __future__ import annotations
import re
CONNECTION_ID_PATTERN = r"^[A-Za-z0-9_][A-Za-z0-9_.-]*$"
RESERVED_CONNECTION_IDS = frozenset({"wf.admin", "wf.mcp"})
"""Source ids reserved by built-in workflow/MCP control surfaces."""
def parse_connection_id(connection_id: str) -> tuple[str, str]:
"""Validate and split one MCP source id into provider/account parts.
Source ids also key persisted auth, registry, and catalog files. Keep this
conservative so unsafe ids are rejected before reaching store boundaries.
"""
if not re.fullmatch(CONNECTION_ID_PATTERN, connection_id):
raise ValueError(
"connection id must start with alphanumeric or underscore and contain "
"only [A-Za-z0-9_.-]"
)
if "." not in connection_id:
raise ValueError("connection id must look like '<server>.<account>'")
server, account = connection_id.split(".", 1)
if not server or not account:
raise ValueError("connection id must look like '<server>.<account>'")
return server, account
__all__ = [
"CONNECTION_ID_PATTERN",
"RESERVED_CONNECTION_IDS",
"parse_connection_id",
]
```
- [ ] **Step 4: Update compatibility imports**
In `src/wf_mcp/connections.py`, remove local `re` and `CONNECTION_ID_PATTERN` / `parse_connection_id` definitions. Import instead:
```python
from wf_sources_mcp.ids import CONNECTION_ID_PATTERN, parse_connection_id
```
Keep `qualify_node_name` and `ConnectionRegistry` in `wf_mcp.connections`.
In `src/wf_mcp/shared/names.py`, replace local `RESERVED_CONNECTION_IDS` with:
```python
from wf_sources_mcp.ids import RESERVED_CONNECTION_IDS
```
Keep `ADMIN_NAMESPACE = "wf.admin"` for MCP frontend namespace logic.
- [ ] **Step 5: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_mcp/test_source_registry.py tests/wf_mcp/test_store.py -q
```
Expected: pass.
---
## Task 2: Move Transport Models Into `wf_sources_mcp.transports`
**Files:**
- Create: `src/wf_sources_mcp/transports.py`
- Modify: `src/wf_sources_mcp/source_registry.py`
- Modify: `src/wf_sources_mcp/__init__.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add failing transport tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from pydantic import TypeAdapter
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
def test_stdio_source_transport_is_typed() -> None:
transport = StdioSourceTransport(
command="uvx",
args=("mcp-server",),
env={"TOKEN": "x"},
)
assert transport.kind == "stdio"
assert transport.command == "uvx"
assert transport.args == ("mcp-server",)
assert transport.env == {"TOKEN": "x"}
def test_http_source_transport_is_typed() -> None:
transport = HttpSourceTransport(url="http://127.0.0.1:8000/mcp")
assert transport.kind == "http"
assert str(transport.url) == "http://127.0.0.1:8000/mcp"
def test_source_transport_discriminated_union_parses() -> None:
adapter = TypeAdapter(SourceTransport)
transport = adapter.validate_python(
{"kind": "stdio", "command": "pnpx", "args": ["-y", "server"]}
)
assert isinstance(transport, StdioSourceTransport)
assert transport.args == ("-y", "server")
```
- [ ] **Step 2: Run failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.transports` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.transports`**
Create `src/wf_sources_mcp/transports.py`:
```python
from __future__ import annotations
from typing import Annotated, Literal
from pydantic import AnyHttpUrl, Field
from wf_api.source_registry import SourceRegistryBaseModel
class StdioSourceTransport(SourceRegistryBaseModel):
kind: Literal["stdio"] = "stdio"
command: str = Field(min_length=1)
args: tuple[str, ...] = ()
env: dict[str, str] = Field(default_factory=dict)
class HttpSourceTransport(SourceRegistryBaseModel):
kind: Literal["http"] = "http"
url: AnyHttpUrl
headers: dict[str, str] = Field(default_factory=dict)
SourceTransport = Annotated[
StdioSourceTransport | HttpSourceTransport,
Field(discriminator="kind"),
]
__all__ = [
"HttpSourceTransport",
"SourceTransport",
"StdioSourceTransport",
]
```
- [ ] **Step 4: Update `source_registry.py` to use canonical transport models**
In `src/wf_sources_mcp/source_registry.py`:
- Remove local `StdioSourceTransport`, `HttpSourceTransport`, and `SourceTransport` definitions.
- Remove now-unused imports `Annotated`, `AnyHttpUrl`.
- Import:
```python
from wf_sources_mcp.ids import RESERVED_CONNECTION_IDS, parse_connection_id
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
```
Keep all three names in `__all__` so existing imports from `wf_sources_mcp.source_registry` continue to work.
- [ ] **Step 5: Update package exports**
In `src/wf_sources_mcp/__init__.py`, export or lazily expose:
```python
HttpSourceTransport
SourceTransport
StdioSourceTransport
```
If direct imports create a circular dependency, use the existing lazy `__getattr__` pattern.
- [ ] **Step 6: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_source_registry.py tests/wf_mcp/test_source_registry.py -q
```
Expected: pass.
---
## Task 3: Add `McpSourceConnection` And Converters
**Files:**
- Create: `src/wf_sources_mcp/connections.py`
- Modify: `src/wf_sources_mcp/__init__.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add failing connection seam tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from wf_sources_mcp.connections import (
McpSourceConnection,
mcp_source_connection_from_connection_config,
mcp_source_connection_from_registry_entry,
)
from wf_sources_mcp.source_registry import McpSourceRegistryEntry
def test_mcp_source_connection_from_registry_entry() -> None:
entry = McpSourceRegistryEntry.model_validate(
{
"id": "github.work",
"provider": "github",
"account": "work",
"profile": "engineering",
"transport": {
"kind": "stdio",
"command": "uvx",
"args": ["github-mcp"],
"env": {"A": "B"},
},
"auth_ref": "github.token",
"metadata": {"team": "platform"},
}
)
connection = mcp_source_connection_from_registry_entry(entry)
assert connection == McpSourceConnection(
id="github.work",
provider="github",
account="work",
enabled=True,
profile="engineering",
transport=StdioSourceTransport(
command="uvx",
args=("github-mcp",),
env={"A": "B"},
),
auth_ref="github.token",
metadata={"team": "platform"},
)
def test_mcp_source_connection_from_legacy_connection_config_stdio() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
enabled=False,
metadata={
"transport": "stdio",
"command": "uvx",
"args": ["github-mcp"],
"env": {"A": "B"},
"auth_ref": "github.token",
"profile": "engineering",
"source_registry": True,
"team": "platform",
},
)
connection = mcp_source_connection_from_connection_config(legacy)
assert connection.id == "github.work"
assert connection.provider == "github"
assert connection.account == "work"
assert connection.enabled is False
assert connection.profile == "engineering"
assert connection.auth_ref == "github.token"
assert connection.metadata == {"source_registry": True, "team": "platform"}
assert isinstance(connection.transport, StdioSourceTransport)
assert connection.transport.command == "uvx"
assert connection.transport.args == ("github-mcp",)
def test_mcp_source_connection_from_legacy_connection_config_http() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
metadata={
"transport": "streamable_http",
"url": "http://127.0.0.1:8000/mcp",
"headers": {"X-Test": "yes"},
},
)
connection = mcp_source_connection_from_connection_config(legacy)
assert isinstance(connection.transport, HttpSourceTransport)
assert str(connection.transport.url) == "http://127.0.0.1:8000/mcp"
assert connection.transport.headers == {"X-Test": "yes"}
def test_mcp_source_connection_rejects_missing_legacy_transport() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
metadata={},
)
with pytest.raises(ValueError, match="requires metadata.transport"):
mcp_source_connection_from_connection_config(legacy)
```
- [ ] **Step 2: Run failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.connections` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.connections`**
Create `src/wf_sources_mcp/connections.py`:
```python
from __future__ import annotations
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
from wf_sources_mcp.ids import parse_connection_id
from wf_sources_mcp.source_registry import McpSourceRegistryEntry
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
if TYPE_CHECKING:
from wf_mcp.broker.models import ConnectionConfig
_FLAT_HTTP_TRANSPORTS = {"http", "streamable-http", "streamable_http", "sse"}
_CONNECTION_METADATA_KEYS = {
"transport",
"command",
"args",
"env",
"cwd",
"url",
"headers",
"profile",
"auth_ref",
}
@dataclass(frozen=True, slots=True)
class McpSourceConnection:
"""Typed runtime-facing MCP source connection.
This is the object runtime/session code should consume. Legacy broker
`ConnectionConfig.metadata` remains at the compatibility edge only.
"""
id: str
provider: str
account: str
transport: SourceTransport
enabled: bool = True
profile: str | None = None
auth_ref: str | None = None
metadata: dict[str, object] = field(default_factory=dict)
def __post_init__(self) -> None:
provider, account = parse_connection_id(self.id)
if not self.provider:
raise ValueError("provider must not be empty")
if not self.account:
raise ValueError("account must not be empty")
if provider != self.provider or account != self.account:
raise ValueError(
"MCP source connection id must match provider/account fields"
)
def mcp_source_connection_from_registry_entry(
entry: McpSourceRegistryEntry,
) -> McpSourceConnection:
"""Adapt persisted desired-source registry state to runtime source shape."""
return McpSourceConnection(
id=entry.id,
provider=entry.provider,
account=entry.account,
enabled=entry.enabled,
profile=entry.profile,
transport=entry.transport,
auth_ref=entry.auth_ref,
metadata=dict(entry.metadata),
)
def mcp_source_connection_from_connection_config(
connection: ConnectionConfig,
) -> McpSourceConnection:
"""Adapt legacy broker connection config into typed source shape.
Keep all metadata-bag reads in this compatibility converter. Runtime/session
code should use `McpSourceConnection.transport` directly.
"""
transport = _transport_from_connection_metadata(connection)
profile = connection.metadata.get("profile")
auth_ref = connection.metadata.get("auth_ref")
metadata = {
str(key): value
for key, value in connection.metadata.items()
if key not in _CONNECTION_METADATA_KEYS
}
return McpSourceConnection(
id=connection.id,
provider=connection.server,
account=connection.account,
enabled=connection.enabled,
profile=profile if isinstance(profile, str) else None,
transport=transport,
auth_ref=auth_ref if isinstance(auth_ref, str) else None,
metadata=metadata,
)
def _transport_from_connection_metadata(connection: ConnectionConfig) -> SourceTransport:
transport = connection.metadata.get("transport")
if isinstance(transport, dict):
kind = transport.get("kind")
if kind == "stdio":
return StdioSourceTransport.model_validate(transport)
if kind == "http":
return HttpSourceTransport.model_validate(transport)
raise ValueError(
f"connection {connection.id!r} has unsupported metadata.transport.kind {kind!r}"
)
if isinstance(transport, str):
if transport == "stdio":
return StdioSourceTransport(
command=str(connection.metadata.get("command", "")),
args=tuple(str(arg) for arg in connection.metadata.get("args", ())),
env={
str(key): str(value)
for key, value in dict(connection.metadata.get("env", {})).items()
},
)
if transport in _FLAT_HTTP_TRANSPORTS:
return HttpSourceTransport(
url=str(connection.metadata.get("url", "")),
headers={
str(key): str(value)
for key, value in dict(
connection.metadata.get("headers", {})
).items()
},
)
raise ValueError(
f"connection {connection.id!r} has unrecognized metadata.transport {transport!r}"
)
raise ValueError(f"connection {connection.id!r} requires metadata.transport")
__all__ = [
"McpSourceConnection",
"mcp_source_connection_from_connection_config",
"mcp_source_connection_from_registry_entry",
]
```
- [ ] **Step 4: Export the new seam from the package**
In `src/wf_sources_mcp/__init__.py`, export or lazily expose:
```python
McpSourceConnection
mcp_source_connection_from_connection_config
mcp_source_connection_from_registry_entry
```
- [ ] **Step 5: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_source_registry.py -q
```
Expected: pass.
---
## Task 4: Update Auth And SDK Protocols To Use The Source Seam
**Files:**
- Modify: `src/wf_sources_mcp/auth.py`
- Modify: `src/wf_sources_mcp/sdk/protocols.py`
- Test: `tests/wf_sources_mcp/test_auth.py`
- Test: `tests/wf_sources_mcp/test_sdk_protocols.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add protocol/conformance tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from typing import Protocol
from wf_sources_mcp.auth import auth_ref_for_connection
from wf_sources_mcp.sdk import BackendAdapter, ToolExecutor
class _ConnectionLike(Protocol):
id: str
auth_ref: str | None
def test_auth_ref_for_typed_mcp_source_connection() -> None:
connection = McpSourceConnection(
id="github.work",
provider="github",
account="work",
transport=StdioSourceTransport(command="uvx"),
auth_ref="github.token",
)
assert auth_ref_for_connection(connection) == "github.token"
def test_sdk_protocols_are_importable_without_broker_connection_config() -> None:
assert BackendAdapter is not None
assert ToolExecutor is not None
```
- [ ] **Step 2: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_auth.py tests/wf_sources_mcp/test_sdk_protocols.py -q
```
Expected: likely fail until auth/protocols stop importing `ConnectionConfig`.
- [ ] **Step 3: Update `auth.py`**
In `src/wf_sources_mcp/auth.py`:
- Remove the `TYPE_CHECKING` import of `wf_mcp.broker.models.ConnectionConfig`.
- Define a small protocol:
```python
class SourceConnectionLike(Protocol):
id: str
auth_ref: str | None
```
- Change:
```python
def auth_ref_for_connection(connection: ConnectionConfig) -> str | None:
auth_ref = connection.metadata.get("auth_ref")
return auth_ref if isinstance(auth_ref, str) else None
```
to:
```python
def auth_ref_for_connection(connection: SourceConnectionLike) -> str | None:
return connection.auth_ref
```
- Change `connection_auth_diagnostic(connection: ConnectionConfig, ...)` to accept `SourceConnectionLike`.
Important: this intentionally means callers that still hold `ConnectionConfig` must convert to `McpSourceConnection` before using auth diagnostics. If current production callers need a compatibility path, use `mcp_source_connection_from_connection_config(connection)` at that call site instead of reintroducing metadata reads in `auth.py`.
- [ ] **Step 4: Update `sdk/protocols.py`**
In `src/wf_sources_mcp/sdk/protocols.py`:
- Remove `TYPE_CHECKING` and `ConnectionConfig`.
- Import:
```python
from wf_sources_mcp.connections import McpSourceConnection
```
- Change all `connection: ConnectionConfig` parameters in `BackendAdapter` and `ToolExecutor` to:
```python
connection: McpSourceConnection
```
This is a type-only protocol change. Production adapters may need a follow-up slice to convert broker DTOs before calling the protocol.
- [ ] **Step 5: Run focused type/tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_auth.py tests/wf_sources_mcp/test_sdk_protocols.py -q
uv run basedpyright --level error src/wf_sources_mcp
```
Expected: pass or reveal call sites that still need explicit conversion. Fix call sites by converting at the broker boundary, not by weakening `McpSourceConnection` back into `metadata`.
---
## Task 5: Update Current Broker Call Sites At The Boundary
**Files:**
- Modify: `src/wf_mcp/broker/service/upstream_transport.py`
- Modify: `src/wf_mcp/broker/service/source_catalog.py`
- Modify: `src/wf_mcp/runtime/factory.py`
- Modify: `src/wf_mcp/runtime/session.py`
- Modify: `src/wf_mcp/runtime/pool.py`
- Modify: `src/wf_mcp/sdk/adapter.py`
- Test: existing focused tests
- [ ] **Step 1: Find all protocol call sites**
Run:
```bash
rg -n 'BackendAdapter|ToolExecutor|call_tool\\(|list_tools\\(|list_resources\\(|list_prompts\\(|read_resource\\(|get_prompt\\(|invoke_method\\(|send_notification\\(' src\\wf_mcp src\\wf_sources_mcp
```
Inspect call sites that pass a `ConnectionConfig` into a `BackendAdapter` or `ToolExecutor`.
- [ ] **Step 2: Convert at the broker edge**
Where broker services call source-provider protocols, convert with:
```python
from wf_sources_mcp.connections import mcp_source_connection_from_connection_config
source_connection = mcp_source_connection_from_connection_config(connection)
```
Then pass `source_connection` to `BackendAdapter` / `ToolExecutor` methods.
Keep `ConnectionConfig` in broker services for registry/config ownership and source catalog behavior. Do not rewrite the whole broker service in this slice.
- [ ] **Step 3: Keep runtime files compiling without moving them**
If `PersistentMcpSession`, `McpRuntimePool`, or `McpSdkAdapter` currently implement `ToolExecutor` / `BackendAdapter`, update their method signatures to accept `McpSourceConnection` where needed.
If their public callers still pass `ConnectionConfig`, convert at the public method boundary and keep a comment:
```python
# Compatibility boundary: broker callers still pass ConnectionConfig. Runtime
# internals use McpSourceConnection so the session code can move to
# wf_sources_mcp in a later slice.
```
- [ ] **Step 4: Run focused tests**
Run:
```bash
uv run pytest tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/service/test_upstream_transport.py tests/wf_mcp/service/test_source_registry_admin.py tests/wf_sources_mcp -q
uv run basedpyright --level error src
```
Expected: pass.
---
## Task 6: Documentation And Verification
**Files:**
- Modify: `docs/current_roadmap.md`
- Modify: `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
- [ ] **Step 1: Update roadmap**
In `docs/current_roadmap.md`, under the MCP source provider / long-lived API section, add a short note:
```markdown
- Completed: typed MCP source connection seam is introduced in `wf_sources_mcp`.
Source IDs, reserved IDs, transport models, and `McpSourceConnection` are now
canonical source-provider concepts. Legacy broker `ConnectionConfig` remains
intact and converts at compatibility edges; runtime/session files are not
moved yet.
```
- [ ] **Step 2: Update long-lived boundary spec**
In `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`, extend the MCP source provider package direction list:
```markdown
6. Complete: typed MCP source connection seam introduced. `McpSourceConnection`
is the runtime-facing source object; legacy `ConnectionConfig` converts at
broker edges.
7. Next: shared MCP session opener, then runtime/session/pool move.
```
- [ ] **Step 3: Run final verification**
Run:
```bash
uv run pytest tests/wf_sources_mcp tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/service/test_upstream_transport.py -q
uv run ruff check src tests
uv run basedpyright --level error src
git diff --check
```
Expected:
- tests pass
- ruff passes
- basedpyright has 0 errors
- no diff whitespace errors
- [ ] **Step 4: Report remaining future slices**
Final report must explicitly state:
- runtime/factory/session/pool were not moved
- proxy/frontend MCP code was not touched
- old `ConnectionConfig` shape is unchanged
- next planned slice is shared session opener using `McpSourceConnection`
---
## Future Slices After This Plan
1. **Shared MCP session opener**
- Create `wf_sources_mcp.sdk.transport.open_mcp_session(connection, auth)`.
- Replace duplicated session opening in `wf_mcp/runtime/factory.py` and `wf_mcp/sdk/adapter.py`.
2. **Move persistent runtime package**
- Move `PersistentSessionFactory`, `PersistentMcpSession`, and `McpRuntimePool` to `wf_sources_mcp.runtime`.
- Keep `wf_mcp.runtime.*` shims.
3. **Move one-shot SDK adapter**
- Move `McpSdkAdapter` to `wf_sources_mcp.sdk.adapter`.
- Keep `wf_mcp.sdk.adapter` shim.
4. **Eventually split MCP frontend transport**
- Move FastMCP server/proxy/admin/workflow tool registration toward `wf_transport_mcp`.
- This is separate from upstream source runtime and should not block persistent connection work.
---
## Self-Review Notes
- This plan intentionally creates focused files before touching runtime. That keeps the first change testable and limits blast radius.
- The plan does not attempt to phase out all 369 `ConnectionConfig` references. It converts only at source-provider protocol edges.
- The likely implementation risk is type fallout after `BackendAdapter` / `ToolExecutor` signatures change. Resolve by explicit conversion at broker boundaries, not by weakening the typed seam.
@@ -0,0 +1,686 @@
# WF MCP Runtime & Source Provider Map
Date: 2026-06-07
## Executive Summary
`wf_mcp` is a monolith containing four distinct responsibilities that should become separate packages:
1. **MCP source provider runtime** (transport opening, session pooling, auth) -- belongs in `wf_sources_mcp`
2. **MCP frontend transport** (FastMCP server, proxy mounts, MCP tool registration) -- belongs in a future `wf_transport_mcp`
3. **Broker service layer** (connection catalog, discovery, source catalog, workflow runtime) -- stays in `wf_mcp` or becomes `wf_server`
4. **Compatibility shims** (re-export modules, `WfMcpService` facade) -- remains in `wf_mcp` until callers migrate
The critical blocker is `ConnectionConfig` (defined at `src/wf_mcp/broker/models.py:37`). This dataclass is consumed by `wf_sources_mcp.sdk.protocols`, `wf_sources_mcp.auth`, `wf_sources_mcp.source_registry`, `wf_sources_mcp.storage.store`, the runtime factory, the adapter, the pool, and every broker service. Until `ConnectionConfig` moves to a neutral package or is replaced by a typed protocol, extraction is blocked.
The next safe slice is **Slice 0: Define a `SourceConnection` protocol in `wf_sources_mcp`** so runtime/session code stops depending on the broker DTO. After that, the transport-opening logic in `runtime/factory.py` and `sdk/adapter.py` can merge into `wf_sources_mcp.sdk` without dragging the broker layer along.
---
## Package Responsibility Map
### `wf_sources_mcp` (source provider core)
Owns: MCP source provider identity, auth, catalog entries, storage, and the SDK adapter/executor protocols.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `auth.py` | MCP auth record, env/header extraction, diagnostic helpers | Canonical. Has TYPE_CHECKING dep on `wf_mcp.broker.models.ConnectionConfig`. |
| `sdk/protocols.py` | `BackendAdapter`, `ToolExecutor`, `ToolCallResult` | Canonical. Has TYPE_CHECKING dep on `ConnectionConfig`. |
| `sdk/converters.py` | MCP tool/resource/prompt conversion | Canonical. No `wf_mcp` deps. |
| `catalog/models.py` | `CatalogSnapshot`, `dump_catalog_snapshot` | Canonical. No `wf_mcp` deps. |
| `catalog/entries.py` | `CatalogNodeEntry`, `CatalogResourceEntry`, `CatalogPromptEntry`, `DiscoveredTool`, etc. | Canonical. No `wf_mcp` deps. |
| `source_registry.py` | `McpSourceRegistryEntry`, `SourceRegistryFile`, conversion helpers | Canonical. Imports `parse_connection_id` from `wf_mcp.connections` and `RESERVED_CONNECTION_IDS` from `wf_mcp.shared.names`. TYPE_CHECKING dep on `ConnectionConfig`. |
| `storage/store.py` | `AuthStore`, `CatalogStore`, `FileAuthStore`, `FileCatalogStore` | Canonical. `FileCatalogStore._connection_path` imports `parse_connection_id` from `wf_mcp.connections` at runtime. |
### Future `wf_transport_mcp` (MCP frontend transport)
Owns: FastMCP server creation, proxy mounting, MCP tool/resource/prompt registration, admin tool registration, workflow surface tool registration.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `proxy/runtime.py` | `ProxyRuntime` -- mount upstream MCP connections into FastMCP | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, `EventBus`, `BrokerConfigManager`. |
| `proxy/mounts.py` | `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy` | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, FastMCP. |
| `proxy/tools.py` | Proxy tool listing/filtering | `wf_mcp` internal. |
| `proxy/safe_names.py` | `SafeToolNames` transform | `wf_mcp` internal. FastMCP transform. |
| `proxy/admin.py` | Proxy admin tools | `wf_mcp` internal. |
| `server/core.py` | `create_server`, `run_server`, `create_server_client` | `wf_mcp` entrypoint. Wires broker service + proxy + workflow surface + admin surface. |
| `admin_surface/tools.py` | `register_service_admin_tools` | `wf_mcp` internal. |
| `admin_surface/handlers/*.py` | Admin tool handlers | `wf_mcp` internal. |
| `workflow_surface/tools.py` | `register_workflow_tools` | `wf_mcp` internal. |
| `workflow_surface/*.py` | Workflow tool models, handlers, lifecycle | `wf_mcp` internal. |
| `shared/names.py` | `ProxyNamespace`, `ADMIN_NAMESPACE`, `RESERVED_CONNECTION_IDS`, namespace helpers | `wf_mcp` internal. FastMCP-specific. |
| `cli.py` | CLI entrypoint | `wf_mcp` entrypoint. |
### `wf_server` / `wf_api` (server core)
Owns: Workflow API, operation context, artifact management, deployment, run lifecycle.
Already extracted. These packages consume `wf_sources_mcp` types and `wf_mcp.broker.service.WfMcpService` through `wf_api.WorkflowApi` and `wf_api.WorkflowRuntimeAdapter`.
### Legacy `wf_mcp` (broker + compatibility shims)
Owns: Connection registry, broker config, broker service coordination, events, and all compatibility re-export shims.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `broker/models.py` | `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots`, `SourceConfigOwnership` | **The core blocker.** All other packages depend on this. |
| `broker/service/core.py` | `WfMcpService` -- compatibility coordinator | Facade. Delegates to focused services. |
| `broker/service/connection_service.py` | `ConnectionService` | Focused service. Depends on `ConnectionConfig`, `SourceCatalogService`. |
| `broker/service/source_catalog.py` | `SourceCatalogService` | Focused service. Depends on `ConnectionConfig`, `McpEvent`, `NodeSpec`. |
| `broker/service/upstream_transport.py` | `UpstreamTransportService` | Focused service. Depends on `ConnectionConfig`, `BackendAdapter`, `ToolExecutor`. |
| `broker/service/workflow_runtime.py` | `WorkflowRuntimeService` | Focused service. Depends on `SourceCatalogService`. |
| `broker/service/content_access.py` | `ContentAccessService` | Focused service. |
| `broker/service/events.py` | `BrokerEventRecorder` | Focused service. Depends on `McpEvent`. |
| `broker/service/adapters.py` | `require_adapter` | Helper. |
| `broker/discovery.py` | `discover_connection_capabilities`, `specs_from_discovered_tools` | Broker logic. |
| `broker/catalog.py` | `snapshot_from_specs`, `CombinedCatalog` | Broker catalog projection. |
| `broker/config.py` | `build_service_from_config`, `load_broker_config`, `broker_config_from_workflow_config` | Config construction. |
| `connections.py` | `ConnectionRegistry`, `parse_connection_id`, `qualify_node_name` | Broker connection registry. |
| `auth.py` | Re-export shim from `wf_sources_mcp.auth` | Compatibility. |
| `models.py` | Re-export shim aggregating broker models | Compatibility. |
| `source_registry.py` | Re-export shim from `wf_sources_mcp.source_registry` | Compatibility. |
| `capabilities.py` | Re-export shim from `wf_sources_mcp.catalog.entries` | Compatibility. |
| `runtime/protocols.py` | Re-export shim from `wf_sources_mcp.sdk` | Compatibility. |
| `events/bus.py` | `EventBus`, `InMemoryEventSink` | Broker-local event fanout. |
| `events/models.py` | `McpEvent`, `make_event` | Broker event model. |
---
## Current Dependency Blockers
### Blocker 1: `ConnectionConfig` origin
**File:** `src/wf_mcp/broker/models.py:37-44`
```python
@dataclass(slots=True)
class ConnectionConfig:
id: str
server: str
account: str
enabled: bool = True
metadata: dict[str, Any] = field(default_factory=dict)
source_config_ownership: SourceConfigOwnership = "locked"
```
This DTO is imported by:
| Consumer | Import path | Import type |
|----------|------------|-------------|
| `wf_sources_mcp.auth` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.sdk.protocols` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.source_registry` | `wf_mcp.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.storage.store` | (none directly, but `FileCatalogStore._connection_path` calls `parse_connection_id` from `wf_mcp.connections`) | runtime |
| `wf_mcp.runtime.factory` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.sdk.adapter` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.runtime.pool` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.runtime.session` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.workflow.wrappers` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.broker.discovery` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.broker.service.*` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.proxy.mounts` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.proxy.runtime` | (indirectly via `BrokerConfig`) | runtime |
**Impact:** Until `ConnectionConfig` moves to a neutral package (or `wf_sources_mcp` defines its own protocol), the runtime code in `wf_sources_mcp` cannot be independent of `wf_mcp`.
### Blocker 2: `parse_connection_id` origin
**File:** `src/wf_mcp/connections.py:11-25`
Used by:
- `wf_sources_mcp.source_registry` (runtime import, line 32)
- `wf_sources_mcp.storage.store` (runtime import inside `_connection_path`, line 132)
- `wf_mcp.connections` (canonical home)
**Impact:** `wf_sources_mcp` has a runtime import dependency on `wf_mcp` for connection ID validation. This function should move to a neutral package or `wf_sources_mcp`.
### Blocker 3: `RESERVED_CONNECTION_IDS` origin
**File:** `src/wf_mcp/shared/names.py:25`
```python
RESERVED_CONNECTION_IDS = frozenset({ADMIN_NAMESPACE, "wf.mcp"})
```
Used by:
- `wf_sources_mcp.source_registry` (runtime import, line 33)
- `wf_mcp.broker.service.connection_service` (runtime import, line 14)
**Impact:** `wf_sources_mcp` imports from `wf_mcp.shared.names` which transitively imports FastMCP transforms (line 8-15 of `shared/names.py`). This creates an unwanted dependency chain.
### Blocker 4: `McpSdkAdapter` transport-opening duplication
**Files:**
- `src/wf_mcp/runtime/factory.py:43-90` (`_create_with_stack`)
- `src/wf_mcp/sdk/adapter.py:34-75` (`_session`)
Both methods contain nearly identical logic for:
- Reading `connection.metadata["transport"]` to select stdio vs streamable HTTP
- Creating `StdioServerParameters` and calling `stdio_client`
- Creating `httpx.AsyncClient` and calling `streamable_http_client`
- Creating `ClientSession` and calling `session.initialize()`
- Applying auth via `mcp_auth_env` / `mcp_auth_headers`
The difference: factory.py owns the session long-term (persistent actor pattern), while adapter.py opens/closes per call (one-shot pattern).
**Impact:** This duplication means both files must be updated together when transport handling changes. They should share a transport-opening helper.
### Blocker 5: `BrokerConfig` in proxy/runtime
**File:** `src/wf_mcp/proxy/runtime.py:73-85`
`ProxyRuntime.__init__` takes `BrokerConfig` directly and reads `config.connections`. The proxy layer is a frontend transport concern that should not depend on the broker config DTO.
**Impact:** Proxy runtime cannot move to `wf_transport_mcp` until it consumes a transport-neutral config shape.
### Blocker 6: `McpEvent` coupling
**File:** `src/wf_mcp/events/models.py`
`McpEvent` is used by:
- `UpstreamTransportService` (event_sink)
- `SourceCatalogService` (emit_event)
- `BrokerEventRecorder` (event_bus)
- `broker/catalog.py` (indirectly through event callbacks)
- `workflow/wrappers.py` (emit_event callback)
**Impact:** `McpEvent` is broker-specific. If source provider code needs to emit events, it should use a protocol or a neutral event type. Currently `wf_sources_mcp` does not import `McpEvent` directly, which is good.
---
## Detailed Findings by Worker Scope
### Worker 1: Runtime/Session Findings
#### Where is MCP session opening duplicated?
The transport-opening logic appears in two places:
1. **`runtime/factory.py:43-90`** (`PersistentSessionFactory._create_with_stack`):
- Opens stdio or streamable HTTP transport
- Creates `ClientSession`
- Calls `session.initialize()`
- Returns session owned by `AsyncExitStack`
- Used for persistent long-lived sessions (actor pattern)
2. **`sdk/adapter.py:34-75`** (`McpSdkAdapter._session`):
- Opens stdio or streamable HTTP transport
- Creates `ClientSession`
- Calls `session.initialize()`
- Yields session in async context manager
- Used for one-shot per-call sessions (discovery, admin operations)
Both import from the same MCP SDK modules:
- `mcp.client.stdio.StdioServerParameters`, `stdio_client`
- `mcp.client.streamable_http.streamable_http_client`
- `mcp.client.session.ClientSession`
- `wf_sources_mcp.auth.mcp_auth_env`, `mcp_auth_headers`
#### Which client operations are supported by one-shot adapter but not persistent runtime?
The `BackendAdapter` protocol (`wf_sources_mcp.sdk.protocols:26-88`) defines:
- `list_tools`, `list_resources`, `list_prompts`
- `get_connection_metadata`
- `read_resource`, `get_prompt`
- `invoke_method`, `send_notification`
- `call_tool`
The `PersistentMcpSession` (`runtime/session.py:19-49`) only exposes:
- `call_tool`
- `close`
The `McpRuntimePool` (`runtime/pool.py:43-96`) only exposes:
- `get_session` (returns `PersistentMcpSession`)
- `call_tool`
- `close_connection`, `close_all`
**Missing from persistent runtime:** `list_tools`, `list_resources`, `list_prompts`, `get_connection_metadata`, `read_resource`, `get_prompt`, `invoke_method`, `send_notification`.
This is intentional -- persistent sessions exist for workflow execution (tool calls only). Discovery and admin operations use one-shot adapters. However, this means the persistent runtime cannot replace the adapter for all operations.
#### What common `McpClientSession` / `McpClientSessionFactory` interface should exist?
Both factory.py and adapter.py share:
1. Transport selection logic (stdio vs streamable HTTP)
2. Auth application (env vars for stdio, headers for HTTP)
3. Session creation and initialization
A shared `open_mcp_session` helper should:
- Accept a connection descriptor (transport type, command/url, env/headers) and optional auth
- Return an initialized `ClientSession` (or yield it)
- Be used by both `PersistentSessionFactory._create_with_stack` and `McpSdkAdapter._session`
The connection descriptor should NOT be `ConnectionConfig` directly -- it should be a transport-specific DTO that `ConnectionConfig` can convert to.
#### What blocks moving this code to `wf_sources_mcp`?
1. `ConnectionConfig` dependency (Blocker 1)
2. `McpEvent` event callbacks in `PersistentSessionFactory` (the `_SessionOwner._run` method does not emit events, but the pool and factory are used by `broker/config.py` which wires events)
3. The `PersistentMcpSession` dataclass holds `connection: ConnectionConfig` and `auth: AuthRecord` -- the `AuthRecord` is already in `wf_sources_mcp`, but `ConnectionConfig` is not
**Recommendation:** Define a `SourceConnection` protocol in `wf_sources_mcp.sdk.protocols` that captures the transport fields `ConnectionConfig` exposes to runtime code. The factory/pool/session code should consume this protocol, not the concrete DTO.
### Worker 2: Broker/Upstream Findings
#### What is truly MCP-source-provider logic vs broker/catalog projection logic?
**MCP-source-provider logic** (belongs in `wf_sources_mcp`):
- Transport opening (stdio, streamable HTTP) -- currently in `factory.py` and `adapter.py`
- Auth application (env vars, headers) -- already in `wf_sources_mcp.auth`
- `BackendAdapter` protocol and `ToolExecutor` protocol -- already in `wf_sources_mcp.sdk`
- `ToolCallResult` dataclass -- already in `wf_sources_mcp.sdk`
- `DiscoveredTool`, `DiscoveredResource`, `DiscoveredPrompt` -- already in `wf_sources_mcp.catalog`
- `CatalogSnapshot` and catalog entries -- already in `wf_sources_mcp.catalog`
- Auth/catalog storage -- already in `wf_sources_mcp.storage`
- Source registry models -- already in `wf_sources_mcp.source_registry`
**Broker/catalog projection logic** (stays in `wf_mcp`):
- `UpstreamTransportService` -- wraps `BackendAdapter` with auth loading, event recording, catalog refresh orchestration
- `SourceCatalogService` -- manages `CapabilitySource` registrations, catalog hydration, source inventory
- `discover_connection_capabilities` -- orchestrates adapter calls and wraps results
- `specs_from_discovered_tools` -- wraps discovered tools into `NodeSpec` with event emission
- `snapshot_from_specs` -- builds `CatalogSnapshot` from `NodeSpec` dict
- `CombinedCatalog` -- aggregates snapshots across connections
- `ConnectionService` -- connection registry lifecycle
- `BrokerEventRecorder` -- broker event fanout
- `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots` -- broker config DTOs
#### What should move to `wf_sources_mcp`?
1. **Transport-opening helper** (`open_mcp_session` or similar) -- extract from `factory.py` and `adapter.py`
2. **`parse_connection_id`** and **`RESERVED_CONNECTION_IDS`** -- move from `wf_mcp.connections` and `wf_mcp.shared.names`
3. **`SourceConnection` protocol** -- new, replacing `ConnectionConfig` in runtime code
#### What should stay in broker compatibility wiring?
1. `UpstreamTransportService` -- it orchestrates broker-specific concerns (events, catalog store, adapter registry)
2. `SourceCatalogService` -- it manages `CapabilitySource` which is `wf_platform`-level
3. `ConnectionService` -- it manages `ConnectionRegistry` which is broker-specific
4. `WfMcpService` -- the compatibility facade
5. All `broker/config.py` construction logic
#### Where are events/catalog/source_catalog dependencies preventing extraction?
- `UpstreamTransportService.refresh_connection_catalog` (lines 203-275) takes `source_catalog: SourceCatalogService` and `record_catalog_change_events` callback. This is broker orchestration, not source provider logic.
- `SourceCatalogService.register_specs` takes `record_catalog_change_events` callback. This is broker event wiring.
- `SourceCatalogService.spec_from_snapshot_entry` (lines 257-298) rebuilds executable `NodeSpec` from stored snapshots, routing calls through `tool_executor_for()`. This is broker hydration, not source provider logic.
None of these prevent `wf_sources_mcp` from owning transport opening -- they just cannot move with it.
### Worker 3: Config/Source/Auth Findings
#### What connection/auth/source DTOs are still too coupled to `wf_mcp`?
1. **`ConnectionConfig`** (`wf_mcp.broker.models:37-44`) -- the primary blocker. Every runtime and broker module depends on it.
2. **`BrokerConfig`** (`wf_mcp.broker.models:47-55`) -- holds `store_root`, `connections`, `store_roots`. Used by proxy/runtime, server/core, broker/config. This is broker-specific and should stay.
3. **`BrokerStoreRoots`** (`wf_mcp.broker.models:11-33`) -- filesystem roots for stores. Broker-specific.
4. **`SourceConfigOwnership`** (`wf_mcp.broker.models:7`) -- `Literal["locked", "seed"]`. Also defined in `wf_config.models:73`. The broker version should be the canonical one since it controls runtime behavior.
5. **`AuthRecord`** (`wf_sources_mcp.auth:22-25`) -- already canonical in `wf_sources_mcp`. The `wf_mcp.auth` module is a re-export shim.
#### What neutral or MCP-source-specific config object should runtime/session code consume instead of `ConnectionConfig.metadata`?
The runtime code (`factory.py`, `adapter.py`) reads these fields from `ConnectionConfig.metadata`:
| Field | Used by | Purpose |
|-------|---------|---------|
| `transport` | `factory.py:49`, `adapter.py:40` | Transport type selector (`"stdio"` or `"streamable_http"`) |
| `command` | `factory.py:56`, `adapter.py:42` | Stdio command |
| `args` | `factory.py:57`, `adapter.py:43` | Stdio arguments |
| `env` | `factory.py:51`, `adapter.py:44` | Stdio environment variables |
| `cwd` | `factory.py:59`, `adapter.py:45` | Stdio working directory |
| `url` | `factory.py:71`, `adapter.py:62` | HTTP transport URL |
A `SourceTransport` union type already exists in `wf_sources_mcp.source_registry:53-69`:
```python
class StdioSourceTransport(SourceRegistryBaseModel):
kind: Literal["stdio"] = "stdio"
command: str
args: tuple[str, ...] = ()
env: dict[str, str] = Field(default_factory=dict)
class HttpSourceTransport(SourceRegistryBaseModel):
kind: Literal["http"] = "http"
url: AnyHttpUrl
headers: dict[str, str] = Field(default_factory=dict)
```
The runtime code should consume a `SourceConnection` protocol that exposes:
- `id: str`
- `transport: StdioSourceTransport | HttpSourceTransport` (or a discriminated union)
- No `metadata: dict[str, Any]` bag
This would eliminate the `connection.metadata.get("transport", "stdio")` pattern scattered across `factory.py` and `adapter.py`.
#### What temporary dependencies remain and how should they be removed?
| Dependency | From | To | How to remove |
|-----------|------|-----|---------------|
| `parse_connection_id` | `wf_sources_mcp.source_registry:32` | `wf_mcp.connections` | Move `parse_connection_id` to `wf_sources_mcp` or a shared `wf_id` package. |
| `RESERVED_CONNECTION_IDS` | `wf_sources_mcp.source_registry:33` | `wf_mcp.shared.names` | Move constant to `wf_sources_mcp.source_registry` or shared package. Remove `wf_mcp.shared.names` import that transitively pulls FastMCP. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.auth:18` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.sdk.protocols:16` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.source_registry:36` | `wf_mcp.models` | Replace with `SourceConnection` protocol or keep as converter-only. |
| `parse_connection_id` runtime | `wf_sources_mcp.storage.store:132` | `wf_mcp.connections` | Move function to `wf_sources_mcp`. |
### Worker 4: Frontend/Compat Findings
#### What is MCP frontend transport vs old compatibility facade?
**MCP frontend transport** (FastMCP server + proxy + tool registration):
- `server/core.py` -- creates `FastMCP` server, wires everything
- `proxy/runtime.py` -- `ProxyRuntime` mounts upstream connections as FastMCP proxies
- `proxy/mounts.py` -- `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy`
- `proxy/tools.py` -- proxy tool listing/filtering
- `proxy/safe_names.py` -- `SafeToolNames` transform for strict clients
- `admin_surface/tools.py` -- registers `wf.admin.*` tools on the server
- `workflow_surface/tools.py` -- registers `wf.workflow.*` tools on the server
- `shared/names.py` -- `ProxyNamespace`, `ADMIN_NAMESPACE`, namespace helpers
**Old compatibility facade** (re-export shims and `WfMcpService`):
- `auth.py` -- re-exports from `wf_sources_mcp.auth`
- `models.py` -- re-exports broker models
- `capabilities.py` -- re-exports from `wf_sources_mcp.catalog.entries`
- `source_registry.py` -- re-exports from `wf_sources_mcp.source_registry`
- `runtime/protocols.py` -- re-exports from `wf_sources_mcp.sdk`
- `broker/service/core.py` -- `WfMcpService` facade
#### What should eventually become `wf_transport_mcp`?
1. `server/core.py` -- `create_server`, `run_server`
2. `proxy/runtime.py` -- `ProxyRuntime`
3. `proxy/mounts.py` -- proxy mount logic
4. `proxy/tools.py` -- proxy tool helpers
5. `proxy/safe_names.py` -- `SafeToolNames`
6. `proxy/admin.py` -- proxy admin tools
7. `admin_surface/tools.py` -- admin tool registration
8. `admin_surface/handlers/*.py` -- admin tool handlers
9. `workflow_surface/tools.py` -- workflow tool registration
10. `workflow_surface/models.py` -- workflow tool models
11. `shared/names.py` -- namespace helpers (minus `RESERVED_CONNECTION_IDS`)
12. `cli.py` -- CLI entrypoint
#### What should remain as legacy `wf_mcp` entrypoints/shims?
1. `auth.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.auth`)
2. `models.py` -- re-export shim (keep until all callers import from `wf_mcp.broker.models` directly)
3. `capabilities.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.catalog`)
4. `source_registry.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.source_registry`)
5. `runtime/protocols.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.sdk`)
6. `runtime/__init__.py` -- re-export `McpRuntimePool`, `PersistentMcpSession`, etc. (keep for backward compat)
#### What should not be touched during upstream-source extraction?
1. `workflow_surface/*` -- workflow tools are MCP-frontend, not source-provider
2. `admin_surface/*` -- admin tools are MCP-frontend, not source-provider
3. `proxy/*` -- proxy mounting is MCP-frontend transport
4. `server/core.py` -- server creation is MCP-frontend
5. `cli.py` -- CLI entrypoint
6. `events/*` -- broker-local event system
7. `broker/service/workflow_runtime.py` -- workflow execution coordination
8. `broker/service/content_access.py` -- resource/prompt access
9. `workflow/wrappers.py` -- NodeSpec wrapping (uses `ToolExecutor` protocol, which is correct)
---
## Recommended Next Slices
Ordered smallest-safe-first.
### Slice 0: Define `SourceConnection` protocol in `wf_sources_mcp`
**Goal:** Break the TYPE_CHECKING dependency from `wf_sources_mcp.sdk.protocols` and `wf_sources_mcp.auth` on `wf_mcp.broker.models.ConnectionConfig` by defining a transport-level protocol that captures what runtime code actually needs.
**Files likely touched:**
- `src/wf_sources_mcp/sdk/protocols.py` -- add `SourceConnection` protocol, update `BackendAdapter` and `ToolExecutor` signatures
- `src/wf_sources_mcp/auth.py` -- update `auth_ref_for_connection` and `connection_auth_diagnostic` to accept protocol
- `src/wf_sources_mcp/source_registry.py` -- update TYPE_CHECKING import
- `src/wf_mcp/broker/models.py` -- make `ConnectionConfig` implement the protocol (no structural change needed, just verify compatibility)
**Tests likely needed:**
- Protocol conformance test: `ConnectionConfig` satisfies `SourceConnection`
- Verify `BackendAdapter` and `ToolExecutor` protocols still typecheck with `ConnectionConfig`
**What must NOT change:**
- `ConnectionConfig` fields/shape
- `BackendAdapter` method signatures (only the type of `connection` parameter changes)
- Any broker service code
- Any proxy/server code
**Migration/shim strategy:**
- `SourceConnection` is a new protocol, not a replacement. `ConnectionConfig` already satisfies it structurally.
- Existing `TYPE_CHECKING` imports in `wf_sources_mcp` become `SourceConnection` protocol imports.
- If any runtime code needs fields beyond the protocol (e.g., `source_config_ownership`), those stay on `ConnectionConfig` and are accessed through a cast or separate parameter.
---
### Slice 1: Move `parse_connection_id` and `RESERVED_CONNECTION_IDS` to `wf_sources_mcp`
**Goal:** Remove the runtime import dependency from `wf_sources_mcp` on `wf_mcp.connections` and `wf_mcp.shared.names`.
**Files likely touched:**
- `src/wf_sources_mcp/source_registry.py` -- replace imports, move validation logic
- `src/wf_sources_mcp/storage/store.py` -- replace `parse_connection_id` import
- `src/wf_mcp/connections.py` -- make `parse_connection_id` a re-export shim
- `src/wf_mcp/shared/names.py` -- make `RESERVED_CONNECTION_IDS` a re-export shim (or keep in `source_registry`)
- `src/wf_mcp/broker/service/connection_service.py` -- update import path
**Tests likely needed:**
- Existing tests for `parse_connection_id` should still pass
- Verify `FileCatalogStore._connection_path` still validates connection IDs
**What must NOT change:**
- Validation logic (same regex, same error messages)
- `RESERVED_CONNECTION_IDS` values
- Any broker service behavior
**Migration/shim strategy:**
- Move `CONNECTION_ID_PATTERN`, `parse_connection_id` to `wf_sources_mcp.source_registry` or a new `wf_sources_mcp.validation` module.
- `wf_mcp.connections` becomes a re-export shim: `from wf_sources_mcp.source_registry import parse_connection_id`
- `RESERVED_CONNECTION_IDS` moves to `wf_sources_mcp.source_registry` (it's already imported there).
- `wf_mcp.shared.names` keeps its own copy or re-exports.
- `wf_mcp.shared.names` can remove the FastMCP import from the top-level module (move `ProxyNamespace` and FastMCP-specific code to a separate submodule if needed).
---
### Slice 2: Extract `open_mcp_session` transport helper to `wf_sources_mcp`
**Goal:** Eliminate the transport-opening duplication between `runtime/factory.py` and `sdk/adapter.py` by extracting a shared helper.
**Files likely touched:**
- New: `src/wf_sources_mcp/sdk/transport.py` -- `open_mcp_session` async context manager
- `src/wf_mcp/runtime/factory.py` -- use `open_mcp_session` in `_create_with_stack`
- `src/wf_mcp/sdk/adapter.py` -- use `open_mcp_session` in `_session`
**Tests likely needed:**
- Unit test for `open_mcp_session` with mock transport
- Verify `PersistentSessionFactory` still creates persistent sessions correctly
- Verify `McpSdkAdapter` still creates one-shot sessions correctly
- Test error handling (unsupported transport, auth failures)
**What must NOT change:**
- `PersistentMcpSession` API
- `McpRuntimePool` behavior
- `McpSdkAdapter` method signatures
- Any broker service code
- Event emission patterns
**Migration/shim strategy:**
- `open_mcp_session` accepts a `SourceConnection` (from Slice 0) and `AuthRecord | None`.
- Returns an async context manager yielding `ClientSession`.
- Both `factory.py` and `adapter.py` call this helper instead of duplicating transport logic.
- The factory wraps it in `AsyncExitStack` for persistent ownership; the adapter uses it directly as a context manager.
---
### Slice 3: Move `PersistentSessionFactory`, `PersistentMcpSession`, `McpRuntimePool` to `wf_sources_mcp`
**Goal:** Move the persistent MCP runtime into the source provider package where it belongs.
**Files likely touched:**
- New: `src/wf_sources_mcp/runtime/__init__.py` -- package init
- New: `src/wf_sources_mcp/runtime/factory.py` -- moved from `wf_mcp/runtime/factory.py`
- New: `src/wf_sources_mcp/runtime/session.py` -- moved from `wf_mcp/runtime/session.py`
- New: `src/wf_sources_mcp/runtime/pool.py` -- moved from `wf_mcp/runtime/pool.py`
- `src/wf_mcp/runtime/__init__.py` -- becomes re-export shim
- `src/wf_mcp/runtime/factory.py` -- becomes re-export shim
- `src/wf_mcp/runtime/session.py` -- becomes re-export shim
- `src/wf_mcp/runtime/pool.py` -- becomes re-export shim
- `src/wf_mcp/broker/config.py` -- update import path
- `src/wf_mcp/broker/service/core.py` -- update import path (if any)
**Tests likely needed:**
- All existing `test_stateful_runtime.py` tests must pass unchanged
- Verify `CrashingSessionFactory` subclass still works
- Verify `McpRuntimePool` fingerprint logic still works
**What must NOT change:**
- `PersistentMcpSession` API (connection, auth, call_tool, close)
- `McpRuntimePool` API (get_session, call_tool, close_connection, close_all)
- `PersistentSessionFactory.create` signature
- `connection_runtime_fingerprint` function
- Any event emission or broker orchestration
**Migration/shim strategy:**
- `wf_mcp.runtime` becomes a re-export shim: `from wf_sources_mcp.runtime import ...`
- `wf_mcp.runtime.protocols.ToolExecutor` already re-exports from `wf_sources_mcp.sdk`
- Tests import from `wf_mcp.runtime` still work via shims
- `broker/config.py` can update to import from `wf_sources_mcp.runtime` directly
---
### Slice 4: Move `McpSdkAdapter` to `wf_sources_mcp.sdk`
**Goal:** Consolidate the one-shot MCP adapter into the source provider SDK.
**Files likely touched:**
- New: `src/wf_sources_mcp/sdk/adapter.py` -- moved from `wf_mcp/sdk/adapter.py`
- `src/wf_mcp/sdk/adapter.py` -- becomes re-export shim
- `src/wf_mcp/sdk/__init__.py` -- update re-exports
- `src/wf_mcp/broker/config.py` -- update import path
- `src/wf_mcp/server/core.py` -- update import path
**Tests likely needed:**
- Existing `test_sdk_adapter.py` tests must pass
- Verify `McpSdkAdapter` still implements `BackendAdapter`
**What must NOT change:**
- `McpSdkAdapter` method signatures
- `BackendAdapter` protocol
- Any broker service code
**Migration/shim strategy:**
- `wf_mcp.sdk.adapter.McpSdkAdapter` re-exports from `wf_sources_mcp.sdk.adapter`
- `wf_mcp.sdk.__init__` keeps exporting `McpSdkAdapter`
---
### Slice 5: Introduce `SourceConnection` dataclass in `wf_sources_mcp` (optional, future)
**Goal:** Replace the `metadata: dict[str, Any]` bag in `ConnectionConfig` with a typed transport DTO for source-provider code.
**Files likely touched:**
- `src/wf_sources_mcp/source_registry.py` -- add `SourceConnection` dataclass
- `src/wf_mcp/runtime/factory.py` -- accept `SourceConnection` instead of `ConnectionConfig`
- `src/wf_mcp/sdk/adapter.py` -- accept `SourceConnection` instead of `ConnectionConfig`
- `src/wf_mcp/runtime/pool.py` -- accept `SourceConnection` in fingerprint
- Conversion helpers in `wf_sources_mcp.source_registry`
**Tests likely needed:**
- Conversion test: `ConnectionConfig` -> `SourceConnection`
- Round-trip test for fingerprint stability
- Verify `McpRuntimePool` fingerprint changes correctly
**What must NOT change:**
- `ConnectionConfig` shape (it's still the broker DTO)
- Broker service behavior
- Proxy/server behavior
**Migration/shim strategy:**
- `SourceConnection` is a new typed DTO in `wf_sources_mcp`.
- `ConnectionConfig` gains a `to_source_connection() -> SourceConnection` method or a standalone converter.
- Runtime code (`factory.py`, `adapter.py`, `pool.py`) accepts `SourceConnection`.
- `McpRuntimePool` fingerprint uses `SourceConnection` fields.
- This slice is optional if Slices 0-4 are sufficient.
---
### Slice 6: Clean up `wf_mcp` re-export shims (final)
**Goal:** Remove all compatibility re-export shims from `wf_mcp` once all callers import from canonical packages.
**Files likely touched:**
- `src/wf_mcp/auth.py` -- delete or leave empty
- `src/wf_mcp/capabilities.py` -- delete or leave empty
- `src/wf_mcp/source_registry.py` -- delete or leave empty
- `src/wf_mcp/runtime/protocols.py` -- delete or leave empty
- `src/wf_mcp/runtime/__init__.py` -- simplify
- `src/wf_mcp/models.py` -- simplify
- `src/wf_mcp/__init__.py` -- simplify
**Tests likely needed:**
- Verify all existing imports still work (or update them)
- `test_compat_imports.py` should pass or be updated
**What must NOT change:**
- Any runtime behavior
- Any broker service behavior
**Migration/shim strategy:**
- This is the final cleanup after all other slices are complete.
- Search for all `from wf_mcp.auth import` etc. and update to canonical paths.
- Leave shims in place for one release cycle, then remove.
---
## Should `runtime/factory.py` Move Now?
**No.** A typed client/session seam must come first.
Reasons:
1. `PersistentSessionFactory._create_with_stack` reads `connection.metadata["transport"]`, `connection.metadata["command"]`, etc. These are `dict[str, Any]` bag accesses that should be replaced by typed protocol access.
2. `PersistentMcpSession` holds `connection: ConnectionConfig` directly. If factory moves to `wf_sources_mcp`, it drags `ConnectionConfig` (and therefore `wf_mcp.broker.models`) into the source provider package at runtime.
3. The `PersistentSessionFactory.create` method returns `PersistentMcpSession` which stores `connection: ConnectionConfig`. Without the protocol, this creates a circular import.
**Correct order:**
1. Slice 0: Define `SourceConnection` protocol
2. Slice 1: Move `parse_connection_id` / `RESERVED_CONNECTION_IDS`
3. Slice 2: Extract `open_mcp_session` helper
4. Slice 3: Move runtime code to `wf_sources_mcp`
5. Slice 4: Move adapter to `wf_sources_mcp.sdk`
6. Slice 5: (Optional) Typed `SourceConnection` dataclass
7. Slice 6: Clean up shims
---
## Test Coverage Summary
Existing tests relevant to the extraction:
| Test file | What it covers | Extraction impact |
|-----------|---------------|-------------------|
| `tests/wf_mcp/test_stateful_runtime.py` | `McpRuntimePool`, `PersistentMcpSession`, `PersistentSessionFactory`, `CrashingSessionFactory` | Must pass unchanged through Slices 0-4 |
| `tests/wf_mcp/test_sdk_adapter.py` | `McpSdkAdapter` one-shot operations | Must pass unchanged through Slices 0-4 |
| `tests/wf_mcp/test_compat_imports.py` | Re-export shim compatibility | Must pass through Slice 6 |
| `tests/wf_mcp/test_workflow_wrappers.py` | `wrap_discovered_tool` with `ToolExecutor` | Must pass unchanged |
| `tests/wf_mcp/test_store.py` | `FileCatalogStore._connection_path` uses `parse_connection_id` | Must pass after Slice 1 |
| `tests/wf_mcp/service/test_connection_service.py` | `ConnectionService` | Unaffected |
| `tests/wf_mcp/service/test_source_registry_admin.py` | Source registry admin tools | Unaffected |
| `tests/wf_mcp/service/test_events.py` | `BrokerEventRecorder` | Unaffected |
| `tests/wf_mcp/service/test_workflow_runtime.py` | `WorkflowRuntimeService` | Unaffected |
---
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Circular import if `wf_sources_mcp` imports `ConnectionConfig` at runtime | High | Use TYPE_CHECKING + protocol pattern (Slice 0) |
| Breaking re-export shims during move | Medium | Keep shims in place, update canonical imports first |
| `McpRuntimePool` fingerprint behavioral change | Medium | Preserve exact same fingerprint computation after move |
| `FileCatalogStore` breaking after `parse_connection_id` move | Low | Move function, keep re-export, run `test_store.py` |
| FastMCP transitive import in `shared/names.py` | Low | Move `RESERVED_CONNECTION_IDS` first, then clean `shared/names.py` |