docs: plan mcp source connection seam

This commit is contained in:
lda
2026-06-07 03:45:51 +07:00 Verified
parent 9960862fc6
commit f79741eb2a
3 changed files with 1628 additions and 4 deletions
+4
View File
@@ -221,6 +221,10 @@ implementation state.
MCP-backed JSON-RPC path. A neutral-config `WorkflowServer` can start an MCP-backed JSON-RPC path. A neutral-config `WorkflowServer` can start an
interrupting run, be rebuilt from the same filesystem stores, inspect the interrupting run, be rebuilt from the same filesystem stores, inspect the
interrupted run, and resume it to completion through `RpcWorkflowApiClient`. interrupted run, and resume it to completion through `RpcWorkflowApiClient`.
- Planned: MCP upstream source runtime cleanup now starts with a typed
`McpSourceConnection` seam in `wf_sources_mcp`, not by moving
`runtime/factory.py` as-is. The active plan is
[2026-06-07 MCP source connection seam](./superpowers/plans/2026-06-07-mcp-source-connection-seam.md).
- Auth/source secrets boundary: keep registry desired state separate from - Auth/source secrets boundary: keep registry desired state separate from
upstream credentials, and surface missing auth as validation diagnostics. upstream credentials, and surface missing auth as validation diagnostics.
The contract is now specified in The contract is now specified in
@@ -0,0 +1,934 @@
# MCP Source Connection Seam Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Introduce a typed MCP source connection seam in `wf_sources_mcp` so runtime/session code can stop depending on `wf_mcp.broker.models.ConnectionConfig.metadata`.
**Architecture:** This is the preparatory slice before moving MCP runtime/session code. Move low-level source ID validation and MCP transport models into focused `wf_sources_mcp` modules, then add `McpSourceConnection` plus explicit converters from the legacy broker DTO and source registry entries. Do not move `runtime/factory.py`, `runtime/session.py`, `runtime/pool.py`, or `McpSdkAdapter` in this plan.
**Tech Stack:** Python 3.14, dataclasses, Pydantic v2, pytest, ruff, basedpyright.
---
## Why This Slice Exists
Persistent MCP connections are the real product need. Stdio MCP startup and HTTP session initialization are expensive and failure-prone, so runtime code needs one clear object that means:
> one configured upstream MCP source account that can be opened, authenticated, catalogued, called, and eventually reused.
Today that object is implicitly `ConnectionConfig` plus `metadata: dict[str, Any]`. That keeps old code working, but it makes runtime/session extraction unsafe. This slice creates the typed seam first:
```text
ConnectionConfig -> McpSourceConnection -> shared MCP session opener -> persistent runtime
```
After this slice, later runtime moves become mechanical instead of dragging broker config bags into `wf_sources_mcp`.
---
## File Structure
Create:
- `src/wf_sources_mcp/ids.py`
- Owns source/connection ID validation for MCP upstream sources.
- Exports `CONNECTION_ID_PATTERN`, `RESERVED_CONNECTION_IDS`, `parse_connection_id`.
- `src/wf_sources_mcp/transports.py`
- Owns `StdioSourceTransport`, `HttpSourceTransport`, `SourceTransport`.
- Replaces transport model definitions currently embedded in `source_registry.py`.
- `src/wf_sources_mcp/connections.py`
- Owns `McpSourceConnection`.
- Owns conversion helpers from legacy broker `ConnectionConfig` and registry entries.
- `tests/wf_sources_mcp/test_connections.py`
- Tests ID validation, transport parsing, legacy conversion, registry conversion, and package exports.
Modify:
- `src/wf_sources_mcp/source_registry.py`
- Import ID helpers from `wf_sources_mcp.ids`.
- Import transport models from `wf_sources_mcp.transports`.
- Keep existing public exports for compatibility.
- `src/wf_sources_mcp/auth.py`
- Make auth helpers accept `McpSourceConnection` / source-like objects instead of directly depending on `ConnectionConfig`.
- `src/wf_sources_mcp/sdk/protocols.py`
- Introduce a source-connection protocol/type alias for backend protocols.
- Remove the `TYPE_CHECKING` dependency on `wf_mcp.broker.models.ConnectionConfig`.
- `src/wf_sources_mcp/__init__.py`
- Export new seam types lazily if needed to avoid circular imports.
- `src/wf_mcp/connections.py`
- Re-export `CONNECTION_ID_PATTERN` and `parse_connection_id` from `wf_sources_mcp.ids`.
- Keep `ConnectionRegistry` and `qualify_node_name` behavior unchanged.
- `src/wf_mcp/shared/names.py`
- Import/re-export `RESERVED_CONNECTION_IDS` from `wf_sources_mcp.ids`.
- Keep FastMCP namespace helpers unchanged.
- `tests/wf_mcp/test_compat_imports.py`
- Add identity checks for moved ID/transport exports where appropriate.
- `docs/current_roadmap.md`
- Add a status note for the typed MCP source connection seam.
- `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
- Add this slice to the MCP source provider package direction list.
---
## Non-Goals
- Do not move `src/wf_mcp/runtime/factory.py`.
- Do not move `src/wf_mcp/runtime/session.py`.
- Do not move `src/wf_mcp/runtime/pool.py`.
- Do not move `src/wf_mcp/sdk/adapter.py`.
- Do not change `ConnectionConfig` field shape.
- Do not change on-disk registry/auth/catalog JSON shapes.
- Do not change MCP proxy/frontend behavior.
---
## Task 1: Move ID Validation Into `wf_sources_mcp.ids`
**Files:**
- Create: `src/wf_sources_mcp/ids.py`
- Modify: `src/wf_mcp/connections.py`
- Modify: `src/wf_mcp/shared/names.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Write failing ID tests**
Create `tests/wf_sources_mcp/test_connections.py` with these tests first:
```python
import pytest
from wf_sources_mcp.ids import (
CONNECTION_ID_PATTERN,
RESERVED_CONNECTION_IDS,
parse_connection_id,
)
def test_parse_connection_id_splits_provider_and_account() -> None:
assert parse_connection_id("github.work") == ("github", "work")
@pytest.mark.parametrize(
"source_id",
["github", ".github.work", "github.", "github/work", "github work"],
)
def test_parse_connection_id_rejects_unsafe_or_unqualified_ids(source_id: str) -> None:
with pytest.raises(ValueError):
parse_connection_id(source_id)
def test_reserved_connection_ids_are_source_provider_constants() -> None:
assert "wf.admin" in RESERVED_CONNECTION_IDS
assert "wf.mcp" in RESERVED_CONNECTION_IDS
assert CONNECTION_ID_PATTERN.startswith("^")
```
- [ ] **Step 2: Run the failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.ids` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.ids`**
Create `src/wf_sources_mcp/ids.py`:
```python
from __future__ import annotations
import re
CONNECTION_ID_PATTERN = r"^[A-Za-z0-9_][A-Za-z0-9_.-]*$"
RESERVED_CONNECTION_IDS = frozenset({"wf.admin", "wf.mcp"})
"""Source ids reserved by built-in workflow/MCP control surfaces."""
def parse_connection_id(connection_id: str) -> tuple[str, str]:
"""Validate and split one MCP source id into provider/account parts.
Source ids also key persisted auth, registry, and catalog files. Keep this
conservative so unsafe ids are rejected before reaching store boundaries.
"""
if not re.fullmatch(CONNECTION_ID_PATTERN, connection_id):
raise ValueError(
"connection id must start with alphanumeric or underscore and contain "
"only [A-Za-z0-9_.-]"
)
if "." not in connection_id:
raise ValueError("connection id must look like '<server>.<account>'")
server, account = connection_id.split(".", 1)
if not server or not account:
raise ValueError("connection id must look like '<server>.<account>'")
return server, account
__all__ = [
"CONNECTION_ID_PATTERN",
"RESERVED_CONNECTION_IDS",
"parse_connection_id",
]
```
- [ ] **Step 4: Update compatibility imports**
In `src/wf_mcp/connections.py`, remove local `re` and `CONNECTION_ID_PATTERN` / `parse_connection_id` definitions. Import instead:
```python
from wf_sources_mcp.ids import CONNECTION_ID_PATTERN, parse_connection_id
```
Keep `qualify_node_name` and `ConnectionRegistry` in `wf_mcp.connections`.
In `src/wf_mcp/shared/names.py`, replace local `RESERVED_CONNECTION_IDS` with:
```python
from wf_sources_mcp.ids import RESERVED_CONNECTION_IDS
```
Keep `ADMIN_NAMESPACE = "wf.admin"` for MCP frontend namespace logic.
- [ ] **Step 5: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_mcp/test_source_registry.py tests/wf_mcp/test_store.py -q
```
Expected: pass.
---
## Task 2: Move Transport Models Into `wf_sources_mcp.transports`
**Files:**
- Create: `src/wf_sources_mcp/transports.py`
- Modify: `src/wf_sources_mcp/source_registry.py`
- Modify: `src/wf_sources_mcp/__init__.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add failing transport tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from pydantic import TypeAdapter
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
def test_stdio_source_transport_is_typed() -> None:
transport = StdioSourceTransport(
command="uvx",
args=("mcp-server",),
env={"TOKEN": "x"},
)
assert transport.kind == "stdio"
assert transport.command == "uvx"
assert transport.args == ("mcp-server",)
assert transport.env == {"TOKEN": "x"}
def test_http_source_transport_is_typed() -> None:
transport = HttpSourceTransport(url="http://127.0.0.1:8000/mcp")
assert transport.kind == "http"
assert str(transport.url) == "http://127.0.0.1:8000/mcp"
def test_source_transport_discriminated_union_parses() -> None:
adapter = TypeAdapter(SourceTransport)
transport = adapter.validate_python(
{"kind": "stdio", "command": "pnpx", "args": ["-y", "server"]}
)
assert isinstance(transport, StdioSourceTransport)
assert transport.args == ("-y", "server")
```
- [ ] **Step 2: Run failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.transports` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.transports`**
Create `src/wf_sources_mcp/transports.py`:
```python
from __future__ import annotations
from typing import Annotated, Literal
from pydantic import AnyHttpUrl, Field
from wf_api.source_registry import SourceRegistryBaseModel
class StdioSourceTransport(SourceRegistryBaseModel):
kind: Literal["stdio"] = "stdio"
command: str = Field(min_length=1)
args: tuple[str, ...] = ()
env: dict[str, str] = Field(default_factory=dict)
class HttpSourceTransport(SourceRegistryBaseModel):
kind: Literal["http"] = "http"
url: AnyHttpUrl
headers: dict[str, str] = Field(default_factory=dict)
SourceTransport = Annotated[
StdioSourceTransport | HttpSourceTransport,
Field(discriminator="kind"),
]
__all__ = [
"HttpSourceTransport",
"SourceTransport",
"StdioSourceTransport",
]
```
- [ ] **Step 4: Update `source_registry.py` to use canonical transport models**
In `src/wf_sources_mcp/source_registry.py`:
- Remove local `StdioSourceTransport`, `HttpSourceTransport`, and `SourceTransport` definitions.
- Remove now-unused imports `Annotated`, `AnyHttpUrl`.
- Import:
```python
from wf_sources_mcp.ids import RESERVED_CONNECTION_IDS, parse_connection_id
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
```
Keep all three names in `__all__` so existing imports from `wf_sources_mcp.source_registry` continue to work.
- [ ] **Step 5: Update package exports**
In `src/wf_sources_mcp/__init__.py`, export or lazily expose:
```python
HttpSourceTransport
SourceTransport
StdioSourceTransport
```
If direct imports create a circular dependency, use the existing lazy `__getattr__` pattern.
- [ ] **Step 6: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_source_registry.py tests/wf_mcp/test_source_registry.py -q
```
Expected: pass.
---
## Task 3: Add `McpSourceConnection` And Converters
**Files:**
- Create: `src/wf_sources_mcp/connections.py`
- Modify: `src/wf_sources_mcp/__init__.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add failing connection seam tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from wf_sources_mcp.connections import (
McpSourceConnection,
mcp_source_connection_from_connection_config,
mcp_source_connection_from_registry_entry,
)
from wf_sources_mcp.source_registry import McpSourceRegistryEntry
def test_mcp_source_connection_from_registry_entry() -> None:
entry = McpSourceRegistryEntry.model_validate(
{
"id": "github.work",
"provider": "github",
"account": "work",
"profile": "engineering",
"transport": {
"kind": "stdio",
"command": "uvx",
"args": ["github-mcp"],
"env": {"A": "B"},
},
"auth_ref": "github.token",
"metadata": {"team": "platform"},
}
)
connection = mcp_source_connection_from_registry_entry(entry)
assert connection == McpSourceConnection(
id="github.work",
provider="github",
account="work",
enabled=True,
profile="engineering",
transport=StdioSourceTransport(
command="uvx",
args=("github-mcp",),
env={"A": "B"},
),
auth_ref="github.token",
metadata={"team": "platform"},
)
def test_mcp_source_connection_from_legacy_connection_config_stdio() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
enabled=False,
metadata={
"transport": "stdio",
"command": "uvx",
"args": ["github-mcp"],
"env": {"A": "B"},
"auth_ref": "github.token",
"profile": "engineering",
"source_registry": True,
"team": "platform",
},
)
connection = mcp_source_connection_from_connection_config(legacy)
assert connection.id == "github.work"
assert connection.provider == "github"
assert connection.account == "work"
assert connection.enabled is False
assert connection.profile == "engineering"
assert connection.auth_ref == "github.token"
assert connection.metadata == {"source_registry": True, "team": "platform"}
assert isinstance(connection.transport, StdioSourceTransport)
assert connection.transport.command == "uvx"
assert connection.transport.args == ("github-mcp",)
def test_mcp_source_connection_from_legacy_connection_config_http() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
metadata={
"transport": "streamable_http",
"url": "http://127.0.0.1:8000/mcp",
"headers": {"X-Test": "yes"},
},
)
connection = mcp_source_connection_from_connection_config(legacy)
assert isinstance(connection.transport, HttpSourceTransport)
assert str(connection.transport.url) == "http://127.0.0.1:8000/mcp"
assert connection.transport.headers == {"X-Test": "yes"}
def test_mcp_source_connection_rejects_missing_legacy_transport() -> None:
from wf_mcp.broker.models import ConnectionConfig
legacy = ConnectionConfig(
id="github.work",
server="github",
account="work",
metadata={},
)
with pytest.raises(ValueError, match="requires metadata.transport"):
mcp_source_connection_from_connection_config(legacy)
```
- [ ] **Step 2: Run failing tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py -q
```
Expected: fail because `wf_sources_mcp.connections` does not exist.
- [ ] **Step 3: Implement `wf_sources_mcp.connections`**
Create `src/wf_sources_mcp/connections.py`:
```python
from __future__ import annotations
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
from wf_sources_mcp.ids import parse_connection_id
from wf_sources_mcp.source_registry import McpSourceRegistryEntry
from wf_sources_mcp.transports import (
HttpSourceTransport,
SourceTransport,
StdioSourceTransport,
)
if TYPE_CHECKING:
from wf_mcp.broker.models import ConnectionConfig
_FLAT_HTTP_TRANSPORTS = {"http", "streamable-http", "streamable_http", "sse"}
_CONNECTION_METADATA_KEYS = {
"transport",
"command",
"args",
"env",
"cwd",
"url",
"headers",
"profile",
"auth_ref",
}
@dataclass(frozen=True, slots=True)
class McpSourceConnection:
"""Typed runtime-facing MCP source connection.
This is the object runtime/session code should consume. Legacy broker
`ConnectionConfig.metadata` remains at the compatibility edge only.
"""
id: str
provider: str
account: str
transport: SourceTransport
enabled: bool = True
profile: str | None = None
auth_ref: str | None = None
metadata: dict[str, object] = field(default_factory=dict)
def __post_init__(self) -> None:
provider, account = parse_connection_id(self.id)
if not self.provider:
raise ValueError("provider must not be empty")
if not self.account:
raise ValueError("account must not be empty")
if provider != self.provider or account != self.account:
raise ValueError(
"MCP source connection id must match provider/account fields"
)
def mcp_source_connection_from_registry_entry(
entry: McpSourceRegistryEntry,
) -> McpSourceConnection:
"""Adapt persisted desired-source registry state to runtime source shape."""
return McpSourceConnection(
id=entry.id,
provider=entry.provider,
account=entry.account,
enabled=entry.enabled,
profile=entry.profile,
transport=entry.transport,
auth_ref=entry.auth_ref,
metadata=dict(entry.metadata),
)
def mcp_source_connection_from_connection_config(
connection: ConnectionConfig,
) -> McpSourceConnection:
"""Adapt legacy broker connection config into typed source shape.
Keep all metadata-bag reads in this compatibility converter. Runtime/session
code should use `McpSourceConnection.transport` directly.
"""
transport = _transport_from_connection_metadata(connection)
profile = connection.metadata.get("profile")
auth_ref = connection.metadata.get("auth_ref")
metadata = {
str(key): value
for key, value in connection.metadata.items()
if key not in _CONNECTION_METADATA_KEYS
}
return McpSourceConnection(
id=connection.id,
provider=connection.server,
account=connection.account,
enabled=connection.enabled,
profile=profile if isinstance(profile, str) else None,
transport=transport,
auth_ref=auth_ref if isinstance(auth_ref, str) else None,
metadata=metadata,
)
def _transport_from_connection_metadata(connection: ConnectionConfig) -> SourceTransport:
transport = connection.metadata.get("transport")
if isinstance(transport, dict):
kind = transport.get("kind")
if kind == "stdio":
return StdioSourceTransport.model_validate(transport)
if kind == "http":
return HttpSourceTransport.model_validate(transport)
raise ValueError(
f"connection {connection.id!r} has unsupported metadata.transport.kind {kind!r}"
)
if isinstance(transport, str):
if transport == "stdio":
return StdioSourceTransport(
command=str(connection.metadata.get("command", "")),
args=tuple(str(arg) for arg in connection.metadata.get("args", ())),
env={
str(key): str(value)
for key, value in dict(connection.metadata.get("env", {})).items()
},
)
if transport in _FLAT_HTTP_TRANSPORTS:
return HttpSourceTransport(
url=str(connection.metadata.get("url", "")),
headers={
str(key): str(value)
for key, value in dict(
connection.metadata.get("headers", {})
).items()
},
)
raise ValueError(
f"connection {connection.id!r} has unrecognized metadata.transport {transport!r}"
)
raise ValueError(f"connection {connection.id!r} requires metadata.transport")
__all__ = [
"McpSourceConnection",
"mcp_source_connection_from_connection_config",
"mcp_source_connection_from_registry_entry",
]
```
- [ ] **Step 4: Export the new seam from the package**
In `src/wf_sources_mcp/__init__.py`, export or lazily expose:
```python
McpSourceConnection
mcp_source_connection_from_connection_config
mcp_source_connection_from_registry_entry
```
- [ ] **Step 5: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_source_registry.py -q
```
Expected: pass.
---
## Task 4: Update Auth And SDK Protocols To Use The Source Seam
**Files:**
- Modify: `src/wf_sources_mcp/auth.py`
- Modify: `src/wf_sources_mcp/sdk/protocols.py`
- Test: `tests/wf_sources_mcp/test_auth.py`
- Test: `tests/wf_sources_mcp/test_sdk_protocols.py`
- Test: `tests/wf_sources_mcp/test_connections.py`
- [ ] **Step 1: Add protocol/conformance tests**
Append to `tests/wf_sources_mcp/test_connections.py`:
```python
from typing import Protocol
from wf_sources_mcp.auth import auth_ref_for_connection
from wf_sources_mcp.sdk import BackendAdapter, ToolExecutor
class _ConnectionLike(Protocol):
id: str
auth_ref: str | None
def test_auth_ref_for_typed_mcp_source_connection() -> None:
connection = McpSourceConnection(
id="github.work",
provider="github",
account="work",
transport=StdioSourceTransport(command="uvx"),
auth_ref="github.token",
)
assert auth_ref_for_connection(connection) == "github.token"
def test_sdk_protocols_are_importable_without_broker_connection_config() -> None:
assert BackendAdapter is not None
assert ToolExecutor is not None
```
- [ ] **Step 2: Run focused tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_auth.py tests/wf_sources_mcp/test_sdk_protocols.py -q
```
Expected: likely fail until auth/protocols stop importing `ConnectionConfig`.
- [ ] **Step 3: Update `auth.py`**
In `src/wf_sources_mcp/auth.py`:
- Remove the `TYPE_CHECKING` import of `wf_mcp.broker.models.ConnectionConfig`.
- Define a small protocol:
```python
class SourceConnectionLike(Protocol):
id: str
auth_ref: str | None
```
- Change:
```python
def auth_ref_for_connection(connection: ConnectionConfig) -> str | None:
auth_ref = connection.metadata.get("auth_ref")
return auth_ref if isinstance(auth_ref, str) else None
```
to:
```python
def auth_ref_for_connection(connection: SourceConnectionLike) -> str | None:
return connection.auth_ref
```
- Change `connection_auth_diagnostic(connection: ConnectionConfig, ...)` to accept `SourceConnectionLike`.
Important: this intentionally means callers that still hold `ConnectionConfig` must convert to `McpSourceConnection` before using auth diagnostics. If current production callers need a compatibility path, use `mcp_source_connection_from_connection_config(connection)` at that call site instead of reintroducing metadata reads in `auth.py`.
- [ ] **Step 4: Update `sdk/protocols.py`**
In `src/wf_sources_mcp/sdk/protocols.py`:
- Remove `TYPE_CHECKING` and `ConnectionConfig`.
- Import:
```python
from wf_sources_mcp.connections import McpSourceConnection
```
- Change all `connection: ConnectionConfig` parameters in `BackendAdapter` and `ToolExecutor` to:
```python
connection: McpSourceConnection
```
This is a type-only protocol change. Production adapters may need a follow-up slice to convert broker DTOs before calling the protocol.
- [ ] **Step 5: Run focused type/tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_connections.py tests/wf_sources_mcp/test_auth.py tests/wf_sources_mcp/test_sdk_protocols.py -q
uv run basedpyright --level error src/wf_sources_mcp
```
Expected: pass or reveal call sites that still need explicit conversion. Fix call sites by converting at the broker boundary, not by weakening `McpSourceConnection` back into `metadata`.
---
## Task 5: Update Current Broker Call Sites At The Boundary
**Files:**
- Modify: `src/wf_mcp/broker/service/upstream_transport.py`
- Modify: `src/wf_mcp/broker/service/source_catalog.py`
- Modify: `src/wf_mcp/runtime/factory.py`
- Modify: `src/wf_mcp/runtime/session.py`
- Modify: `src/wf_mcp/runtime/pool.py`
- Modify: `src/wf_mcp/sdk/adapter.py`
- Test: existing focused tests
- [ ] **Step 1: Find all protocol call sites**
Run:
```bash
rg -n 'BackendAdapter|ToolExecutor|call_tool\\(|list_tools\\(|list_resources\\(|list_prompts\\(|read_resource\\(|get_prompt\\(|invoke_method\\(|send_notification\\(' src\\wf_mcp src\\wf_sources_mcp
```
Inspect call sites that pass a `ConnectionConfig` into a `BackendAdapter` or `ToolExecutor`.
- [ ] **Step 2: Convert at the broker edge**
Where broker services call source-provider protocols, convert with:
```python
from wf_sources_mcp.connections import mcp_source_connection_from_connection_config
source_connection = mcp_source_connection_from_connection_config(connection)
```
Then pass `source_connection` to `BackendAdapter` / `ToolExecutor` methods.
Keep `ConnectionConfig` in broker services for registry/config ownership and source catalog behavior. Do not rewrite the whole broker service in this slice.
- [ ] **Step 3: Keep runtime files compiling without moving them**
If `PersistentMcpSession`, `McpRuntimePool`, or `McpSdkAdapter` currently implement `ToolExecutor` / `BackendAdapter`, update their method signatures to accept `McpSourceConnection` where needed.
If their public callers still pass `ConnectionConfig`, convert at the public method boundary and keep a comment:
```python
# Compatibility boundary: broker callers still pass ConnectionConfig. Runtime
# internals use McpSourceConnection so the session code can move to
# wf_sources_mcp in a later slice.
```
- [ ] **Step 4: Run focused tests**
Run:
```bash
uv run pytest tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/service/test_upstream_transport.py tests/wf_mcp/service/test_source_registry_admin.py tests/wf_sources_mcp -q
uv run basedpyright --level error src
```
Expected: pass.
---
## Task 6: Documentation And Verification
**Files:**
- Modify: `docs/current_roadmap.md`
- Modify: `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
- [ ] **Step 1: Update roadmap**
In `docs/current_roadmap.md`, under the MCP source provider / long-lived API section, add a short note:
```markdown
- Completed: typed MCP source connection seam is introduced in `wf_sources_mcp`.
Source IDs, reserved IDs, transport models, and `McpSourceConnection` are now
canonical source-provider concepts. Legacy broker `ConnectionConfig` remains
intact and converts at compatibility edges; runtime/session files are not
moved yet.
```
- [ ] **Step 2: Update long-lived boundary spec**
In `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`, extend the MCP source provider package direction list:
```markdown
6. Complete: typed MCP source connection seam introduced. `McpSourceConnection`
is the runtime-facing source object; legacy `ConnectionConfig` converts at
broker edges.
7. Next: shared MCP session opener, then runtime/session/pool move.
```
- [ ] **Step 3: Run final verification**
Run:
```bash
uv run pytest tests/wf_sources_mcp tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/service/test_upstream_transport.py -q
uv run ruff check src tests
uv run basedpyright --level error src
git diff --check
```
Expected:
- tests pass
- ruff passes
- basedpyright has 0 errors
- no diff whitespace errors
- [ ] **Step 4: Report remaining future slices**
Final report must explicitly state:
- runtime/factory/session/pool were not moved
- proxy/frontend MCP code was not touched
- old `ConnectionConfig` shape is unchanged
- next planned slice is shared session opener using `McpSourceConnection`
---
## Future Slices After This Plan
1. **Shared MCP session opener**
- Create `wf_sources_mcp.sdk.transport.open_mcp_session(connection, auth)`.
- Replace duplicated session opening in `wf_mcp/runtime/factory.py` and `wf_mcp/sdk/adapter.py`.
2. **Move persistent runtime package**
- Move `PersistentSessionFactory`, `PersistentMcpSession`, and `McpRuntimePool` to `wf_sources_mcp.runtime`.
- Keep `wf_mcp.runtime.*` shims.
3. **Move one-shot SDK adapter**
- Move `McpSdkAdapter` to `wf_sources_mcp.sdk.adapter`.
- Keep `wf_mcp.sdk.adapter` shim.
4. **Eventually split MCP frontend transport**
- Move FastMCP server/proxy/admin/workflow tool registration toward `wf_transport_mcp`.
- This is separate from upstream source runtime and should not block persistent connection work.
---
## Self-Review Notes
- This plan intentionally creates focused files before touching runtime. That keeps the first change testable and limits blast radius.
- The plan does not attempt to phase out all 369 `ConnectionConfig` references. It converts only at source-provider protocol edges.
- The likely implementation risk is type fallout after `BackendAdapter` / `ToolExecutor` signatures change. Resolve by explicit conversion at broker boundaries, not by weakening the typed seam.
@@ -0,0 +1,686 @@
# WF MCP Runtime & Source Provider Map
Date: 2026-06-07
## Executive Summary
`wf_mcp` is a monolith containing four distinct responsibilities that should become separate packages:
1. **MCP source provider runtime** (transport opening, session pooling, auth) -- belongs in `wf_sources_mcp`
2. **MCP frontend transport** (FastMCP server, proxy mounts, MCP tool registration) -- belongs in a future `wf_transport_mcp`
3. **Broker service layer** (connection catalog, discovery, source catalog, workflow runtime) -- stays in `wf_mcp` or becomes `wf_server`
4. **Compatibility shims** (re-export modules, `WfMcpService` facade) -- remains in `wf_mcp` until callers migrate
The critical blocker is `ConnectionConfig` (defined at `src/wf_mcp/broker/models.py:37`). This dataclass is consumed by `wf_sources_mcp.sdk.protocols`, `wf_sources_mcp.auth`, `wf_sources_mcp.source_registry`, `wf_sources_mcp.storage.store`, the runtime factory, the adapter, the pool, and every broker service. Until `ConnectionConfig` moves to a neutral package or is replaced by a typed protocol, extraction is blocked.
The next safe slice is **Slice 0: Define a `SourceConnection` protocol in `wf_sources_mcp`** so runtime/session code stops depending on the broker DTO. After that, the transport-opening logic in `runtime/factory.py` and `sdk/adapter.py` can merge into `wf_sources_mcp.sdk` without dragging the broker layer along.
---
## Package Responsibility Map
### `wf_sources_mcp` (source provider core)
Owns: MCP source provider identity, auth, catalog entries, storage, and the SDK adapter/executor protocols.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `auth.py` | MCP auth record, env/header extraction, diagnostic helpers | Canonical. Has TYPE_CHECKING dep on `wf_mcp.broker.models.ConnectionConfig`. |
| `sdk/protocols.py` | `BackendAdapter`, `ToolExecutor`, `ToolCallResult` | Canonical. Has TYPE_CHECKING dep on `ConnectionConfig`. |
| `sdk/converters.py` | MCP tool/resource/prompt conversion | Canonical. No `wf_mcp` deps. |
| `catalog/models.py` | `CatalogSnapshot`, `dump_catalog_snapshot` | Canonical. No `wf_mcp` deps. |
| `catalog/entries.py` | `CatalogNodeEntry`, `CatalogResourceEntry`, `CatalogPromptEntry`, `DiscoveredTool`, etc. | Canonical. No `wf_mcp` deps. |
| `source_registry.py` | `McpSourceRegistryEntry`, `SourceRegistryFile`, conversion helpers | Canonical. Imports `parse_connection_id` from `wf_mcp.connections` and `RESERVED_CONNECTION_IDS` from `wf_mcp.shared.names`. TYPE_CHECKING dep on `ConnectionConfig`. |
| `storage/store.py` | `AuthStore`, `CatalogStore`, `FileAuthStore`, `FileCatalogStore` | Canonical. `FileCatalogStore._connection_path` imports `parse_connection_id` from `wf_mcp.connections` at runtime. |
### Future `wf_transport_mcp` (MCP frontend transport)
Owns: FastMCP server creation, proxy mounting, MCP tool/resource/prompt registration, admin tool registration, workflow surface tool registration.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `proxy/runtime.py` | `ProxyRuntime` -- mount upstream MCP connections into FastMCP | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, `EventBus`, `BrokerConfigManager`. |
| `proxy/mounts.py` | `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy` | `wf_mcp` internal. Depends on `BrokerConfig`, `ConnectionConfig`, FastMCP. |
| `proxy/tools.py` | Proxy tool listing/filtering | `wf_mcp` internal. |
| `proxy/safe_names.py` | `SafeToolNames` transform | `wf_mcp` internal. FastMCP transform. |
| `proxy/admin.py` | Proxy admin tools | `wf_mcp` internal. |
| `server/core.py` | `create_server`, `run_server`, `create_server_client` | `wf_mcp` entrypoint. Wires broker service + proxy + workflow surface + admin surface. |
| `admin_surface/tools.py` | `register_service_admin_tools` | `wf_mcp` internal. |
| `admin_surface/handlers/*.py` | Admin tool handlers | `wf_mcp` internal. |
| `workflow_surface/tools.py` | `register_workflow_tools` | `wf_mcp` internal. |
| `workflow_surface/*.py` | Workflow tool models, handlers, lifecycle | `wf_mcp` internal. |
| `shared/names.py` | `ProxyNamespace`, `ADMIN_NAMESPACE`, `RESERVED_CONNECTION_IDS`, namespace helpers | `wf_mcp` internal. FastMCP-specific. |
| `cli.py` | CLI entrypoint | `wf_mcp` entrypoint. |
### `wf_server` / `wf_api` (server core)
Owns: Workflow API, operation context, artifact management, deployment, run lifecycle.
Already extracted. These packages consume `wf_sources_mcp` types and `wf_mcp.broker.service.WfMcpService` through `wf_api.WorkflowApi` and `wf_api.WorkflowRuntimeAdapter`.
### Legacy `wf_mcp` (broker + compatibility shims)
Owns: Connection registry, broker config, broker service coordination, events, and all compatibility re-export shims.
| Module | Responsibility | Status |
|--------|---------------|--------|
| `broker/models.py` | `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots`, `SourceConfigOwnership` | **The core blocker.** All other packages depend on this. |
| `broker/service/core.py` | `WfMcpService` -- compatibility coordinator | Facade. Delegates to focused services. |
| `broker/service/connection_service.py` | `ConnectionService` | Focused service. Depends on `ConnectionConfig`, `SourceCatalogService`. |
| `broker/service/source_catalog.py` | `SourceCatalogService` | Focused service. Depends on `ConnectionConfig`, `McpEvent`, `NodeSpec`. |
| `broker/service/upstream_transport.py` | `UpstreamTransportService` | Focused service. Depends on `ConnectionConfig`, `BackendAdapter`, `ToolExecutor`. |
| `broker/service/workflow_runtime.py` | `WorkflowRuntimeService` | Focused service. Depends on `SourceCatalogService`. |
| `broker/service/content_access.py` | `ContentAccessService` | Focused service. |
| `broker/service/events.py` | `BrokerEventRecorder` | Focused service. Depends on `McpEvent`. |
| `broker/service/adapters.py` | `require_adapter` | Helper. |
| `broker/discovery.py` | `discover_connection_capabilities`, `specs_from_discovered_tools` | Broker logic. |
| `broker/catalog.py` | `snapshot_from_specs`, `CombinedCatalog` | Broker catalog projection. |
| `broker/config.py` | `build_service_from_config`, `load_broker_config`, `broker_config_from_workflow_config` | Config construction. |
| `connections.py` | `ConnectionRegistry`, `parse_connection_id`, `qualify_node_name` | Broker connection registry. |
| `auth.py` | Re-export shim from `wf_sources_mcp.auth` | Compatibility. |
| `models.py` | Re-export shim aggregating broker models | Compatibility. |
| `source_registry.py` | Re-export shim from `wf_sources_mcp.source_registry` | Compatibility. |
| `capabilities.py` | Re-export shim from `wf_sources_mcp.catalog.entries` | Compatibility. |
| `runtime/protocols.py` | Re-export shim from `wf_sources_mcp.sdk` | Compatibility. |
| `events/bus.py` | `EventBus`, `InMemoryEventSink` | Broker-local event fanout. |
| `events/models.py` | `McpEvent`, `make_event` | Broker event model. |
---
## Current Dependency Blockers
### Blocker 1: `ConnectionConfig` origin
**File:** `src/wf_mcp/broker/models.py:37-44`
```python
@dataclass(slots=True)
class ConnectionConfig:
id: str
server: str
account: str
enabled: bool = True
metadata: dict[str, Any] = field(default_factory=dict)
source_config_ownership: SourceConfigOwnership = "locked"
```
This DTO is imported by:
| Consumer | Import path | Import type |
|----------|------------|-------------|
| `wf_sources_mcp.auth` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.sdk.protocols` | `wf_mcp.broker.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.source_registry` | `wf_mcp.models.ConnectionConfig` | `TYPE_CHECKING` |
| `wf_sources_mcp.storage.store` | (none directly, but `FileCatalogStore._connection_path` calls `parse_connection_id` from `wf_mcp.connections`) | runtime |
| `wf_mcp.runtime.factory` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.sdk.adapter` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.runtime.pool` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.runtime.session` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.workflow.wrappers` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.broker.discovery` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.broker.service.*` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.proxy.mounts` | `wf_mcp.models.ConnectionConfig` | runtime |
| `wf_mcp.proxy.runtime` | (indirectly via `BrokerConfig`) | runtime |
**Impact:** Until `ConnectionConfig` moves to a neutral package (or `wf_sources_mcp` defines its own protocol), the runtime code in `wf_sources_mcp` cannot be independent of `wf_mcp`.
### Blocker 2: `parse_connection_id` origin
**File:** `src/wf_mcp/connections.py:11-25`
Used by:
- `wf_sources_mcp.source_registry` (runtime import, line 32)
- `wf_sources_mcp.storage.store` (runtime import inside `_connection_path`, line 132)
- `wf_mcp.connections` (canonical home)
**Impact:** `wf_sources_mcp` has a runtime import dependency on `wf_mcp` for connection ID validation. This function should move to a neutral package or `wf_sources_mcp`.
### Blocker 3: `RESERVED_CONNECTION_IDS` origin
**File:** `src/wf_mcp/shared/names.py:25`
```python
RESERVED_CONNECTION_IDS = frozenset({ADMIN_NAMESPACE, "wf.mcp"})
```
Used by:
- `wf_sources_mcp.source_registry` (runtime import, line 33)
- `wf_mcp.broker.service.connection_service` (runtime import, line 14)
**Impact:** `wf_sources_mcp` imports from `wf_mcp.shared.names` which transitively imports FastMCP transforms (line 8-15 of `shared/names.py`). This creates an unwanted dependency chain.
### Blocker 4: `McpSdkAdapter` transport-opening duplication
**Files:**
- `src/wf_mcp/runtime/factory.py:43-90` (`_create_with_stack`)
- `src/wf_mcp/sdk/adapter.py:34-75` (`_session`)
Both methods contain nearly identical logic for:
- Reading `connection.metadata["transport"]` to select stdio vs streamable HTTP
- Creating `StdioServerParameters` and calling `stdio_client`
- Creating `httpx.AsyncClient` and calling `streamable_http_client`
- Creating `ClientSession` and calling `session.initialize()`
- Applying auth via `mcp_auth_env` / `mcp_auth_headers`
The difference: factory.py owns the session long-term (persistent actor pattern), while adapter.py opens/closes per call (one-shot pattern).
**Impact:** This duplication means both files must be updated together when transport handling changes. They should share a transport-opening helper.
### Blocker 5: `BrokerConfig` in proxy/runtime
**File:** `src/wf_mcp/proxy/runtime.py:73-85`
`ProxyRuntime.__init__` takes `BrokerConfig` directly and reads `config.connections`. The proxy layer is a frontend transport concern that should not depend on the broker config DTO.
**Impact:** Proxy runtime cannot move to `wf_transport_mcp` until it consumes a transport-neutral config shape.
### Blocker 6: `McpEvent` coupling
**File:** `src/wf_mcp/events/models.py`
`McpEvent` is used by:
- `UpstreamTransportService` (event_sink)
- `SourceCatalogService` (emit_event)
- `BrokerEventRecorder` (event_bus)
- `broker/catalog.py` (indirectly through event callbacks)
- `workflow/wrappers.py` (emit_event callback)
**Impact:** `McpEvent` is broker-specific. If source provider code needs to emit events, it should use a protocol or a neutral event type. Currently `wf_sources_mcp` does not import `McpEvent` directly, which is good.
---
## Detailed Findings by Worker Scope
### Worker 1: Runtime/Session Findings
#### Where is MCP session opening duplicated?
The transport-opening logic appears in two places:
1. **`runtime/factory.py:43-90`** (`PersistentSessionFactory._create_with_stack`):
- Opens stdio or streamable HTTP transport
- Creates `ClientSession`
- Calls `session.initialize()`
- Returns session owned by `AsyncExitStack`
- Used for persistent long-lived sessions (actor pattern)
2. **`sdk/adapter.py:34-75`** (`McpSdkAdapter._session`):
- Opens stdio or streamable HTTP transport
- Creates `ClientSession`
- Calls `session.initialize()`
- Yields session in async context manager
- Used for one-shot per-call sessions (discovery, admin operations)
Both import from the same MCP SDK modules:
- `mcp.client.stdio.StdioServerParameters`, `stdio_client`
- `mcp.client.streamable_http.streamable_http_client`
- `mcp.client.session.ClientSession`
- `wf_sources_mcp.auth.mcp_auth_env`, `mcp_auth_headers`
#### Which client operations are supported by one-shot adapter but not persistent runtime?
The `BackendAdapter` protocol (`wf_sources_mcp.sdk.protocols:26-88`) defines:
- `list_tools`, `list_resources`, `list_prompts`
- `get_connection_metadata`
- `read_resource`, `get_prompt`
- `invoke_method`, `send_notification`
- `call_tool`
The `PersistentMcpSession` (`runtime/session.py:19-49`) only exposes:
- `call_tool`
- `close`
The `McpRuntimePool` (`runtime/pool.py:43-96`) only exposes:
- `get_session` (returns `PersistentMcpSession`)
- `call_tool`
- `close_connection`, `close_all`
**Missing from persistent runtime:** `list_tools`, `list_resources`, `list_prompts`, `get_connection_metadata`, `read_resource`, `get_prompt`, `invoke_method`, `send_notification`.
This is intentional -- persistent sessions exist for workflow execution (tool calls only). Discovery and admin operations use one-shot adapters. However, this means the persistent runtime cannot replace the adapter for all operations.
#### What common `McpClientSession` / `McpClientSessionFactory` interface should exist?
Both factory.py and adapter.py share:
1. Transport selection logic (stdio vs streamable HTTP)
2. Auth application (env vars for stdio, headers for HTTP)
3. Session creation and initialization
A shared `open_mcp_session` helper should:
- Accept a connection descriptor (transport type, command/url, env/headers) and optional auth
- Return an initialized `ClientSession` (or yield it)
- Be used by both `PersistentSessionFactory._create_with_stack` and `McpSdkAdapter._session`
The connection descriptor should NOT be `ConnectionConfig` directly -- it should be a transport-specific DTO that `ConnectionConfig` can convert to.
#### What blocks moving this code to `wf_sources_mcp`?
1. `ConnectionConfig` dependency (Blocker 1)
2. `McpEvent` event callbacks in `PersistentSessionFactory` (the `_SessionOwner._run` method does not emit events, but the pool and factory are used by `broker/config.py` which wires events)
3. The `PersistentMcpSession` dataclass holds `connection: ConnectionConfig` and `auth: AuthRecord` -- the `AuthRecord` is already in `wf_sources_mcp`, but `ConnectionConfig` is not
**Recommendation:** Define a `SourceConnection` protocol in `wf_sources_mcp.sdk.protocols` that captures the transport fields `ConnectionConfig` exposes to runtime code. The factory/pool/session code should consume this protocol, not the concrete DTO.
### Worker 2: Broker/Upstream Findings
#### What is truly MCP-source-provider logic vs broker/catalog projection logic?
**MCP-source-provider logic** (belongs in `wf_sources_mcp`):
- Transport opening (stdio, streamable HTTP) -- currently in `factory.py` and `adapter.py`
- Auth application (env vars, headers) -- already in `wf_sources_mcp.auth`
- `BackendAdapter` protocol and `ToolExecutor` protocol -- already in `wf_sources_mcp.sdk`
- `ToolCallResult` dataclass -- already in `wf_sources_mcp.sdk`
- `DiscoveredTool`, `DiscoveredResource`, `DiscoveredPrompt` -- already in `wf_sources_mcp.catalog`
- `CatalogSnapshot` and catalog entries -- already in `wf_sources_mcp.catalog`
- Auth/catalog storage -- already in `wf_sources_mcp.storage`
- Source registry models -- already in `wf_sources_mcp.source_registry`
**Broker/catalog projection logic** (stays in `wf_mcp`):
- `UpstreamTransportService` -- wraps `BackendAdapter` with auth loading, event recording, catalog refresh orchestration
- `SourceCatalogService` -- manages `CapabilitySource` registrations, catalog hydration, source inventory
- `discover_connection_capabilities` -- orchestrates adapter calls and wraps results
- `specs_from_discovered_tools` -- wraps discovered tools into `NodeSpec` with event emission
- `snapshot_from_specs` -- builds `CatalogSnapshot` from `NodeSpec` dict
- `CombinedCatalog` -- aggregates snapshots across connections
- `ConnectionService` -- connection registry lifecycle
- `BrokerEventRecorder` -- broker event fanout
- `ConnectionConfig`, `BrokerConfig`, `BrokerStoreRoots` -- broker config DTOs
#### What should move to `wf_sources_mcp`?
1. **Transport-opening helper** (`open_mcp_session` or similar) -- extract from `factory.py` and `adapter.py`
2. **`parse_connection_id`** and **`RESERVED_CONNECTION_IDS`** -- move from `wf_mcp.connections` and `wf_mcp.shared.names`
3. **`SourceConnection` protocol** -- new, replacing `ConnectionConfig` in runtime code
#### What should stay in broker compatibility wiring?
1. `UpstreamTransportService` -- it orchestrates broker-specific concerns (events, catalog store, adapter registry)
2. `SourceCatalogService` -- it manages `CapabilitySource` which is `wf_platform`-level
3. `ConnectionService` -- it manages `ConnectionRegistry` which is broker-specific
4. `WfMcpService` -- the compatibility facade
5. All `broker/config.py` construction logic
#### Where are events/catalog/source_catalog dependencies preventing extraction?
- `UpstreamTransportService.refresh_connection_catalog` (lines 203-275) takes `source_catalog: SourceCatalogService` and `record_catalog_change_events` callback. This is broker orchestration, not source provider logic.
- `SourceCatalogService.register_specs` takes `record_catalog_change_events` callback. This is broker event wiring.
- `SourceCatalogService.spec_from_snapshot_entry` (lines 257-298) rebuilds executable `NodeSpec` from stored snapshots, routing calls through `tool_executor_for()`. This is broker hydration, not source provider logic.
None of these prevent `wf_sources_mcp` from owning transport opening -- they just cannot move with it.
### Worker 3: Config/Source/Auth Findings
#### What connection/auth/source DTOs are still too coupled to `wf_mcp`?
1. **`ConnectionConfig`** (`wf_mcp.broker.models:37-44`) -- the primary blocker. Every runtime and broker module depends on it.
2. **`BrokerConfig`** (`wf_mcp.broker.models:47-55`) -- holds `store_root`, `connections`, `store_roots`. Used by proxy/runtime, server/core, broker/config. This is broker-specific and should stay.
3. **`BrokerStoreRoots`** (`wf_mcp.broker.models:11-33`) -- filesystem roots for stores. Broker-specific.
4. **`SourceConfigOwnership`** (`wf_mcp.broker.models:7`) -- `Literal["locked", "seed"]`. Also defined in `wf_config.models:73`. The broker version should be the canonical one since it controls runtime behavior.
5. **`AuthRecord`** (`wf_sources_mcp.auth:22-25`) -- already canonical in `wf_sources_mcp`. The `wf_mcp.auth` module is a re-export shim.
#### What neutral or MCP-source-specific config object should runtime/session code consume instead of `ConnectionConfig.metadata`?
The runtime code (`factory.py`, `adapter.py`) reads these fields from `ConnectionConfig.metadata`:
| Field | Used by | Purpose |
|-------|---------|---------|
| `transport` | `factory.py:49`, `adapter.py:40` | Transport type selector (`"stdio"` or `"streamable_http"`) |
| `command` | `factory.py:56`, `adapter.py:42` | Stdio command |
| `args` | `factory.py:57`, `adapter.py:43` | Stdio arguments |
| `env` | `factory.py:51`, `adapter.py:44` | Stdio environment variables |
| `cwd` | `factory.py:59`, `adapter.py:45` | Stdio working directory |
| `url` | `factory.py:71`, `adapter.py:62` | HTTP transport URL |
A `SourceTransport` union type already exists in `wf_sources_mcp.source_registry:53-69`:
```python
class StdioSourceTransport(SourceRegistryBaseModel):
kind: Literal["stdio"] = "stdio"
command: str
args: tuple[str, ...] = ()
env: dict[str, str] = Field(default_factory=dict)
class HttpSourceTransport(SourceRegistryBaseModel):
kind: Literal["http"] = "http"
url: AnyHttpUrl
headers: dict[str, str] = Field(default_factory=dict)
```
The runtime code should consume a `SourceConnection` protocol that exposes:
- `id: str`
- `transport: StdioSourceTransport | HttpSourceTransport` (or a discriminated union)
- No `metadata: dict[str, Any]` bag
This would eliminate the `connection.metadata.get("transport", "stdio")` pattern scattered across `factory.py` and `adapter.py`.
#### What temporary dependencies remain and how should they be removed?
| Dependency | From | To | How to remove |
|-----------|------|-----|---------------|
| `parse_connection_id` | `wf_sources_mcp.source_registry:32` | `wf_mcp.connections` | Move `parse_connection_id` to `wf_sources_mcp` or a shared `wf_id` package. |
| `RESERVED_CONNECTION_IDS` | `wf_sources_mcp.source_registry:33` | `wf_mcp.shared.names` | Move constant to `wf_sources_mcp.source_registry` or shared package. Remove `wf_mcp.shared.names` import that transitively pulls FastMCP. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.auth:18` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.sdk.protocols:16` | `wf_mcp.broker.models` | Replace with `SourceConnection` protocol. |
| `ConnectionConfig` TYPE_CHECKING | `wf_sources_mcp.source_registry:36` | `wf_mcp.models` | Replace with `SourceConnection` protocol or keep as converter-only. |
| `parse_connection_id` runtime | `wf_sources_mcp.storage.store:132` | `wf_mcp.connections` | Move function to `wf_sources_mcp`. |
### Worker 4: Frontend/Compat Findings
#### What is MCP frontend transport vs old compatibility facade?
**MCP frontend transport** (FastMCP server + proxy + tool registration):
- `server/core.py` -- creates `FastMCP` server, wires everything
- `proxy/runtime.py` -- `ProxyRuntime` mounts upstream connections as FastMCP proxies
- `proxy/mounts.py` -- `ProxyMountRegistry`, `create_proxy_mount`, `ResilientFastMCPProxy`
- `proxy/tools.py` -- proxy tool listing/filtering
- `proxy/safe_names.py` -- `SafeToolNames` transform for strict clients
- `admin_surface/tools.py` -- registers `wf.admin.*` tools on the server
- `workflow_surface/tools.py` -- registers `wf.workflow.*` tools on the server
- `shared/names.py` -- `ProxyNamespace`, `ADMIN_NAMESPACE`, namespace helpers
**Old compatibility facade** (re-export shims and `WfMcpService`):
- `auth.py` -- re-exports from `wf_sources_mcp.auth`
- `models.py` -- re-exports broker models
- `capabilities.py` -- re-exports from `wf_sources_mcp.catalog.entries`
- `source_registry.py` -- re-exports from `wf_sources_mcp.source_registry`
- `runtime/protocols.py` -- re-exports from `wf_sources_mcp.sdk`
- `broker/service/core.py` -- `WfMcpService` facade
#### What should eventually become `wf_transport_mcp`?
1. `server/core.py` -- `create_server`, `run_server`
2. `proxy/runtime.py` -- `ProxyRuntime`
3. `proxy/mounts.py` -- proxy mount logic
4. `proxy/tools.py` -- proxy tool helpers
5. `proxy/safe_names.py` -- `SafeToolNames`
6. `proxy/admin.py` -- proxy admin tools
7. `admin_surface/tools.py` -- admin tool registration
8. `admin_surface/handlers/*.py` -- admin tool handlers
9. `workflow_surface/tools.py` -- workflow tool registration
10. `workflow_surface/models.py` -- workflow tool models
11. `shared/names.py` -- namespace helpers (minus `RESERVED_CONNECTION_IDS`)
12. `cli.py` -- CLI entrypoint
#### What should remain as legacy `wf_mcp` entrypoints/shims?
1. `auth.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.auth`)
2. `models.py` -- re-export shim (keep until all callers import from `wf_mcp.broker.models` directly)
3. `capabilities.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.catalog`)
4. `source_registry.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.source_registry`)
5. `runtime/protocols.py` -- re-export shim (keep until all callers import from `wf_sources_mcp.sdk`)
6. `runtime/__init__.py` -- re-export `McpRuntimePool`, `PersistentMcpSession`, etc. (keep for backward compat)
#### What should not be touched during upstream-source extraction?
1. `workflow_surface/*` -- workflow tools are MCP-frontend, not source-provider
2. `admin_surface/*` -- admin tools are MCP-frontend, not source-provider
3. `proxy/*` -- proxy mounting is MCP-frontend transport
4. `server/core.py` -- server creation is MCP-frontend
5. `cli.py` -- CLI entrypoint
6. `events/*` -- broker-local event system
7. `broker/service/workflow_runtime.py` -- workflow execution coordination
8. `broker/service/content_access.py` -- resource/prompt access
9. `workflow/wrappers.py` -- NodeSpec wrapping (uses `ToolExecutor` protocol, which is correct)
---
## Recommended Next Slices
Ordered smallest-safe-first.
### Slice 0: Define `SourceConnection` protocol in `wf_sources_mcp`
**Goal:** Break the TYPE_CHECKING dependency from `wf_sources_mcp.sdk.protocols` and `wf_sources_mcp.auth` on `wf_mcp.broker.models.ConnectionConfig` by defining a transport-level protocol that captures what runtime code actually needs.
**Files likely touched:**
- `src/wf_sources_mcp/sdk/protocols.py` -- add `SourceConnection` protocol, update `BackendAdapter` and `ToolExecutor` signatures
- `src/wf_sources_mcp/auth.py` -- update `auth_ref_for_connection` and `connection_auth_diagnostic` to accept protocol
- `src/wf_sources_mcp/source_registry.py` -- update TYPE_CHECKING import
- `src/wf_mcp/broker/models.py` -- make `ConnectionConfig` implement the protocol (no structural change needed, just verify compatibility)
**Tests likely needed:**
- Protocol conformance test: `ConnectionConfig` satisfies `SourceConnection`
- Verify `BackendAdapter` and `ToolExecutor` protocols still typecheck with `ConnectionConfig`
**What must NOT change:**
- `ConnectionConfig` fields/shape
- `BackendAdapter` method signatures (only the type of `connection` parameter changes)
- Any broker service code
- Any proxy/server code
**Migration/shim strategy:**
- `SourceConnection` is a new protocol, not a replacement. `ConnectionConfig` already satisfies it structurally.
- Existing `TYPE_CHECKING` imports in `wf_sources_mcp` become `SourceConnection` protocol imports.
- If any runtime code needs fields beyond the protocol (e.g., `source_config_ownership`), those stay on `ConnectionConfig` and are accessed through a cast or separate parameter.
---
### Slice 1: Move `parse_connection_id` and `RESERVED_CONNECTION_IDS` to `wf_sources_mcp`
**Goal:** Remove the runtime import dependency from `wf_sources_mcp` on `wf_mcp.connections` and `wf_mcp.shared.names`.
**Files likely touched:**
- `src/wf_sources_mcp/source_registry.py` -- replace imports, move validation logic
- `src/wf_sources_mcp/storage/store.py` -- replace `parse_connection_id` import
- `src/wf_mcp/connections.py` -- make `parse_connection_id` a re-export shim
- `src/wf_mcp/shared/names.py` -- make `RESERVED_CONNECTION_IDS` a re-export shim (or keep in `source_registry`)
- `src/wf_mcp/broker/service/connection_service.py` -- update import path
**Tests likely needed:**
- Existing tests for `parse_connection_id` should still pass
- Verify `FileCatalogStore._connection_path` still validates connection IDs
**What must NOT change:**
- Validation logic (same regex, same error messages)
- `RESERVED_CONNECTION_IDS` values
- Any broker service behavior
**Migration/shim strategy:**
- Move `CONNECTION_ID_PATTERN`, `parse_connection_id` to `wf_sources_mcp.source_registry` or a new `wf_sources_mcp.validation` module.
- `wf_mcp.connections` becomes a re-export shim: `from wf_sources_mcp.source_registry import parse_connection_id`
- `RESERVED_CONNECTION_IDS` moves to `wf_sources_mcp.source_registry` (it's already imported there).
- `wf_mcp.shared.names` keeps its own copy or re-exports.
- `wf_mcp.shared.names` can remove the FastMCP import from the top-level module (move `ProxyNamespace` and FastMCP-specific code to a separate submodule if needed).
---
### Slice 2: Extract `open_mcp_session` transport helper to `wf_sources_mcp`
**Goal:** Eliminate the transport-opening duplication between `runtime/factory.py` and `sdk/adapter.py` by extracting a shared helper.
**Files likely touched:**
- New: `src/wf_sources_mcp/sdk/transport.py` -- `open_mcp_session` async context manager
- `src/wf_mcp/runtime/factory.py` -- use `open_mcp_session` in `_create_with_stack`
- `src/wf_mcp/sdk/adapter.py` -- use `open_mcp_session` in `_session`
**Tests likely needed:**
- Unit test for `open_mcp_session` with mock transport
- Verify `PersistentSessionFactory` still creates persistent sessions correctly
- Verify `McpSdkAdapter` still creates one-shot sessions correctly
- Test error handling (unsupported transport, auth failures)
**What must NOT change:**
- `PersistentMcpSession` API
- `McpRuntimePool` behavior
- `McpSdkAdapter` method signatures
- Any broker service code
- Event emission patterns
**Migration/shim strategy:**
- `open_mcp_session` accepts a `SourceConnection` (from Slice 0) and `AuthRecord | None`.
- Returns an async context manager yielding `ClientSession`.
- Both `factory.py` and `adapter.py` call this helper instead of duplicating transport logic.
- The factory wraps it in `AsyncExitStack` for persistent ownership; the adapter uses it directly as a context manager.
---
### Slice 3: Move `PersistentSessionFactory`, `PersistentMcpSession`, `McpRuntimePool` to `wf_sources_mcp`
**Goal:** Move the persistent MCP runtime into the source provider package where it belongs.
**Files likely touched:**
- New: `src/wf_sources_mcp/runtime/__init__.py` -- package init
- New: `src/wf_sources_mcp/runtime/factory.py` -- moved from `wf_mcp/runtime/factory.py`
- New: `src/wf_sources_mcp/runtime/session.py` -- moved from `wf_mcp/runtime/session.py`
- New: `src/wf_sources_mcp/runtime/pool.py` -- moved from `wf_mcp/runtime/pool.py`
- `src/wf_mcp/runtime/__init__.py` -- becomes re-export shim
- `src/wf_mcp/runtime/factory.py` -- becomes re-export shim
- `src/wf_mcp/runtime/session.py` -- becomes re-export shim
- `src/wf_mcp/runtime/pool.py` -- becomes re-export shim
- `src/wf_mcp/broker/config.py` -- update import path
- `src/wf_mcp/broker/service/core.py` -- update import path (if any)
**Tests likely needed:**
- All existing `test_stateful_runtime.py` tests must pass unchanged
- Verify `CrashingSessionFactory` subclass still works
- Verify `McpRuntimePool` fingerprint logic still works
**What must NOT change:**
- `PersistentMcpSession` API (connection, auth, call_tool, close)
- `McpRuntimePool` API (get_session, call_tool, close_connection, close_all)
- `PersistentSessionFactory.create` signature
- `connection_runtime_fingerprint` function
- Any event emission or broker orchestration
**Migration/shim strategy:**
- `wf_mcp.runtime` becomes a re-export shim: `from wf_sources_mcp.runtime import ...`
- `wf_mcp.runtime.protocols.ToolExecutor` already re-exports from `wf_sources_mcp.sdk`
- Tests import from `wf_mcp.runtime` still work via shims
- `broker/config.py` can update to import from `wf_sources_mcp.runtime` directly
---
### Slice 4: Move `McpSdkAdapter` to `wf_sources_mcp.sdk`
**Goal:** Consolidate the one-shot MCP adapter into the source provider SDK.
**Files likely touched:**
- New: `src/wf_sources_mcp/sdk/adapter.py` -- moved from `wf_mcp/sdk/adapter.py`
- `src/wf_mcp/sdk/adapter.py` -- becomes re-export shim
- `src/wf_mcp/sdk/__init__.py` -- update re-exports
- `src/wf_mcp/broker/config.py` -- update import path
- `src/wf_mcp/server/core.py` -- update import path
**Tests likely needed:**
- Existing `test_sdk_adapter.py` tests must pass
- Verify `McpSdkAdapter` still implements `BackendAdapter`
**What must NOT change:**
- `McpSdkAdapter` method signatures
- `BackendAdapter` protocol
- Any broker service code
**Migration/shim strategy:**
- `wf_mcp.sdk.adapter.McpSdkAdapter` re-exports from `wf_sources_mcp.sdk.adapter`
- `wf_mcp.sdk.__init__` keeps exporting `McpSdkAdapter`
---
### Slice 5: Introduce `SourceConnection` dataclass in `wf_sources_mcp` (optional, future)
**Goal:** Replace the `metadata: dict[str, Any]` bag in `ConnectionConfig` with a typed transport DTO for source-provider code.
**Files likely touched:**
- `src/wf_sources_mcp/source_registry.py` -- add `SourceConnection` dataclass
- `src/wf_mcp/runtime/factory.py` -- accept `SourceConnection` instead of `ConnectionConfig`
- `src/wf_mcp/sdk/adapter.py` -- accept `SourceConnection` instead of `ConnectionConfig`
- `src/wf_mcp/runtime/pool.py` -- accept `SourceConnection` in fingerprint
- Conversion helpers in `wf_sources_mcp.source_registry`
**Tests likely needed:**
- Conversion test: `ConnectionConfig` -> `SourceConnection`
- Round-trip test for fingerprint stability
- Verify `McpRuntimePool` fingerprint changes correctly
**What must NOT change:**
- `ConnectionConfig` shape (it's still the broker DTO)
- Broker service behavior
- Proxy/server behavior
**Migration/shim strategy:**
- `SourceConnection` is a new typed DTO in `wf_sources_mcp`.
- `ConnectionConfig` gains a `to_source_connection() -> SourceConnection` method or a standalone converter.
- Runtime code (`factory.py`, `adapter.py`, `pool.py`) accepts `SourceConnection`.
- `McpRuntimePool` fingerprint uses `SourceConnection` fields.
- This slice is optional if Slices 0-4 are sufficient.
---
### Slice 6: Clean up `wf_mcp` re-export shims (final)
**Goal:** Remove all compatibility re-export shims from `wf_mcp` once all callers import from canonical packages.
**Files likely touched:**
- `src/wf_mcp/auth.py` -- delete or leave empty
- `src/wf_mcp/capabilities.py` -- delete or leave empty
- `src/wf_mcp/source_registry.py` -- delete or leave empty
- `src/wf_mcp/runtime/protocols.py` -- delete or leave empty
- `src/wf_mcp/runtime/__init__.py` -- simplify
- `src/wf_mcp/models.py` -- simplify
- `src/wf_mcp/__init__.py` -- simplify
**Tests likely needed:**
- Verify all existing imports still work (or update them)
- `test_compat_imports.py` should pass or be updated
**What must NOT change:**
- Any runtime behavior
- Any broker service behavior
**Migration/shim strategy:**
- This is the final cleanup after all other slices are complete.
- Search for all `from wf_mcp.auth import` etc. and update to canonical paths.
- Leave shims in place for one release cycle, then remove.
---
## Should `runtime/factory.py` Move Now?
**No.** A typed client/session seam must come first.
Reasons:
1. `PersistentSessionFactory._create_with_stack` reads `connection.metadata["transport"]`, `connection.metadata["command"]`, etc. These are `dict[str, Any]` bag accesses that should be replaced by typed protocol access.
2. `PersistentMcpSession` holds `connection: ConnectionConfig` directly. If factory moves to `wf_sources_mcp`, it drags `ConnectionConfig` (and therefore `wf_mcp.broker.models`) into the source provider package at runtime.
3. The `PersistentSessionFactory.create` method returns `PersistentMcpSession` which stores `connection: ConnectionConfig`. Without the protocol, this creates a circular import.
**Correct order:**
1. Slice 0: Define `SourceConnection` protocol
2. Slice 1: Move `parse_connection_id` / `RESERVED_CONNECTION_IDS`
3. Slice 2: Extract `open_mcp_session` helper
4. Slice 3: Move runtime code to `wf_sources_mcp`
5. Slice 4: Move adapter to `wf_sources_mcp.sdk`
6. Slice 5: (Optional) Typed `SourceConnection` dataclass
7. Slice 6: Clean up shims
---
## Test Coverage Summary
Existing tests relevant to the extraction:
| Test file | What it covers | Extraction impact |
|-----------|---------------|-------------------|
| `tests/wf_mcp/test_stateful_runtime.py` | `McpRuntimePool`, `PersistentMcpSession`, `PersistentSessionFactory`, `CrashingSessionFactory` | Must pass unchanged through Slices 0-4 |
| `tests/wf_mcp/test_sdk_adapter.py` | `McpSdkAdapter` one-shot operations | Must pass unchanged through Slices 0-4 |
| `tests/wf_mcp/test_compat_imports.py` | Re-export shim compatibility | Must pass through Slice 6 |
| `tests/wf_mcp/test_workflow_wrappers.py` | `wrap_discovered_tool` with `ToolExecutor` | Must pass unchanged |
| `tests/wf_mcp/test_store.py` | `FileCatalogStore._connection_path` uses `parse_connection_id` | Must pass after Slice 1 |
| `tests/wf_mcp/service/test_connection_service.py` | `ConnectionService` | Unaffected |
| `tests/wf_mcp/service/test_source_registry_admin.py` | Source registry admin tools | Unaffected |
| `tests/wf_mcp/service/test_events.py` | `BrokerEventRecorder` | Unaffected |
| `tests/wf_mcp/service/test_workflow_runtime.py` | `WorkflowRuntimeService` | Unaffected |
---
## Risk Assessment
| Risk | Severity | Mitigation |
|------|----------|------------|
| Circular import if `wf_sources_mcp` imports `ConnectionConfig` at runtime | High | Use TYPE_CHECKING + protocol pattern (Slice 0) |
| Breaking re-export shims during move | Medium | Keep shims in place, update canonical imports first |
| `McpRuntimePool` fingerprint behavioral change | Medium | Preserve exact same fingerprint computation after move |
| `FileCatalogStore` breaking after `parse_connection_id` move | Low | Move function, keep re-export, run `test_store.py` |
| FastMCP transitive import in `shared/names.py` | Low | Move `RESERVED_CONNECTION_IDS` first, then clean `shared/names.py` |