refactor: move mcp runtime to wf_sources_mcp

2026-06-07 16:40:04 +07:00
parent 008cc6d139
commit b4fd4a6f54
15 changed files with 1126 additions and 376 deletions
@@ -0,0 +1,588 @@
+# MCP Runtime Package Move Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Move persistent MCP runtime ownership from `wf_mcp.runtime` to `wf_sources_mcp.runtime` while preserving old imports as compatibility shims.
+
+**Architecture:** The typed `McpSourceConnection` seam and shared `open_mcp_session()` now exist. This slice makes `wf_sources_mcp.runtime` canonical for persistent MCP sessions, pool reuse, and connection fingerprinting. `wf_mcp.runtime.*` should become thin re-export shims only; behavior should not change and persistent runtime remains tool-call-only.
+
+**Tech Stack:** Python 3.14, dataclasses, asyncio actor/queue pattern, MCP Python SDK `ClientSession`, pytest, ruff, basedpyright.
+
+---
+
+## Current State
+
+Canonical source-provider code already exists:
+
+- `wf_sources_mcp.connections.McpSourceConnection`
+- `wf_sources_mcp.client.open_mcp_session`
+- `wf_sources_mcp.sdk.ToolCallResult`
+- `wf_sources_mcp.sdk.converters.tool_result_to_call_result`
+
+Old runtime files still live in `wf_mcp`:
+
+- `src/wf_mcp/runtime/factory.py`
+- `src/wf_mcp/runtime/session.py`
+- `src/wf_mcp/runtime/pool.py`
+
+The current `McpRuntimePool` has temporary compatibility glue:
+
+```text
+McpSourceConnection -> _legacy_connection_config() -> PersistentSessionFactory
+```
+
+After this plan, that back-conversion should disappear. The canonical runtime factory should accept `McpSourceConnection` directly.
+
+---
+
+## Non-Goals
+
+- Do not broaden persistent runtime beyond `call_tool`.
+- Do not add persistent `read_resource`, `get_prompt`, `invoke_method`, or `send_notification`.
+- Do not move `McpSdkAdapter`.
+- Do not touch MCP proxy/frontend transport.
+- Do not change workflow runtime semantics.
+- Do not change on-disk auth/catalog/source registry formats.
+
+---
+
+## Target File Structure
+
+Create:
+
+- `src/wf_sources_mcp/runtime/__init__.py`
+- `src/wf_sources_mcp/runtime/session.py`
+- `src/wf_sources_mcp/runtime/factory.py`
+- `src/wf_sources_mcp/runtime/pool.py`
+- `tests/wf_sources_mcp/test_runtime.py`
+
+Modify:
+
+- `src/wf_mcp/runtime/__init__.py` -> re-export shim
+- `src/wf_mcp/runtime/session.py` -> re-export shim
+- `src/wf_mcp/runtime/factory.py` -> re-export shim
+- `src/wf_mcp/runtime/pool.py` -> re-export shim
+- `src/wf_mcp/broker/config.py` -> canonical import from `wf_sources_mcp.runtime`
+- any other production imports found by `rg 'wf_mcp\\.runtime' src`
+- `tests/wf_mcp/test_compat_imports.py` -> shim identity tests
+- `docs/current_roadmap.md`
+- `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
+
+---
+
+## Task 1: Create Canonical Runtime Session
+
+**Files:**
+- Create: `src/wf_sources_mcp/runtime/session.py`
+- Test: `tests/wf_sources_mcp/test_runtime.py`
+
+- [ ] **Step 1: Add session tests**
+
+Create `tests/wf_sources_mcp/test_runtime.py` with:
+
+```python
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import pytest
+from mcp.types import CallToolResult, TextContent
+
+from wf_sources_mcp.auth import AuthRecord
+from wf_sources_mcp.connections import McpSourceConnection
+from wf_sources_mcp.runtime import PersistentMcpSession
+from wf_sources_mcp.transports import StdioSourceTransport
+
+
+def _connection() -> McpSourceConnection:
+    return McpSourceConnection(
+        id="demo.personal",
+        provider="demo",
+        account="personal",
+        transport=StdioSourceTransport(command="fake"),
+    )
+
+
+@pytest.mark.asyncio
+async def test_persistent_session_call_callback_normalizes_tool_result() -> None:
+    async def call_tool(tool_name: str, payload: dict[str, Any]) -> CallToolResult:
+        assert tool_name == "echo"
+        assert payload == {"text": "hi"}
+        return CallToolResult(
+            content=[TextContent(type="text", text="ok")],
+            structuredContent={"echoed": "hi"},
+        )
+
+    session = PersistentMcpSession(
+        connection=_connection(),
+        auth=AuthRecord(connection_id="demo.personal", scheme="none"),
+        call_callback=call_tool,
+    )
+
+    result = await session.call_tool("echo", {"text": "hi"})
+
+    assert result.outcome == "ok"
+    assert result.output == {"echoed": "hi"}
+
+
+@pytest.mark.asyncio
+async def test_persistent_session_raises_without_transport() -> None:
+    session = PersistentMcpSession(connection=_connection(), auth=None)
+
+    with pytest.raises(RuntimeError, match="no tool call transport"):
+        await session.call_tool("echo", {})
+```
+
+- [ ] **Step 2: Run failing tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_mcp/test_runtime.py -q
+```
+
+Expected: fail because `wf_sources_mcp.runtime` does not exist.
+
+- [ ] **Step 3: Implement canonical session**
+
+Create `src/wf_sources_mcp/runtime/session.py` by moving the implementation from `src/wf_mcp/runtime/session.py`, but change imports/types:
+
+- Import `AuthRecord` from `wf_sources_mcp.auth`.
+- Import `McpSourceConnection` from `wf_sources_mcp.connections`.
+- Import `ToolCallResult` and `tool_result_to_call_result` from `wf_sources_mcp`.
+- `PersistentMcpSession.connection` must be `McpSourceConnection`, not `ConnectionConfig`.
+
+Keep:
+
+- `RawToolCaller`
+- `client` injection path
+- `call_callback` path
+- `close_callback`
+- error message `"persistent MCP session has no tool call transport"`
+
+- [ ] **Step 4: Run session tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_mcp/test_runtime.py -q
+uv run basedpyright --level error src/wf_sources_mcp/runtime
+```
+
+Expected: pass.
+
+---
+
+## Task 2: Create Canonical Runtime Factory
+
+**Files:**
+- Create: `src/wf_sources_mcp/runtime/factory.py`
+- Modify: `tests/wf_sources_mcp/test_runtime.py`
+
+- [ ] **Step 1: Add factory owner-task tests**
+
+Append to `tests/wf_sources_mcp/test_runtime.py`:
+
+```python
+from wf_sources_mcp.runtime.factory import PersistentSessionFactory
+
+
+class _FakeFactory(PersistentSessionFactory):
+    def __init__(self) -> None:
+        self.calls: list[tuple[str, dict[str, object]]] = []
+        self.closed = False
+
+    async def _call_tool(self, tool_name: str, payload: dict[str, object]):
+        self.calls.append((tool_name, payload))
+        return CallToolResult(
+            content=[TextContent(type="text", text="ok")],
+            structuredContent={"echoed": payload["text"]},
+        )
+
+    async def _close(self) -> None:
+        self.closed = True
+
+    async def _create_with_stack(self, stack, connection, auth):
+        class _FakeClient:
+            async def call_tool(self, tool_name, payload):
+                return await self_factory._call_tool(tool_name, payload)
+
+        self_factory = self
+        return _FakeClient()
+
+
+@pytest.mark.asyncio
+async def test_persistent_session_factory_serializes_tool_calls() -> None:
+    factory = _FakeFactory()
+    session = await factory.create(_connection(), None)
+
+    first = await session.call_tool("echo", {"text": "one"})
+    second = await session.call_tool("echo", {"text": "two"})
+    await session.close()
+
+    assert first.output == {"echoed": "one"}
+    assert second.output == {"echoed": "two"}
+    assert factory.calls == [
+        ("echo", {"text": "one"}),
+        ("echo", {"text": "two"}),
+    ]
+```
+
+- [ ] **Step 2: Implement factory**
+
+Create `src/wf_sources_mcp/runtime/factory.py` by moving the implementation from `src/wf_mcp/runtime/factory.py`, but change types/imports:
+
+- `PersistentSessionFactory.create(connection: McpSourceConnection, auth: AuthRecord | None)`.
+- `_SessionOwner.connection: McpSourceConnection`.
+- `_create_with_stack(stack, connection: McpSourceConnection, auth)` uses:
+
+```python
+session = await stack.enter_async_context(open_mcp_session(connection, auth))
+return session
+```
+
+- Remove any `ConnectionConfig` import.
+- Keep `_ToolCallRequest` and `_SessionOwner` private.
+- Keep actor/queue behavior unchanged.
+
+- [ ] **Step 3: Run factory tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_mcp/test_runtime.py -q
+uv run basedpyright --level error src/wf_sources_mcp/runtime
+```
+
+Expected: pass.
+
+---
+
+## Task 3: Create Canonical Runtime Pool
+
+**Files:**
+- Create: `src/wf_sources_mcp/runtime/pool.py`
+- Modify: `src/wf_sources_mcp/runtime/__init__.py`
+- Modify: `tests/wf_sources_mcp/test_runtime.py`
+
+- [ ] **Step 1: Add pool tests**
+
+Append to `tests/wf_sources_mcp/test_runtime.py`:
+
+```python
+from wf_sources_mcp.runtime import McpRuntimePool, connection_runtime_fingerprint
+
+
+@pytest.mark.asyncio
+async def test_runtime_pool_reuses_unchanged_connection() -> None:
+    created: list[McpSourceConnection] = []
+
+    async def create_session(connection: McpSourceConnection, auth: AuthRecord | None):
+        created.append(connection)
+        return PersistentMcpSession(
+            connection=connection,
+            auth=auth,
+            call_callback=lambda tool_name, payload: CallToolResult(
+                content=[TextContent(type="text", text="ok")],
+                structuredContent={"echoed": payload["text"]},
+            ),
+        )
+
+    pool = McpRuntimePool(session_factory=create_session)
+    connection = _connection()
+
+    await pool.call_tool(connection, None, "echo", {"text": "one"})
+    await pool.call_tool(connection, None, "echo", {"text": "two"})
+
+    assert created == [connection]
+
+
+def test_runtime_fingerprint_changes_when_transport_changes() -> None:
+    original = _connection()
+    changed = McpSourceConnection(
+        id="demo.personal",
+        provider="demo",
+        account="personal",
+        transport=StdioSourceTransport(command="changed"),
+    )
+
+    assert connection_runtime_fingerprint(original) != connection_runtime_fingerprint(
+        changed
+    )
+```
+
+- [ ] **Step 2: Implement pool**
+
+Create `src/wf_sources_mcp/runtime/pool.py` by moving the implementation from `src/wf_mcp/runtime/pool.py`, but make it canonical:
+
+- `RuntimeConnection` should be `McpSourceConnection`.
+- `SessionFactory` should accept `McpSourceConnection`, not `ConnectionConfig`.
+- Remove `_legacy_connection_config`.
+- Remove imports from `wf_mcp`.
+- `connection_runtime_fingerprint` should accept `McpSourceConnection`.
+- Keep reuse/close behavior unchanged.
+
+Create `src/wf_sources_mcp/runtime/__init__.py`:
+
+```python
+from .factory import PersistentSessionFactory
+from .pool import McpRuntimePool, connection_runtime_fingerprint
+from .session import PersistentMcpSession
+
+__all__ = [
+    "McpRuntimePool",
+    "PersistentMcpSession",
+    "PersistentSessionFactory",
+    "connection_runtime_fingerprint",
+]
+```
+
+- [ ] **Step 3: Run runtime tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_mcp/test_runtime.py -q
+uv run basedpyright --level error src/wf_sources_mcp/runtime
+```
+
+Expected: pass.
+
+---
+
+## Task 4: Turn `wf_mcp.runtime` Into Compatibility Shims
+
+**Files:**
+- Replace: `src/wf_mcp/runtime/session.py`
+- Replace: `src/wf_mcp/runtime/factory.py`
+- Replace: `src/wf_mcp/runtime/pool.py`
+- Modify: `src/wf_mcp/runtime/__init__.py`
+- Test: `tests/wf_mcp/test_compat_imports.py`
+
+- [ ] **Step 1: Add shim identity tests**
+
+In `tests/wf_mcp/test_compat_imports.py`, add:
+
+```python
+def test_runtime_shims_reexport_wf_sources_mcp_runtime() -> None:
+    from wf_mcp.runtime import (
+        McpRuntimePool as OldMcpRuntimePool,
+        PersistentMcpSession as OldPersistentMcpSession,
+        PersistentSessionFactory as OldPersistentSessionFactory,
+        connection_runtime_fingerprint as old_connection_runtime_fingerprint,
+    )
+    from wf_sources_mcp.runtime import (
+        McpRuntimePool,
+        PersistentMcpSession,
+        PersistentSessionFactory,
+        connection_runtime_fingerprint,
+    )
+
+    assert OldMcpRuntimePool is McpRuntimePool
+    assert OldPersistentMcpSession is PersistentMcpSession
+    assert OldPersistentSessionFactory is PersistentSessionFactory
+    assert old_connection_runtime_fingerprint is connection_runtime_fingerprint
+```
+
+- [ ] **Step 2: Replace old runtime files with shims**
+
+`src/wf_mcp/runtime/session.py`:
+
+```python
+"""Compatibility shim for the canonical MCP source runtime session."""
+
+from wf_sources_mcp.runtime.session import PersistentMcpSession, RawToolCaller
+
+__all__ = ["PersistentMcpSession", "RawToolCaller"]
+```
+
+`src/wf_mcp/runtime/factory.py`:
+
+```python
+"""Compatibility shim for the canonical MCP source runtime factory."""
+
+from wf_sources_mcp.runtime.factory import PersistentSessionFactory
+
+__all__ = ["PersistentSessionFactory"]
+```
+
+`src/wf_mcp/runtime/pool.py`:
+
+```python
+"""Compatibility shim for the canonical MCP source runtime pool."""
+
+from wf_sources_mcp.runtime.pool import (
+    McpRuntimePool,
+    SessionFactory,
+    connection_runtime_fingerprint,
+)
+
+__all__ = [
+    "McpRuntimePool",
+    "SessionFactory",
+    "connection_runtime_fingerprint",
+]
+```
+
+Keep `src/wf_mcp/runtime/protocols.py` as-is if it already shims `ToolExecutor`.
+
+Update `src/wf_mcp/runtime/__init__.py` to re-export from `wf_sources_mcp.runtime` plus `ToolExecutor`:
+
+```python
+from wf_sources_mcp.runtime import (
+    McpRuntimePool,
+    PersistentMcpSession,
+    PersistentSessionFactory,
+    connection_runtime_fingerprint,
+)
+
+from .protocols import ToolExecutor
+
+__all__ = [
+    "McpRuntimePool",
+    "PersistentMcpSession",
+    "PersistentSessionFactory",
+    "ToolExecutor",
+    "connection_runtime_fingerprint",
+]
+```
+
+- [ ] **Step 3: Run shim tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_compat_imports.py tests/wf_sources_mcp/test_runtime.py -q
+```
+
+Expected: pass.
+
+---
+
+## Task 5: Update Production Imports To Canonical Runtime
+
+**Files:**
+- Modify: `src/wf_mcp/broker/config.py`
+- Search all source files
+
+- [ ] **Step 1: Find old runtime imports**
+
+Run:
+
+```bash
+rg -n 'wf_mcp\.runtime|from \.\.runtime|from \.runtime' src tests
+```
+
+- [ ] **Step 2: Update production imports**
+
+For production code outside shim files, import canonical runtime from `wf_sources_mcp.runtime`.
+
+Likely file:
+
+`src/wf_mcp/broker/config.py`
+
+Replace:
+
+```python
+from wf_mcp.runtime import McpRuntimePool, PersistentSessionFactory
+```
+
+or relative equivalents with:
+
+```python
+from wf_sources_mcp.runtime import McpRuntimePool, PersistentSessionFactory
+```
+
+Do not update tests that intentionally verify compatibility shims.
+
+- [ ] **Step 3: Run focused production tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/server/test_config.py::test_server_reuses_real_upstream_session_across_workflow_requests -q
+uv run basedpyright --level error src
+```
+
+Expected: pass.
+
+---
+
+## Task 6: Preserve Existing Stateful Runtime Tests
+
+**Files:**
+- Modify: `tests/wf_mcp/test_stateful_runtime.py` only if required
+- Test: `tests/wf_mcp/test_stateful_runtime.py`
+
+- [ ] **Step 1: Run existing tests unchanged first**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_stateful_runtime.py -q
+```
+
+Expected: should pass through shims. If it fails only because helper subclasses still type `ConnectionConfig`, update the tests to import canonical runtime but keep behavior assertions unchanged.
+
+- [ ] **Step 2: Do not weaken behavior assertions**
+
+The following behavior must remain tested:
+
+- pool reuses unchanged connection fingerprint
+- pool replaces changed connection fingerprint
+- session owner serializes calls through one owner task
+- closed sessions are closed via callback
+- crashing factory surfaces errors to queued calls
+
+If any of these tests need edits, preserve the same assertions and explain why in final report.
+
+---
+
+## Task 7: Documentation And Verification
+
+**Files:**
+- Modify: `docs/current_roadmap.md`
+- Modify: `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
+
+- [ ] **Step 1: Update docs**
+
+Roadmap/spec should say:
+
+- persistent MCP runtime moved to `wf_sources_mcp.runtime`
+- `wf_mcp.runtime.*` are compatibility shims
+- runtime remains tool-call-only
+- next slice is moving `McpSdkAdapter` to `wf_sources_mcp.sdk.adapter`
+
+- [ ] **Step 2: Final verification**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_mcp tests/wf_mcp/test_stateful_runtime.py tests/wf_mcp/test_compat_imports.py tests/wf_mcp/server/test_config.py::test_server_reuses_real_upstream_session_across_workflow_requests -q
+uv run ruff check src tests
+uv run basedpyright --level error src
+git diff --check
+```
+
+Expected:
+
+- focused tests pass
+- ruff passes
+- basedpyright has 0 errors
+- no whitespace errors
+
+If `ruff check src tests` finds unrelated pre-existing errors, do not fix unrelated files in this slice. Report exact files/errors and run `ruff check` on changed files instead.
+
+---
+
+## Final Report Requirements
+
+The final report must state:
+
+- runtime files moved or shimmed
+- no behavior expansion beyond `call_tool`
+- `wf_mcp.runtime` compatibility status
+- whether any tests had to change and why
+- exact verification commands and outputs
+