refactor: move mcp catalog dtos to wf_sources_mcp

This commit is contained in:
lda
2026-06-07 00:48:23 +07:00 Verified
parent 83a3793534
commit 9bef0b0f63
25 changed files with 1117 additions and 146 deletions
+4 -1
View File
@@ -279,10 +279,13 @@ implementation state.
Keep `wf_mcp` re-export shims for compatibility and add import-direction
tests so `wf_sources_mcp` does not depend on workflow/admin surface,
frontend server, or proxy modules.
Second `wf_sources_mcp` slice complete: MCP desired source registry
Second `wf_sources_mcp` slice complete: MCP desired source registry
models, file store, and conversion helpers now live in
`wf_sources_mcp.source_registry`, with `wf_mcp.source_registry` retained
as a compatibility shim.
Third `wf_sources_mcp` slice complete: upstream MCP catalog/discovery DTOs
and catalog snapshot dumping now live in `wf_sources_mcp.catalog`, with
`wf_mcp.capabilities` and `wf_mcp.catalog.models` retained as shims.
The `wf-mcp` script is now a legacy/special-purpose MCP entrypoint, not the
preferred durable workflow server. New product paths should target
`wf-rpc-server` plus neutral `wf_config`/`wf_server` composition, then keep
@@ -0,0 +1,751 @@
# wf_sources_mcp Catalog DTOs Slice Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Move upstream MCP discovery/catalog DTOs from `wf_mcp` into canonical `wf_sources_mcp.catalog` modules while preserving `wf_mcp` compatibility shims.
**Architecture:** `wf_sources_mcp` owns MCP-as-upstream-source data shapes. `wf_mcp` keeps old import paths and frontend/entrypoint compatibility. This slice moves DTOs only; it does not move catalog services, discovery I/O, SDK adapters, runtime/session pools, or broker orchestration.
**Tech Stack:** Python 3.14, dataclasses, pytest, Ruff, basedpyright, `src/` package layout.
---
## Boundaries
Move only DTO/model code:
- `DiscoveredTool`
- `DiscoveredResource`
- `DiscoveredPrompt`
- `CatalogNodeEntry`
- `CatalogResourceEntry`
- `CatalogPromptEntry`
- `CatalogSnapshot`
- `dump_catalog_snapshot`
Do not move:
- `wf_mcp.broker.catalog.CombinedCatalog`
- `wf_mcp.broker.catalog.snapshot_from_specs`
- `wf_mcp.broker.discovery`
- `wf_mcp.sdk.*`
- `wf_mcp.runtime.*`
- `SourceCatalogService`
- `UpstreamTransportService`
- MCP frontend/admin/workflow tools
Rationale: this removes current temporary DTO dependencies from `wf_sources_mcp.storage.store` without dragging live upstream I/O into this slice.
## File Map
Create:
- `src/wf_sources_mcp/catalog/__init__.py` — exports canonical catalog DTOs.
- `src/wf_sources_mcp/catalog/entries.py` — discovered tool/resource/prompt DTOs and catalog entry DTOs.
- `src/wf_sources_mcp/catalog/models.py``CatalogSnapshot` and `dump_catalog_snapshot`.
- `tests/wf_sources_mcp/test_catalog_dtos.py` — canonical DTO tests.
Modify:
- `src/wf_sources_mcp/storage/store.py` — import catalog DTOs from `wf_sources_mcp.catalog`.
- `src/wf_sources_mcp/__init__.py` — optionally lazy-export catalog DTOs if direct root exports are already expected.
- `src/wf_mcp/capabilities.py` — replace with compatibility shim.
- `src/wf_mcp/catalog/models.py` — replace with compatibility shim.
- `src/wf_mcp/catalog/__init__.py` — re-export shim symbols.
- `src/wf_mcp/models.py` — canonical import from `wf_sources_mcp.catalog.models`.
- Production imports that currently use `wf_mcp.capabilities` or `wf_mcp.catalog.models` for DTOs may be changed to `wf_sources_mcp.catalog`.
- `tests/wf_mcp/test_compat_imports.py` — shim identity tests.
- `tests/wf_sources_mcp/test_import_direction_guard.py` — ensure catalog DTOs do not import forbidden MCP frontend/proxy modules.
- `docs/current_roadmap.md` — mark catalog DTO slice complete.
- `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md` — mark catalog DTO slice complete.
After implementation, move this plan to:
- `docs/historical/superpowers/plans/2026-06-07-wf-sources-mcp-catalog-dtos-slice.md`
---
### Task 1: Create Canonical Catalog DTO Modules
**Files:**
- Create: `src/wf_sources_mcp/catalog/entries.py`
- Create: `src/wf_sources_mcp/catalog/models.py`
- Create: `src/wf_sources_mcp/catalog/__init__.py`
- Test: `tests/wf_sources_mcp/test_catalog_dtos.py`
- [ ] **Step 1: Create `entries.py`**
Create `src/wf_sources_mcp/catalog/entries.py` with the current contents of `src/wf_mcp/capabilities.py`:
```python
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
@dataclass(slots=True)
class DiscoveredTool:
"""Tool snapshot after converting from an upstream MCP SDK tool."""
name: str
title: str | None
description: str | None
input_schema: dict[str, Any]
output_schema: dict[str, Any]
outcomes: tuple[str, ...] = ("ok",)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredResource:
"""Resource snapshot after converting from an upstream MCP SDK resource."""
uri: str
name: str
title: str | None
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredPrompt:
"""Prompt snapshot after converting from an upstream MCP SDK prompt."""
name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogNodeEntry:
"""Namespaced tool entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
outcomes: tuple[str, ...]
input_schema: dict[str, Any]
output_schema: dict[str, Any]
@dataclass(slots=True)
class CatalogResourceEntry:
"""Namespaced resource entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
uri: str
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogPromptEntry:
"""Namespaced prompt entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
]
```
- [ ] **Step 2: Create `models.py`**
Create `src/wf_sources_mcp/catalog/models.py`:
```python
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from typing import Any
from .entries import CatalogNodeEntry, CatalogPromptEntry, CatalogResourceEntry
@dataclass(slots=True)
class CatalogSnapshot:
"""Stored upstream MCP catalog snapshot for one source connection."""
connection_id: str
fetched_at_epoch_ms: int
max_age_seconds: int
nodes: list[CatalogNodeEntry] = field(default_factory=list)
resources: list[CatalogResourceEntry] = field(default_factory=list)
prompts: list[CatalogPromptEntry] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
def is_stale(self, now_epoch_ms: int) -> bool:
age_ms = now_epoch_ms - self.fetched_at_epoch_ms
return age_ms > self.max_age_seconds * 1000
def dump_catalog_snapshot(snapshot: CatalogSnapshot) -> dict[str, Any]:
return {
"connection_id": snapshot.connection_id,
"fetched_at_epoch_ms": snapshot.fetched_at_epoch_ms,
"max_age_seconds": snapshot.max_age_seconds,
"nodes": [asdict(node) for node in snapshot.nodes],
"resources": [asdict(resource) for resource in snapshot.resources],
"prompts": [asdict(prompt) for prompt in snapshot.prompts],
"metadata": snapshot.metadata,
}
__all__ = [
"CatalogSnapshot",
"dump_catalog_snapshot",
]
```
- [ ] **Step 3: Create package exports**
Create `src/wf_sources_mcp/catalog/__init__.py`:
```python
from __future__ import annotations
from .entries import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
)
from .models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"CatalogSnapshot",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
"dump_catalog_snapshot",
]
```
- [ ] **Step 4: Add canonical DTO tests**
Create `tests/wf_sources_mcp/test_catalog_dtos.py`:
```python
from __future__ import annotations
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
CatalogSnapshot,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
dump_catalog_snapshot,
)
def test_discovered_tool_default_outcome_and_metadata() -> None:
tool = DiscoveredTool(
name="echo",
title=None,
description="Echo input",
input_schema={"type": "object"},
output_schema={"type": "object"},
)
assert tool.outcomes == ("ok",)
assert tool.metadata == {}
def test_discovered_resource_and_prompt_keep_structural_fields() -> None:
resource = DiscoveredResource(
uri="docs://guide",
name="guide",
title="Guide",
description="Read me",
mime_type="text/markdown",
)
prompt = DiscoveredPrompt(
name="summarize",
title=None,
description="Summarize",
arguments=[{"name": "topic"}],
)
assert resource.uri == "docs://guide"
assert resource.mime_type == "text/markdown"
assert prompt.arguments == [{"name": "topic"}]
def test_catalog_snapshot_staleness_and_dump_shape() -> None:
snapshot = CatalogSnapshot(
connection_id="demo.default",
fetched_at_epoch_ms=1_000,
max_age_seconds=2,
nodes=[
CatalogNodeEntry(
qualified_name="demo.default.echo",
connection_id="demo.default",
local_name="echo",
title=None,
description="Echo",
outcomes=("ok",),
input_schema={"type": "object"},
output_schema={"type": "object"},
)
],
resources=[
CatalogResourceEntry(
qualified_name="demo.default.guide",
connection_id="demo.default",
local_name="guide",
title=None,
uri="docs://guide",
description="Guide",
)
],
prompts=[
CatalogPromptEntry(
qualified_name="demo.default.summarize",
connection_id="demo.default",
local_name="summarize",
title=None,
description="Summarize",
)
],
metadata={"source": "test"},
)
assert snapshot.is_stale(3_001) is True
dumped = dump_catalog_snapshot(snapshot)
assert dumped["connection_id"] == "demo.default"
assert dumped["nodes"][0]["qualified_name"] == "demo.default.echo"
assert dumped["resources"][0]["uri"] == "docs://guide"
assert dumped["prompts"][0]["local_name"] == "summarize"
assert dumped["metadata"] == {"source": "test"}
```
- [ ] **Step 5: Run canonical DTO tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_catalog_dtos.py -q
```
Expected: 3 tests pass.
---
### Task 2: Replace Old DTO Modules With Shims
**Files:**
- Modify: `src/wf_mcp/capabilities.py`
- Modify: `src/wf_mcp/catalog/models.py`
- Modify: `src/wf_mcp/catalog/__init__.py`
- Modify: `tests/wf_mcp/test_compat_imports.py`
- [ ] **Step 1: Replace `wf_mcp.capabilities` with shim**
Replace `src/wf_mcp/capabilities.py` with:
```python
"""Compatibility shim for MCP upstream catalog entry DTOs.
Canonical implementation lives in `wf_sources_mcp.catalog.entries`.
"""
from __future__ import annotations
from wf_sources_mcp.catalog.entries import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
)
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
]
```
- [ ] **Step 2: Replace `wf_mcp.catalog.models` with shim**
Replace `src/wf_mcp/catalog/models.py` with:
```python
"""Compatibility shim for MCP upstream catalog snapshot DTOs.
Canonical implementation lives in `wf_sources_mcp.catalog.models`.
"""
from __future__ import annotations
from wf_sources_mcp.catalog.models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"CatalogSnapshot",
"dump_catalog_snapshot",
]
```
- [ ] **Step 3: Keep `wf_mcp.catalog` package re-exporting**
Set `src/wf_mcp/catalog/__init__.py` to:
```python
from __future__ import annotations
from .models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"CatalogSnapshot",
"dump_catalog_snapshot",
]
```
- [ ] **Step 4: Add shim identity tests**
Append to `tests/wf_mcp/test_compat_imports.py`:
```python
def test_wf_mcp_capabilities_shim_reexports_wf_sources_mcp_catalog_entries() -> None:
from wf_mcp.capabilities import CatalogNodeEntry as CompatCatalogNodeEntry
from wf_mcp.capabilities import DiscoveredTool as CompatDiscoveredTool
from wf_sources_mcp.catalog import CatalogNodeEntry, DiscoveredTool
assert CompatCatalogNodeEntry is CatalogNodeEntry
assert CompatDiscoveredTool is DiscoveredTool
def test_wf_mcp_catalog_models_shim_reexports_wf_sources_mcp_catalog_models() -> None:
from wf_mcp.catalog.models import CatalogSnapshot as CompatCatalogSnapshot
from wf_mcp.catalog.models import dump_catalog_snapshot as compat_dump
from wf_sources_mcp.catalog import CatalogSnapshot, dump_catalog_snapshot
assert CompatCatalogSnapshot is CatalogSnapshot
assert compat_dump is dump_catalog_snapshot
```
- [ ] **Step 5: Run compatibility tests**
Run:
```bash
uv run pytest tests/wf_mcp/test_compat_imports.py tests/wf_mcp/test_store.py -q
```
Expected: tests pass, proving old imports still work.
---
### Task 3: Update Canonical Imports in `wf_sources_mcp` and Production Code
**Files:**
- Modify: `src/wf_sources_mcp/storage/store.py`
- Modify: `src/wf_mcp/models.py`
- Modify: selected production files under `src/wf_mcp/`
- [ ] **Step 1: Update `wf_sources_mcp.storage.store` imports**
Change the `TYPE_CHECKING` import:
```python
if TYPE_CHECKING:
from wf_sources_mcp.catalog.models import CatalogSnapshot
```
Change `save_catalog()` lazy import:
```python
from wf_sources_mcp.catalog.models import dump_catalog_snapshot
```
Change `load_catalog()` lazy imports:
```python
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
CatalogSnapshot as CatalogSnapshotType,
)
```
- [ ] **Step 2: Update `wf_mcp.models` canonical import**
Change `src/wf_mcp/models.py`:
```python
from wf_sources_mcp.catalog.models import CatalogSnapshot, dump_catalog_snapshot
```
Keep existing `AuthRecord`, `BrokerConfig`, and `ConnectionConfig` exports unchanged.
- [ ] **Step 3: Rewrite production DTO imports where safe**
Change production files that directly import DTOs to canonical package imports:
```python
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
CatalogSnapshot,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
dump_catalog_snapshot,
)
```
Likely files:
- `src/wf_mcp/broker/catalog.py`
- `src/wf_mcp/broker/discovery.py`
- `src/wf_mcp/broker/service/core.py`
- `src/wf_mcp/broker/service/events.py`
- `src/wf_mcp/broker/service/source_catalog.py`
- `src/wf_mcp/broker/service/upstream_transport.py`
- `src/wf_mcp/sdk/adapter.py`
- `src/wf_mcp/sdk/base.py`
- `src/wf_mcp/sdk/converters.py`
- `src/wf_mcp/workflow/wrappers.py`
Do not chase every test import. Tests importing `wf_mcp.capabilities` are useful compatibility coverage unless the test is specifically about the canonical package.
- [ ] **Step 4: Confirm production imports no longer depend on shims**
Run:
```bash
rg -n "from wf_mcp\\.capabilities|from wf_mcp\\.catalog\\.models|from \\.\\.capabilities|from \\.capabilities|from \\.\\.models import CatalogSnapshot|from wf_mcp\\.models import CatalogSnapshot" src
```
Expected: no production imports use old DTO paths except shim files and possibly `wf_mcp.__init__` facade exports. If `wf_mcp.__init__` imports `DiscoveredTool` from `.capabilities`, leave it as facade behavior.
- [ ] **Step 5: Run focused production tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp tests/wf_mcp/test_store.py tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/service/test_catalog.py tests/wf_mcp/service/test_upstream_transport.py tests/wf_mcp/test_workflow_wrappers.py -q
```
Expected: all focused tests pass.
---
### Task 4: Strengthen Import-Direction Guard
**Files:**
- Modify: `tests/wf_sources_mcp/test_import_direction_guard.py`
- Test: `tests/wf_sources_mcp/test_import_direction_guard.py`
- [ ] **Step 1: Keep forbidden frontend/proxy prefixes**
Ensure `FORBIDDEN_WF_MCP_PREFIXES` still includes:
```python
FORBIDDEN_WF_MCP_PREFIXES = (
"wf_mcp.admin_surface",
"wf_mcp.workflow_surface",
"wf_mcp.server",
"wf_mcp.proxy",
"wf_mcp.cli",
)
```
- [ ] **Step 2: Update comment for current temporary imports**
Update the comment above the prefix tuple:
```python
# Temporary low-level wf_mcp imports are allowed for connection id parsing,
# reserved names, and broker DTO conversion. Catalog DTOs should now be local
# to wf_sources_mcp. Frontend/proxy/workflow-surface imports are forbidden
# because wf_sources_mcp is upstream-source code.
```
- [ ] **Step 3: Add a targeted no-old-catalog-import assertion**
Add this test:
```python
def test_wf_sources_mcp_does_not_import_wf_mcp_catalog_dtos() -> None:
root = Path(__file__).resolve().parents[2] / "src" / "wf_sources_mcp"
forbidden = {
"wf_mcp.capabilities",
"wf_mcp.catalog",
"wf_mcp.catalog.models",
}
violations: list[str] = []
for py_file in sorted(root.rglob("*.py")):
rel = py_file.relative_to(root.parent)
module = str(rel.with_suffix("")).replace("/", ".").replace("\\", ".")
tree = ast.parse(py_file.read_text(encoding="utf-8"), filename=str(py_file))
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module in forbidden:
violations.append(f"{module}:{node.lineno}: from {node.module} import ...")
elif isinstance(node, ast.Import):
for alias in node.names:
if alias.name in forbidden:
violations.append(f"{module}:{node.lineno}: import {alias.name}")
assert violations == [], (
"wf_sources_mcp still imports old wf_mcp catalog DTO modules:\n"
+ "\n".join(f" {violation}" for violation in violations)
)
```
- [ ] **Step 4: Run guard tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp/test_import_direction_guard.py -q
```
Expected: guard tests pass.
---
### Task 5: Docs Status and Plan Archival
**Files:**
- Modify: `docs/current_roadmap.md`
- Modify: `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`
- Move: `docs/superpowers/plans/2026-06-07-wf-sources-mcp-catalog-dtos-slice.md` to `docs/historical/superpowers/plans/2026-06-07-wf-sources-mcp-catalog-dtos-slice.md`
- [ ] **Step 1: Update roadmap**
In `docs/current_roadmap.md`, under the MCP package split section, add:
```markdown
Third `wf_sources_mcp` slice complete: upstream MCP catalog/discovery DTOs
and catalog snapshot dumping now live in `wf_sources_mcp.catalog`, with
`wf_mcp.capabilities` and `wf_mcp.catalog.models` retained as shims.
```
- [ ] **Step 2: Update long-lived API boundary spec**
In `docs/superpowers/specs/2026-06-03-long-lived-workflow-api-boundary.md`, update the `wf_sources_mcp` status list so it includes:
```markdown
3. Complete: upstream MCP catalog/discovery DTOs moved to `wf_sources_mcp.catalog`, with `wf_mcp.capabilities` and `wf_mcp.catalog.models` retained as shims.
4. Upstream transport/discovery/session services.
```
- [ ] **Step 3: Move completed plan to historical**
Run:
```bash
git mv docs/superpowers/plans/2026-06-07-wf-sources-mcp-catalog-dtos-slice.md docs/historical/superpowers/plans/2026-06-07-wf-sources-mcp-catalog-dtos-slice.md
```
Expected: `git status --short` shows an `R` rename for this plan.
---
### Task 6: Final Verification
**Files:**
- All changed files
- [ ] **Step 1: Run focused extraction tests**
Run:
```bash
uv run pytest tests/wf_sources_mcp tests/wf_mcp/test_compat_imports.py tests/wf_mcp/test_store.py tests/wf_mcp/test_sdk_adapter.py tests/wf_mcp/service/test_catalog.py tests/wf_mcp/service/test_upstream_transport.py tests/wf_mcp/test_workflow_wrappers.py -q
```
Expected: all focused tests pass.
- [ ] **Step 2: Run lint and type checks**
Run:
```bash
uv run ruff check src tests
uv run basedpyright --level error src
```
Expected: Ruff reports `All checks passed!`; basedpyright reports `0 errors`.
- [ ] **Step 3: Run full suite**
Run:
```bash
uv run pytest -q
```
Expected: full suite passes with current skip/xfail counts.
- [ ] **Step 4: Review remaining old DTO imports**
Run:
```bash
rg -n "from wf_mcp\\.capabilities|from wf_mcp\\.catalog\\.models|from wf_mcp\\.models import CatalogSnapshot" src tests
```
Expected: remaining occurrences are compatibility shims, facade exports, or tests intentionally exercising old import paths.
- [ ] **Step 5: Report**
Report:
- files created/modified
- focused/full verification output
- whether `wf_sources_mcp.storage.store` now imports catalog DTOs from `wf_sources_mcp.catalog`
- whether compatibility shims remain
- deviations from this plan
Do not commit unless the user explicitly asks. If committing, use:
```bash
git add -A
git commit -m "refactor: move mcp catalog dtos to wf_sources_mcp"
```
---
## Self-Review
- Spec coverage: covers catalog DTO and snapshot cache dependency cleanup before upstream session/runtime moves.
- Placeholder scan: no `TODO`, `TBD`, or unspecified test steps.
- Type consistency: canonical symbols keep the same names and dataclass field shapes as existing `wf_mcp` DTOs, preserving compatibility.
@@ -90,7 +90,8 @@ First slices should move leaf modules only and leave `wf_mcp` re-export shims:
1. Complete: MCP auth helpers and focused auth/catalog stores moved to
`wf_sources_mcp`, with `wf_mcp` shims preserved.
2. Complete: MCP source registry models/conversion moved to `wf_sources_mcp.source_registry`, with `wf_mcp.source_registry` retained as a shim.
3. Upstream transport/discovery/session services.
3. Complete: upstream MCP catalog/discovery DTOs moved to `wf_sources_mcp.catalog`, with `wf_mcp.capabilities` and `wf_mcp.catalog.models` retained as shims.
4. Upstream transport/discovery/session services.
Each slice should add import-direction tests so the new source-provider package
does not depend on `wf_mcp.workflow_surface`, `wf_mcp.admin_surface`,
+3 -3
View File
@@ -4,16 +4,16 @@ from dataclasses import dataclass, field
from typing import Any
from wf_authoring import NodeCatalog, NodeSpec
from ..capabilities import (
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
DiscoveredPrompt,
DiscoveredResource,
)
from wf_sources_mcp.catalog.models import CatalogSnapshot
from ..connections import qualify_node_name
from ..models import CatalogSnapshot
from ..sdk.converters import workflow_output_schema_from_mcp_tool_schema
+1 -1
View File
@@ -8,9 +8,9 @@ from mcp import McpError
from mcp.types import METHOD_NOT_FOUND
from wf_authoring import NodeSpec
from wf_sources_mcp.catalog import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..auth import AuthRecord
from ..capabilities import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..models import ConnectionConfig
from ..runtime import ToolExecutor
from ..sdk import BackendAdapter
+5 -5
View File
@@ -17,15 +17,15 @@ from wf_core import (
RunState,
Workflow,
)
from wf_mcp.capabilities import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
from wf_platform import (
CapabilitySource,
)
from wf_sources_mcp.auth import AuthRecord
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
from wf_sources_mcp.source_registry import SourceRegistryStore
from wf_sources_mcp.storage import AuthStore, CatalogStore, Store
+1 -1
View File
@@ -4,7 +4,7 @@ from dataclasses import dataclass
from typing import Any, cast
from wf_mcp.events import EventBus, McpEvent, make_event
from wf_mcp.models import CatalogSnapshot
from wf_sources_mcp.catalog.models import CatalogSnapshot
@dataclass(slots=True)
+5 -5
View File
@@ -8,11 +8,6 @@ from typing import Any
from pydantic import BaseModel
from wf_authoring import NodeReturn, NodeSpec
from wf_mcp.capabilities import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
from wf_platform import (
CapabilityBuckets,
CapabilitySource,
@@ -23,6 +18,11 @@ from wf_platform import (
page_items,
)
from wf_sources_mcp.auth import AuthRecord
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
from wf_sources_mcp.storage import CatalogStore
from ...connections import ConnectionConfig, qualify_node_name
@@ -23,11 +23,12 @@ from wf_mcp.broker.discovery import (
specs_from_discovered_tools,
)
from wf_mcp.events import McpEvent, make_event
from wf_mcp.models import CatalogSnapshot, ConnectionConfig
from wf_mcp.models import ConnectionConfig
from wf_mcp.runtime import ToolExecutor
from wf_mcp.sdk import BackendAdapter
from wf_mcp.shared.errors import error_payload
from wf_sources_mcp.auth import AuthRecord, connection_auth_diagnostic
from wf_sources_mcp.catalog.models import CatalogSnapshot
from wf_sources_mcp.storage import AuthStore, CatalogStore
from .adapters import require_adapter
+21 -78
View File
@@ -1,81 +1,24 @@
"""Compatibility shim for MCP upstream catalog entry DTOs.
Canonical implementation lives in `wf_sources_mcp.catalog.entries`.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
from wf_sources_mcp.catalog.entries import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
)
@dataclass(slots=True)
class DiscoveredTool:
"""Tool snapshot after converting from an upstream MCP SDK tool."""
name: str
title: str | None
description: str | None
input_schema: dict[str, Any]
output_schema: dict[str, Any]
outcomes: tuple[str, ...] = ("ok",)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredResource:
"""Resource snapshot after converting from an upstream MCP SDK resource."""
uri: str
name: str
title: str | None
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredPrompt:
"""Prompt snapshot after converting from an upstream MCP SDK prompt."""
name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogNodeEntry:
"""Namespaced tool entry stored in the broker catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
outcomes: tuple[str, ...]
input_schema: dict[str, Any]
output_schema: dict[str, Any]
@dataclass(slots=True)
class CatalogResourceEntry:
"""Namespaced resource entry stored in the broker catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
uri: str
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogPromptEntry:
"""Namespaced prompt entry stored in the broker catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
]
+2
View File
@@ -1,3 +1,5 @@
from __future__ import annotations
from .models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
+6 -36
View File
@@ -1,41 +1,11 @@
"""Compatibility shim for MCP upstream catalog snapshot DTOs.
Canonical implementation lives in `wf_sources_mcp.catalog.models`.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from typing import Any
from wf_mcp.capabilities import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
@dataclass(slots=True)
class CatalogSnapshot:
connection_id: str
fetched_at_epoch_ms: int
max_age_seconds: int
nodes: list[CatalogNodeEntry] = field(default_factory=list)
resources: list[CatalogResourceEntry] = field(default_factory=list)
prompts: list[CatalogPromptEntry] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
def is_stale(self, now_epoch_ms: int) -> bool:
age_ms = now_epoch_ms - self.fetched_at_epoch_ms
return age_ms > self.max_age_seconds * 1000
def dump_catalog_snapshot(snapshot: CatalogSnapshot) -> dict[str, Any]:
return {
"connection_id": snapshot.connection_id,
"fetched_at_epoch_ms": snapshot.fetched_at_epoch_ms,
"max_age_seconds": snapshot.max_age_seconds,
"nodes": [asdict(node) for node in snapshot.nodes],
"resources": [asdict(resource) for resource in snapshot.resources],
"prompts": [asdict(prompt) for prompt in snapshot.prompts],
"metadata": snapshot.metadata,
}
from wf_sources_mcp.catalog.models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"CatalogSnapshot",
+1 -1
View File
@@ -8,7 +8,7 @@ from wf_mcp.broker.models import (
ConnectionConfig,
SourceConfigOwnership,
)
from wf_mcp.catalog.models import CatalogSnapshot, dump_catalog_snapshot
from wf_sources_mcp.catalog.models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"AuthRecord",
+1 -1
View File
@@ -18,8 +18,8 @@ from mcp.types import (
from pydantic import AnyUrl
from wf_sources_mcp.auth import AuthRecord, mcp_auth_env, mcp_auth_headers
from wf_sources_mcp.catalog import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..capabilities import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..models import ConnectionConfig
from .base import BackendAdapter, ToolCallResult
from .converters import (
+2 -1
View File
@@ -3,8 +3,9 @@ from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Protocol
from wf_sources_mcp.catalog import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..auth import AuthRecord
from ..capabilities import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from ..models import ConnectionConfig
+2 -1
View File
@@ -7,7 +7,8 @@ from mcp.types import Prompt as McpPrompt
from mcp.types import Resource as McpResource
from mcp.types import Tool as McpTool
from ..capabilities import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from wf_sources_mcp.catalog import DiscoveredPrompt, DiscoveredResource, DiscoveredTool
from .base import ToolCallResult
+1 -1
View File
@@ -9,9 +9,9 @@ from pydantic import BaseModel, ConfigDict, Field, create_model
from wf_authoring import NodeReturn, NodeSpec
from wf_core import RuntimeContext
from wf_mcp.broker.events import McpEvent, make_event
from wf_sources_mcp.catalog import DiscoveredTool
from ..auth import AuthRecord
from ..capabilities import DiscoveredTool
from ..models import ConnectionConfig
from ..runtime import ToolExecutor
+22
View File
@@ -0,0 +1,22 @@
from __future__ import annotations
from .entries import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
)
from .models import CatalogSnapshot, dump_catalog_snapshot
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"CatalogSnapshot",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
"dump_catalog_snapshot",
]
+91
View File
@@ -0,0 +1,91 @@
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
@dataclass(slots=True)
class DiscoveredTool:
"""Tool snapshot after converting from an upstream MCP SDK tool."""
name: str
title: str | None
description: str | None
input_schema: dict[str, Any]
output_schema: dict[str, Any]
outcomes: tuple[str, ...] = ("ok",)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredResource:
"""Resource snapshot after converting from an upstream MCP SDK resource."""
uri: str
name: str
title: str | None
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class DiscoveredPrompt:
"""Prompt snapshot after converting from an upstream MCP SDK prompt."""
name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogNodeEntry:
"""Namespaced tool entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
outcomes: tuple[str, ...]
input_schema: dict[str, Any]
output_schema: dict[str, Any]
@dataclass(slots=True)
class CatalogResourceEntry:
"""Namespaced resource entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
uri: str
description: str | None
mime_type: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(slots=True)
class CatalogPromptEntry:
"""Namespaced prompt entry stored in an MCP upstream catalog snapshot."""
qualified_name: str
connection_id: str
local_name: str
title: str | None
description: str | None
arguments: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
__all__ = [
"CatalogNodeEntry",
"CatalogPromptEntry",
"CatalogResourceEntry",
"DiscoveredPrompt",
"DiscoveredResource",
"DiscoveredTool",
]
+41
View File
@@ -0,0 +1,41 @@
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from typing import Any
from .entries import CatalogNodeEntry, CatalogPromptEntry, CatalogResourceEntry
@dataclass(slots=True)
class CatalogSnapshot:
"""Stored upstream MCP catalog snapshot for one source connection."""
connection_id: str
fetched_at_epoch_ms: int
max_age_seconds: int
nodes: list[CatalogNodeEntry] = field(default_factory=list)
resources: list[CatalogResourceEntry] = field(default_factory=list)
prompts: list[CatalogPromptEntry] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
def is_stale(self, now_epoch_ms: int) -> bool:
age_ms = now_epoch_ms - self.fetched_at_epoch_ms
return age_ms > self.max_age_seconds * 1000
def dump_catalog_snapshot(snapshot: CatalogSnapshot) -> dict[str, Any]:
return {
"connection_id": snapshot.connection_id,
"fetched_at_epoch_ms": snapshot.fetched_at_epoch_ms,
"max_age_seconds": snapshot.max_age_seconds,
"nodes": [asdict(node) for node in snapshot.nodes],
"resources": [asdict(resource) for resource in snapshot.resources],
"prompts": [asdict(prompt) for prompt in snapshot.prompts],
"metadata": snapshot.metadata,
}
__all__ = [
"CatalogSnapshot",
"dump_catalog_snapshot",
]
+7 -6
View File
@@ -1,8 +1,7 @@
"""MCP upstream-source auth and catalog file stores.
These stores preserve the current MCP compatibility JSON shapes. Catalog entry
types still come from `wf_mcp` until catalog DTOs finish moving to a neutral or
source-provider package.
types come from `wf_sources_mcp.catalog`.
"""
from __future__ import annotations
@@ -16,7 +15,7 @@ from wf_api.auth import validate_auth_id
from wf_sources_mcp.auth import AuthRecord, mcp_auth_from_neutral, neutral_auth_from_mcp
if TYPE_CHECKING:
from wf_mcp.catalog.models import CatalogSnapshot
from wf_sources_mcp.catalog.models import CatalogSnapshot
class AuthStore:
@@ -142,7 +141,7 @@ class FileCatalogStore(CatalogStore):
return path
def save_catalog(self, snapshot: CatalogSnapshot) -> None:
from wf_mcp.catalog.models import dump_catalog_snapshot
from wf_sources_mcp.catalog.models import dump_catalog_snapshot
self._catalog_path(snapshot.connection_id).write_text(
json.dumps(dump_catalog_snapshot(snapshot), indent=2),
@@ -150,12 +149,14 @@ class FileCatalogStore(CatalogStore):
)
def load_catalog(self, connection_id: str) -> CatalogSnapshot | None:
from wf_mcp.capabilities import (
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
)
from wf_mcp.catalog.models import CatalogSnapshot as CatalogSnapshotType
from wf_sources_mcp.catalog import (
CatalogSnapshot as CatalogSnapshotType,
)
path = self._catalog_path(connection_id)
if not path.exists():
+18
View File
@@ -84,3 +84,21 @@ def test_wf_mcp_source_registry_shim_reexports_wf_sources_mcp_registry() -> None
assert CompatFileStore is FileSourceRegistryStore
assert CompatEntry is McpSourceRegistryEntry
assert CompatFile is SourceRegistryFile
def test_wf_mcp_capabilities_shim_reexports_wf_sources_mcp_catalog_entries() -> None:
from wf_mcp.capabilities import CatalogNodeEntry as CompatCatalogNodeEntry
from wf_mcp.capabilities import DiscoveredTool as CompatDiscoveredTool
from wf_sources_mcp.catalog import CatalogNodeEntry, DiscoveredTool
assert CompatCatalogNodeEntry is CatalogNodeEntry
assert CompatDiscoveredTool is DiscoveredTool
def test_wf_mcp_catalog_models_shim_reexports_wf_sources_mcp_catalog_models() -> None:
from wf_mcp.catalog.models import CatalogSnapshot as CompatCatalogSnapshot
from wf_mcp.catalog.models import dump_catalog_snapshot as compat_dump
from wf_sources_mcp.catalog import CatalogSnapshot, dump_catalog_snapshot
assert CompatCatalogSnapshot is CatalogSnapshot
assert compat_dump is dump_catalog_snapshot
@@ -1,7 +1,6 @@
from __future__ import annotations
from wf_api.auth import AuthRecord as NeutralAuthRecord
from wf_mcp.catalog.models import CatalogSnapshot
from wf_sources_mcp.auth import (
AuthRecord,
mcp_auth_env,
@@ -9,6 +8,7 @@ from wf_sources_mcp.auth import (
mcp_auth_headers,
neutral_auth_from_mcp,
)
from wf_sources_mcp.catalog import CatalogSnapshot
from wf_sources_mcp.storage import FileAuthStore, FileCatalogStore, FileStore
+93
View File
@@ -0,0 +1,93 @@
from __future__ import annotations
from wf_sources_mcp.catalog import (
CatalogNodeEntry,
CatalogPromptEntry,
CatalogResourceEntry,
CatalogSnapshot,
DiscoveredPrompt,
DiscoveredResource,
DiscoveredTool,
dump_catalog_snapshot,
)
def test_discovered_tool_default_outcome_and_metadata() -> None:
tool = DiscoveredTool(
name="echo",
title=None,
description="Echo input",
input_schema={"type": "object"},
output_schema={"type": "object"},
)
assert tool.outcomes == ("ok",)
assert tool.metadata == {}
def test_discovered_resource_and_prompt_keep_structural_fields() -> None:
resource = DiscoveredResource(
uri="docs://guide",
name="guide",
title="Guide",
description="Read me",
mime_type="text/markdown",
)
prompt = DiscoveredPrompt(
name="summarize",
title=None,
description="Summarize",
arguments=[{"name": "topic"}],
)
assert resource.uri == "docs://guide"
assert resource.mime_type == "text/markdown"
assert prompt.arguments == [{"name": "topic"}]
def test_catalog_snapshot_staleness_and_dump_shape() -> None:
snapshot = CatalogSnapshot(
connection_id="demo.default",
fetched_at_epoch_ms=1_000,
max_age_seconds=2,
nodes=[
CatalogNodeEntry(
qualified_name="demo.default.echo",
connection_id="demo.default",
local_name="echo",
title=None,
description="Echo",
outcomes=("ok",),
input_schema={"type": "object"},
output_schema={"type": "object"},
)
],
resources=[
CatalogResourceEntry(
qualified_name="demo.default.guide",
connection_id="demo.default",
local_name="guide",
title=None,
uri="docs://guide",
description="Guide",
)
],
prompts=[
CatalogPromptEntry(
qualified_name="demo.default.summarize",
connection_id="demo.default",
local_name="summarize",
title=None,
description="Summarize",
)
],
metadata={"source": "test"},
)
assert snapshot.is_stale(3_001) is True
dumped = dump_catalog_snapshot(snapshot)
assert dumped["connection_id"] == "demo.default"
assert dumped["nodes"][0]["qualified_name"] == "demo.default.echo"
assert dumped["resources"][0]["uri"] == "docs://guide"
assert dumped["prompts"][0]["local_name"] == "summarize"
assert dumped["metadata"] == {"source": "test"}
@@ -4,8 +4,9 @@ import ast
from pathlib import Path
# Temporary low-level wf_mcp imports are allowed for connection id parsing,
# reserved names, and broker DTO conversion. Frontend/proxy/workflow-surface
# imports are forbidden because wf_sources_mcp is upstream-source code.
# reserved names, and broker DTO conversion. Catalog DTOs should now be local
# to wf_sources_mcp. Frontend/proxy/workflow-surface imports are forbidden
# because wf_sources_mcp is upstream-source code.
FORBIDDEN_WF_MCP_PREFIXES = (
"wf_mcp.admin_surface",
"wf_mcp.workflow_surface",
@@ -40,3 +41,34 @@ def test_wf_sources_mcp_does_not_import_frontend_mcp_modules() -> None:
"wf_sources_mcp imports frontend/proxy MCP modules:\n"
+ "\n".join(f" {violation}" for violation in violations)
)
def test_wf_sources_mcp_does_not_import_wf_mcp_catalog_dtos() -> None:
root = Path(__file__).resolve().parents[2] / "src" / "wf_sources_mcp"
forbidden = {
"wf_mcp.capabilities",
"wf_mcp.catalog",
"wf_mcp.catalog.models",
}
violations: list[str] = []
for py_file in sorted(root.rglob("*.py")):
rel = py_file.relative_to(root.parent)
module = str(rel.with_suffix("")).replace("/", ".").replace("\\", ".")
tree = ast.parse(py_file.read_text(encoding="utf-8"), filename=str(py_file))
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module in forbidden:
violations.append(
f"{module}:{node.lineno}: from {node.module} import ..."
)
elif isinstance(node, ast.Import):
for alias in node.names:
if alias.name in forbidden:
violations.append(
f"{module}:{node.lineno}: import {alias.name}"
)
assert violations == [], (
"wf_sources_mcp still imports old wf_mcp catalog DTO modules:\n"
+ "\n".join(f" {violation}" for violation in violations)
)