docs: plan source registry next slices
This commit is contained in:
@@ -123,6 +123,11 @@ implementation state.
|
||||
- First source registry implementation slice complete: validated registry
|
||||
models plus `FileSourceRegistryStore` exist, but startup merge and mutation
|
||||
commands are still deferred.
|
||||
- Next source registry slices are planned in
|
||||
[2026-06-03 source registry next slices](./superpowers/plans/2026-06-03-source-registry-next-slices.md):
|
||||
first split generic registry mechanics into `wf_api`, then add MCP entry
|
||||
conversion, startup merge, desired-registry admin reads, and finally safe
|
||||
mutation commands.
|
||||
- Longer term: make the MCP frontend an adapter over these neutral workflow,
|
||||
source-admin, and config-admin surfaces so the old `wf_mcp` server entry
|
||||
point can shrink or retire.
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
# Source Registry Next Slices Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Move the source registry toward a generic platform/API boundary, then wire startup merge and finally mutation without locking MCP-specific assumptions into `wf_api`.
|
||||
|
||||
**Architecture:** Split generic registry mechanics from MCP-specific source entries. `wf_api` should own the generic registry file/store concepts once they no longer import MCP validators. `wf_mcp` should own MCP source entry models and conversion into `ConnectionConfig` / broker services. Startup merge happens after the split so config-vs-store precedence is implemented against the right abstractions.
|
||||
|
||||
**Tech Stack:** Python 3.14, Pydantic v2, existing `wf_mcp.source_registry`, `wf_api`, `wf_config`, `WfMcpService`, pytest, ruff, basedpyright.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
Slice 1 created `src/wf_mcp/source_registry.py` with:
|
||||
|
||||
- `SourceRegistryModel`
|
||||
- `StdioSourceTransport`
|
||||
- `HttpSourceTransport`
|
||||
- `McpSourceRegistryEntry`
|
||||
- `SourceRegistryFile`
|
||||
- `SourceRegistryStore`
|
||||
- `FileSourceRegistryStore`
|
||||
|
||||
This is useful and tested, but still MCP-shaped:
|
||||
|
||||
- id validation uses `wf_mcp.connections.parse_connection_id`
|
||||
- reserved ids come from `wf_mcp.shared.names`
|
||||
- source entry type is `McpSourceRegistryEntry`
|
||||
- transport definitions are MCP transports
|
||||
|
||||
That is acceptable for Slice 1, but should be resolved before startup merge and mutation commands make the shape more permanent.
|
||||
|
||||
## Slice Order
|
||||
|
||||
1. **Slice 2A: Generic Registry Mechanics**
|
||||
- Move generic validation/store/file mechanics to `wf_api`.
|
||||
- Keep MCP entry/transport validation in `wf_mcp`.
|
||||
- Prefer boring helpers over deep Pydantic generics where that keeps the
|
||||
boundary easier to type-check.
|
||||
- Do not change runtime behavior.
|
||||
|
||||
2. **Slice 2B: MCP Entry Conversion**
|
||||
- Add conversion helpers between `McpSourceRegistryEntry` and
|
||||
`ConnectionConfig`.
|
||||
- Keep config merge out of scope.
|
||||
|
||||
3. **Slice 3: Startup Merge**
|
||||
- Load registry at broker/server startup.
|
||||
- Merge config and registry with config precedence.
|
||||
- Emit events/diagnostics for shadowed registry entries.
|
||||
|
||||
4. **Slice 4: Read Desired Registry Through Admin**
|
||||
- Expose desired registry entries separately from observed source inventory.
|
||||
- Make the difference visible in API/CLI docs.
|
||||
|
||||
5. **Slice 5: Mutation Commands**
|
||||
- Add add/update/enable/disable/remove operations.
|
||||
- Use registry store, validation, and atomic writes.
|
||||
- Keep auth/catalog cleanup deferred.
|
||||
|
||||
---
|
||||
|
||||
## Slice 2A: Generic Registry Mechanics
|
||||
|
||||
### Goal
|
||||
|
||||
Move generic registry store mechanics out of `wf_mcp` without pretending MCP
|
||||
source entries are generic.
|
||||
|
||||
### Target Shape
|
||||
|
||||
Create `src/wf_api/source_registry.py` with protocol-neutral mechanics only:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Generic, Protocol, TypeVar
|
||||
from uuid import uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
|
||||
class SourceRegistryBaseModel(BaseModel):
|
||||
"""Base model for persisted source registry state; reject misspelled fields."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
SOURCE_REGISTRY_ID_PATTERN = r"^[A-Za-z0-9_][A-Za-z0-9_.-]*$"
|
||||
|
||||
|
||||
def validate_source_registry_id(value: str) -> str:
|
||||
"""Validate ids that are safe as registry keys and filesystem path segments.
|
||||
|
||||
This helper intentionally does not parse provider/account meaning. MCP can
|
||||
layer stricter `parse_connection_id` validation on top while other future
|
||||
source families can reuse the safe-id rule.
|
||||
"""
|
||||
|
||||
if not re.fullmatch(SOURCE_REGISTRY_ID_PATTERN, value):
|
||||
raise ValueError(
|
||||
"source id must start with alphanumeric or underscore and contain "
|
||||
"only [A-Za-z0-9_.-]"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def validate_unique_source_ids(entries: list[object]) -> None:
|
||||
"""Reject duplicate `id` fields without owning the entry model shape."""
|
||||
|
||||
seen: set[str] = set()
|
||||
for entry in entries:
|
||||
source_id = getattr(entry, "id", None)
|
||||
if not isinstance(source_id, str):
|
||||
raise ValueError("source registry entries must expose string id")
|
||||
if source_id in seen:
|
||||
raise ValueError(f"duplicate source id {source_id!r}")
|
||||
seen.add(source_id)
|
||||
|
||||
|
||||
RegistryT = TypeVar("RegistryT", bound=BaseModel)
|
||||
|
||||
|
||||
class SourceRegistryStore(Protocol[RegistryT]):
|
||||
def load_registry(self) -> RegistryT: ...
|
||||
|
||||
def save_registry(self, registry: RegistryT) -> None: ...
|
||||
|
||||
|
||||
class AtomicJsonRegistryStore(Generic[RegistryT]):
|
||||
"""Filesystem implementation for small desired-registry documents."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
root: Path,
|
||||
*,
|
||||
filename: str,
|
||||
registry_type: type[RegistryT],
|
||||
empty_factory: Callable[[], RegistryT],
|
||||
corrupt_label: str,
|
||||
) -> None:
|
||||
self.root = root
|
||||
self.filename = filename
|
||||
self.registry_type = registry_type
|
||||
self.empty_factory = empty_factory
|
||||
self.corrupt_label = corrupt_label
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
return self.root / self.filename
|
||||
|
||||
def load_registry(self) -> RegistryT:
|
||||
if not self.path.exists():
|
||||
return self.empty_factory()
|
||||
try:
|
||||
data = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"{self.corrupt_label} is corrupted: {self.path}") from exc
|
||||
return self.registry_type.model_validate(data)
|
||||
|
||||
def save_registry(self, registry: RegistryT) -> None:
|
||||
validated = self.registry_type.model_validate(registry.model_dump(mode="json"))
|
||||
payload = json.dumps(validated.model_dump(mode="json"), indent=2)
|
||||
tmp_path = self.path.with_name(f"{self.path.name}.{uuid4().hex}.tmp")
|
||||
tmp_path.write_text(payload, encoding="utf-8")
|
||||
tmp_path.replace(self.path)
|
||||
```
|
||||
|
||||
Then `src/wf_mcp/source_registry.py` should import these generic pieces and keep:
|
||||
|
||||
- `StdioSourceTransport`
|
||||
- `HttpSourceTransport`
|
||||
- `SourceTransport`
|
||||
- `McpSourceRegistryEntry`
|
||||
- `SourceRegistryFile` or `McpSourceRegistryFile` as the MCP registry document
|
||||
model, using `validate_unique_source_ids(self.sources)`
|
||||
- `FileSourceRegistryStore` as the MCP concrete store wrapper around
|
||||
`AtomicJsonRegistryStore[SourceRegistryFile]`
|
||||
|
||||
### Tests
|
||||
|
||||
Move generic tests to `tests/wf_api/test_source_registry.py`:
|
||||
|
||||
- safe id validation accepts normal registry ids and rejects unsafe ids
|
||||
- duplicate id helper rejects repeated ids using a fake entry object
|
||||
- missing file returns empty generic registry
|
||||
- save/load round trip
|
||||
- corrupted JSON wraps as `ValueError`
|
||||
|
||||
Keep MCP tests in `tests/wf_mcp/test_source_registry.py`:
|
||||
|
||||
- MCP id validation
|
||||
- reserved id rejection
|
||||
- stdio/http transport models
|
||||
- `McpFileSourceRegistryStore` round trip
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- `wf_api.source_registry` imports no `wf_mcp`.
|
||||
- Existing `wf_mcp.source_registry` public behavior remains compatible.
|
||||
- Tests pass.
|
||||
- No startup/runtime behavior changes.
|
||||
|
||||
---
|
||||
|
||||
## Slice 2B: MCP Entry Conversion
|
||||
|
||||
### Goal
|
||||
|
||||
Add explicit conversion helpers so startup merge can convert registry entries
|
||||
into broker connection configs without duplicating field logic.
|
||||
|
||||
### New Helpers
|
||||
|
||||
In `src/wf_mcp/source_registry.py`:
|
||||
|
||||
```python
|
||||
def registry_entry_to_connection_config(
|
||||
entry: McpSourceRegistryEntry,
|
||||
) -> ConnectionConfig:
|
||||
return ConnectionConfig(
|
||||
id=entry.id,
|
||||
server=entry.provider,
|
||||
account=entry.account,
|
||||
enabled=entry.enabled,
|
||||
metadata={
|
||||
"profile": entry.profile,
|
||||
"transport": entry.transport.model_dump(mode="json"),
|
||||
"source_registry": True,
|
||||
**entry.metadata,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
And optionally:
|
||||
|
||||
```python
|
||||
def connection_config_to_registry_entry(
|
||||
connection: ConnectionConfig,
|
||||
) -> McpSourceRegistryEntry | None:
|
||||
...
|
||||
```
|
||||
|
||||
Only add reverse conversion if an implementation needs it. Do not guess unknown
|
||||
transport metadata.
|
||||
|
||||
### Tests
|
||||
|
||||
- registry entry converts to `ConnectionConfig`
|
||||
- provider/account/profile/transport metadata are preserved
|
||||
- disabled entry creates disabled connection config
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Conversion is explicit and tested.
|
||||
- No startup/runtime behavior changes yet.
|
||||
|
||||
---
|
||||
|
||||
## Slice 3: Startup Merge
|
||||
|
||||
### Goal
|
||||
|
||||
Load desired dynamic registry state during service construction and merge it
|
||||
with config-defined connections/sources.
|
||||
|
||||
### Merge Rules
|
||||
|
||||
1. Built-in reserved ids always win.
|
||||
2. Config-defined entries win over registry entries with the same id.
|
||||
3. Registry entries fill ids not present in config.
|
||||
|
||||
### Implementation Direction
|
||||
|
||||
- Broker config construction should create a `McpFileSourceRegistryStore`.
|
||||
- `WfMcpService` or `ConnectionService` should accept an optional registry store.
|
||||
- On startup/reload:
|
||||
- load config connections
|
||||
- load registry entries
|
||||
- convert registry entries to `ConnectionConfig`
|
||||
- merge with config precedence
|
||||
- register merged connections
|
||||
- emit an event for registry entries shadowed by config
|
||||
|
||||
### Tests
|
||||
|
||||
- absent registry preserves current config-only behavior
|
||||
- registry-only connection appears after service construction
|
||||
- config shadows same-id registry entry
|
||||
- invalid registry fails startup clearly
|
||||
- disabled registry source hydrates disabled
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Existing legacy config behavior is unchanged when no registry file exists.
|
||||
- Dynamic registry entries persist across service recreation.
|
||||
- Shadowed entries do not silently override config.
|
||||
|
||||
---
|
||||
|
||||
## Slice 4: Read Desired Registry Through Admin
|
||||
|
||||
### Goal
|
||||
|
||||
Expose desired registry entries distinctly from observed source inventory.
|
||||
|
||||
### Why
|
||||
|
||||
`wf source list` currently reports runtime/observed source inventory. Registry
|
||||
entries are desired server-owned configuration. Users need both views when a
|
||||
source exists in registry but is disabled, shadowed, or not hydrated.
|
||||
|
||||
### Candidate Commands
|
||||
|
||||
- `wf admin registry list`
|
||||
- `wf admin registry inspect SOURCE_ID`
|
||||
|
||||
or:
|
||||
|
||||
- `wf source registry list`
|
||||
- `wf source registry inspect SOURCE_ID`
|
||||
|
||||
Pick one naming shape in the implementation plan.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Desired registry view is not confused with observed source inventory.
|
||||
- Shadowed/disabled state is visible.
|
||||
- No mutation yet.
|
||||
|
||||
---
|
||||
|
||||
## Slice 5: Mutation Commands
|
||||
|
||||
### Goal
|
||||
|
||||
Add safe registry mutation.
|
||||
|
||||
### Operations
|
||||
|
||||
- add source
|
||||
- update source
|
||||
- enable source
|
||||
- disable source
|
||||
- remove source
|
||||
|
||||
### Rules
|
||||
|
||||
- Validate full registry before commit.
|
||||
- Write atomically.
|
||||
- Do not mutate config files.
|
||||
- Do not delete auth/catalog files in v1.
|
||||
- Prefer disable over remove for sources referenced by deployments.
|
||||
- Optional live validation can be a flag, not required for save.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- RPC methods exist.
|
||||
- CLI commands exist.
|
||||
- Mutations persist across process restart.
|
||||
- Validation errors are actionable.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- This is a multi-slice roadmap, not a single execution plan for all mutation work.
|
||||
- Slice 2A is the immediate next implementation target and resolves the location problem.
|
||||
- Startup merge is intentionally after generic/MCP split and conversion helpers.
|
||||
- No mutation commands are planned before persistence and merge rules are implemented.
|
||||
@@ -229,20 +229,44 @@ of bloating workflow lifecycle APIs.
|
||||
- Atomic write for filesystem store.
|
||||
- No runtime wiring yet.
|
||||
|
||||
### Slice 2: Startup Merge
|
||||
Status: complete. The first implementation lives in `wf_mcp.source_registry`
|
||||
because the entry shape is MCP-specific: connection-id validation, reserved
|
||||
source ids, and concrete MCP transports all come from the MCP broker layer.
|
||||
|
||||
### Slice 2A: Generic Registry Mechanics
|
||||
|
||||
- Move protocol-neutral registry mechanics to `wf_api.source_registry`.
|
||||
- Keep MCP source entries, MCP transports, connection-id parsing, and reserved
|
||||
id rules in `wf_mcp.source_registry`.
|
||||
- Generic mechanics include safe registry-id validation, duplicate-id checking,
|
||||
a registry-store protocol, and a small atomic JSON model store.
|
||||
- Do not wire registry state into startup yet.
|
||||
|
||||
This resolves the current location tension without pretending an MCP source
|
||||
entry is a generic workflow source. Future non-MCP source families should reuse
|
||||
the generic mechanics and define their own entry models.
|
||||
|
||||
### Slice 2B: MCP Entry Conversion
|
||||
|
||||
- Add explicit conversion from `McpSourceRegistryEntry` to `ConnectionConfig`.
|
||||
- Preserve structural fields such as provider, account, profile, transport,
|
||||
enabled state, auth reference, and metadata.
|
||||
- Do not merge config and registry yet.
|
||||
|
||||
### Slice 3: Startup Merge
|
||||
|
||||
- Load registry during server/broker construction.
|
||||
- Merge config + registry deterministically.
|
||||
- Emit diagnostics/events for ignored shadowed entries.
|
||||
- Preserve existing config-only behavior when registry file is absent.
|
||||
|
||||
### Slice 3: Read Registry Through Admin
|
||||
### Slice 4: Read Registry Through Admin
|
||||
|
||||
- Add admin read method for desired registry entries if needed.
|
||||
- Keep current source inventory list as runtime/observed source inventory.
|
||||
- Document difference between desired registry and observed source catalog.
|
||||
|
||||
### Slice 4: Mutating RPC/CLI
|
||||
### Slice 5: Mutating RPC/CLI
|
||||
|
||||
- Add add/update/enable/disable/remove operations.
|
||||
- Add JSON-RPC methods.
|
||||
@@ -263,9 +287,10 @@ of bloating workflow lifecycle APIs.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Implement Slice 1 first. Keep it independent from runtime startup so the model
|
||||
and file persistence rules become solid before they affect source hydration.
|
||||
Implement Slice 2A next. Keep it independent from runtime startup so the
|
||||
generic/MCP split becomes solid before it affects source hydration.
|
||||
|
||||
The existing `wf_mcp` store can host the first file-backed implementation, but
|
||||
the interface should be neutral enough to move later. Catalog/auth remain
|
||||
observed/secret state; the new registry is desired source configuration state.
|
||||
Catalog/auth remain observed/secret state; the new registry is desired source
|
||||
configuration state. Generic registry mechanics can live in `wf_api`, but
|
||||
provider-specific entries and conversion into broker runtime connections stay
|
||||
with the package that owns that provider.
|
||||
|
||||
Reference in New Issue
Block a user