lda-wf/docs/historical/superpowers/plans/2026-06-03-source-registry-next-slices.md

# Source Registry Next Slices Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Move the source registry toward a generic platform/API boundary, then wire startup merge and finally mutation without locking MCP-specific assumptions into `wf_api`.

**Architecture:** Split generic registry mechanics from MCP-specific source entries. `wf_api` should own the generic registry file/store concepts once they no longer import MCP validators. `wf_mcp` should own MCP source entry models and conversion into `ConnectionConfig` / broker services. Startup merge happens after the split so config-vs-store precedence is implemented against the right abstractions.

**Tech Stack:** Python 3.14, Pydantic v2, existing `wf_mcp.source_registry`, `wf_api`, `wf_config`, `WfMcpService`, pytest, ruff, basedpyright.

---

## Current State

Slice 1 created `src/wf_mcp/source_registry.py` with:

- `SourceRegistryModel`
- `StdioSourceTransport`
- `HttpSourceTransport`
- `McpSourceRegistryEntry`
- `SourceRegistryFile`
- `SourceRegistryStore`
- `FileSourceRegistryStore`

This is useful and tested, but still MCP-shaped:

- id validation uses `wf_mcp.connections.parse_connection_id`
- reserved ids come from `wf_mcp.shared.names`
- source entry type is `McpSourceRegistryEntry`
- transport definitions are MCP transports

Slice 2A then moved generic registry mechanics to `wf_api.source_registry`,
while `wf_mcp.source_registry` kept MCP-specific entries and transports. Slice
2B added `registry_entry_to_connection_config()`.

The next executable slice is startup merge:
`docs/superpowers/plans/2026-06-03-source-registry-startup-merge.md`.

## Slice Order

1. **Slice 2A: Generic Registry Mechanics**
   - **Status: complete.**
   - Move generic validation/store/file mechanics to `wf_api`.
   - Keep MCP entry/transport validation in `wf_mcp`.
   - Prefer boring helpers over deep Pydantic generics where that keeps the
     boundary easier to type-check.
   - Do not change runtime behavior.

2. **Slice 2B: MCP Entry Conversion**
   - **Status: complete.**
   - Add conversion helpers between `McpSourceRegistryEntry` and
     `ConnectionConfig`.
   - Keep config merge out of scope.

3. **Slice 3: Startup Merge**
   - **Status: complete.**
   - Load registry at broker/server startup.
   - Merge config and registry with config precedence.
   - Emit events/diagnostics for shadowed registry entries.

4. **Slice 4: Read Desired Registry Through Admin**
   - **Status: complete.**
   - Expose desired registry entries separately from observed source inventory.
   - `WorkflowSourceRegistryApi` provides neutral read-only access.
   - JSON-RPC methods `workflow.admin.source_registry.list` / `.inspect`.
   - CLI commands `wf admin registry list` / `wf admin registry inspect`.
   - Local/static servers report unavailable instead of empty.
   - Concrete MCP-backed `WorkflowServer` construction remains future work.

5. **Slice 5: Mutation Commands**
   - **Status: complete.**
   - Add add/update/enable/disable/remove operations.
   - Use registry store, validation, and atomic writes.
   - Keep auth/catalog cleanup deferred.
   - JSON-RPC/CLI calls work for targets that expose the registry-admin surface;
     local/static servers report unavailable and concrete MCP-backed
     `WorkflowServer` construction remains future work.

6. **Slice 6: Config Ownership Policy**
   - **Status: complete.**
   - Replace implicit config-shadowing with explicit `locked` / `seed`
     ownership policy.
   - `locked` config entries remain operator-owned and shadow/reject registry
     mutation for the same id.
   - `seed` config entries bootstrap missing store entries, then the store owns
     later admin changes.
   - Update startup merge diagnostics and registry admin payloads so users can
     see why a source is mutable or shadowed.

---

## Slice 2A: Generic Registry Mechanics

### Goal

Move generic registry store mechanics out of `wf_mcp` without pretending MCP
source entries are generic.

### Target Shape

Create `src/wf_api/source_registry.py` with protocol-neutral mechanics only:

```python
from __future__ import annotations

import json
import re
from collections.abc import Callable
from pathlib import Path
from typing import Generic, Protocol, TypeVar
from uuid import uuid4

from pydantic import BaseModel, ConfigDict


class SourceRegistryBaseModel(BaseModel):
    """Base model for persisted source registry state; reject misspelled fields."""

    model_config = ConfigDict(extra="forbid")


SOURCE_REGISTRY_ID_PATTERN = r"^[A-Za-z0-9_][A-Za-z0-9_.-]*$"


def validate_source_registry_id(value: str) -> str:
    """Validate ids that are safe as registry keys and filesystem path segments.

    This helper intentionally does not parse provider/account meaning. MCP can
    layer stricter `parse_connection_id` validation on top while other future
    source families can reuse the safe-id rule.
    """

    if not re.fullmatch(SOURCE_REGISTRY_ID_PATTERN, value):
        raise ValueError(
            "source id must start with alphanumeric or underscore and contain "
            "only [A-Za-z0-9_.-]"
        )
    return value


def validate_unique_source_ids(entries: list[object]) -> None:
    """Reject duplicate `id` fields without owning the entry model shape."""

    seen: set[str] = set()
    for entry in entries:
        source_id = getattr(entry, "id", None)
        if not isinstance(source_id, str):
            raise ValueError("source registry entries must expose string id")
        if source_id in seen:
            raise ValueError(f"duplicate source id {source_id!r}")
        seen.add(source_id)


RegistryT = TypeVar("RegistryT", bound=BaseModel)


class SourceRegistryStore(Protocol[RegistryT]):
    def load_registry(self) -> RegistryT: ...

    def save_registry(self, registry: RegistryT) -> None: ...


class AtomicJsonRegistryStore(Generic[RegistryT]):
    """Filesystem implementation for small desired-registry documents."""

    def __init__(
        self,
        root: Path,
        *,
        filename: str,
        registry_type: type[RegistryT],
        empty_factory: Callable[[], RegistryT],
        corrupt_label: str,
    ) -> None:
        self.root = root
        self.filename = filename
        self.registry_type = registry_type
        self.empty_factory = empty_factory
        self.corrupt_label = corrupt_label
        self.root.mkdir(parents=True, exist_ok=True)

    @property
    def path(self) -> Path:
        return self.root / self.filename

    def load_registry(self) -> RegistryT:
        if not self.path.exists():
            return self.empty_factory()
        try:
            data = json.loads(self.path.read_text(encoding="utf-8"))
        except json.JSONDecodeError as exc:
            raise ValueError(f"{self.corrupt_label} is corrupted: {self.path}") from exc
        return self.registry_type.model_validate(data)

    def save_registry(self, registry: RegistryT) -> None:
        validated = self.registry_type.model_validate(registry.model_dump(mode="json"))
        payload = json.dumps(validated.model_dump(mode="json"), indent=2)
        tmp_path = self.path.with_name(f"{self.path.name}.{uuid4().hex}.tmp")
        tmp_path.write_text(payload, encoding="utf-8")
        tmp_path.replace(self.path)
```

Then `src/wf_mcp/source_registry.py` should import these generic pieces and keep:

- `StdioSourceTransport`
- `HttpSourceTransport`
- `SourceTransport`
- `McpSourceRegistryEntry`
- `SourceRegistryFile` or `McpSourceRegistryFile` as the MCP registry document
  model, using `validate_unique_source_ids(self.sources)`
- `FileSourceRegistryStore` as the MCP concrete store wrapper around
  `AtomicJsonRegistryStore[SourceRegistryFile]`

### Tests

Move generic tests to `tests/wf_api/test_source_registry.py`:

- safe id validation accepts normal registry ids and rejects unsafe ids
- duplicate id helper rejects repeated ids using a fake entry object
- missing file returns empty generic registry
- save/load round trip
- corrupted JSON wraps as `ValueError`

Keep MCP tests in `tests/wf_mcp/test_source_registry.py`:

- MCP id validation
- reserved id rejection
- stdio/http transport models
- `McpFileSourceRegistryStore` round trip

### Acceptance Criteria

- `wf_api.source_registry` imports no `wf_mcp`.
- Existing `wf_mcp.source_registry` public behavior remains compatible.
- Tests pass.
- No startup/runtime behavior changes.

---

## Slice 2B: MCP Entry Conversion

### Goal

Add explicit conversion helpers so startup merge can convert registry entries
into broker connection configs without duplicating field logic.

Status: complete. The implemented helper preserves entry metadata, `auth_ref`,
profile, transport details, and a `source_registry` marker.

### New Helpers

In `src/wf_mcp/source_registry.py`:

```python
def registry_entry_to_connection_config(
    entry: McpSourceRegistryEntry,
) -> ConnectionConfig:
    return ConnectionConfig(
        id=entry.id,
        server=entry.provider,
        account=entry.account,
        enabled=entry.enabled,
        metadata={
            **entry.metadata,
            "auth_ref": entry.auth_ref,
            "profile": entry.profile,
            "transport": entry.transport.model_dump(mode="json"),
            "source_registry": True,
        },
    )
```

And optionally:

```python
def connection_config_to_registry_entry(
    connection: ConnectionConfig,
) -> McpSourceRegistryEntry | None:
    ...
```

Only add reverse conversion if an implementation needs it. Do not guess unknown
transport metadata.

### Tests

- registry entry converts to `ConnectionConfig`
- provider/account/profile/transport metadata are preserved
- disabled entry creates disabled connection config

### Acceptance Criteria

- Conversion is explicit and tested.
- No startup/runtime behavior changes yet.

---

## Slice 3: Startup Merge

### Goal

Load desired dynamic registry state during service construction and merge it
with config-defined connections/sources.

### Merge Rules

1. Built-in reserved ids always win.
2. Config-defined entries win over registry entries with the same id.
3. Registry entries fill ids not present in config.

### Implementation Direction

- Broker config construction should create a `McpFileSourceRegistryStore`.
- `WfMcpService` or `ConnectionService` should accept an optional registry store.
- On startup/reload:
  - load config connections
  - load registry entries
  - convert registry entries to `ConnectionConfig`
  - merge with config precedence
  - register merged connections
  - emit an event for registry entries shadowed by config

### Tests

- absent registry preserves current config-only behavior
- registry-only connection appears after service construction
- config shadows same-id registry entry
- invalid registry fails startup clearly
- disabled registry source hydrates disabled

### Acceptance Criteria

- Existing legacy config behavior is unchanged when no registry file exists.
- Dynamic registry entries persist across service recreation.
- Shadowed entries do not silently override config.

---

## Slice 4: Read Desired Registry Through Admin

### Goal

Expose desired registry entries distinctly from observed source inventory.

Status: complete for API/transport/CLI plumbing. `WorkflowSourceRegistryApi`
provides neutral read-only access. JSON-RPC methods
`workflow.admin.source_registry.list` / `.inspect` are registered. CLI commands
`wf admin registry list` / `wf admin registry inspect` are available for targets
that expose the surface. Local/static servers report
`source_registry_unavailable`. Concrete MCP-backed `WorkflowServer` construction
remains future work.

### Why

`wf source list` currently reports runtime/observed source inventory. Registry
entries are desired server-owned configuration. Users need both views when a
source exists in registry but is disabled, shadowed, or not hydrated.

### Candidate Commands

- `wf admin registry list`
- `wf admin registry inspect SOURCE_ID`

or:

- `wf source registry list`
- `wf source registry inspect SOURCE_ID`

Pick one naming shape in the implementation plan.

### Acceptance Criteria

- Desired registry view is not confused with observed source inventory.
- Shadowed/disabled state is visible.
- No mutation yet.

---

## Slice 5: Mutation Commands

### Goal

Add safe registry mutation.

Status: complete. Implementation:
[2026-06-04 source registry mutations](../plans/2026-06-04-source-registry-mutations.md).

### Operations

- add source
- update source
- enable source
- disable source
- remove source

### Rules

- Validate full registry before commit.
- Write atomically.
- Do not mutate config files.
- Do not delete auth/catalog files in v1.
- Prefer disable over remove for sources referenced by deployments.
- Optional live validation can be a flag, not required for save.

### Acceptance Criteria

- RPC methods exist.
- CLI commands exist.
- Mutations persist across process restart for targets backed by a registry
  store; local/static servers report unavailable.
- Validation errors are actionable.

---

## Self-Review

- This is a multi-slice roadmap, not a single execution plan for all mutation work.
- Slice 2A is the immediate next implementation target and resolves the location problem.
- Startup merge is intentionally after generic/MCP split and conversion helpers.
- Config ownership policy is intentionally after mutation commands, because it
  changes precedence semantics rather than introducing persistence.