docs: plan source registry store

docs/current_roadmap.md

@@ -118,6 +118,8 @@ implementation state.
       source registry is designed. Config can bootstrap sources, but server-owned
       dynamic source changes need persistence, validation, and auth rules before
       they are safe.
+   - The store-backed source registry design is recorded in
+      [2026-06-03 store-backed source registry](./superpowers/specs/2026-06-03-store-backed-source-registry-design.md).
    - Longer term: make the MCP frontend an adapter over these neutral workflow,
       source-admin, and config-admin surfaces so the old `wf_mcp` server entry
       point can shrink or retire.

docs/superpowers/plans/2026-06-03-source-registry-store-slice-1.md

@@ -0,0 +1,433 @@
+# Source Registry Store Slice 1 Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add validated source registry models and a filesystem store for desired server-owned source configuration, without wiring it into startup or mutation commands yet.
+
+**Architecture:** The new registry is desired configuration state and stays separate from existing auth/catalog storage. Models live in `wf_mcp` for this first slice because current connection/source registry semantics are still MCP-provider-specific; the file-store interface is small enough to move later. Runtime merge, RPC/CLI mutation, and config reconciliation are explicitly deferred.
+
+**Tech Stack:** Python 3.14, Pydantic v2, existing `wf_mcp.connections.parse_connection_id`, existing `RESERVED_CONNECTION_IDS`, pytest, ruff, basedpyright.
+
+---
+
+## Scope
+
+In scope:
+
+- Typed `SourceRegistryFile` model.
+- Typed `McpSourceRegistryEntry` model.
+- Typed `StdioSourceTransport` and `HttpSourceTransport` models.
+- Duplicate id validation.
+- Reserved id validation.
+- ID validation using existing connection id rules.
+- `SourceRegistryStore` protocol.
+- `FileSourceRegistryStore` using `<store_root>/source_registry.json`.
+- Atomic filesystem writes.
+- Tests for load missing file, save/load round trip, validation errors, and path.
+
+Out of scope:
+
+- Startup merge with config.
+- Runtime hydration from registry.
+- Mutating API/CLI/RPC commands.
+- Auth record changes.
+- Catalog deletion or cleanup.
+- SQL/remote stores.
+
+## File Structure
+
+- Create `src/wf_mcp/source_registry.py`
+  - Pydantic models and validation.
+  - Protocol and file store.
+  - No dependency on `WfMcpService`.
+
+- Modify `src/wf_mcp/storage/__init__.py`
+  - No change in this slice unless the implementor decides a store export is needed.
+  - Prefer exporting from `wf_mcp.source_registry`, not overloading `wf_mcp.storage`.
+
+- Create `tests/wf_mcp/test_source_registry.py`
+  - Direct model/store tests.
+
+- Modify `docs/current_roadmap.md`
+  - Mark Slice 1 complete after implementation.
+
+---
+
+### Task 1: Add failing source registry model tests
+
+**Files:**
+- Create: `tests/wf_mcp/test_source_registry.py`
+
+- [ ] **Step 1: Write model validation tests**
+
+Create `tests/wf_mcp/test_source_registry.py`:
+
+```python
+from __future__ import annotations
+
+import pytest
+
+from wf_mcp.source_registry import (
+    HttpSourceTransport,
+    McpSourceRegistryEntry,
+    SourceRegistryFile,
+    StdioSourceTransport,
+)
+
+
+def _entry(source_id: str = "github.work") -> McpSourceRegistryEntry:
+    return McpSourceRegistryEntry(
+        id=source_id,
+        provider="github",
+        account="work",
+        transport=StdioSourceTransport(
+            command="npx",
+            args=("-y", "@modelcontextprotocol/server-github"),
+            env={"GITHUB_TOKEN": "${GITHUB_TOKEN}"},
+        ),
+        auth_ref=source_id,
+        metadata={"purpose": "tests"},
+    )
+
+
+def test_source_registry_entry_keeps_identity_and_transport_structural() -> None:
+    entry = _entry()
+
+    assert entry.id == "github.work"
+    assert entry.provider == "github"
+    assert entry.account == "work"
+    assert entry.profile is None
+    assert entry.transport.kind == "stdio"
+    assert entry.transport.command == "npx"
+    assert entry.auth_ref == "github.work"
+
+
+def test_source_registry_accepts_http_transport() -> None:
+    entry = McpSourceRegistryEntry(
+        id="github.http",
+        provider="github",
+        account="work",
+        transport=HttpSourceTransport(url="https://example.test/mcp"),
+    )
+
+    assert entry.transport.kind == "http"
+    assert str(entry.transport.url) == "https://example.test/mcp"
+
+
+def test_source_registry_rejects_duplicate_ids() -> None:
+    with pytest.raises(ValueError, match="duplicate source id 'github.work'"):
+        SourceRegistryFile(sources=[_entry("github.work"), _entry("github.work")])
+
+
+def test_source_registry_rejects_reserved_ids() -> None:
+    with pytest.raises(ValueError, match="reserved"):
+        _entry("wf.admin")
+
+
+def test_source_registry_rejects_unsafe_ids() -> None:
+    with pytest.raises(ValueError, match="connection id"):
+        _entry("../bad")
+```
+
+- [ ] **Step 2: Run tests and verify failure**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_source_registry.py -q
+```
+
+Expected: FAIL because `wf_mcp.source_registry` does not exist yet.
+
+---
+
+### Task 2: Implement source registry models
+
+**Files:**
+- Create: `src/wf_mcp/source_registry.py`
+- Test: `tests/wf_mcp/test_source_registry.py`
+
+- [ ] **Step 1: Add models and validators**
+
+Create `src/wf_mcp/source_registry.py`:
+
+```python
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import Annotated, Literal, Protocol
+
+from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, field_validator, model_validator
+
+from .connections import parse_connection_id
+from .shared.names import RESERVED_CONNECTION_IDS
+
+
+class SourceRegistryModel(BaseModel):
+    """Base model for persisted source registry state; reject misspelled fields."""
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class StdioSourceTransport(SourceRegistryModel):
+    kind: Literal["stdio"] = "stdio"
+    command: str = Field(min_length=1)
+    args: tuple[str, ...] = ()
+    env: dict[str, str] = Field(default_factory=dict)
+
+
+class HttpSourceTransport(SourceRegistryModel):
+    kind: Literal["http"] = "http"
+    url: AnyHttpUrl
+    headers: dict[str, str] = Field(default_factory=dict)
+
+
+SourceTransport = Annotated[
+    StdioSourceTransport | HttpSourceTransport,
+    Field(discriminator="kind"),
+]
+
+
+class McpSourceRegistryEntry(SourceRegistryModel):
+    """Desired MCP source configuration persisted by server-owned mutation."""
+
+    id: str
+    kind: Literal["mcp"] = "mcp"
+    enabled: bool = True
+    provider: str = Field(min_length=1)
+    account: str = Field(min_length=1)
+    profile: str | None = None
+    transport: SourceTransport
+    auth_ref: str | None = None
+    metadata: dict[str, object] = Field(default_factory=dict)
+
+    @field_validator("id")
+    @classmethod
+    def validate_id(cls, value: str) -> str:
+        parse_connection_id(value)
+        if value in RESERVED_CONNECTION_IDS:
+            raise ValueError(f"source id {value!r} is reserved")
+        return value
+
+
+class SourceRegistryFile(SourceRegistryModel):
+    version: Literal[1] = 1
+    sources: list[McpSourceRegistryEntry] = Field(default_factory=list)
+
+    @model_validator(mode="after")
+    def validate_unique_source_ids(self) -> SourceRegistryFile:
+        seen: set[str] = set()
+        for source in self.sources:
+            if source.id in seen:
+                raise ValueError(f"duplicate source id {source.id!r}")
+            seen.add(source.id)
+        return self
+
+    def source_map(self) -> dict[str, McpSourceRegistryEntry]:
+        return {source.id: source for source in self.sources}
+
+
+class SourceRegistryStore(Protocol):
+    """Persistence boundary for desired server-owned source configuration."""
+
+    def load_registry(self) -> SourceRegistryFile:
+        """Return the stored registry, or an empty registry when absent."""
+        ...
+
+    def save_registry(self, registry: SourceRegistryFile) -> None:
+        """Persist one validated registry atomically."""
+        ...
+```
+
+- [ ] **Step 2: Run model tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_source_registry.py -q
+```
+
+Expected: model tests PASS except store tests are not added yet.
+
+---
+
+### Task 3: Add failing file store tests
+
+**Files:**
+- Modify: `tests/wf_mcp/test_source_registry.py`
+
+- [ ] **Step 1: Append file store tests**
+
+Append:
+
+```python
+from pathlib import Path
+
+from wf_mcp.source_registry import FileSourceRegistryStore
+
+
+def test_file_source_registry_store_loads_empty_registry_when_missing(
+    tmp_path: Path,
+) -> None:
+    store = FileSourceRegistryStore(tmp_path)
+
+    registry = store.load_registry()
+
+    assert registry.version == 1
+    assert registry.sources == []
+    assert store.path == tmp_path / "source_registry.json"
+
+
+def test_file_source_registry_store_round_trips_registry(tmp_path: Path) -> None:
+    store = FileSourceRegistryStore(tmp_path)
+    registry = SourceRegistryFile(sources=[_entry("github.work")])
+
+    store.save_registry(registry)
+    loaded = store.load_registry()
+
+    assert loaded.source_map()["github.work"].provider == "github"
+    assert loaded.source_map()["github.work"].transport.kind == "stdio"
+
+
+def test_file_source_registry_store_validates_loaded_registry(tmp_path: Path) -> None:
+    store = FileSourceRegistryStore(tmp_path)
+    store.path.write_text(
+        '{"version": 1, "sources": [{"id": "wf.admin", "provider": "wf", '
+        '"account": "admin", "transport": {"kind": "stdio", "command": "x"}}]}',
+        encoding="utf-8",
+    )
+
+    with pytest.raises(ValueError, match="reserved"):
+        store.load_registry()
+```
+
+- [ ] **Step 2: Run tests and verify failure**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_source_registry.py -q
+```
+
+Expected: FAIL because `FileSourceRegistryStore` does not exist yet.
+
+---
+
+### Task 4: Implement file store
+
+**Files:**
+- Modify: `src/wf_mcp/source_registry.py`
+- Test: `tests/wf_mcp/test_source_registry.py`
+
+- [ ] **Step 1: Add file store implementation**
+
+Append to `src/wf_mcp/source_registry.py`:
+
+```python
+class FileSourceRegistryStore:
+    """Filesystem implementation for desired source registry state."""
+
+    def __init__(self, root: Path) -> None:
+        self.root = root
+        self.root.mkdir(parents=True, exist_ok=True)
+
+    @property
+    def path(self) -> Path:
+        return self.root / "source_registry.json"
+
+    def load_registry(self) -> SourceRegistryFile:
+        if not self.path.exists():
+            return SourceRegistryFile()
+        data = json.loads(self.path.read_text(encoding="utf-8"))
+        return SourceRegistryFile.model_validate(data)
+
+    def save_registry(self, registry: SourceRegistryFile) -> None:
+        # Validate again at the store boundary so callers cannot persist stale or
+        # partially constructed model-like objects after mutation.
+        validated = SourceRegistryFile.model_validate(
+            registry.model_dump(mode="json")
+        )
+        payload = json.dumps(validated.model_dump(mode="json"), indent=2)
+        tmp_path = self.path.with_name(f"{self.path.name}.tmp")
+        tmp_path.write_text(payload, encoding="utf-8")
+        tmp_path.replace(self.path)
+```
+
+- [ ] **Step 2: Add `__all__`**
+
+At the bottom of `src/wf_mcp/source_registry.py`, add:
+
+```python
+__all__ = [
+    "FileSourceRegistryStore",
+    "HttpSourceTransport",
+    "McpSourceRegistryEntry",
+    "SourceRegistryFile",
+    "SourceRegistryStore",
+    "SourceTransport",
+    "StdioSourceTransport",
+]
+```
+
+- [ ] **Step 3: Run tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_source_registry.py -q
+```
+
+Expected: PASS.
+
+---
+
+### Task 5: Documentation and verification
+
+**Files:**
+- Modify: `docs/current_roadmap.md`
+
+- [ ] **Step 1: Update roadmap**
+
+In `docs/current_roadmap.md`, near the store-backed source registry note, add:
+
+```markdown
+- First source registry implementation slice complete: validated registry
+  models plus `FileSourceRegistryStore` exist, but startup merge and mutation
+  commands are still deferred.
+```
+
+- [ ] **Step 2: Run verification**
+
+Run:
+
+```bash
+uv run pytest tests/wf_mcp/test_source_registry.py -q
+uv run ruff check src/wf_mcp/source_registry.py tests/wf_mcp/test_source_registry.py
+uv run ruff format --check src/wf_mcp/source_registry.py tests/wf_mcp/test_source_registry.py
+uv run basedpyright --level error src/wf_mcp/source_registry.py tests/wf_mcp/test_source_registry.py
+```
+
+Expected:
+
+- pytest PASS
+- ruff check PASS
+- ruff format PASS
+- basedpyright 0 errors
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add src/wf_mcp/source_registry.py tests/wf_mcp/test_source_registry.py docs/current_roadmap.md
+git commit -m "feat: add source registry file store"
+```
+
+---
+
+## Self-Review
+
+- Spec coverage: Slice 1 only is covered: models, validation, file store, tests, docs.
+- Placeholder scan: no TBD/TODO placeholders.
+- Type consistency: `SourceRegistryFile`, `McpSourceRegistryEntry`, `SourceRegistryStore`, and `FileSourceRegistryStore` names are consistent.
+- Deferred work is explicit: startup merge, runtime hydration, RPC/CLI mutation, and auth/catalog cleanup are not part of this slice.

docs/superpowers/specs/2026-06-03-store-backed-source-registry-design.md

@@ -0,0 +1,271 @@
+# Store-Backed Source Registry Design
+
+## Purpose
+
+The workflow server now has neutral read-only workflow, source-admin, and
+admin/config surfaces. The next platform step is safe mutation: adding,
+updating, disabling, and removing server-owned sources/connections without
+editing config files by hand.
+
+This spec defines the persistence and merge rules before mutation commands are
+implemented.
+
+## Current State
+
+`wf_mcp.storage.FileStore` persists:
+
+- auth records under `auth/<connection_id>.json`
+- catalog snapshots under `catalog/<connection_id>.json`
+
+It does not persist the connection/source registry itself. Current connection
+definitions come from config and live in memory through `ConnectionService`.
+
+Neutral workflow config currently has:
+
+- `client.target`: local or JSON-RPC HTTP target
+- `server.store`: filesystem store root
+- `server.transports`: server-hosted transports
+- `server.sources`: static built-in sources such as `wf.std` / `wf.recipes`
+
+Legacy MCP config still has `connections`.
+
+## Goals
+
+- Persist server-owned dynamic source/connection changes across process restarts.
+- Keep config useful as bootstrap and deployment-time infrastructure.
+- Preserve structural source identity. Source ids are explicit ids, not parsed
+  from dotted display names.
+- Make mutation validation explicit and fail-fast.
+- Keep disabled/missing sources visible as diagnostics, not silent deletion from
+  deployments or runs.
+- Avoid turning catalog snapshots into source definitions. Catalogs are observed
+  capability state; registry entries are desired configuration state.
+
+## Non-Goals
+
+- No UI design.
+- No auth-secret format redesign beyond referencing existing auth records.
+- No SQL store in the first implementation.
+- No automatic source id inference from provider/account strings.
+- No workflow lifecycle changes.
+
+## Registry Model
+
+The persisted registry stores desired source/connection definitions:
+
+```json
+{
+  "version": 1,
+  "sources": [
+    {
+      "id": "github.work",
+      "kind": "mcp",
+      "enabled": true,
+      "provider": "github",
+      "account": "work",
+      "profile": null,
+      "transport": {
+        "kind": "stdio",
+        "command": "npx",
+        "args": ["-y", "@modelcontextprotocol/server-github"],
+        "env": {}
+      },
+      "auth_ref": "github.work",
+      "metadata": {}
+    }
+  ]
+}
+```
+
+### Source Identity
+
+`id` is the stable workflow-facing concrete source id. Examples:
+
+- `github.work`
+- `github.personal`
+- `everything.default`
+- `wf.std`
+
+The id is not decomposed for meaning. The structured fields carry meaning:
+
+- `provider`: logical provider/server family, such as `github`
+- `account`: account/workspace name, such as `work`
+- `profile`: optional variant under one provider/account
+- `transport`: concrete connectivity details
+
+Transport belongs to the concrete source entry, not only to the provider. The
+same provider/account may need different connection transports in different
+server deployments, and one source id must carry the exact runtime transport the
+server should use.
+
+### Source Kinds
+
+Initial persisted kind:
+
+- `mcp`: upstream MCP source
+
+Bootstrap-only / built-in kinds can remain config/code-owned for now:
+
+- `stdlib`
+- `docs`
+- `admin`
+
+Do not persist built-in `wf.std`, `wf.recipes`, or `wf.admin` in the dynamic
+registry unless/until there is a real need.
+
+## Store Shape
+
+Add a registry store beside existing auth/catalog files:
+
+```text
+<store_root>/
+  auth/
+  catalog/
+  source_registry.json
+```
+
+Why one file first:
+
+- The registry is small.
+- Writes can be atomic by writing a temp file then replacing it.
+- Whole-file validation is simple.
+- It matches config-like semantics.
+
+Future stores can expose the same interface over SQL or another durable backend.
+
+## Merge Rules
+
+On server startup:
+
+1. Load built-in sources from code/config.
+2. Load config-defined connections/sources.
+3. Load `source_registry.json`.
+4. Merge into one desired source map.
+5. Hydrate runtime `ConnectionService` / `SourceCatalogService` from that map.
+
+Precedence:
+
+1. Built-in reserved ids always win for reserved ids.
+2. Config-defined entries win over dynamic registry entries with the same id.
+3. Dynamic registry entries fill in ids not present in config.
+
+Rationale: config is deployment bootstrap and operator-controlled. Dynamic store
+state should not secretly override source definitions checked into deployment
+config.
+
+Duplicate behavior:
+
+- Duplicate ids inside one config or one registry file are validation errors.
+- A registry entry with the same id as config is allowed but ignored with a
+  diagnostic/event.
+- A registry entry using a reserved id is invalid.
+
+## Mutation Rules
+
+Initial mutation commands should target the registry only, not config:
+
+- add source
+- update source
+- enable/disable source
+- remove source
+
+Rules:
+
+- Mutations validate the full registry before saving.
+- Mutations write one atomic registry replacement.
+- Enabling a source requires validation of source shape and transport config.
+- Optional live validation can be requested, but ordinary connection failure
+  should not corrupt registry state.
+- Removing a source deletes desired registry state only. It does not delete old
+  catalog/auth files in the first pass.
+- Disable is preferred over remove when deployments may still reference the
+  source.
+
+## Runtime Semantics
+
+If a deployment references a missing/disabled/unreachable source:
+
+- validation reports diagnostics (`source_missing`, `source_disabled`,
+  `source_unreachable`)
+- run start fails or is blocked by validation
+- existing deployments are not rewritten
+- ordinary dead tools/sources do not become interrupts or pauses
+
+## Events
+
+Registry mutations should emit broker/server events:
+
+- `source_registered`
+- `source_updated`
+- `source_enabled`
+- `source_disabled`
+- `source_removed`
+- `source_registry_ignored_config_shadow`
+
+Events are read-only through the existing `WorkflowAdminApi`.
+
+## API Surfaces
+
+Do not add mutation to `WorkflowApiSurface`.
+
+Likely surfaces:
+
+- `WorkflowSourceAdminSurface`: read-only source list/inspect already exists
+- future `WorkflowSourceRegistrySurface`: mutating registry operations
+- `WorkflowAdminSurface`: read-only connections/status/events already exists
+
+The mutation surface may live in `wf_api` if it remains protocol-neutral. If it
+becomes transport/provider-heavy, split it into a platform admin package instead
+of bloating workflow lifecycle APIs.
+
+## First Implementation Slices
+
+### Slice 1: Registry Models and File Store
+
+- Add Pydantic registry models.
+- Add `SourceRegistryStore` protocol.
+- Add `FileSourceRegistryStore`.
+- Validate duplicate ids and reserved ids.
+- Atomic write for filesystem store.
+- No runtime wiring yet.
+
+### Slice 2: Startup Merge
+
+- Load registry during server/broker construction.
+- Merge config + registry deterministically.
+- Emit diagnostics/events for ignored shadowed entries.
+- Preserve existing config-only behavior when registry file is absent.
+
+### Slice 3: Read Registry Through Admin
+
+- Add admin read method for desired registry entries if needed.
+- Keep current source inventory list as runtime/observed source inventory.
+- Document difference between desired registry and observed source catalog.
+
+### Slice 4: Mutating RPC/CLI
+
+- Add add/update/enable/disable/remove operations.
+- Add JSON-RPC methods.
+- Add CLI commands.
+- Validate before commit.
+- Do not delete auth/catalog on remove in v1.
+
+## Open Questions
+
+- Should dynamic registry entries support non-MCP transports in v1, or only MCP
+  stdio/HTTP?
+- Should auth references be required for sources that need auth, or optional
+  until live validation?
+- Should config shadowing registry entries be a warning only, or should server
+  startup fail in strict mode?
+- Should disabled registry entries still hydrate as disabled sources so inspect
+  can explain them, or stay only in registry/admin output?
+
+## Recommendation
+
+Implement Slice 1 first. Keep it independent from runtime startup so the model
+and file persistence rules become solid before they affect source hydration.
+
+The existing `wf_mcp` store can host the first file-backed implementation, but
+the interface should be neutral enough to move later. Catalog/auth remain
+observed/secret state; the new registry is desired source configuration state.