docs: plan source registry startup merge

docs/current_roadmap.md

@@ -125,9 +125,11 @@ implementation state.
       commands are still deferred.
    - Next source registry slices are planned in
       [2026-06-03 source registry next slices](./superpowers/plans/2026-06-03-source-registry-next-slices.md):
-      first split generic registry mechanics into `wf_api`, then add MCP entry
+      generic registry mechanics now live in `wf_api`, MCP entry conversion is
-      conversion, startup merge, desired-registry admin reads, and finally safe
+      explicit and tested, and the next executable slice is
-      mutation commands.
+      [startup merge](./superpowers/plans/2026-06-03-source-registry-startup-merge.md).
+      Desired-registry admin reads and safe mutation commands remain later
+      slices.
    - Longer term: make the MCP frontend an adapter over these neutral workflow,
       source-admin, and config-admin surfaces so the old `wf_mcp` server entry
       point can shrink or retire.

docs/superpowers/plans/2026-06-03-source-registry-next-slices.md

@@ -29,11 +29,17 @@ This is useful and tested, but still MCP-shaped:
 - source entry type is `McpSourceRegistryEntry`
 - transport definitions are MCP transports
 
-That is acceptable for Slice 1, but should be resolved before startup merge and mutation commands make the shape more permanent.
+Slice 2A then moved generic registry mechanics to `wf_api.source_registry`,
+while `wf_mcp.source_registry` kept MCP-specific entries and transports. Slice
+2B added `registry_entry_to_connection_config()`.
+
+The next executable slice is startup merge:
+`docs/superpowers/plans/2026-06-03-source-registry-startup-merge.md`.
 
 ## Slice Order
 
 1. **Slice 2A: Generic Registry Mechanics**
+   - **Status: complete.**
    - Move generic validation/store/file mechanics to `wf_api`.
    - Keep MCP entry/transport validation in `wf_mcp`.
    - Prefer boring helpers over deep Pydantic generics where that keeps the
@@ -41,11 +47,13 @@ That is acceptable for Slice 1, but should be resolved before startup merge and
    - Do not change runtime behavior.
 
 2. **Slice 2B: MCP Entry Conversion**
+   - **Status: complete.**
    - Add conversion helpers between `McpSourceRegistryEntry` and
      `ConnectionConfig`.
    - Keep config merge out of scope.
 
 3. **Slice 3: Startup Merge**
+   - **Status: planned.**
    - Load registry at broker/server startup.
    - Merge config and registry with config precedence.
    - Emit events/diagnostics for shadowed registry entries.
@@ -216,6 +224,9 @@ Keep MCP tests in `tests/wf_mcp/test_source_registry.py`:
 Add explicit conversion helpers so startup merge can convert registry entries
 into broker connection configs without duplicating field logic.
 
+Status: complete. The implemented helper preserves entry metadata, `auth_ref`,
+profile, transport details, and a `source_registry` marker.
+
 ### New Helpers
 
 In `src/wf_mcp/source_registry.py`:
@@ -230,10 +241,11 @@ def registry_entry_to_connection_config(
         account=entry.account,
         enabled=entry.enabled,
         metadata={
+            **entry.metadata,
+            "auth_ref": entry.auth_ref,
             "profile": entry.profile,
             "transport": entry.transport.model_dump(mode="json"),
             "source_registry": True,
-            **entry.metadata,
         },
     )
 ```

docs/superpowers/plans/2026-06-03-source-registry-startup-merge.md

@@ -0,0 +1,351 @@
+# Source Registry Startup Merge Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Load persisted dynamic MCP source registry entries during broker/server construction and merge them with config-defined connections without changing behavior when the registry file is absent.
+
+**Architecture:** Keep parsing/loading config separate from runtime hydration. `wf_mcp.source_registry` owns MCP registry entries and conversion to `ConnectionConfig`; `ConnectionService` owns connection reconciliation and source-catalog hydration; `build_service_from_config()` wires the default file store from `BrokerConfig.store_root`.
+
+**Tech Stack:** Python 3.14, dataclasses, Pydantic v2, `wf_mcp.source_registry`, `ConnectionService`, `WfMcpService`, `build_service_from_config`, pytest, ruff, basedpyright.
+
+---
+
+## Preconditions
+
+Completed before this plan:
+
+- Generic registry mechanics exist in `wf_api.source_registry`.
+- MCP-specific registry models remain in `wf_mcp.source_registry`.
+- `registry_entry_to_connection_config()` exists and is tested.
+- No startup/runtime merge uses the registry yet.
+
+## Merge Rules
+
+1. Config-defined connections win over registry entries with the same id.
+2. Registry entries fill ids not present in config.
+3. Reserved ids remain rejected by existing `ConnectionService` validation.
+4. Shadowed registry entries must be visible through an event/diagnostic, not silently ignored.
+5. Missing `source_registry.json` must preserve current config-only behavior.
+
+Do not mutate config files. Do not delete auth/catalog files.
+
+---
+
+## Task 1: Add Merge Helper Tests First
+
+- [ ] Add focused tests in `tests/wf_mcp/service/test_connection_service.py`.
+- [ ] Import:
+
+```python
+from wf_mcp.source_registry import (
+    FileSourceRegistryStore,
+    McpSourceRegistryEntry,
+    SourceRegistryFile,
+    StdioSourceTransport,
+)
+```
+
+- [ ] Add helper:
+
+```python
+def _registry_entry(
+    source_id: str = "demo.registry",
+    *,
+    enabled: bool = True,
+) -> McpSourceRegistryEntry:
+    return McpSourceRegistryEntry(
+        id=source_id,
+        kind="mcp",
+        enabled=enabled,
+        provider="demo",
+        account=source_id.rsplit(".", 1)[-1],
+        transport=StdioSourceTransport(command="demo-server"),
+    )
+```
+
+- [ ] Add `test_connection_service_sync_merges_registry_entries`.
+
+Expected behavior:
+
+```python
+store = FileSourceRegistryStore(tmp_path)
+store.save_registry(SourceRegistryFile(sources=[_registry_entry()]))
+
+service.sync_connections_from_config(
+    BrokerConfig(store_root=tmp_path, connections=[]),
+    source_registry_store=store,
+)
+
+assert [connection.id for connection in service.list_all()] == ["demo.registry"]
+assert "demo.registry" in catalog.capability_sources
+```
+
+- [ ] Add `test_connection_service_sync_config_shadows_registry_entry`.
+
+Expected behavior:
+
+```python
+store.save_registry(SourceRegistryFile(sources=[_registry_entry("demo.same")]))
+service.sync_connections_from_config(
+    BrokerConfig(
+        store_root=tmp_path,
+        connections=[
+            ConnectionConfig(id="demo.same", server="demo", account="config"),
+        ],
+    ),
+    source_registry_store=store,
+)
+
+assert service.get("demo.same").account == "config"
+assert service.events.list_events()[-1].kind == "source_registry_ignored_config_shadow"
+assert service.events.list_events()[-1].connection_id == "demo.same"
+```
+
+- [ ] Add `test_connection_service_sync_registry_disabled_entry_hydrates_disabled_source`.
+
+Expected behavior:
+
+```python
+store.save_registry(SourceRegistryFile(sources=[_registry_entry(enabled=False)]))
+service.sync_connections_from_config(
+    BrokerConfig(store_root=tmp_path, connections=[]),
+    source_registry_store=store,
+)
+
+assert service.get("demo.registry").enabled is False
+assert catalog.capability_sources["demo.registry"].enabled is False
+```
+
+- [ ] Run:
+
+```bash
+uv run pytest tests/wf_mcp/service/test_connection_service.py -q
+```
+
+Expected: new tests fail until Task 2.
+
+---
+
+## Task 2: Implement Registry-Aware Connection Reconciliation
+
+- [ ] Update `src/wf_mcp/broker/service/connection_service.py`.
+- [ ] Import:
+
+```python
+from ...source_registry import (
+    SourceRegistryStore,
+    registry_entry_to_connection_config,
+)
+```
+
+- [ ] Change `sync_connections_from_config` signature:
+
+```python
+def sync_connections_from_config(
+    self,
+    config: BrokerConfig,
+    *,
+    source_registry_store: SourceRegistryStore | None = None,
+) -> None:
+```
+
+- [ ] Build a merged connection list before existing remove/update/register logic:
+
+```python
+connections = list(config.connections)
+config_ids = {connection.id for connection in connections}
+if source_registry_store is not None:
+    registry = source_registry_store.load_registry()
+    for entry in registry.sources:
+        if entry.id in config_ids:
+            self.events.record_kind(
+                "source_registry_ignored_config_shadow",
+                connection_id=entry.id,
+                payload={
+                    "server": entry.provider,
+                    "account": entry.account,
+                    "reason": "config_connection_takes_precedence",
+                },
+            )
+            continue
+        connections.append(registry_entry_to_connection_config(entry))
+```
+
+- [ ] Reuse the existing reconciliation logic against the merged `connections` list.
+- [ ] Keep `BrokerConfig` immutable; do not mutate `config.connections`.
+- [ ] Keep existing event behavior for registered/updated/removed connections.
+- [ ] Add a short comment near the merge explaining config precedence.
+
+- [ ] Run:
+
+```bash
+uv run pytest tests/wf_mcp/service/test_connection_service.py -q
+```
+
+Expected: all connection service tests pass.
+
+---
+
+## Task 3: Wire WfMcpService Facade
+
+- [ ] Update `src/wf_mcp/broker/service/core.py`.
+- [ ] Import `SourceRegistryStore` from `wf_mcp.source_registry`.
+- [ ] Change facade method signature:
+
+```python
+def sync_connections_from_config(
+    self,
+    config: BrokerConfig,
+    *,
+    source_registry_store: SourceRegistryStore | None = None,
+) -> None:
+    self.connection_service.sync_connections_from_config(
+        config,
+        source_registry_store=source_registry_store,
+    )
+```
+
+- [ ] Add/adjust a facade test in `tests/wf_mcp/service/test_connection_service.py` proving `WfMcpService.sync_connections_from_config(..., source_registry_store=store)` delegates and hydrates a registry connection.
+
+---
+
+## Task 4: Wire Build From Config
+
+- [ ] Update `src/wf_mcp/broker/config.py`.
+- [ ] Import `FileSourceRegistryStore`.
+- [ ] Construct the store in `build_service_from_config(config)`:
+
+```python
+source_registry_store = FileSourceRegistryStore(config.store_root)
+```
+
+- [ ] Replace the manual connection loop with one registry-aware sync call:
+
+```python
+service.sync_connections_from_config(
+    config,
+    source_registry_store=source_registry_store,
+)
+for connection in service.connections.list_all():
+    if connection.server not in service.adapters:
+        service.register_adapter(connection.server, McpSdkAdapter())
+```
+
+Important:
+
+- Preserve adapter registration for config-defined and registry-defined connections.
+- Do not load the registry in `load_broker_config`; it should only parse config files.
+- Do not save the registry during startup.
+
+---
+
+## Task 5: Add Build-Service Integration Tests
+
+- [ ] Update `tests/wf_mcp/test_broker_server.py` or `tests/wf_mcp/server/test_config.py`.
+- [ ] Add `test_build_service_from_config_loads_source_registry_entries`.
+
+Setup:
+
+```python
+config = BrokerConfig(store_root=tmp_path, connections=[])
+FileSourceRegistryStore(tmp_path).save_registry(
+    SourceRegistryFile(sources=[_registry_entry("fixture.registry")])
+)
+service = build_service_from_config(config)
+```
+
+Assertions:
+
+```python
+assert service.connections.get("fixture.registry").server == "fixture"
+assert "fixture" in service.adapters
+assert "fixture.registry" in service.capability_sources
+```
+
+- [ ] Add `test_build_service_from_config_config_shadows_registry`.
+
+Assertions:
+
+```python
+assert service.connections.get("fixture.same").account == "config"
+assert any(
+    event.kind == "source_registry_ignored_config_shadow"
+    and event.connection_id == "fixture.same"
+    for event in service.list_events()
+)
+```
+
+- [ ] Add `test_build_service_from_config_absent_registry_preserves_existing_behavior` if no existing test already covers this. It can extend the current `test_build_service_from_config_registers_connections`.
+
+---
+
+## Task 6: Update Docs
+
+- [ ] Update `docs/superpowers/specs/2026-06-03-store-backed-source-registry-design.md`.
+
+Mark startup merge as complete or in-progress, depending on final status:
+
+```md
+### Slice 3: Startup Merge
+
+Status: complete. Broker/service construction now loads `source_registry.json`,
+merges config-defined connections with dynamic registry entries, preserves config
+precedence, and emits `source_registry_ignored_config_shadow` for shadowed
+registry entries.
+```
+
+- [ ] Update `docs/current_roadmap.md`.
+
+Replace the source-registry note with:
+
+```md
+- Source registry startup merge is implemented: absent registry preserves
+  config-only behavior, registry-only entries hydrate as dynamic connections,
+  and config entries shadow same-id registry entries with an event.
+```
+
+- [ ] Update `docs/superpowers/plans/2026-06-03-source-registry-next-slices.md`.
+
+Mark Slice 2A and Slice 2B complete if not already done, and mark Slice 3 complete after implementation.
+
+---
+
+## Task 7: Verify
+
+- [ ] Run focused tests:
+
+```bash
+uv run pytest tests/wf_mcp/service/test_connection_service.py tests/wf_mcp/test_broker_server.py tests/wf_mcp/server/test_config.py -q
+```
+
+- [ ] Run source registry tests:
+
+```bash
+uv run pytest tests/wf_api/test_source_registry.py tests/wf_mcp/test_source_registry.py -q
+```
+
+- [ ] Run quality checks:
+
+```bash
+uv run ruff check src/wf_mcp/source_registry.py src/wf_mcp/broker/config.py src/wf_mcp/broker/service/connection_service.py src/wf_mcp/broker/service/core.py tests/wf_mcp/service/test_connection_service.py tests/wf_mcp/test_broker_server.py tests/wf_mcp/server/test_config.py
+uv run basedpyright --level error src/wf_mcp/source_registry.py src/wf_mcp/broker/config.py src/wf_mcp/broker/service/connection_service.py src/wf_mcp/broker/service/core.py tests/wf_mcp/service/test_connection_service.py tests/wf_mcp/test_broker_server.py tests/wf_mcp/server/test_config.py
+```
+
+- [ ] If focused tests pass, run the full suite if time permits:
+
+```bash
+uv run pytest -q
+```
+
+---
+
+## Acceptance Criteria
+
+- Missing registry file keeps existing config-only behavior.
+- Registry-only entries become registered connections and hydrated source catalog entries.
+- Config entries shadow same-id registry entries.
+- Shadowing emits `source_registry_ignored_config_shadow`.
+- Disabled registry entries hydrate disabled connection/source state.
+- `load_broker_config()` remains config-only parsing.
+- No registry mutation commands are added in this slice.
+- `wf_api` still imports no `wf_mcp`.

docs/superpowers/specs/2026-06-03-store-backed-source-registry-design.md

@@ -235,6 +235,9 @@ source ids, and concrete MCP transports all come from the MCP broker layer.
 
 ### Slice 2A: Generic Registry Mechanics
 
+Status: complete. Generic registry mechanics live in `wf_api.source_registry`;
+MCP source entries and transports remain in `wf_mcp.source_registry`.
+
 - Move protocol-neutral registry mechanics to `wf_api.source_registry`.
 - Keep MCP source entries, MCP transports, connection-id parsing, and reserved
   id rules in `wf_mcp.source_registry`.
@@ -248,6 +251,10 @@ the generic mechanics and define their own entry models.
 
 ### Slice 2B: MCP Entry Conversion
 
+Status: complete. `registry_entry_to_connection_config()` converts MCP registry
+entries to broker `ConnectionConfig` values while preserving metadata, `auth_ref`,
+profile, transport details, enabled state, and source-registry origin.
+
 - Add explicit conversion from `McpSourceRegistryEntry` to `ConnectionConfig`.
 - Preserve structural fields such as provider, account, profile, transport,
   enabled state, auth reference, and metadata.
@@ -255,6 +262,9 @@ the generic mechanics and define their own entry models.
 
 ### Slice 3: Startup Merge
 
+Implementation plan:
+[2026-06-03 source registry startup merge](../plans/2026-06-03-source-registry-startup-merge.md).
+
 - Load registry during server/broker construction.
 - Merge config + registry deterministically.
 - Emit diagnostics/events for ignored shadowed entries.