lda-wf/docs/superpowers/specs/2026-06-03-store-backed-source-registry-design.md

Store-Backed Source Registry Design

Related:

• Runtime source lifecycle

Purpose

The workflow server now has neutral read-only workflow, source-admin, and admin/config surfaces. The next platform step is safe mutation: adding, updating, disabling, and removing server-owned sources/connections without editing config files by hand.

This spec defines the persistence and merge rules before mutation commands are implemented.

Current State

wf_mcp.storage.FileStore persists:

• auth records under auth/<connection_id>.json
• catalog snapshots under catalog/<connection_id>.json

It does not persist the connection/source registry itself. Current connection definitions come from config and live in memory through ConnectionService.

Neutral workflow config currently has:

• client.target: local or JSON-RPC HTTP target
• server.store: filesystem store root
• server.transports: server-hosted transports
• server.sources: static built-in sources such as wf.std / wf.recipes

Legacy MCP config still has connections.

Goals

• Persist server-owned dynamic source/connection changes across process restarts.
• Keep config useful as bootstrap and deployment-time infrastructure.
• Preserve structural source identity. Source ids are explicit ids, not parsed from dotted display names.
• Make mutation validation explicit and fail-fast.
• Keep disabled/missing sources visible as diagnostics, not silent deletion from deployments or runs.
• Avoid turning catalog snapshots into source definitions. Catalogs are observed capability state; registry entries are desired configuration state.

Non-Goals

• No UI design.
• No auth-secret format redesign beyond referencing existing auth records.
• No SQL store in the first implementation.
• No automatic source id inference from provider/account strings.
• No workflow lifecycle changes.

Registry Model

The persisted registry stores desired source/connection definitions:

{
  "version": 1,
  "sources": [
    {
      "id": "github.work",
      "kind": "mcp",
      "enabled": true,
      "provider": "github",
      "account": "work",
      "profile": null,
      "transport": {
        "kind": "stdio",
        "command": "npx",
        "args": ["-y", "@modelcontextprotocol/server-github"],
        "env": {}
      },
      "auth_ref": "github.work",
      "metadata": {}
    }
  ]
}

Source Identity

id is the stable workflow-facing concrete source id. Examples:

• github.work
• github.personal
• everything.default
• wf.std

The id is not decomposed for meaning. The structured fields carry meaning:

• provider: logical provider/server family, such as github
• account: account/workspace name, such as work
• profile: optional variant under one provider/account
• transport: concrete connectivity details

Transport belongs to the concrete source entry, not only to the provider. The same provider/account may need different connection transports in different server deployments, and one source id must carry the exact runtime transport the server should use.

Source Kinds

Initial persisted kind:

• mcp: upstream MCP source

Bootstrap-only / built-in kinds can remain config/code-owned for now:

• stdlib
• docs
• admin

Do not persist built-in wf.std, wf.recipes, or wf.admin in the dynamic registry unless/until there is a real need.

Store Shape

Add a registry store beside existing auth/catalog files:

<store_root>/
  auth/
  catalog/
  source_registry.json

Why one file first:

• The registry is small.
• Writes can be atomic by writing a temp file then replacing it.
• Whole-file validation is simple.
• It matches config-like semantics.

Future stores can expose the same interface over SQL or another durable backend.

Merge Rules

On server startup:

1. Load built-in sources from code/config.
2. Load config-defined connections/sources.
3. Load source_registry.json.
4. Merge into one desired source map.
5. Hydrate runtime ConnectionService / SourceCatalogService from that map.

Current v1 precedence:

1. Built-in reserved ids always win for reserved ids.
2. Config-defined entries win over dynamic registry entries with the same id.
3. Dynamic registry entries fill in ids not present in config.

Rationale: this first pass is conservative. Config is deployment bootstrap and operator-controlled, so dynamic store state should not secretly override source definitions checked into deployment config before source ownership policy exists.

Long-term ownership policy:

• Config remains valid for bootstrap, dev/test portability, disaster recovery, and operator-managed static sources.
• Store-backed registry is the normal mutable desired-state authority for server-owned dynamic sources.
• Config should not win forever by default. Future source definitions should distinguish at least two config ownership modes:
  • locked: config owns the id. Registry add/update for that id is rejected or remains explicitly shadowed.
  • seed: config creates or hydrates the initial store entry when missing, then the store owns later admin changes.

Until that policy exists in models and startup merge, same-id config entries continue to shadow store entries for safety.

Duplicate behavior:

• Duplicate ids inside one config or one registry file are validation errors.
• A registry entry with the same id as config is allowed but ignored with a diagnostic/event in v1. Future seed config entries should instead allow the store entry to become authoritative after seeding.
• A registry entry using a reserved id is invalid.

Mutation Rules

Initial mutation commands should target the registry only, not config:

• add source
• update source
• enable/disable source
• remove source

Rules:

• Mutations validate the full registry before saving.
• Mutations write one atomic registry replacement.
• Enabling a source requires validation of source shape and transport config.
• Optional live validation can be requested, but ordinary connection failure should not corrupt registry state.
• Removing a source deletes desired registry state only. It does not delete old catalog/auth files in the first pass.
• Disable is preferred over remove when deployments may still reference the source.

Runtime Semantics

If a deployment references a missing/disabled/unreachable source:

• validation reports diagnostics (source_missing, source_disabled, source_unreachable)
• run start fails or is blocked by validation
• existing deployments are not rewritten
• ordinary dead tools/sources do not become interrupts or pauses

Events

Registry mutations should emit broker/server events:

• source_registered
• source_updated
• source_enabled
• source_disabled
• source_removed
• source_registry_ignored_config_shadow

Events are read-only through the existing WorkflowAdminApi.

API Surfaces

Do not add mutation to WorkflowApiSurface.

Likely surfaces:

• WorkflowSourceAdminSurface: read-only source list/inspect already exists
• future WorkflowSourceRegistrySurface: mutating registry operations
• WorkflowAdminSurface: read-only connections/status/events already exists

The mutation surface may live in wf_api if it remains protocol-neutral. If it becomes transport/provider-heavy, split it into a platform admin package instead of bloating workflow lifecycle APIs.

First Implementation Slices

Slice 1: Registry Models and File Store

• Add Pydantic registry models.
• Add SourceRegistryStore protocol.
• Add FileSourceRegistryStore.
• Validate duplicate ids and reserved ids.
• Atomic write for filesystem store.
• No runtime wiring yet.

Status: complete. The first implementation lives in wf_mcp.source_registry because the entry shape is MCP-specific: connection-id validation, reserved source ids, and concrete MCP transports all come from the MCP broker layer.

Slice 2A: Generic Registry Mechanics

Status: complete. Generic registry mechanics live in wf_api.source_registry; MCP source entries and transports remain in wf_mcp.source_registry.

• Move protocol-neutral registry mechanics to wf_api.source_registry.
• Keep MCP source entries, MCP transports, connection-id parsing, and reserved id rules in wf_mcp.source_registry.
• Generic mechanics include safe registry-id validation, duplicate-id checking, a registry-store protocol, and a small atomic JSON model store.
• Do not wire registry state into startup yet.

This resolves the current location tension without pretending an MCP source entry is a generic workflow source. Future non-MCP source families should reuse the generic mechanics and define their own entry models.

Slice 2B: MCP Entry Conversion

Status: complete. registry_entry_to_connection_config() converts MCP registry entries to broker ConnectionConfig values while preserving metadata, auth_ref, profile, transport details, enabled state, and source-registry origin.

• Add explicit conversion from McpSourceRegistryEntry to ConnectionConfig.
• Preserve structural fields such as provider, account, profile, transport, enabled state, auth reference, and metadata.
• Do not merge config and registry yet.

Slice 3: Startup Merge

Status: complete. Broker/service construction now loads source_registry.json, merges config-defined connections with dynamic registry entries, preserves config precedence, and emits source_registry_ignored_config_shadow for shadowed registry entries.

Slice 4: Read Registry Through Admin

Status: complete for API/transport/CLI plumbing. WorkflowSourceRegistryApi provides neutral read-only access to desired registry entries. JSON-RPC methods workflow.admin.source_registry.list and .inspect are registered. CLI commands wf admin registry list and wf admin registry inspect are available for targets that expose the surface. Local/static servers report source_registry_unavailable instead of pretending to have an empty registry. No mutations added. wf source list behavior remains unchanged. Concrete MCP-backed WorkflowServer construction remains future work.

Slice 5: Mutating RPC/CLI

Status: complete. Add/update/enable/disable/remove operations are available through WorkflowSourceRegistryApi, JSON-RPC methods, and CLI commands. Mutations target persisted desired registry state only; config files, auth records, and catalog snapshots are not mutated. Config-shadowed add is rejected in v1. Remove requires --confirm in CLI. Local/static servers report unavailable for mutation commands. Concrete MCP-backed WorkflowServer construction remains future work.

Slice 6: Config Ownership Policy

Status: complete. MCP broker config connections now support source_config_ownership="locked" | "seed". locked preserves v1 shadowing. seed materializes missing store entries and lets existing registry entries own future runtime state.

Apply Semantics

Registry mutation commands write desired persisted state. They do not implicitly change the running server. apply_registry_changes is the explicit boundary that reconciles desired registry state with the current runtime source graph.

The apply operation mirrors config reload reconciliation by calling the same connection/source merge logic. It preserves locked config shadowing and seed config handoff rules. It does not mutate config files, remount public MCP proxy providers, or handle upstream credential prompts.

Open Questions

• Should dynamic registry entries support non-MCP transports in v1, or only MCP stdio/HTTP?
• Should auth references be required for sources that need auth, or optional until live validation?
• Should strict mode fail on locked config/store conflicts, or keep them as explicit shadow diagnostics?
• Should disabled registry entries still hydrate as disabled sources so inspect can explain them, or stay only in registry/admin output?

Recommendation

Implement Slice 2A next. Keep it independent from runtime startup so the generic/MCP split becomes solid before it affects source hydration.

Catalog/auth remain observed/secret state; the new registry is desired source configuration state. Generic registry mechanics can live in wf_api, but provider-specific entries and conversion into broker runtime connections stay with the package that owns that provider.