lda-wf/docs/superpowers/specs/2026-06-09-runtime-source-lifecycle.md

Runtime Source Lifecycle

Date: 2026-06-09

Status: design direction for next source families

Related:

• Workflow config targets and sources
• Store-backed source registry
• Long-lived workflow API boundary
• Project map

Purpose

The current mutable source registry and apply/reload path are concrete for MCP sources. The next source families should not be forced through MCP ConnectionConfig language. This spec defines the generic lifecycle that MCP, Python, HTTP/OpenAPI, and future source providers should share.

The rule:

Admin mutation changes desired source state. Apply/reload reconciles desired source state into live runtime sources. wf_api keeps seeing only CapabilitySource, NodeSpec, resources, prompts, and protocol surfaces.

Current State

MCP has a working implementation:

source registry desired MCP entries
  -> wf admin registry apply
  -> WfMcpService connections/adapters/catalog
  -> WorkflowApi capability/source surfaces

This works, but it is still MCP-shaped internally:

• registry entries are McpSourceRegistryEntry
• runtime apply mutates ConnectionService
• execution uses McpRuntimePool
• compatibility glue still translates through legacy ConnectionConfig

That is acceptable for MCP. It should not become the required model for Python or HTTP sources.

Desired Generic Lifecycle

All source families should fit this lifecycle:

config seed sources + store-backed desired sources
  -> source provider manager
  -> live source instances
  -> capability inventory and executable NodeSpecs
  -> WorkflowApi and transports

Definitions:

• Source config: deployment bootstrap in wf_config.
• Source registry: mutable desired state stored by the server.
• Source provider: implementation package for one source family, such as wf_sources_mcp, wf_sources_python, or wf_sources_openapi.
• Live source instance: in-memory runtime object created from desired state.
• Apply/reload: reconciliation from desired state to live source instances.
• Capability projection: CapabilitySource plus executable NodeSpecs, resources, prompts, reducers, or workflows exposed to wf_api.

Provider Boundary

A future neutral provider boundary should be closer to this shape than to MCP connections:

from collections.abc import Mapping, Sequence
from typing import Any, Protocol

from wf_authoring import NodeSpec
from wf_platform import CapabilitySource


class SourceEntryLike(Protocol):
    id: str
    kind: str
    enabled: bool


class SourceProvider(Protocol):
    kind: str

    async def apply_sources(
        self,
        *,
        desired: Sequence[SourceEntryLike],
    ) -> dict[str, Any]:
        """Reconcile desired entries into live source instances."""
        ...

    @property
    def capability_sources(self) -> Mapping[str, CapabilitySource]:
        """Return current workflow-visible source inventory."""
        ...

    def get_qualified_spec(self, qualified_name: str) -> NodeSpec[Any, Any]:
        """Return the executable spec for one qualified capability."""
        ...

This is not an immediate API contract. It is the direction: the provider owns source-family semantics, while wf_api remains source-family neutral.

Source Family Expectations

MCP Sources

MCP sources already have:

• desired registry entries
• config seed/locked ownership
• explicit apply/reload
• persistent runtime sessions through McpRuntimePool
• catalog snapshots for tools/resources/prompts

MCP may continue to use ConnectionConfig behind its compatibility boundary until the old wf_mcp facade is retired.

Python Sources

Python sources should be the first non-MCP proof that the architecture is not MCP-shaped.

Expected config example:

{
  "kind": "python",
  "id": "local.ops",
  "module": "my_project.workflow_ops",
  "registry": "registry"
}

Expected behavior:

• load a module/object containing NodeSpecs or Python functions convertible to NodeSpecs
• expose them under the configured source id
• execute in-process without MCP sessions
• support reload by re-importing/re-reading the configured registry

Python source reload should be explicit. Do not silently reload modules on every capability call.

HTTP/OpenAPI Sources

HTTP/OpenAPI sources should be a separate source family, not a special MCP transport.

Expected behavior:

• parse OpenAPI or explicit operation definitions
• expose operations as NodeSpecs
• use auth records where appropriate
• define a clear HTTP status/outcome policy before implementation
• keep large response/body handling bounded for CLI and workflow safety

This source family needs more design than Python because error semantics, timeouts, idempotency, and output normalization are product decisions.

Config vs Registry

Config and registry should keep their current distinction:

• Config is bootstrap, deployment intent, and portable local setup.
• Registry is mutable server-owned desired state.
• locked config entries own their ids and shadow registry entries.
• seed config entries initialize missing registry entries, then registry owns later admin changes.

For future source families, this ownership model should apply to source ids independently of MCP provider/account fields.

Apply Semantics

Apply/reload should:

• validate desired source entries before mutating live state
• add newly desired sources
• update changed source instances
• disable or remove deleted desired sources according to provider policy
• preserve diagnostics for referenced missing/disabled sources
• avoid mutating config files
• emit admin-visible events

Apply/reload should not:

• delete auth records by default
• delete old catalog/cache files by default
• rewrite deployments or artifacts
• turn live source failure into workflow interrupts

Workflow API Boundary

wf_api should not know whether a capability came from MCP, Python, HTTP, or a future source family. It should depend on:

• WorkflowSpecProvider
• CapabilitySource
• NodeSpec
• store/admin protocols

If a future source needs a family-specific operation, put it behind that provider package or admin surface. Do not add source-family-specific branches to workflow run execution.

First Implementation Direction

Recommended next implementation slice:

1. Add PythonSourceConfig to wf_config as a tagged server.sources entry.
2. Create wf_sources_python with a loader for module:object registries.
3. Project loaded NodeSpecs into CapabilitySource.
4. Wire local/static WorkflowServer construction to include Python sources.
5. Prove wf cap list, wf cap call, and deployment run start work without importing wf_mcp.

This slice should not implement source registry mutation for Python yet. Static config first proves the provider shape. Mutable registry/apply can follow once the provider interface is real.

Open Questions

• Should generic source registry entries live in wf_api, wf_server, or a new wf_sources package?
• Should provider managers be composed by wf_server.config or by a separate source runtime package?
• How much hot-reload behavior should Python sources support in development?
• What are the safe defaults for HTTP timeouts, retries, and large response handling?