docs: record static python source support

2026-06-11 18:54:34 +07:00
parent 6c10ef59f7
commit 7735d2012e
4 changed files with 25 additions and 3 deletions
@@ -0,0 +1,758 @@
+# Static Python Sources Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add static config-driven Python workflow sources so local/project Python `NodeSpec`s can be exposed through `WorkflowServer`, `wf cap list`, `wf cap call`, and workflow runs without MCP.
+
+**Architecture:** `wf_authoring` continues to own `NodeSpec` and `@node`. New package `wf_sources_python` owns module/object loading and projection into `CapabilitySource`. `wf_config` gains `kind: "python"` source entries, and `wf_server.config` composes Python sources into the local/static `WorkflowServer` path.
+
+**Tech Stack:** Python 3.14, Pydantic v2 discriminated unions, `importlib`, `wf_authoring.NodeSpec`, `wf_platform.CapabilitySource`, pytest/pytest-asyncio, ruff, basedpyright.
+
+---
+
+## File Structure
+
+- Create `src/wf_sources_python/__init__.py`
+  - Public exports for `PythonSourceConfigLike`, `load_python_source`, `python_capability_source`.
+- Create `src/wf_sources_python/loader.py`
+  - Import `module`, resolve `registry`, normalize supported registry shapes.
+- Create `tests/wf_sources_python/test_loader.py`
+  - Unit tests for loading mappings, sequences, callables, qualification, and errors.
+- Modify `src/wf_config/models.py`
+  - Add `PythonSourceConfig` to `SourceConfig` union.
+- Modify `src/wf_config/__init__.py`
+  - Export `PythonSourceConfig`.
+- Modify `src/wf_server/context.py`
+  - Let `build_local_static_workflow_server(...)` accept extra capability sources.
+- Modify `src/wf_server/config.py`
+  - Detect `kind: "python"` sources and pass loaded sources to local/static server construction.
+- Modify tests:
+  - `tests/wf_config/test_config_models.py`
+  - `tests/wf_server/test_config_composition.py`
+  - Add or update an RPC/client smoke test proving `cap call` works.
+- Modify docs:
+  - `docs/current_roadmap.md`
+  - `docs/superpowers/specs/2026-06-11-python-source-provider.md`
+
+---
+
+## Task 1: Config Model
+
+**Files:**
+
+- Modify: `src/wf_config/models.py`
+- Modify: `src/wf_config/__init__.py`
+- Modify: `tests/wf_config/test_config_models.py`
+
+- [ ] **Step 1: Add failing config test**
+
+Add to `tests/wf_config/test_config_models.py`:
+
+```python
+def test_workflow_config_parses_python_source() -> None:
+    config = WorkflowConfigFile.model_validate(
+        {
+            "version": 1,
+            "server": {
+                "sources": [
+                    {
+                        "kind": "python",
+                        "id": "local.ops",
+                        "module": "tests.fixtures.python_source_ops",
+                        "registry": "registry",
+                    }
+                ]
+            },
+        }
+    )
+
+    source = config.server.sources[0]
+    assert source.kind == "python"
+    assert source.id == "local.ops"
+    assert source.module == "tests.fixtures.python_source_ops"
+    assert source.registry == "registry"
+```
+
+- [ ] **Step 2: Run failing test**
+
+Run:
+
+```bash
+uv run pytest tests/wf_config/test_config_models.py::test_workflow_config_parses_python_source -q
+```
+
+Expected: fail because `PythonSourceConfig` is not in the source union.
+
+- [ ] **Step 3: Add `PythonSourceConfig`**
+
+In `src/wf_config/models.py`, add after `StdlibSourceConfig`:
+
+```python
+class PythonSourceConfig(WorkflowConfigModel):
+    """Static config for trusted in-process Python workflow sources."""
+
+    kind: Literal["python"] = "python"
+    id: str
+    enabled: bool = True
+    module: str = Field(min_length=1)
+    registry: str = Field(default="registry", min_length=1)
+
+    @field_validator("id")
+    @classmethod
+    def validate_source_id(cls, value: str) -> str:
+        if not re.fullmatch(SOURCE_ID_PATTERN, value):
+            raise ValueError(
+                "source id must start with alphanumeric or underscore and contain "
+                "only [A-Za-z0-9_.-]"
+            )
+        if "." not in value:
+            raise ValueError("source id must look like '<namespace>.<name>'")
+        return value
+```
+
+Update `SourceConfig`:
+
+```python
+SourceConfig = Annotated[
+    StdlibSourceConfig | PythonSourceConfig | McpSourceConfig,
+    Field(discriminator="kind"),
+]
+```
+
+Update `src/wf_config/__init__.py` to export `PythonSourceConfig`.
+
+- [ ] **Step 4: Run config tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_config/test_config_models.py -q
+```
+
+Expected: pass.
+
+- [ ] **Step 5: Commit config slice**
+
+Run:
+
+```bash
+git add src/wf_config/models.py src/wf_config/__init__.py tests/wf_config/test_config_models.py
+git commit -m "feat: add python source config model"
+```
+
+---
+
+## Task 2: Python Source Loader
+
+**Files:**
+
+- Create: `src/wf_sources_python/__init__.py`
+- Create: `src/wf_sources_python/loader.py`
+- Create: `tests/wf_sources_python/test_loader.py`
+- Create: `tests/fixtures/python_source_ops.py`
+
+- [ ] **Step 1: Add fixture module**
+
+Create `tests/fixtures/python_source_ops.py`:
+
+```python
+from __future__ import annotations
+
+from pydantic import BaseModel
+
+from wf_authoring import node
+
+
+class EchoInput(BaseModel):
+    text: str
+
+
+class EchoOutput(BaseModel):
+    echoed: str
+
+
+@node(name="echo")
+def echo(payload: EchoInput) -> EchoOutput:
+    return EchoOutput(echoed=payload.text)
+
+
+@node(name="authoring.upper")
+def upper(payload: EchoInput) -> EchoOutput:
+    return EchoOutput(echoed=payload.text.upper())
+
+
+registry = [echo, upper]
+```
+
+- [ ] **Step 2: Add failing loader tests**
+
+Create `tests/wf_sources_python/test_loader.py`:
+
+```python
+from __future__ import annotations
+
+import pytest
+
+from wf_sources_python import load_python_source
+
+
+def test_load_python_source_from_sequence_registry() -> None:
+    source = load_python_source(
+        source_id="local.ops",
+        module="tests.fixtures.python_source_ops",
+        registry="registry",
+    )
+
+    assert source.id == "local.ops"
+    assert source.kind == "python"
+    assert set(source.capabilities.node_specs) == {
+        "local.ops.echo",
+        "local.ops.upper",
+    }
+    assert source.permissions.safe_for_workflow is True
+
+
+def test_load_python_source_rejects_missing_registry() -> None:
+    with pytest.raises(ValueError, match="missing registry object"):
+        load_python_source(
+            source_id="local.ops",
+            module="tests.fixtures.python_source_ops",
+            registry="missing",
+        )
+
+
+def test_load_python_source_rejects_non_node_spec() -> None:
+    with pytest.raises(TypeError, match="expected NodeSpec"):
+        load_python_source(
+            source_id="local.ops",
+            module="math",
+            registry="pi",
+        )
+```
+
+- [ ] **Step 3: Run failing tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_python/test_loader.py -q
+```
+
+Expected: fail because `wf_sources_python` does not exist.
+
+- [ ] **Step 4: Implement loader**
+
+Create `src/wf_sources_python/loader.py`:
+
+```python
+from __future__ import annotations
+
+from collections.abc import Callable, Mapping, Sequence
+from importlib import import_module
+from typing import Any, Protocol
+
+from wf_authoring import NodeSpec, node
+from wf_platform import (
+    CapabilityBuckets,
+    CapabilitySource,
+    SourcePermissions,
+    SourceVisibility,
+)
+
+
+class PythonSourceConfigLike(Protocol):
+    id: str
+    module: str
+    registry: str
+    enabled: bool
+
+
+def load_python_source(
+    *,
+    source_id: str,
+    module: str,
+    registry: str = "registry",
+    enabled: bool = True,
+) -> CapabilitySource:
+    """Load a trusted in-process Python source from a module registry object."""
+    module_obj = import_module(module)
+    if not hasattr(module_obj, registry):
+        raise ValueError(f"missing registry object {registry!r} in module {module!r}")
+    raw_registry = getattr(module_obj, registry)
+    if callable(raw_registry) and not isinstance(raw_registry, NodeSpec):
+        raw_registry = raw_registry()
+    specs = _coerce_specs(raw_registry)
+    qualified = [_qualify_spec(source_id, spec) for spec in specs]
+    names = [spec.name for spec in qualified]
+    if len(names) != len(set(names)):
+        raise ValueError(f"duplicate NodeSpec names in Python source {source_id!r}")
+    return CapabilitySource(
+        id=source_id,
+        kind="python",
+        enabled=enabled,
+        capabilities=CapabilityBuckets(
+            node_specs={spec.name: spec for spec in qualified},
+        ),
+        visibility=SourceVisibility(
+            planner=True,
+            mcp_client=True,
+            admin_dashboard=True,
+        ),
+        permissions=SourcePermissions(safe_for_workflow=True),
+        description=f"Python source loaded from {module}:{registry}.",
+    )
+
+
+def python_capability_source(config: PythonSourceConfigLike) -> CapabilitySource:
+    return load_python_source(
+        source_id=config.id,
+        module=config.module,
+        registry=config.registry,
+        enabled=config.enabled,
+    )
+
+
+def _coerce_specs(raw_registry: object) -> list[NodeSpec[Any, Any]]:
+    if isinstance(raw_registry, Mapping):
+        values = list(raw_registry.values())
+    elif isinstance(raw_registry, Sequence) and not isinstance(raw_registry, str):
+        values = list(raw_registry)
+    else:
+        values = [raw_registry]
+
+    specs: list[NodeSpec[Any, Any]] = []
+    for value in values:
+        if not isinstance(value, NodeSpec):
+            raise TypeError(f"expected NodeSpec in Python source registry, got {type(value).__name__}")
+        specs.append(value)
+    return specs
+
+
+def _qualify_spec(source_id: str, spec: NodeSpec[Any, Any]) -> NodeSpec[Any, Any]:
+    local_name = spec.name.removeprefix("authoring.")
+    if local_name.startswith(f"{source_id}."):
+        return spec
+    return node(spec, name=f"{source_id}.{local_name}")
+```
+
+Create `src/wf_sources_python/__init__.py`:
+
+```python
+from __future__ import annotations
+
+from .loader import PythonSourceConfigLike, load_python_source, python_capability_source
+
+__all__ = [
+    "PythonSourceConfigLike",
+    "load_python_source",
+    "python_capability_source",
+]
+```
+
+- [ ] **Step 5: Run loader tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_python/test_loader.py -q
+```
+
+Expected: pass.
+
+- [ ] **Step 6: Commit loader slice**
+
+Run:
+
+```bash
+git add src/wf_sources_python tests/wf_sources_python tests/fixtures/python_source_ops.py
+git commit -m "feat: add python source loader"
+```
+
+---
+
+## Task 3: Compose Python Sources Into Local WorkflowServer
+
+**Files:**
+
+- Modify: `src/wf_server/context.py`
+- Modify: `src/wf_server/config.py`
+- Modify: `tests/wf_server/test_config_composition.py`
+
+- [ ] **Step 1: Add failing server composition test**
+
+Add to `tests/wf_server/test_config_composition.py`:
+
+```python
+from wf_config import PythonSourceConfig
+
+
+def test_workflow_config_with_python_source_exposes_capability(tmp_path) -> None:
+    config = WorkflowConfigFile.model_validate(
+        {
+            "version": 1,
+            "server": {
+                "store": {"kind": "filesystem", "root": str(tmp_path / "store")},
+                "sources": [
+                    {
+                        "kind": "python",
+                        "id": "local.ops",
+                        "module": "tests.fixtures.python_source_ops",
+                        "registry": "registry",
+                    }
+                ],
+            },
+        }
+    )
+
+    server = build_workflow_server_from_workflow_config(config)
+
+    assert "local.ops" in server.specs.capability_sources
+    assert (
+        "local.ops.echo"
+        in server.specs.capability_sources["local.ops"].capabilities.node_specs
+    )
+```
+
+Use existing imports in that file; only add missing names.
+
+- [ ] **Step 2: Run failing test**
+
+Run:
+
+```bash
+uv run pytest tests/wf_server/test_config_composition.py::test_workflow_config_with_python_source_exposes_capability -q
+```
+
+Expected: fail because Python sources are ignored or unsupported.
+
+- [ ] **Step 3: Extend `build_local_static_workflow_server`**
+
+In `src/wf_server/context.py`, update `build_local_static_workflow_server` to accept extra sources:
+
+```python
+def build_local_static_workflow_server(
+    store_root: Path,
+    *,
+    extra_sources: Mapping[str, CapabilitySource] | None = None,
+) -> WorkflowServer:
+```
+
+Inside the function, merge sources after built-ins:
+
+```python
+    sources = builtin_sources()
+    if extra_sources:
+        overlap = set(sources) & set(extra_sources)
+        if overlap:
+            raise ValueError(f"duplicate workflow source ids: {sorted(overlap)}")
+        sources.update(extra_sources)
+```
+
+Use `sources` where the function currently passes `builtin_sources()` into `StaticWorkflowSpecProvider`.
+
+- [ ] **Step 4: Wire Python sources in `wf_server.config`**
+
+In `src/wf_server/config.py`, import:
+
+```python
+from wf_sources_python import python_capability_source
+```
+
+Add helper:
+
+```python
+def _python_sources(config: WorkflowConfigFile) -> dict[str, CapabilitySource]:
+    return {
+        source.id: python_capability_source(source)
+        for source in config.server.sources
+        if getattr(source, "kind", None) == "python"
+    }
+```
+
+Import `CapabilitySource` from `wf_platform` for the return annotation.
+
+In `build_workflow_server_from_workflow_config`, change:
+
+```python
+    return build_local_static_workflow_server(store.root)
+```
+
+to:
+
+```python
+    return build_local_static_workflow_server(
+        store.root,
+        extra_sources=_python_sources(config),
+    )
+```
+
+- [ ] **Step 5: Run server composition tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_server/test_config_composition.py tests/wf_server/test_local_static_server.py -q
+```
+
+Expected: pass.
+
+- [ ] **Step 6: Commit composition slice**
+
+Run:
+
+```bash
+git add src/wf_server/context.py src/wf_server/config.py tests/wf_server/test_config_composition.py
+git commit -m "feat: compose python sources in workflow server"
+```
+
+---
+
+## Task 4: Prove CLI/RPC Capability Call And Workflow Run
+
+**Files:**
+
+- Modify: `tests/wf_transport_rpc_http/test_app.py` or `tests/wf_server/test_config_composition.py`
+
+- [ ] **Step 1: Add RPC capability call test**
+
+Add to `tests/wf_transport_rpc_http/test_app.py`:
+
+```python
+async def test_rpc_calls_python_source_capability(tmp_path) -> None:
+    config = WorkflowConfigFile.model_validate(
+        {
+            "version": 1,
+            "server": {
+                "store": {"kind": "filesystem", "root": str(tmp_path / "store")},
+                "sources": [
+                    {
+                        "kind": "python",
+                        "id": "local.ops",
+                        "module": "tests.fixtures.python_source_ops",
+                        "registry": "registry",
+                    }
+                ],
+            },
+        }
+    )
+    server = build_workflow_server_from_workflow_config(config)
+    app = create_rpc_app(server)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        listed = await _rpc(
+            client,
+            "workflow.capabilities.list",
+            {"source_id": "local.ops", "limit": 10},
+        )
+        called = await _rpc(
+            client,
+            "workflow.capabilities.call",
+            {
+                "qualified_name": "local.ops.echo",
+                "payload": {"text": "hello python"},
+            },
+        )
+
+    assert listed["result"]["total"] == 2
+    assert called["result"]["outcome"] == "ok"
+    assert called["result"]["output"] == {"echoed": "hello python"}
+```
+
+Add imports if missing:
+
+```python
+from wf_config import WorkflowConfigFile
+from wf_server.config import build_workflow_server_from_workflow_config
+```
+
+- [ ] **Step 2: Run RPC test**
+
+Run:
+
+```bash
+uv run pytest tests/wf_transport_rpc_http/test_app.py::test_rpc_calls_python_source_capability -q
+```
+
+Expected: pass.
+
+- [ ] **Step 3: Add workflow run test if not covered by capability call**
+
+If the capability call test passes but does not exercise deployment execution, add a direct server/API run test in `tests/wf_server/test_config_composition.py`:
+
+```python
+async def test_python_source_capability_runs_in_deployment(tmp_path) -> None:
+    config = WorkflowConfigFile.model_validate(
+        {
+            "version": 1,
+            "server": {
+                "store": {"kind": "filesystem", "root": str(tmp_path / "store")},
+                "sources": [
+                    {
+                        "kind": "python",
+                        "id": "local.ops",
+                        "module": "tests.fixtures.python_source_ops",
+                    }
+                ],
+            },
+        }
+    )
+    server = build_workflow_server_from_workflow_config(config)
+    await server.api.create_artifact_from_capability(
+        artifact_id="python_echo",
+        version=1,
+        capability_name="local.ops.echo",
+        title="Python Echo",
+        input_map={"text": "input.text"},
+        output_map={"echoed": "state.echoed"},
+        source_bindings={"local.ops": "local.ops"},
+        outcomes=("ok",),
+    )
+    await server.api.save_deployment(
+        {
+            "id": "python_echo.default",
+            "artifact_id": "python_echo",
+            "artifact_version": 1,
+            "bindings": [{"logical_source": "local.ops", "concrete_source": "local.ops"}],
+        }
+    )
+
+    result = await server.api.run_deployment(
+        deployment_id="python_echo.default",
+        workflow_input={"text": "run me"},
+    )
+
+    assert result["status"] == "completed"
+    assert result["output"]["echoed"] == "run me"
+```
+
+If `create_artifact_from_capability` has a different signature, follow the existing closest test in `tests/wf_api/test_capability_api.py` or `tests/wf_transport_rpc_http/test_app.py`.
+
+- [ ] **Step 4: Run focused integration tests**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_python tests/wf_config/test_config_models.py tests/wf_server/test_config_composition.py tests/wf_transport_rpc_http/test_app.py::test_rpc_calls_python_source_capability -q
+```
+
+Expected: pass.
+
+- [ ] **Step 5: Commit integration slice**
+
+Run:
+
+```bash
+git add tests/wf_transport_rpc_http/test_app.py tests/wf_server/test_config_composition.py
+git commit -m "test: prove python source capabilities over rpc"
+```
+
+---
+
+## Task 5: Docs And Verification
+
+**Files:**
+
+- Modify: `docs/current_roadmap.md`
+- Modify: `docs/superpowers/specs/2026-06-11-python-source-provider.md`
+- Move: `docs/superpowers/plans/2026-06-11-static-python-sources.md` to `docs/historical/superpowers/plans/2026-06-11-static-python-sources.md`
+
+- [ ] **Step 1: Update roadmap**
+
+In `docs/current_roadmap.md`, under Priority 3, add:
+
+```markdown
+- Completed: static config `kind: "python"` sources can load trusted local
+  `NodeSpec` registries and expose them through WorkflowServer. Implementation:
+  [`static Python sources`](historical/superpowers/plans/2026-06-11-static-python-sources.md).
+```
+
+- [ ] **Step 2: Update spec status**
+
+In `docs/superpowers/specs/2026-06-11-python-source-provider.md`, change:
+
+```markdown
+Status: planned first non-MCP source family
+```
+
+to:
+
+```markdown
+Status: first static config slice implemented
+```
+
+Add:
+
+```markdown
+## Implementation Status
+
+Implemented:
+
+- `wf_config` accepts `server.sources[]` entries with `kind: "python"`.
+- `wf_sources_python` loads trusted in-process `NodeSpec` registries from
+  `module:registry`.
+- `wf_server.config` composes Python sources into local/static servers.
+- Capability listing/calling works over JSON-RPC.
+
+Still deferred:
+
+- mutable source registry/apply support
+- hot reload
+- reducer exports
+- sandboxing/untrusted code
+```
+
+- [ ] **Step 3: Move completed plan**
+
+Run:
+
+```bash
+git mv docs/superpowers/plans/2026-06-11-static-python-sources.md docs/historical/superpowers/plans/2026-06-11-static-python-sources.md
+```
+
+- [ ] **Step 4: Run verification**
+
+Run:
+
+```bash
+uv run pytest tests/wf_sources_python tests/wf_config/test_config_models.py tests/wf_server/test_config_composition.py tests/wf_server/test_local_static_server.py tests/wf_transport_rpc_http/test_app.py -q
+uv run ruff check src/wf_sources_python src/wf_config src/wf_server tests/wf_sources_python tests/wf_config/test_config_models.py tests/wf_server/test_config_composition.py tests/wf_transport_rpc_http/test_app.py
+uv run ruff format --check src/wf_sources_python src/wf_config src/wf_server tests/wf_sources_python tests/wf_config/test_config_models.py tests/wf_server/test_config_composition.py tests/wf_transport_rpc_http/test_app.py
+uv run basedpyright --level error src/wf_sources_python src/wf_config src/wf_server tests/wf_sources_python tests/wf_config/test_config_models.py tests/wf_server/test_config_composition.py tests/wf_transport_rpc_http/test_app.py
+git diff --check
+```
+
+Expected:
+
+- Tests pass.
+- Ruff passes.
+- Basedpyright reports 0 errors for changed files.
+- `git diff --check` reports no whitespace errors; CRLF warnings on Windows are acceptable.
+
+- [ ] **Step 5: Final commit**
+
+Run:
+
+```bash
+git add src tests docs
+git commit -m "docs: record static python source support"
+```
+
+If earlier task commits were skipped, use one final commit instead:
+
+```bash
+git add src tests docs
+git commit -m "feat: add static python sources"
+```
+
+---
+
+## Acceptance Criteria
+
+- `wf_config` parses `kind: "python"` source entries.
+- `wf_sources_python` imports a trusted module registry and projects `NodeSpec`s under the configured source id.
+- `wf_server.config` includes Python sources in local/static `WorkflowServer` construction.
+- Built-in `wf.std` and `wf.recipes` remain unchanged.
+- No `wf_mcp` dependency is introduced.
+- `workflow.capabilities.list` and `workflow.capabilities.call` work for Python source capabilities over JSON-RPC.
+- Docs make clear this is trusted in-process code and not hot-reloaded or registry-mutable yet.
+