lda-wf/docs/wf_mcp_plan.md

wf_mcp Plan

This document describes the intended direction of wf_mcp.

The important change in scope is this:

• wf_mcp is not just a tool wrapper layer
• wf_mcp is a namespaced MCP capability broker plus workflow build/run layer

That means it should be able to face:

• human users
• client LLMs
• workflow execution services

without forcing every MCP capability to immediately become a workflow node.

Goals

wf_mcp should:

• manage multiple named MCP backend connections
• persist auth/session state
• discover and cache MCP capabilities per connection
• expose namespaced catalogs to humans and LLMs
• wrap callable tool capabilities into workflow-executable NodeSpecs
• compile and run workflows against those capabilities
• preserve traceability between client actions, MCP backend calls, and workflow runs

Non-goals for the first phase

• full persistence of workflow runs/jobs
• parallel workflow execution semantics beyond what wf_core already supports
• turning every MCP capability type into a workflow node immediately
• hiding protocol complexity by inventing vague magic abstractions

Layering

wf_core

Owns:

• workflow model
• validation
• runtime semantics
• frames
• trace
• interrupts
• foreach

Does not own:

• MCP connections
• auth/session persistence
• discovery caching
• capability brokerage

wf_authoring

Owns:

• @node
• NodeSpec
• WorkflowBuilder
• condition DSL
• subgraph wrapping

Does not own:

• MCP transport/client logic

wf_mcp

Owns:

• connection registry
• auth store
• capability discovery/cache
• namespaced capability catalog
• MCP backend adapter layer
• tool-to-NodeSpec wrapping
• raw plan compilation and workflow execution entrypoints
• future user-facing service/API surface

Mental model

Think of wf_mcp as having two planes.

1. Capability proxy plane

This plane exposes what a backend MCP connection offers.

Capabilities include:

• tools
• resources
• prompts
• notifications/events
• auth metadata
• tasks
• elicitations
• app/server metadata

This plane is about discovery, namespacing, caching, and proxying.

2. Workflow execution plane

This plane decides which capabilities can be used inside workflows and how.

In the first phase:

• tools become executable workflow nodes
• resources and prompts are exposed in catalog/proxy APIs first
• tasks, notifications, and elicitations are modeled and surfaced first, then integrated into workflows later

Connection identity

Connections must be first-class and stable.

Expected shape:

• <server>.<account>

Examples:

• google.personal
• google.work
• github.main
• everything.default

The same backend/server type may have multiple configured connections with different auth, tool availability, or environment.

Namespacing

All exposed capabilities should be explicitly namespaced.

Examples:

• google.personal.tool.list_files
• google.personal.prompt.summarize_folder
• google.personal.resource.drive://folder/abc
• everything.default.tool.echo_tool

The exact string shape can evolve, but these properties should hold:

• globally unique
• reversible back to connection id + local capability id
• understandable by both humans and LLMs

Current implemented slice

Already present:

• connection config and registry
• pluggable file-backed auth/catalog store
• tool catalog snapshots
• raw workflow plan compilation
• workflow execution via async runtime
• lightweight adapter protocol
• fake adapter tests
• real MCP SDK adapter for stdio and streamable_http

This is a good first vertical slice for tool execution.

What "good proxy" means

For wf_mcp, "proxy" should not only mean "call tools on behalf of a client."

It should mean:

• discover backend MCP capabilities and expose them with stable namespacing
• preserve enough metadata that a client can understand what exists and how to use it
• forward or mirror backend-facing events and responses in a traceable way
• keep auth/session boundaries explicit per connection
• separate "what the backend offers" from "what becomes a workflow node"

So the proxy layer should eventually broker:

• tools
• resources
• prompts
• notifications/events
• auth-related state markers
• tasks / long-running operations
• elicitations / human-input capabilities
• app/server metadata

Not all of these need workflow semantics immediately, but they should still have a place in the package model.

Transparent MCP proxy mode

The broker also needs a more protocol-native mode than the current generic broker tools.

The current broker tools are useful for catalog and control-plane inspection:

• get_catalog
• read_broker_resource
• render_broker_prompt
• invoke_broker_method

But these do not make upstream capabilities appear as first-class MCP capabilities to a client like Inspector.

For a real proxy experience, the broker should eventually mirror upstream capabilities into the broker's own MCP surfaces:

• upstream tools/list entries appear in broker tools/list
• upstream tools/call requests route back to the owning connection
• upstream resources/list entries appear in broker resources/list
• upstream resources/read requests route back to the owning connection
• upstream prompts/list entries appear in broker prompts/list
• upstream prompts/get requests route back to the owning connection

Example desired tool entry:

{
  "name": "everything.default.echo",
  "title": "Everything Reference Server: Echo Tool",
  "description": "Echoes back the input string",
  "inputSchema": {
    "type": "object",
    "properties": {
      "message": {
        "type": "string",
        "description": "Message to echo"
      }
    },
    "required": ["message"],
    "additionalProperties": false,
    "$schema": "http://json-schema.org/draft-07/schema#"
  },
  "execution": {
    "taskSupport": "forbidden"
  }
}

Mirrored tools are actual MCP tools exposed by the broker/proxy surface. Do not reintroduce a generic call_tool wrapper for upstream tools; it duplicates the proxy plane and is the wrong abstraction for stateful MCP servers.

Official protocol boundary

The proxy layer should prefer official MCP boundary types over local mirror models.

Use MCP SDK types at the proxy boundary:

• mcp.types.Tool
• mcp.types.Resource
• mcp.types.Prompt
• mcp.types.CallToolResult
• mcp.types.GetPromptResult
• mcp.types.ReadResourceResult

Local models such as DiscoveredTool, CatalogNodeEntry, and NodeSpec remain useful, but they are not the wire contract a standard MCP client understands. They are internal broker/workflow models.

Practical rule:

• MCP proxy plane speaks official MCP types
• workflow plane compiles selected tools into wf_authoring.NodeSpec
• catalog/UI plane can use normalized local models as a persisted view

This keeps protocol compatibility separate from workflow-specific concepts like outcome.

Name mapping

Mirrored capability names must be globally unique and reversible.

Recommended tool name shape:

• <connection_id>.<local_tool_name>

Example:

• everything.default.echo
• github.work.create_issue

This matches the MCP tool-name guidance: ASCII letters, digits, _, -, and . are valid.

The proxy should keep a mapping table:

• broker name -> connection id
• broker name -> upstream kind
• broker name -> upstream local name or URI
• broker name -> display title
• broker name -> enabled/disabled state

This mapping should eventually support UI/admin controls:

• enable/disable a connection
• enable/disable specific tools/resources/prompts
• logout / clear auth for a connection
• refresh a connection
• inspect cached vs live capability state

The UI can come later, but the data model should make room for these states.

FastMCP middleware path

FastMCP's documented tool injection middleware is promising for the first transparent tool mirror.

The relevant hooks are:

• ToolInjectionMiddleware.on_list_tools
• ToolInjectionMiddleware.on_call_tool

These are exactly the two operations needed for first-class mirrored tools:

• inject upstream tools into broker tools/list
• intercept calls to injected tool names and route them upstream

Implementation should verify whether the installed MCP/FastMCP dependency exposes the documented middleware API. If it does, use it. If it does not, treat this as a dependency/version problem and either:

• add the compatible fastmcp dependency explicitly
• or implement the proxy at a lower MCP server/provider layer

Do not encode the proxy as a pile of static decorators. The proxy surface is connection/catalog driven and must be able to change after refresh.

Sampling and elicitation

Sampling and elicitation make protocol-transparent proxying much harder than simple tool forwarding.

Simple request flow:

• client -> broker -> upstream -> broker -> client

Sampling/elicitation flow:

• upstream server may ask the client for a model sample or user input
• broker must forward that request to the original client
• broker must route the client's answer back to the upstream server

For a client like Inspector to show an elicitation form from an upstream server, the proxy must preserve this bidirectional protocol behavior.

Near-term stance:

• mirror normal tools/resources/prompts first
• keep raw method invocation only for explicit protocol/admin diagnostics
• test the Everything Reference Server elicitation tool as the first serious proxy pressure test
• only claim elicitation/sampling support once a live Inspector flow works end to end

Relationship to workflow execution

Transparent proxying and workflow execution are separate concerns.

Proxy path:

• expose upstream tools/resources/prompts to MCP clients
• preserve official MCP semantics
• forward protocol interactions

Workflow path:

• turn selected tools into NodeSpecs
• expose them to wf_core
• add workflow-specific state mapping, outcomes, tracing, retries, interrupts

The proxy must not leak workflow concepts like outcome into MCP tools/list. outcome belongs to workflow execution, not the MCP wire contract.

Next capability expansion

The next architectural move should be broadening the capability model beyond tools.

Add models for

• DiscoveredResource
• DiscoveredPrompt
• DiscoveredNotification
• DiscoveredTaskCapability
• DiscoveredElicitationCapability
• maybe DiscoveredAppMetadata

These do not all need execution semantics immediately.

They do need:

• namespacing
• storage/caching
• catalog exposure

Catalog direction

We should move from a tool-only catalog to a unified capability catalog.

The catalog should be able to expose, per connection:

• tools
• resources
• prompts
• capability metadata
• auth requirements or state markers
• freshness metadata

The client-facing catalog payload should be usable by:

• a human inspector UI
• an LLM that builds workflows
• internal service code

Near-term implementation order

This is the intended order of work from here.

Milestone 1: Broaden discovery models

Add normalized discovery models for:

• resources
• prompts
• notifications
• tasks
• elicitations
• app/server metadata

Outcome:

• wf_mcp can describe more of an MCP backend than just tools
• namespacing rules apply uniformly across capability types

Milestone 2: Unified capability snapshots

Move from tool-only snapshots to connection-scoped capability snapshots that can store:

• tools
• resources
• prompts
• capability metadata
• freshness timestamps
• auth-state hints

Outcome:

• the catalog becomes useful to a real client or inspector
• capability caching is no longer tool-specific

Milestone 3: Service APIs for non-tool capabilities

Expose service methods that let a client:

• inspect resources
• inspect prompts
• read capability metadata
• refresh one connection or all connections

Outcome:

• wf_mcp becomes a usable broker, not only a workflow launcher

Milestone 4: Trace/event correlation

Add a light event model that can connect:

• client action
• backend MCP request/response
• workflow run/step trace

Outcome:

• better observability
• cleaner debugging
• future notification/task support has a home

Milestone 5: Selective workflow mapping

Only after the above is stable:

• keep tools as workflow nodes
• evaluate whether prompts/resources should become helper nodes or stay catalog-only
• map elicitation to workflow interrupts carefully

Outcome:

• workflow integration grows from understood protocol behavior, not guesses

Workflow integration policy

Not every capability becomes a workflow node right away.

Immediate workflow nodes

• tools

Catalog/proxy first, workflow later

• resources
• prompts
• notifications
• tasks
• elicitations

Why:

• tools already match the node call model well
• the others need more careful semantic mapping

Elicitations and interrupts

Elicitations are especially interesting because they align with the workflow interrupt model.

Planned stance:

• expose MCP elicitation capability in catalogs first
• understand the backend semantics first
• later map appropriate elicitation flows into workflow InterruptNode behavior

This should be done deliberately, because workflow interrupt semantics are stronger and more structured than generic protocol-level elicitation.

Tasks and long-running work

Tasks likely align with future run/job persistence, but they should not force that design immediately.

Near-term stance:

• surface task capability in capability catalogs
• understand the task API shape
• leave run persistence mostly out of scope for now

Future direction:

• scheduled jobs
• persisted runs
• polling or event-driven task monitoring

Notifications and traceability

Traceability is a major requirement.

We should eventually distinguish:

• workflow execution trace
• backend MCP call trace
• client-facing event stream

The important design rule is that these should be related, but not collapsed into one giant anonymous log.

Concrete next files

If we follow the plan above, the next likely modules are:

• wf_mcp/discovery.py
  • orchestrate refresh and cache policy for all capability types
• wf_mcp/capabilities.py
  • broader normalized capability models if models.py starts getting crowded
• wf_mcp/events.py
  • event / trace correlation shapes
• wf_mcp/resources.py
  • resource-facing service helpers
• wf_mcp/prompts.py
  • prompt-facing service helpers

Whether these stay as separate files or fold back into models.py / service.py should be driven by clarity, not purity.

Auth and storage

Auth should remain behind a pluggable store interface.

First implementation:

• file-backed store

Expected future replacements:

• database-backed store
• encrypted local store
• secret-manager-backed store

The store should persist:

• auth records
• cached capability snapshots

It may later persist:

• saved plans/workflows
• job specs
• run metadata

Execution model

Near-term execution stance:

• async-first at the MCP layer
• use existing async workflow runtime
• keep workflow runs mostly in memory
• leave room for future scheduled/offline execution

The important offline use case is:

• build workflow once
• execute it later without the LLM in the loop

This is closer to scheduled automation than to interactive planning.

Public API direction

wf_mcp should expose two explicit entrypoints.

1. Convenient/build-style API

For human or higher-level service use.

Examples:

• build workflow from selected catalog items
• helper methods around namespaced tools/resources/prompts

2. Raw plan API

For client LLM use.

This should accept plans that:

• reference namespaced capabilities directly
• avoid raw NodeDef authoring
• still compile down to wf_core.Workflow

Test organization direction

The current test suite is still small enough to live in two top-level files, but it will get noisy if wf_mcp grows beyond tools.

Recommended direction once the next capability work starts:

• tests/wf_core/
• tests/wf_authoring/
• tests/wf_mcp/
• tests/fixtures/

For wf_mcp, likely split by concern:

• tests/wf_mcp/test_store.py
• tests/wf_mcp/test_catalog.py
• tests/wf_mcp/test_service.py
• tests/wf_mcp/test_adapters.py
• tests/wf_mcp/test_sdk_adapter.py

This does not need to happen immediately, but it should happen before one test_wf_mcp.py turns into a junk drawer.

Working rule for the next phase

Before making a new MCP capability executable inside workflows, first answer:

• is this primarily catalog/proxy surface, or workflow surface?
• what is the stable namespaced identifier?
• what is the trace/event story?
• does it need auth/session context?
• does it map cleanly to wf_core, or does it stay above it?

That rule should keep us from forcing everything through the node model too early.