config docs
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,470 @@
|
||||
# Workflow Config Targets and Sources
|
||||
|
||||
Date: 2026-06-03
|
||||
|
||||
Status: design note
|
||||
|
||||
Related:
|
||||
|
||||
- [Long-lived workflow API boundary](./2026-06-03-long-lived-workflow-api-boundary.md)
|
||||
- [Persisted run/resume contract](./2026-06-03-persisted-run-resume-contract.md)
|
||||
- [Current roadmap](../../current_roadmap.md)
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the next workflow config shape before remote CLI targeting and source
|
||||
provider config expand. The key distinctions are:
|
||||
|
||||
- `client.target` chooses where a client sends workflow operations by default.
|
||||
- `server` declares what a long-lived process can host.
|
||||
- `server.sources` declare capability providers available to that server.
|
||||
|
||||
These must not be collapsed into one concept.
|
||||
|
||||
## Client Target
|
||||
|
||||
`client.target.kind` answers: where should this client send workflow
|
||||
operations by default?
|
||||
|
||||
It is a control-plane setting for clients such as `wf`. It should be
|
||||
overridable by CLI flags, for example:
|
||||
|
||||
```bash
|
||||
wf --config wf.json --url http://127.0.0.1:8765/rpc run start demo.default
|
||||
wf --config wf.json --local run start demo.default
|
||||
```
|
||||
|
||||
CLI overrides should affect only the current invocation. They should not mutate
|
||||
the config file unless a dedicated config-write command is used.
|
||||
|
||||
Example local target:
|
||||
|
||||
```json
|
||||
{
|
||||
"client": {
|
||||
"target": {
|
||||
"kind": "local"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This means the CLI builds an in-process `WorkflowApi` from local stores and
|
||||
local source providers.
|
||||
|
||||
Example remote target:
|
||||
|
||||
```json
|
||||
{
|
||||
"client": {
|
||||
"target": {
|
||||
"kind": "rpc_http",
|
||||
"url": "http://127.0.0.1:8765/rpc",
|
||||
"timeout_seconds": 30
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This means the CLI should call the long-lived JSON-RPC server. It should not
|
||||
construct local runtime services for execution unless `--local` or an equivalent
|
||||
override is supplied.
|
||||
|
||||
Recommended Pydantic shape:
|
||||
|
||||
```python
|
||||
from typing import Annotated, Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class LocalTargetConfig(BaseModel):
|
||||
kind: Literal["local"] = "local"
|
||||
|
||||
|
||||
class RpcHttpTargetConfig(BaseModel):
|
||||
kind: Literal["rpc_http"]
|
||||
url: str
|
||||
timeout_seconds: float = 30.0
|
||||
|
||||
|
||||
TargetConfig = Annotated[
|
||||
LocalTargetConfig | RpcHttpTargetConfig,
|
||||
Field(discriminator="kind"),
|
||||
]
|
||||
```
|
||||
|
||||
Do not add `client.target.kind = "mcp"` for the workflow CLI path. MCP can be
|
||||
hosted by the server as one transport, but `wf` should use JSON-RPC for
|
||||
first-class workflow operations.
|
||||
|
||||
## Server Hosting
|
||||
|
||||
`server` answers: what can this process host if started as a long-lived
|
||||
workflow server?
|
||||
|
||||
A server can host multiple transports at the same time. This is required so a
|
||||
user can point `wf` at JSON-RPC while another client points MCP at the same
|
||||
server process.
|
||||
|
||||
Recommended shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"server": {
|
||||
"store": {
|
||||
"kind": "filesystem",
|
||||
"root": ".wf_store"
|
||||
},
|
||||
"transports": [
|
||||
{
|
||||
"kind": "rpc_http",
|
||||
"host": "127.0.0.1",
|
||||
"port": 8765,
|
||||
"path": "/rpc"
|
||||
},
|
||||
{
|
||||
"kind": "mcp_http",
|
||||
"host": "127.0.0.1",
|
||||
"port": 8765,
|
||||
"path": "/mcp"
|
||||
}
|
||||
],
|
||||
"sources": [
|
||||
{"kind": "stdlib", "id": "wf.std"}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Recommended Pydantic shape:
|
||||
|
||||
```python
|
||||
class RpcHttpTransportConfig(BaseModel):
|
||||
kind: Literal["rpc_http"]
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 8765
|
||||
path: str = "/rpc"
|
||||
|
||||
|
||||
class McpHttpTransportConfig(BaseModel):
|
||||
kind: Literal["mcp_http"]
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 8765
|
||||
path: str = "/mcp"
|
||||
|
||||
|
||||
ServerTransportConfig = Annotated[
|
||||
RpcHttpTransportConfig | McpHttpTransportConfig,
|
||||
Field(discriminator="kind"),
|
||||
]
|
||||
|
||||
|
||||
class ServerConfig(BaseModel):
|
||||
store: StoreConfig = StoreConfig(kind="filesystem", root=Path(".wf_store"))
|
||||
transports: list[ServerTransportConfig] = []
|
||||
sources: list[SourceConfig] = []
|
||||
```
|
||||
|
||||
Transport configs describe what the server hosts. They are not workflow
|
||||
sources.
|
||||
|
||||
## Store
|
||||
|
||||
`server.store` answers: where does the server persist workflow platform state?
|
||||
|
||||
The current implementation can start with a filesystem store:
|
||||
|
||||
```json
|
||||
{
|
||||
"server": {
|
||||
"store": {
|
||||
"kind": "filesystem",
|
||||
"root": ".wf_store"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Recommended Pydantic shape for the first slice:
|
||||
|
||||
```python
|
||||
class FilesystemStoreConfig(BaseModel):
|
||||
kind: Literal["filesystem"]
|
||||
root: Path = Path(".wf_store")
|
||||
|
||||
|
||||
class SqliteStoreConfig(BaseModel):
|
||||
kind: Literal["sqlite"]
|
||||
url: str
|
||||
|
||||
|
||||
StoreConfig = Annotated[
|
||||
FilesystemStoreConfig | SqliteStoreConfig,
|
||||
Field(discriminator="kind"),
|
||||
]
|
||||
```
|
||||
|
||||
Only `filesystem` needs implementation immediately. The tagged union exists so
|
||||
the config does not bake the store concept into a single `store_root` field.
|
||||
|
||||
Relative filesystem paths should resolve relative to the config file directory.
|
||||
SQL-backed stores are future work.
|
||||
|
||||
The store should eventually own mutable workflow platform registries:
|
||||
|
||||
- artifact records
|
||||
- deployment records
|
||||
- draft workspaces
|
||||
- run/checkpoint records
|
||||
- source registry entries
|
||||
- source catalog snapshots
|
||||
- source liveness/status metadata
|
||||
- auth records or auth references
|
||||
|
||||
This matters for daemon/server mode. A source added dynamically through an admin
|
||||
tool or UI should survive process restart because it was written to the store,
|
||||
not only held in process memory.
|
||||
|
||||
## Sources
|
||||
|
||||
`sources[].kind` answers: what type of capability provider is this?
|
||||
|
||||
Sources are execution/catalog configuration owned by the server runtime. They
|
||||
are not client target configuration and not transport configuration.
|
||||
|
||||
`server.sources` are bootstrap source declarations. They are loaded at server
|
||||
startup and should be treated as desired startup state, not the only possible
|
||||
source registry.
|
||||
|
||||
Long-term mutable sources should live in a store-backed source registry. The
|
||||
effective source catalog is:
|
||||
|
||||
```text
|
||||
effective sources = config bootstrap sources + stored source registry entries
|
||||
```
|
||||
|
||||
Conflict rules must be explicit:
|
||||
|
||||
- duplicate source ids between config and store should fail fast in the first
|
||||
implementation
|
||||
- config bootstrap sources can be treated as immutable unless copied into the
|
||||
store by an explicit admin action
|
||||
- store-backed sources can be added, disabled, refreshed, or deleted by future
|
||||
admin/UI flows
|
||||
- server startup validates the effective source set before accepting workflow
|
||||
runs
|
||||
|
||||
Recommended shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"server": {
|
||||
"sources": [
|
||||
{
|
||||
"kind": "stdlib",
|
||||
"id": "wf.std"
|
||||
},
|
||||
{
|
||||
"kind": "mcp",
|
||||
"id": "github.work",
|
||||
"server": "mcp-github",
|
||||
"account": "work",
|
||||
"transport": {
|
||||
"kind": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"]
|
||||
},
|
||||
"auth": {
|
||||
"kind": "env",
|
||||
"name": "GITHUB_WORK_TOKEN"
|
||||
}
|
||||
},
|
||||
{
|
||||
"kind": "mcp",
|
||||
"id": "github.personal",
|
||||
"server": "github-2",
|
||||
"account": "personal",
|
||||
"transport": {
|
||||
"kind": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"]
|
||||
},
|
||||
"auth": {
|
||||
"kind": "env",
|
||||
"name": "GITHUB_PERSONAL_TOKEN"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Recommended Pydantic shape:
|
||||
|
||||
```python
|
||||
class StdlibSourceConfig(BaseModel):
|
||||
kind: Literal["stdlib"]
|
||||
id: str
|
||||
|
||||
|
||||
class McpStdioTransportConfig(BaseModel):
|
||||
kind: Literal["stdio"]
|
||||
command: str
|
||||
args: list[str] = []
|
||||
cwd: str | None = None
|
||||
env: dict[str, str] = {}
|
||||
|
||||
|
||||
class AuthEnvRef(BaseModel):
|
||||
kind: Literal["env"]
|
||||
name: str
|
||||
|
||||
|
||||
class McpSourceConfig(BaseModel):
|
||||
kind: Literal["mcp"]
|
||||
id: str
|
||||
server: str
|
||||
account: str | None = None
|
||||
transport: McpStdioTransportConfig
|
||||
auth: AuthEnvRef | None = None
|
||||
|
||||
|
||||
class OpenApiSourceConfig(BaseModel):
|
||||
kind: Literal["openapi"]
|
||||
id: str
|
||||
spec_url: str | None = None
|
||||
spec_path: str | None = None
|
||||
auth: AuthEnvRef | None = None
|
||||
|
||||
|
||||
SourceConfig = Annotated[
|
||||
StdlibSourceConfig | McpSourceConfig | OpenApiSourceConfig,
|
||||
Field(discriminator="kind"),
|
||||
]
|
||||
```
|
||||
|
||||
Exact transport/auth subtypes can expand later. The important decision is that
|
||||
the top-level source union is discriminated by `kind`.
|
||||
|
||||
## Identity Rules
|
||||
|
||||
`id` is the primary source identity.
|
||||
|
||||
Rules:
|
||||
|
||||
- `sources[].id` must be unique across the config.
|
||||
- Artifact and deployment bindings refer to source ids, not `server` or
|
||||
`account`.
|
||||
- `server` and `account` are provider-specific metadata/config, not identity.
|
||||
- Same `server` with different `account` is allowed.
|
||||
- Same `server` and same `account` with different `id` is allowed only when
|
||||
the implementation can prove or report why the entries differ.
|
||||
- No code should split a source id to infer `server`, `account`, provider, or
|
||||
version.
|
||||
- Dotted ids are display-friendly names, not parseable structure.
|
||||
|
||||
This matters for cases like:
|
||||
|
||||
```text
|
||||
github.work -> MCP source using server "mcp-github", account "work"
|
||||
github.personal -> MCP source using server "github-2", account "personal"
|
||||
```
|
||||
|
||||
Both are valid even though their display ids share a prefix. Runtime validation
|
||||
must use the explicit struct fields.
|
||||
|
||||
## Deployment Binding Relationship
|
||||
|
||||
Artifacts describe required logical sources. Deployments bind those logical
|
||||
sources to concrete source ids available on the execution target.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"required_sources": ["github.work"],
|
||||
"deployment": {
|
||||
"bindings": [
|
||||
{
|
||||
"logical_source": "github.work",
|
||||
"concrete_source": "github.personal"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Before run, validation should check:
|
||||
|
||||
- the concrete source id exists on the selected execution target
|
||||
- the concrete source exposes the required capability keys
|
||||
- the pinned artifact/deployment dependencies are still runnable
|
||||
|
||||
If `client.target.kind = "rpc_http"`, the remote server is the source of truth
|
||||
for execution-time source availability. The local CLI config should not pretend
|
||||
its local `server.sources` prove remote runnability unless the same process is
|
||||
also serving that target.
|
||||
|
||||
## Config File Direction
|
||||
|
||||
The current `wf_mcp.config.json` shape is MCP-specific:
|
||||
|
||||
```json
|
||||
{
|
||||
"store_root": ".wf_mcp_store",
|
||||
"connections": []
|
||||
}
|
||||
```
|
||||
|
||||
The workflow platform needs a neutral config shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"client": {
|
||||
"target": {
|
||||
"kind": "local"
|
||||
}
|
||||
},
|
||||
"server": {
|
||||
"store": {
|
||||
"kind": "filesystem",
|
||||
"root": ".wf_store"
|
||||
},
|
||||
"transports": [],
|
||||
"sources": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Compatibility can keep loading old MCP configs for MCP commands, but new
|
||||
workflow CLI/server config should move toward the neutral shape.
|
||||
|
||||
Relative paths should resolve relative to the config file directory.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not redesign artifact/deployment binding models in this note.
|
||||
- Do not implement source provider lifecycle here.
|
||||
- Do not store production secrets directly in JSON config.
|
||||
- Do not make `client.target.kind` influence workflow source identity.
|
||||
- Do not derive identity by splitting dotted source ids.
|
||||
- Do not use MCP as the first-class `wf` CLI target.
|
||||
|
||||
## Next Implementation Slice
|
||||
|
||||
First config implementation should be small:
|
||||
|
||||
1. Add neutral config models for `version`, `client.target`, and `server`.
|
||||
2. Support `client.target.kind = "local"` and
|
||||
`client.target.kind = "rpc_http"`.
|
||||
3. Support CLI overrides for `--local` and `--url` without mutating config.
|
||||
4. Support `server.transports.kind = "rpc_http"` for `wf-rpc-server`.
|
||||
5. Support `server.store.kind = "filesystem"` and map it to existing file
|
||||
stores.
|
||||
6. Support `server.sources.kind = "stdlib"` only, plus a compatibility bridge
|
||||
for the existing MCP config where needed.
|
||||
7. Update `wf_cli.context` so local/remote target selection is explicit.
|
||||
8. Keep store-backed source registry, full MCP source config migration, SQL
|
||||
stores, and `/mcp` hosting for later slices.
|
||||
Reference in New Issue
Block a user