lda-wf/docs/current_roadmap.md

Current roadmap

This roadmap records the product shape, active implementation order, and constraints that must survive future work. Completed narratives and executable plans live under historical/.

Use these references for orientation:

• project_map.md: package map and entry points
• wf_api_architecture.md: application programming interface (API), server, transport, and source boundaries
• wf_core_architecture.md: workflow model and runtime architecture
• wf_cli.md: current command-line interface (CLI) usage

Current product shape

The durable product path uses a neutral server composition behind local or remote clients:

wf_client / wf_cli / web console
  -> local WorkflowApi or JSON-RPC client
  -> wf_server.WorkflowServer
  -> wf_api application and administration surfaces
  -> wf_core / wf_artifacts / source providers

wf-rpc-server is the durable remote entry point. The old wf-mcp entry point remains a legacy or special-purpose Model Context Protocol (MCP) surface.

The Python client covers capability discovery, local graph authoring, remote validation, immutable artifact save, deployment selection, and durable runs. Draft workspaces remain a separate administration surface. The primary saved workflow lifecycle stays:

author -> validate -> save artifact -> deploy -> run -> inspect or resume

Active runtime sequence

The next two slices build on the foreach control-region, structured-context, scheduler, lineage, barrier, and persisted run step budget foundations in this order.

1. Consolidate runtime identity resolution

Introduce one internal resolver for a frame, lineage, runtime scope, and foreach activation environment. The resolver should validate the canonical identity chain once so fork/gather code does not pass related identifiers independently or repeat ownership walks.

2. Implement explicit fork and gather

Reuse the scheduler, activation, lineage, and reducer-aware barrier machinery:

• ADR-0006: explicit fork and topology-driven gather
• Draft contract and verification gates

Production planning remains gated on correlation analysis, merge-order semantics, and closure of the draft's owner-completion, failure-cleanup, scope-boundary, occurrence-state, and checkpoint-integration decisions. The draft records two reproduced reference-model defects and points to archived runnable research; passing that prototype's tests is not approval to copy it into the runtime.

The isolated reference verification plan closes occurrence regressions and gathers executable decision evidence first; it does not authorize production fork/gather changes.

Outcomes continue to choose one transition. Forks create concurrent branch activations. Gathers wait on declared incoming topology, merge compatible lineages according to policy, and emit one continuation.

Runtime work after fork and gather

Defer these slices until the active sequence exposes a concrete need:

• Add optional per-use child deployment overrides and clearer child trace inspection for native subgraphs
• Investigate protocol-native progress or streaming only if polling through wf run watch proves inadequate
• Profile run latency before adding instrumentation: separate test setup, graph execution, validation/serialization, and result printing in the slow rewrite example. Capture a flame graph or time-oriented profile before attributing cost to model_validate. Use the measurements to scope per-node timing, wall-clock correlation timestamps, and possible OpenTelemetry integration; preserve monotonic duration measurement as a separate concern.
• Continue OpenAPI capability sources when a real non-MCP source requires them

Scheduling follow-ups

These are separate improvement slices, not reopened scheduling blockers or new prerequisites for the active fork/gather sequence. Plan them explicitly rather than expanding review-cleanup waves.

• Persistence and concurrency design: concentrate ownership, transaction, and cancellation responsibilities behind explicit guarantees. Keep filesystem roots and lock mechanics inside the file-backed implementation; preserve durable admission, attempt identity, scoped recovery, and join-before-ownership-release tests. Do not introduce an ORM or silently optional transaction guarantees as part of this design work.
• Responsiveness and scale: first address synchronous schedule-admin work on the event loop with cancellation-safe, joined execution and shutdown coordination. Then measure historical-run scans before planning occurrence indexes or active-run indexes; any index needs its own recovery contract.
• Python scheduling ergonomics: design client-owned occurrence objects and pages, computed next-run information, and the placement of schedule mutation methods. Keep calendar behavior library-owned and distinguish a predicted next occurrence from guaranteed execution under capacity and overlap policies.

Thesis and evidence work

Continue the system design and implementation thesis alongside runtime work. Preserve its focus on application purpose, general UI/UX, and execution semantics before introducing implementation details; explain artifacts and deployments in that progression. Revisit diagrams for standalone clarity and keep implemented capabilities separate from future product directions.

The companion shell-backed assistant has a local execution backend; its chat interface is under development. Workflow-client integration and agent-driven authoring remain future evaluation work. Rerun the agent evaluation when that integration is ready; retain the old 36-trial material as historical evidence rather than presenting it as validation of the new experience.

Resolve the missing generated thesis PDF fixtures separately: decide whether tests generate them or the reproducible build supplies them, then document and test that policy. Do not weaken asset checks merely to make CI green.

Durable platform constraints

These rules describe current boundaries. New work should preserve them.

Run and resume correctness

Persisted interrupted runs, bounded trace reads, dependency revalidation, process-rebuild resume, and same-process resume serialization exist.

• Broken pinned dependencies produce blocked readiness and diagnostics
• Live tool or source failures produce failed runs, not implicit pauses
• Store-level concurrency must follow the store transaction boundary
• The current contracts are persisted run and resume and durable workflow runs

Source, authentication, and configuration boundaries

Source registry state, static configuration, runtime source sessions, and authentication records remain separate concerns.

• Keep configuration bootstrap separate from mutable source registry state
• Keep secret payload values write-only; inspection may expose metadata and payload keys
• Keep role-specific stores filesystem-backed until a real database or secret manager slice is planned
• Add new source families through the generic runtime source lifecycle instead of forcing them through MCP connection configuration
• Preserve the server and transport boundary

Current source contracts:

• workflow configuration and sources
• store-backed source registry
• authentication and source secrets

MCP package ownership

wf_sources_mcp owns upstream MCP source implementation. Keep durable server and transport packages independent of the combined wf_mcp facade. Retain compatibility shims only for real callers, move code only when ownership is clear, and keep MCP application or widget metadata out of durable workflow transports.

Established runtime baseline

The active sequence can assume these foundations:

• Native subgraph scopes and durable return to the parent node
• Concurrent foreach with activation barriers and reducer-aware lineage merges
• Validated foreach back-edges with one static control region per node use
• Structured runtime context shared by execution, expressions, validation, and authoring references
• Removal of the pass-through JoinNode; future GatherNode starts with its actual synchronization contract and no placeholder compatibility
• Durable stopped-run inspection and resume
• Persisted run-wide step budget (RunLimits, steps_executed, computed steps_remaining) covering every frame and subgraph scope, surviving checkpoint and resume, with exhaustion as a failed run
• Python client reconstruction of capabilities, artifacts, deployments, and runs through the API

The active sequence can also assume deployment scheduling: an opt-in same-server scheduler for local/static servers only starts ordinary deployment runs without a connected client (one-shot and recurring cron, durable admission, coalesced missed-start recovery, bounded parallel runs, occurrence inspection, and API/Python-client administration). Scheduling is disabled by default and enabled per server; MCP-backed servers reject scheduler enablement. The current contract is deployment scheduling; operator usage is under deployment scheduling operations.

The current foreach return contract is foreach back-edge design. The current context contract is structured runtime context.

Historical entry points

Use these completed roadmaps when implementation history matters:

• API extraction roadmap
• source registry slices
• MCP source connection seam
• MCP runtime session reuse