diff --git a/docs/historical/thesis/2026-09-07-retired-agent-evaluation.md b/docs/historical/thesis/2026-09-07-retired-agent-evaluation.md new file mode 100644 index 00000000..a6e25242 --- /dev/null +++ b/docs/historical/thesis/2026-09-07-retired-agent-evaluation.md @@ -0,0 +1,755 @@ +# Retired thesis evaluation and appendices + + + +Archived on September 7, 2026, before removing the earlier CLI campaign +from the active thesis. This is historical evidence, not an evaluation of +the current Python client or a future shell-backed application. + +The sections below preserve the previous wording, including claims and +paths that were valid only for their recorded revisions. Bare repository +paths are rooted at the repository, not this archive directory. + +Supporting artifacts remain at their existing locations to preserve the +reproducible evaluation tooling: + +- [Generated results](../../thesis/agent-challenge-results.md) +- [Cohort manifest](../../thesis/agent-challenge-cohort.json) +- [Evaluation generator](../../thesis/generate_agent_challenge_evaluation.py) +- [Challenge harness](../../../examples/agent_challenges/) +- [Bibliography for historical citation keys](../../thesis/references.bib) + +# Evaluation + +The evaluation uses concrete evidence: automated tests, live smoke tests, and +the deterministic case study. The evidence claim is that the prototype +demonstrates the architecture and workflow lifecycle under controlled examples. + +The current design account is being revised against the September 7, 2026 +implementation. The recorded campaign and verification snapshots below concern +earlier revisions and retain their original dates and interfaces. In particular, +the CLI-based campaign does not evaluate the newer Python client. Later +capabilities require their own evidence. + +## Prototype Conformance Criteria + +The evaluation is organized around prototype conformance criteria derived from +the research question. These criteria test whether the implemented substrate has +the intended lifecycle, validation, source, and inspection behavior under +controlled examples; they do not constitute a broad reliability or user study. +The later Agent Instruction Layer section explains why CLI/API conformance is +necessary but not sufficient for broad agent-success claims. + +| Criterion | Question | Evidence Type | +| --- | ---- | --- | +| Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests | +| Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests | +| Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests | +| Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow and browser-click case studies | +| Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests | +| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests | +| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness | + +: Prototype conformance criteria used for evaluation. {#tbl:prototype-conformance} + +This is a prototype system evaluation, not a broad user study or reliability +benchmark. + +## Separating Design Comparison from Evaluation + +The earlier comparison of n8n, Zapier, and LangGraph explains different +authoring and execution choices. It is not a comparative experiment. The +same task has not been implemented and measured across those systems under +matched conditions, so the evidence here cannot rank their usability, +reliability, or performance against this prototype. + +There are also two distinct questions within the prototype. Execution tests +ask whether contracts, routing, state updates, and persistence behave as +specified. Interaction evaluation asks whether a person or agent can discover +operations, express a procedure, understand errors, and recover without +inspecting implementation code. Passing the first kind of test does not +answer the second. + +The report walkthrough exposes concrete interaction costs, such as explicit +bindings and deployment selection, but does not measure whether those costs +are acceptable to new users. The recorded agent campaign supplies narrower +interaction evidence for its own tasks, interfaces, and instruction profiles. +Neither is a substitute for a comparative user study. + +## Current Walkthrough Check + +During this revision on September 7, 2026, all five Python blocks in the +report walkthrough were extracted from the document and executed in order. +The check substituted an in-process client port for the HTTP connection and +used the example server configuration with an isolated temporary store. +The resulting run completed, its report assertions passed, and its trace +contained four steps, including the explicit end node. + +This checks the documented Python calls against the current service API. It +does not test network startup, HTTP transport, or an independent author +following the instructions. The check was an editorial smoke test, not a new +agent-campaign trial or a permanent regression test. The documentation and +report-example test files were also rerun together: 15 tests passed. + +The historical campaign results and verification snapshot below are retained +with their original scope. These current checks do not update their sample +sizes or retroactively evaluate the Python client. + +## Formative Agent-Trial Findings + +Before the checked 36-trial campaign, exploratory agent runs were used as +design feedback. Prompts, product behavior, workspace isolation, and enabled +tools changed during this period, so these runs are not pooled into the outcome, +duration, or token statistics. They instead provide process-tracing evidence: +recurring agent failures exposed public-surface gaps, and subsequent slices +addressed those gaps. + +| Formative observation | Product or harness response | Engineering interpretation | +| --- | --- | --- | +| Agents could build raw plans through the Python API but could not import them through the public CLI/RPC lifecycle | Added JSON-RPC and `wf artifact create-from-plan` support | A working internal API is insufficient when the agent-facing front door omits it | +| Agents inspected source and tests to infer raw-plan and component shapes | Added the compact and verbose `wf schema` catalog and expanded workflow skills | Public schema discovery is part of the product contract | +| Local CLI mode silently omitted configured Python sources | Routed local CLI composition through the configuration-aware server builder | Equivalent CLI targets must compose equivalent source inventories | +| Output bindings failed when destination schemas or referenced `$defs` were absent | Added capability-aware schema projection, generalized `wf draft bind`, and workflow-output editing | Binding helpers must propagate known schemas rather than force agents to reproduce JSON Schema internals | +| Forward routes failed while the target step had not yet been added | Preserved invalid intermediate drafts and returned direct route-repair guidance | Mutable authoring state must tolerate repairable incompleteness | +| Draft bootstrap bound optional inputs that were absent at run time | Changed capability bootstrap to bind required inputs only and report optional inputs as notes | Best-effort synthesis should avoid inventing runtime requirements | +| Agents misreported source reads or returned reports only as files | Added tool-evidence policy checks, explicit instruction profiles, inline-report requirements, and authoritative manual audit | Agent self-reports are evidence inputs, not final evaluation truth | + +: Formative agent-trial observations that shaped product and harness changes. {#tbl:formative-agent-findings} + +These findings support the design of the operation, repair, and instruction +surfaces. They do not estimate how frequently a new agent or model will encounter +the same failures. + +## Evidence Package + +The evidence supporting the thesis claims is summarized below. + +- **Deployment validation catches source drift.** Evidence: + `test_validation.py`. It asserts that missing, disabled, or changed + capabilities produce diagnostics. Result: pass in the focused test suite. +- **Interrupted runs resume at explicit boundaries.** Evidence: + `test_run_api.py` and resume-concurrency tests. They assert that stopped run + state is persisted and resumed through the run API. Result: pass in the + focused test suite. +- **Python source lifecycle works.** Evidence: + `test_report_workflow_example.py`. It asserts that a Python capability can be + loaded, saved as an artifact, deployed, and executed. Result: pass in the + focused test suite. +- **Serial multi-node workflow works.** Evidence: + `test_browser_click_workflow_example.py`. It asserts that `open_click_page`, + `wait_for_click`, and `collect_snapshots` complete with before/after evidence. + Result: pass in the focused test suite. +- **Bounded agent-operability campaign is checkable.** Evidence: + `agent-challenge-cohort.json`, generated results and figures, local report + hashes, and Appendix C. It asserts that two challenges, two models, three + instruction profiles, and three audited repetitions per cell are explicitly + recorded. Result: 36 audited trials: 27 pass, 8 invalid, 1 fail. +- **CLI and JSON-RPC share the API surface.** Evidence: + `tests/wf_transport_rpc_http/` and `tests/wf_cli/`. They assert that transport + and CLI operations delegate to the same workflow API surface. Result: pass in + the focused test suite. + +The bullet list summarizes repository evidence verified at the recorded commit. + +Historical generated results: see the linked results document above. + +## Verification Snapshot + +This report records one focused verification snapshot to make the evidence +claims auditable from the text. + +| Field | Value | +| --- | --------- | +| Date run | 2026-06-16 | +| Baseline commit | `e24f2892` before subsequent document-polish edits | +| Result | `72 passed in 9.22s` | +| Environment | Local Windows development environment, Python via `uv` | +| Scope | Documentation links, report workflow, browser-click workflow, challenge harness, deployment validation, and run API tests | + +: Focused verification snapshot recorded during document preparation. {#tbl:verification-snapshot} + +Command: + +```powershell +uv run pytest tests/docs tests/examples/test_report_workflow_example.py ` + tests/examples/test_browser_click_workflow_example.py ` + tests/examples/test_opencode_browser_click_challenge.py ` + tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q +``` + +## Implemented Scope Matrix + +| Area | Implemented evidence | Not claimed | Future work | +| --- | ---- | ---- | ---- | +| Workflow lifecycle | Draft, artifact, deployment, run, trace, and list/inspect/resume surfaces | Exactly-once execution or arbitrary mid-node crash recovery | Transactional stores and richer run debugging | +| Source providers | Built-in, MCP, and Python source families | Symmetric feature depth across all providers | Provider add/update/remove/reload lifecycle | +| Execution model | Outcome-routed graph with node, condition, foreach, subgraph, join, interrupt, and end steps | General fork/gather programming model | Parallel fork/gather and aggregation | +| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output, and a bounded 36-trial campaign | Broad model generalization, controlled profile effects, or token reduction | Broader challenge suite and controlled comparative evaluation | +| Auth/security | Auth record plumbing and source diagnostics | Production security, encrypted-at-rest secrets, RBAC, sandboxing | Secret-manager integration and policy enforcement | + +: Implemented scope, explicit non-claims, and future work. {#tbl:implemented-scope} + +### Architecture And Code Walkthrough + +The four-layer architecture (core, API surface, server composition, transport) +is implemented in separate packages with clear boundaries. The Workflow API +Surface is protocol-neutral; JSON-RPC and CLI are transport implementations +that delegate to the same `WorkflowApi` facade. + +### Workflow Lifecycle Tests + +Automated tests cover artifact creation, deployment validation, run execution, +run inspection, and trace retrieval. These tests exercise the full lifecycle +from plan to completed run. + +Evidence: + +- `tests/wf_api/test_artifact_api.py` +- `tests/wf_api/test_run_api.py` + +### Validation And Diagnostics Tests + +Tests verify that draft validation catches schema violations, deployment +validation detects source drift, and diagnostics include repair hints. The +validation tests demonstrate that failed states are machine-readable and include +repair guidance. + +Evidence: + +- `tests/artifacts/test_validation.py` +- `tests/wf_api/test_source_admin_api.py` + +### Source Provider Tests + +MCP source provider tests cover tool discovery, resource listing, prompt +inventory, stateful session reuse, and auth binding. Python source provider +tests cover module import, `NodeSpec` projection, and capability calling. The +tests exercise the source-provider boundary across different source families. + +Evidence: + +- `tests/wf_sources_mcp/test_runtime.py` +- `tests/wf_sources_python/test_loader.py` +- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` + +### Stateful MCP Session Tests + +MCP-backed server tests verify that stateful sessions are reused across +workflow calls rather than creating fresh one-off clients. This demonstrates +source-provider correctness for providers whose behavior depends on session +state. + +Evidence: + +- `tests/wf_sources_mcp/test_runtime.py` +- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` + +### Python Source Case Study + +The report workflow example demonstrates the source abstraction is not +MCP-only. A Python source with three typed capabilities is loaded and exposed +through the source inventory; the automated lifecycle test runs the +deterministic three-node report pipeline through artifact, deployment, and run +records. The browser-click example complements this with a serial three-node +Python workflow. + +Evidence: + +- `examples/report_workflow/` +- `examples/browser_click_workflow/` +- `tests/examples/test_report_workflow_example.py` +- `tests/examples/test_browser_click_workflow_example.py` + +### CLI And Transport Tests + +CLI and transport tests verify that the surface intended for external-agent +operation is exposed through JSON-RPC. Structured output, validation commands, +and inspect commands produce machine-readable responses. + +Evidence: + +- `tests/wf_cli/` +- `tests/wf_transport_rpc_http/` + +### Config Validation + +Config validation catches import and path errors before server startup. This +prevents the server from starting with broken source configurations and +provides earlier, structured failure feedback. + +(Evidence: `src/wf_config/`.) + +## Planner-Efficiency Design Hypothesis, Not Measured Outcome + +The platform targets planner efficiency and operational clarity rather than +runtime throughput. The design hypothesis is that typed contracts, validation, +diagnostics, compact outputs, and traces are intended to reduce blind retries: + +- Validation calls return structured diagnostics with repair hints. +- Source catalogs let agents discover available capabilities without probing. +- Compact JSON output is intended to reduce token usage compared to raw + provider payloads. +- Next-action guidance provides a suggested next step without the agent having + to reconstruct lifecycle state. + +A before/after comparison is illustrative: in early ad-hoc agent/tool +interaction, an agent might spend multiple attempts discovering a valid tool +sequence through trial and error. With the typed lifecycle, the agent validates +a draft, reads the diagnostic, fixes the specific issue, and proceeds. This +report evaluates whether the diagnostic and lifecycle surfaces exist and are +actionable; it does not measure retry reduction, token savings, or +convergence rates across agents. + +The tradeoff is that this lifecycle can require more authoring turns up front: +an agent may discover capabilities, create or patch a draft, validate, save an +artifact, bind a deployment, and validate again before the first production +run. The intended exchange is higher authoring overhead for more deterministic, +inspectable, and reusable runtime execution. + +Threat to validity: the audited agent campaign tests product operability, not +planner efficiency. It changed alongside the prototype and prompt rules and has +no direct-tool-loop baseline. Claims regarding convergence, retry reduction, or +token savings should therefore be interpreted as design hypotheses rather than +experimentally validated results. + +## Agent Instruction Layer + +The product-facing CLI and JSON-RPC surfaces are not sufficient by themselves +for agent operability. External agents also need an instruction layer: skills, +runbooks, and prompt templates that explain the lifecycle, valid command paths, +plan shapes, validation workflow, and failure rules without requiring the agent +to inspect implementation code. + +This became visible in early browser-click challenge trials. When the prompt +or skills did not clearly explain the raw-plan and draft-authoring paths, +agents sometimes looked at tests, source files, prior trial artifacts, or +existing example stores to infer the correct shape. That behavior may still +produce a successful workflow run, but it weakens the evaluation because the +trial no longer measures whether the public product surface and instruction +layer were sufficient. + +For this reason, the challenge report schema tracks read-behavior flags such as +skills, docs, product code, adjacent attempts, prior stores, and existing +solutions. These flags are not moral judgments about an agent; they are audit +metadata. They distinguish product-surface success from success that depended +on reverse-engineering implementation details or reading nearby answers. + +The design implication is that agent-facing infrastructure has three layers: +the operation surface (`wf` and JSON-RPC), the repair surface (validation +diagnostics, traces, compact output, and next actions), and the instruction +surface (skills and runbooks). The bounded campaign measures their combined +operability but does not causally isolate any one layer. + +## Falsifiability Criteria + +The design would fail its own criteria if: + +- source providers routinely required changes to `wf_core`; +- deployments could not detect missing or drifted source requirements before + execution; +- run records could not be inspected or resumed at explicit interruption + boundaries; +- external agents had to import implementation internals rather than using the + public CLI/API lifecycle for ordinary authoring and execution. + +## Evaluation Questions + +The implementation addresses these evaluation questions: + +1. Can a source capability be discovered, called, saved into a workflow, + deployed, and run? --- Demonstrated in controlled tests by the Python source + case study and its automated tests. + +2. Can an interrupted run persist at an explicit interruption boundary and + resume? --- Demonstrated in controlled tests by run persistence and resume + tests. + +3. Can the same server be used through CLI and JSON-RPC transport? --- + Demonstrated in controlled tests: the CLI and transport tests exercise both + surfaces against the same server composition. + +4. Can a new source family be added without changing `wf_core`? --- + Demonstrated for the implemented built-in, MCP, and Python split: the + source-provider boundary is in `wf_platform` and `wf_server`, not in the + core. Future source families should fit this pattern if they can be + projected into the same capability/source contract. + +5. Are large raw provider payloads bounded in CLI output? --- Partially. Source + inventory previews are bounded by `SOURCE_PREVIEW_LIMIT`; `wf cap call` + offers compact/text rendering with `--max-output-chars`. Raw JSON output + remains intentionally lossless. + +6. Can platform sources such as `wf.std` be used without self-bindings? --- + Demonstrated in validation tests: platform sources have + `binding_required: False` in their source policy, and deployment validation + rejects unnecessary platform source bindings. + +7. Can source resources be referenced by logical source and dereferenced + through a bounded helper? --- Demonstrated for `wf.source.read_resource`, + which resolves logical source refs through runtime context with bounded + output policy. + +8. Does the structured surface reduce failed attempts before success? --- Not + measured in this report. The validation diagnostics, compact output, and + next-action guidance are designed for this purpose, but retry reduction + remains future evaluation work. + +9. Do validation and deployment validation catch source drift? --- Demonstrated + in controlled validation tests: deployment validation reports unrunnable + state with diagnostics instead of silently executing against incompatible + capabilities. + +# Case Study Command Transcript + +The following commands demonstrate the full lifecycle of the report workflow +case study. All commands assume execution from the repository root. + +## Config Validation + +```powershell +uv run wf config validate examples/report_workflow/wf.config.json +``` + +## Server Startup + +```powershell +uv run wf-rpc-server --config examples/report_workflow/wf.config.json +``` + +## Status Check + +```powershell +uv run wf --config examples/report_workflow/wf.config.json status +``` + +## Capability Discovery + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` + cap list --source local.report +``` + +## Capability Call + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +cap call local.report.extract_report ` +--input-file examples/report_workflow/cap-input.json --format compact +``` + +## Draft Bootstrap And Focused Edits + +`wf draft create --capability` is a best-effort bootstrap. It creates a one-step +draft from the selected capability's wrapper hints. Focused commands then cover +common edits without requiring the agent to write RFC 6902 patches by hand. + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +draft create report_ws --capability local.report.extract_report ` + --name report_case_study --title "Report Case Study" + +uv run wf --config examples/report_workflow/wf.config.json ` +draft set-name report_ws --revision 1 --name report_case_study + +uv run wf --config examples/report_workflow/wf.config.json ` +draft set-input report_ws --revision 2 --step call ` +--map input.text=text + +uv run wf --config examples/report_workflow/wf.config.json ` +draft set-output report_ws --revision 3 --step call ` +--map title=state.title --map summary=state.summary +``` + +For structural growth, prefer focused helpers such as `draft add-step`, +`draft branch`, `draft handle`, and `draft bind` when they cover the intended +edit. Use `draft patch` only as the low-level fallback, or import a complete +raw plan when the full graph is already available. + +## Draft Validation + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +draft validate report_ws +``` + +## Artifact Saving + +The tested case-study artifact imports the complete three-node plan: + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +artifact create-from-plan examples/report_workflow/workflow.plan.json ` +--artifact report_case_study --version 1 ` +--title "Report Case Study" --outcome ok ` +--binding local.report=local.report +``` + +## Deployment Saving + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +deploy save report_case_study.default --artifact report_case_study ` +--version 1 --binding local.report=local.report +``` + +## Deployment Validation + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +deploy validate report_case_study.default +``` + +## Run Execution + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +run start report_case_study.default ` +--input-file examples/report_workflow/run-input.json ` +--trace-from 0 --trace-limit 5 +``` + +## Run Inspection + +```powershell +uv run wf --config examples/report_workflow/wf.config.json run list --limit 5 +uv run wf --config examples/report_workflow/wf.config.json run inspect +``` + +## Run Trace + +```powershell +uv run wf --config examples/report_workflow/wf.config.json ` +run trace --from 0 --limit 5 +``` + +# Evidence Index + +This appendix maps thesis claims to implementation evidence. It is a guardrail +against unsupported claims and complements the focused verification snapshot in +the Evaluation section. + +## Core Workflow Lifecycle + +Claim: The platform separates mutable drafts, immutable artifacts, deployments, +runs, and traces. + +Evidence: + +- `src/wf_artifacts/models.py`: artifact/deployment models. +- `src/wf_artifacts/runs/`: run records and run store. +- `src/wf_api/service.py`: facade for workflow lifecycle operations. +- `tests/wf_api/test_artifact_api.py` +- `tests/wf_api/test_run_api.py` + +## Source Provider Boundary + +Claim: Workflow execution consumes source-provided capabilities without making +the core runtime MCP-specific. + +Evidence: + +- `src/wf_platform/sources.py`: neutral source DTOs and source policy. +- `src/wf_server/config.py`: server composition for configured sources. +- `src/wf_sources_mcp/`: MCP source family. +- `src/wf_sources_python/`: Python source family. +- `docs/source_architecture.md` + +## Agent-Operable Surface + +Claim: The workflow lifecycle is designed to be operated by external agents +through stable CLI/API surfaces. + +Evidence: + +- `src/wf_cli/` +- `src/wf_transport_rpc_http/` +- `tests/wf_cli/` +- `tests/wf_transport_rpc_http/` +- `docs/wf_cli.md` +- `examples/agent_challenges/browser_click_challenge/`: challenge harness for + CLI-operability trials. +- `docs/thesis/agent-challenge-cohort.json`: explicit 36-trial audited cohort. + +## Validation And Diagnostics + +Claim: Validation and diagnostics make failed workflow states machine-readable +and include repair hints. + +Evidence: + +- `src/wf_artifacts/validation.py` +- `src/wf_api/next_actions.py` +- `src/wf_api/source_admin.py` +- `tests/artifacts/test_validation.py` +- `tests/wf_api/test_source_admin_api.py` + +## Stateful MCP Source Correctness + +Claim: MCP-backed sources can preserve stateful sessions across workflow calls. + +Evidence: + +- `src/wf_sources_mcp/runtime/` +- `src/wf_sources_mcp/client/` +- `tests/wf_sources_mcp/test_runtime.py` +- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` + +## Python Source Case Study + +Claim: The source-provider model is not MCP-only. + +Evidence: + +- `examples/report_workflow/` +- `src/wf_sources_python/` +- `tests/examples/test_report_workflow_example.py` +- `tests/wf_sources_python/test_loader.py` +- `examples/browser_click_workflow/` +- `tests/examples/test_browser_click_workflow_example.py` +- `examples/agent_challenges/browser_click_challenge/` +- `tests/examples/test_opencode_browser_click_challenge.py` + +## Agent Challenge Evaluation Protocol + +Claim: The project has a repeatable protocol and a bounded audited campaign for +evaluating whether external agents can use the product-facing CLI lifecycle. + +Evidence: + +- `examples/agent_challenges/browser_click_challenge/challenge.yaml`: + browser-click challenge manifest with success assertions. +- `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`: + task-specific prompt for the browser-click challenge. +- `examples/agent_challenges/report_workflow_challenge/challenge.yaml`: + report-workflow challenge manifest with success assertions. +- `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`: + task-specific prompt for the report-workflow challenge. +- `examples/agent_challenges/run_trials.py`: + central trial runner accepting any challenge manifest and instruction profile. +- `examples/agent_challenges/manifests.py`: + generic manifest loading and path resolution. +- `examples/agent_challenges/prompts.py`: + prompt composition with base, profile, and challenge fragments. +- `tests/examples/test_opencode_browser_click_challenge.py` +- `tests/examples/test_report_workflow_challenge.py` +- `tests/examples/test_agent_challenge_harness_v2.py` + +Two data-driven challenges exist (browser-click and report-workflow), both +supporting `none`, `skills`, and `all` instruction profiles. The checked cohort +contains three manually audited repetitions for each challenge/model/profile +cell. Its explicit manifest and generated figures appear in the Evaluation +chapter. Because repository snapshots and prompt rules changed across waves, +the results are longitudinal engineering evidence rather than a controlled +model leaderboard. + +## Limitations + +Claim: This is a prototype platform substrate, not a finished automation +product. + +Evidence: + +- `docs/historical/thesis/thesis-outline.md` +- `docs/current_roadmap.md` +- Absence of scheduler, visual-editor, and secret-manager production packages + in the current source tree. + +# Agent Challenge Harness + +## Shared Challenge Protocol + +The agent-challenge harness is an evaluation instrument for the CLI surface +intended for external-agent operation. It deliberately evaluates the +product-facing lifecycle rather than general Python programmability. A valid +solution uses `uv run wf ...` commands for artifact creation, deployment saving, +and run execution. Importing `WorkflowApi`, building `WorkflowServer` directly, +calling source functions directly, or solving the task as a standalone script is +treated as a bypass even if the visible output is correct. + +Both checked challenges accept two product-facing authoring paths: + +1. **Draft path.** Create a draft from one capability, apply focused draft edits + or an RFC 6902 patch, validate, save, deploy, and run. +2. **Raw-plan path.** Write a `RawWorkflowPlan` and load it with + `wf artifact create-from-plan`, then deploy and run. + +The challenge report is an inline YAML self-report with fields for product-path +use, helper-script use, workflow file, deployment id, run id, read-behavior +flags, attempt counts, missed requirements, and challenge-specific assertions. +The harness uses that block for automatic convenience classification, but the +official outcome is manually reviewed. + +## Browser-Click Challenge + +The browser-click challenge asks an external agent to build and successfully run +a workflow that opens a local page with a visible button. The workflow records a +before-click snapshot, performs or waits for a click, records an after-click +snapshot, and returns both snapshots from a deployed workflow run. Its success +contract requires `before_clicked: false`, `after_clicked: true`, no failed run, +and no leftover browser or HTTP-server process. + +## Report-Workflow Challenge + +The report-workflow challenge asks an external agent to build and successfully +run a three-step workflow over a local Python source: `read_notes`, +`extract_report`, and `render_markdown_report`. Its success contract requires a +deployed workflow run, a title matching the expected report title, rendered +Markdown output, and no helper-script or direct-API bypass. + +## Manual Audit Rubric + +Manual review checks the command transcript, the workflow file, the deployment +id, the run id, the run output or trace, and whether the agent read product +source code, adjacent attempts, prior stores, or existing solutions. The +decision precedence is: + +| Condition | Official outcome | Reason | +| --- | --- | --- | +| Product path completed and the audit trail has no disqualifying reads or bypasses | Pass | Supports product-surface operability | +| Product path completed but the agent used a disqualifying source, prior artifact, adjacent attempt, or hidden answer | Invalid | The output exists but cannot support clean evaluation | +| No product-path artifact, deployment, and run evidence | Fail | The task contract was not established | +| Product path exists, but a helper script or direct API bypass materially contributed | Invalid | Output exists, but the trial is contaminated | +| No product-path artifact, deployment, and run evidence; task solved through a helper script or direct API | Fail | The product-facing challenge contract was not established | + +: Manual audit decision rules for agent-challenge trials. {#tbl:agent-challenge-audit-rubric} + +This distinction is intentional. Agent benchmark literature and practice show +that automated scores and self-reports can be misleading when an agent can +inspect hidden answers, prior artifacts, source code, or evaluator state +[@nist-agent-cheating-2025; @openai-swebench-audit-2026]. The harness therefore +records possible invalidation flags such as helper-script bypass, +adjacent-attempt leakage, prior-store reuse, product-code dependency, false YAML +claims, timeouts, parse failures, and missing run evidence. + +## Cohort Manifest And Reproducibility + +The harness, both challenge workflows, and the 36-trial checked cohort are +implemented and manually audited. The Evaluation chapter reports official +outcomes, automatic/manual disagreement, duration, and recorded token totals. +It does not claim controlled model superiority, normalized throughput, or retry +reduction because the product, prompts, and hosted service conditions were not +held constant across waves. + +```{=latex} +\clearpage +``` + +Evidence: + +- `examples/browser_click_workflow/` +- `examples/agent_challenges/browser_click_challenge/` +- `examples/agent_challenges/report_workflow_challenge/` +- `docs/thesis/agent-challenge-cohort.json` +- `docs/thesis/agent-challenge-results.md` +- `tests/examples/test_browser_click_workflow_example.py` +- `tests/examples/test_opencode_browser_click_challenge.py` +- `tests/examples/test_report_workflow_challenge.py` + +```{.mermaid #fig:agent-challenge-audit width=50% latex-placement="H" caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."} +flowchart TB + Transcript[Agent transcript and files] --> YAML[YAML self-report] + YAML --> Classifier[Automatic convenience classification] + Transcript --> Audit[Manual audit] + Classifier --> Audit + Audit --> Outcome[Official outcome] +``` diff --git a/docs/thesis/.gitignore b/docs/thesis/.gitignore index 86a8fdd9..9c582338 100644 --- a/docs/thesis/.gitignore +++ b/docs/thesis/.gitignore @@ -4,3 +4,5 @@ *.typ !figures/*.svg !title-pages.tex +!header-includes.tex +!header-includes.html \ No newline at end of file diff --git a/docs/thesis/evidence-index.md b/docs/thesis/evidence-index.md index a9e0c57d..a4a70e6b 100644 --- a/docs/thesis/evidence-index.md +++ b/docs/thesis/evidence-index.md @@ -1,14 +1,19 @@ # Thesis Evidence Index The claim-to-evidence map now lives inline in -[`system-design-implementation.md`](system-design-implementation.md), Appendix B: +[`system-design-implementation.md`](system-design-implementation.md), Appendix A: Evidence Index. This file remains as a stable pointer for older roadmap and project-map links. -For the external-agent challenge evaluation workflow, including trial profiles, -manual audits, and report interpretation, see +The earlier CLI campaign is retired from the active thesis. Its preserved +text and supporting data are indexed in the +[historical evaluation](../historical/thesis/2026-09-07-retired-agent-evaluation.md). +No replacement shell-backed agent results are claimed yet. + +For the retained challenge harness, including trial profiles, manual audits, +and report interpretation, see [`../runbooks/agent-challenge-evaluation.md`](../runbooks/agent-challenge-evaluation.md). -The primary checked cohort and generated aggregate are +The historical cohort and generated aggregate remain at [`agent-challenge-cohort.json`](agent-challenge-cohort.json) and [`agent-challenge-results.md`](agent-challenge-results.md). diff --git a/docs/thesis/generate.ps1 b/docs/thesis/generate.ps1 index 8428e2ad..e9158d6a 100644 --- a/docs/thesis/generate.ps1 +++ b/docs/thesis/generate.ps1 @@ -2,8 +2,16 @@ # use pandoc -M to change the key: diagram:engine:mermaid:outputFormat to svg or pdf if output is html or pdf. # use the script at stuff/pandoc-diagram.ps1 to set the env vars and pass the filter to pandoc. param( + [Parameter(Mandatory = $true, HelpMessage = "The output type to generate. Must be either 'html' or 'pdf'.")] [string]$type, - [Parameter(ValueFromRemainingArguments = $true)] + [Parameter(Mandatory = $true, HelpMessage = "The input Markdown file to process.")] + [string]$InputFile = (Join-Path $PSScriptRoot "system-design-implementation.md"), + [string]$OutputFile = (Join-Path $PSScriptRoot "system-design-implementation.$type"), + [Parameter(ValueFromRemainingArguments = $true, HelpMessage = "Additional arguments to pass to pandoc. + + Notable use: + - passing --resource-path to specify the directory where images and other resources are located, + - passing --pdf-engine=xelatex to specify the PDF engine for PDF output.")] [string[]]$RemainingArgs ) function New-PandocDiagramMetadata([string] $outputFormat) { @@ -100,24 +108,61 @@ if ($type -eq "pdf") { $title_pages_args = @("--include-in-header", $title_pages_header) } +$tex_preamble_header = Join-Path $PSScriptRoot "header-includes.tex" +if (-not (Test-Path $tex_preamble_header)) { + Write-Error "header-includes.tex is missing. Make sure the thesis preamble header exists." + exit 1 +} +$html_preamble_header = Join-Path $PSScriptRoot "header-includes.html" +if (-not (Test-Path $html_preamble_header)) { + Write-Error "header-includes.html is missing. Make sure the thesis preamble header exists." + exit 1 +} + +$preamble_args = @() +if ($type -eq "pdf") { + $preamble_args += @("--include-in-header", $tex_preamble_header) +} +elseif ($type -eq "html") { + $preamble_args += @("--include-in-header", $html_preamble_header) +} + +$info_md = Join-Path $PSScriptRoot "info.md" +if (-not (Test-Path $info_md)) { + Write-Error "info.md is missing. Make sure the thesis info.md file exists." + exit 1 +} + +$input_files = @($info_md, $InputFile) $metatempfile = New-TemporaryFile $pandocExitCode = 0 +$pandoc_args = $( + @( + "--lua-filter", $include_markdown_filter, + "--lua-filter", $diagram_filter, + "--lua-filter", $figure_format_filter, + "--filter=pandoc-crossref", + "--pdf-engine=xelatex" + ) + + $title_pages_args + + $preamble_args + + @( + "--metadata", "thesisFigureFormat=$outputFormat", + "--metadata", "thesisAgentResults=$agent_results", + "--metadata-file", $metatempfile, + "--embed-resources", "--standalone", "--citeproc" + ) + + $input_files + + @("--output", $OutputFile) + + $RemainingArgs +).Where({ [string]::IsNullOrWhiteSpace($_) -eq $false }) + try { Set-Content -Path $metatempfile -Value $metadata - pandoc ` - --lua-filter $include_markdown_filter ` - --lua-filter $diagram_filter ` - --lua-filter $figure_format_filter ` - --filter=pandoc-crossref ` - --pdf-engine=xelatex ` - @title_pages_args ` - --metadata thesisFigureFormat=$outputFormat ` - --metadata thesisAgentResults=$agent_results ` - --metadata-file=$metatempfile ` - --embed-resources --standalone --citeproc ` - @RemainingArgs + Write-Host "Running pandoc for file $($InputFile | Split-Path -Leaf) to generate $($OutputFile | Split-Path -Leaf)..." + pandoc @pandoc_args $pandocExitCode = $LASTEXITCODE } finally { diff --git a/docs/thesis/gengen.ps1 b/docs/thesis/gengen.ps1 index 4c320266..d95f84b8 100644 --- a/docs/thesis/gengen.ps1 +++ b/docs/thesis/gengen.ps1 @@ -1,6 +1,6 @@ # Generate both HTML and PDF outputs for one Markdown file. param( - [string]$file = $(throw "File is required."), + [string]$file = (Join-Path $PSScriptRoot "system-design-implementation.md"), [Parameter(ValueFromRemainingArguments = $true)] [string[]]$RemainingArgs = @() ) @@ -37,6 +37,6 @@ function Get-OutputFilenames([string] $file, [string] $type) { Write-Host "Generating HTML and PDF for $file..." Write-Debug "Remaining args: $($RemainingArgs | Format-List)" -& $PSScriptRoot\generate.ps1 -type html -- -i $file -o (Get-OutputFilenames $file "html") --resource-path $resourcePath @RemainingArgs +& $PSScriptRoot\generate.ps1 -type html -i $file -o (Get-OutputFilenames $file "html") -- --resource-path $resourcePath @RemainingArgs -& $PSScriptRoot\generate.ps1 -type pdf -- -i $file -o (Get-OutputFilenames $file "pdf") --resource-path $resourcePath @RemainingArgs +& $PSScriptRoot\generate.ps1 -type pdf -i $file -o (Get-OutputFilenames $file "pdf") -- --resource-path $resourcePath @RemainingArgs diff --git a/docs/thesis/header-includes.html b/docs/thesis/header-includes.html new file mode 100644 index 00000000..4e2a99e0 --- /dev/null +++ b/docs/thesis/header-includes.html @@ -0,0 +1,6 @@ + diff --git a/docs/thesis/header-includes.tex b/docs/thesis/header-includes.tex new file mode 100644 index 00000000..9ae53c97 --- /dev/null +++ b/docs/thesis/header-includes.tex @@ -0,0 +1,29 @@ +\usepackage{graphicx} +% \usepackage{booktabs} % exists in the template +\usepackage{tabulary} +\usepackage{hyperref} +\usepackage{hyperxmp} +\usepackage[dvipsnames]{xcolor} +\usepackage{fancyhdr} +\usepackage{float} +\pagestyle{fancy} +\usepackage{seqsplit} + +% Pandoc emits inline code as \texttt{...}. This blunt wrapper keeps long +% paths and commands from overflowing PDF table cells. +\let\origtexttt\texttt +\renewcommand{\texttt}[1]{{\origtexttt{\seqsplit{#1}}}} + +\usepackage{fvextra} +\fvset{breaklines=true, breaknonspaceingroup=true, breakanywhere=true} +\fancyhead[L]{\small lda.chat} +\fancyhead[R]{\small\leftmark} +\fancyfoot[C]{\thepage} +\setlength{\parskip}{0.6em} +\setlength{\parindent}{0pt} +\setkeys{Gin}{width=\linewidth,height=0.55\textheight,keepaspectratio} +\renewcommand{\arraystretch}{1.3} + +% \hypersetup{pdfauthor={lda.chat}, pdftitle={Design and Implementation of lda.chat}} +\usepackage{newfloat} +\DeclareFloatingEnvironment{none} diff --git a/docs/thesis/info.md b/docs/thesis/info.md new file mode 100644 index 00000000..0969152e --- /dev/null +++ b/docs/thesis/info.md @@ -0,0 +1,15 @@ +--- +# this file supplies info, and it is prepend(!) into the other file. +author: "Lương Đức Anh" +student-id: "23BI14021" +--- + +# Acknowledgements {.unnumbered} + +I would like to express my sincere gratitude to **Eng. Trần Văn Trường** at **Dicom Technology Co. Ltd.** for the trust, autonomy, and practical space to pursue this project and develop its technical direction independently. His thoughtful input, critical perspective, and support throughout the development process helped shape the project into a concrete and technically grounded engineering project. + +I am also grateful to **Dr. Nghiêm Thị Phương** at the **University of Science and Technology of Hanoi** for coordinating the university--company requirements of the thesis, providing academic guidance on the submission process, and helping ensure that the final report met the university's formal academic requirements. + +I would also like to thank the **University of Science and Technology of Hanoi**, the **Department of Information and Communication Technology**, and **Dicom Technology Co. Ltd.** for providing the academic and professional environment in which this thesis could be carried out. + +Finally, I am deeply thankful to my family and friends for their encouragement, patience, and support during the development and writing of this thesis. diff --git a/docs/thesis/references.bib b/docs/thesis/references.bib index 0b9d336a..1cb980a6 100644 --- a/docs/thesis/references.bib +++ b/docs/thesis/references.bib @@ -1,3 +1,27 @@ +@online{n8n-merge-2026, + title = {Merge}, + author = {{n8n}}, + year = {2026}, + url = {https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.merge/}, + urldate = {2026-09-07} +} + +@online{zapier-paths-2026, + title = {Add branching logic to Zap workflows with Paths}, + author = {{Zapier}}, + year = {2026}, + url = {https://help.zapier.com/hc/en-us/articles/8496288555917-Add-branching-logic-to-Zap-workflows-with-Paths}, + urldate = {2026-09-07} +} + +@online{langgraph-graph-api-2026, + title = {Graph API overview}, + author = {{LangChain}}, + year = {2026}, + url = {https://docs.langchain.com/oss/python/langgraph/graph-api}, + urldate = {2026-09-07} +} + @online{mcp-tools-2025, title = {Tools}, author = {{Model Context Protocol}}, diff --git a/docs/thesis/system-design-implementation.md b/docs/thesis/system-design-implementation.md index 9514c758..6da82a87 100644 --- a/docs/thesis/system-design-implementation.md +++ b/docs/thesis/system-design-implementation.md @@ -1,8 +1,6 @@ --- title: "Design and Implementation of lda.chat: An AI Agent for Automating and Creating Workspace Workflows" subtitle: "" -author: "Lương Đức Anh" -student-id: "23BI14021" date: "July 1, 2026" lang: "en-US" documentclass: report @@ -41,54 +39,13 @@ keywords: - MCP - Python sources header-includes: - - | - - - \usepackage{graphicx} - - \usepackage{booktabs} - - \usepackage{tabulary} - - \usepackage{hyperref} - - \usepackage{hyperxmp} - - \usepackage[dvipsnames]{xcolor} - - \usepackage{fancyhdr} - - \usepackage{float} - - \pagestyle{fancy} - - \usepackage{seqsplit} - # Pandoc emits inline code as \texttt{...}. This blunt wrapper keeps long - # paths and commands from overflowing PDF table cells. - - | - \let\origtexttt\texttt - \renewcommand{\texttt}[1]{{\origtexttt{\seqsplit{#1}}}} - - \usepackage{fvextra} - - \fvset{breaklines=true, breaknonspaceingroup=true, breakanywhere=true} - - \fancyhead[L]{\small lda.chat} - - \fancyhead[R]{\small\leftmark} - - \fancyfoot[C]{\thepage} - - \setlength{\parskip}{0.6em} - - \setlength{\parindent}{0pt} - - \setkeys{Gin}{width=\linewidth,height=0.55\textheight,keepaspectratio} - - \renewcommand{\arraystretch}{1.3} - # - \hypersetup{pdfauthor={lda.chat}, pdftitle={Design and Implementation of lda.chat}} + # you can not specify -H and this at the same time. diagram: engine: mermaid: theme: neutral --- -# Acknowledgements {.unnumbered} - -I would like to express my sincere gratitude to **Eng. Trần Văn Trường** at **Dicom Technology Co. Ltd.** for the trust, autonomy, and practical space to pursue this project and develop its technical direction independently. His thoughtful input, critical perspective, and support throughout the development process helped shape the project into a concrete and technically grounded engineering project. - -I am also grateful to **Dr. Nghiêm Thị Phương** at the **University of Science and Technology of Hanoi** for coordinating the university--company requirements of the thesis, providing academic guidance on the submission process, and helping ensure that the final report met the university's formal academic requirements. - -I would also like to thank the **University of Science and Technology of Hanoi**, the **Department of Information and Communication Technology**, and **Dicom Technology Co. Ltd.** for providing the academic and professional environment in which this thesis could be carried out. - -Finally, I am deeply thankful to my family and friends for their encouragement, patience, and support during the development and writing of this thesis. - # List of Abbreviations {.unnumbered} | Abbreviation | Meaning | @@ -106,1638 +63,1503 @@ Finally, I am deeply thankful to my family and friends for their encouragement, # Abstract {.unnumbered} -External large language model agents can assemble sequences of tool calls, but -reusable workspace automation also requires lifecycle state, validation, -deployment binding, persistence, and inspectable execution. This thesis presents -the design and implementation of `lda.chat`, a prototype workflow substrate that -separates agent planning from typed runtime execution. The system represents -workflows as outcome-routed graphs and manages them through a -Draft--Artifact--Deployment--Run lifecycle. A neutral source-provider boundary -projects built-in, Model Context Protocol, and Python capabilities into the same -workflow surface, while structured diagnostics and repair guidance support -agent-operable authoring through CLI and JSON-RPC interfaces. An external agent -interface can be layered over these operations; this thesis focuses on the -lower-level substrate that makes such an interface useful rather than proposing -a new autonomous planning algorithm. +Preparing reports, transforming documents, and collecting workspace information +often involve procedures that must be repeated with new inputs. An AI assistant +can help perform such work, but a successful conversation does not itself +preserve an executable procedure. This thesis presents `lda.chat`, a +programmable workflow platform for defining, checking, running, and inspecting +reusable workspace procedures. -The implementation is evaluated through automated conformance tests, a -deterministic three-node report workflow, a browser-interaction workflow, and a -manually audited external-agent campaign. The campaign contains 36 trials across -two challenges, two hosted models, three instruction profiles, and three -longitudinal waves. Manual audit, performed by the author, classified 27 trials -as clean product-path passes under the campaign rules, eight as invalid -evaluation samples, and one as a failure. These counts are not a -model-success-rate estimate. The disagreement -between automatic completion and manual outcomes demonstrates why successful -execution alone is insufficient evidence when agents can inspect implementation -files, prior artifacts, or evaluator state. +A workflow describes the operations to perform, the data they exchange, and +the decisions that select the next step. An **artifact** is an immutable saved +version of that workflow. A **deployment** connects a saved version to the +concrete services it will use. A **run** records one execution, including its +inputs, status, result, and trace. These distinctions separate revising a +procedure, configuring where it operates, and examining what happened. -The contribution is architectural rather than algorithmic: a typed workflow -lifecycle, a provider-neutral capability boundary, and an agent-operable -validation and inspection surface implemented as a working prototype. The study -does not establish production security, broad model generalization, or reduced -token use; the agent campaign records evolving product and prompt snapshots and -is therefore longitudinal engineering evidence rather than a controlled model -comparison. +The design addresses both authoring experience and execution behavior. Authors +need to discover operations, connect their inputs and outputs, understand +validation errors, and inspect results. The runtime needs corresponding rules +for data contracts, control flow, state updates, and interruption. The thesis +compares these concerns with other workflow systems and explains the prototype's +choice to separate a node's data output from its routing outcome. + +The implementation provides Python authoring objects, a workflow service, and +adapters for trusted Python functions and external tools. Evidence includes +a deterministic case study and focused conformance tests. These establish +specific engineering results, not broad usability, model superiority, or +reduced token use. The system remains in +development, with further work needed on authoring ergonomics and user +experience. # Introduction -`lda.chat` is positioned as an AI-agent-facing workflow platform. An agent -interface can be implemented as a surrounding layer that combines a chat or web -front end, a planner graph, and `wf` CLI/API operations exposed as tools. This -thesis focuses on the workflow substrate beneath that layer: typed lifecycle -records, source bindings, validation, execution, diagnostics, traces, and -resumability boundaries. The contribution is therefore the infrastructure that -lets external agents and human operators create reusable workspace workflows, -not a new autonomous planning algorithm. +Preparing a weekly project report involves collecting notes, extracting actions +and risks, checking the result, and producing a document that other people can +use. A person can do this manually, write a script, or ask an AI assistant for +help. When the procedure becomes recurring, the workspace also needs a way to +preserve it, supply different inputs, and inspect unsuccessful attempts. -This report assumes a setting in which external LLM agents are used as workflow -authors and operators, and asks what platform substrate they need for reusable -workspace automation. It describes the design and implementation of `lda.chat`, -a prototype platform where agents can author, validate, execute, and inspect -reusable workspace workflows without making the LLM itself responsible for -runtime state, validation, source binding, or persistence. +`lda.chat` is a programmable workflow platform for that recurring work. +It allows a human or external agent to assemble available operations into a +saved procedure and execute it through a service. The procedure exists +independently of the conversation or programming session that created it. +Its executions can be inspected separately, and explicit requests for additional +input can be resumed from saved state. -The central claim is that agent-facing workflow automation should separate -planning from execution. The LLM or human author can propose and revise workflow -structure, while the platform owns artifacts, deployments, runs, source -inventory, validation diagnostics, traces, and resumability. +For the report task, the initial procedure is short: -The research question guiding this work is: how can an AI-agent-facing workflow -platform represent, validate, execute, and persist reusable workspace -automations while keeping planning separate from deterministic execution? +Read the notes, extract a structured report, and render it as Markdown. -The short version of the thesis is: the LLM plans; the runtime executes; source -providers expose capabilities; stores preserve persisted lifecycle records. The -implementation demonstrates this model across controlled built-in, MCP, and -Python source examples. +The operator supplies new notes each week rather than rebuilding those steps. +If the requirements change, the author revises the procedure and saves another +version. If execution fails, the operator needs to identify the affected step +and its inputs, not merely observe that no report appeared. -**Scope of claims.** This report does not claim production security, broad or -representative external-agent evaluation, arbitrary mid-node crash recovery, -scheduling, role-based access control, general workflow parallelism, or a -bundled autonomous planning layer. It reports a bounded, manually audited -36-trial agent-operability campaign. Claims about planner efficiency remain -design hypotheses: the campaign was not a controlled retry-reduction or token -efficiency experiment. +## From a useful procedure to a usable system -## Contributions +The intended beneficiary is a workspace operator who needs repeatable work. +Its author may be that person, a developer, or an external agent acting on their +behalf. Authoring a procedure and operating a saved one are different activities; +neither requires an LLM to participate in every execution. -This work makes five architectural and systems-engineering contributions: +A workflow system must help its author answer practical questions: which +operations are available, what information they require, and how one operation's +result becomes another's input. During operation, it must distinguish an invalid +definition, a missing service, a failed execution, and a request for more input. +These are interaction-design concerns as well as runtime concerns. An interface +that draws a branch without explaining whether one path or both will execute +leaves a consequential rule implicit. -1. It defines a typed Draft--Artifact--Deployment--Run lifecycle for workflows - authored and operated by external agents. -2. It separates planner decisions from runtime execution, persisted state, - validation, and trace collection. -3. It defines a provider-neutral capability boundary through which built-in, - MCP, and Python sources share one workflow model without provider logic in - the core runtime. -4. It exposes structured validation diagnostics, repair hints, next-action - guidance, and inspection surfaces intended for agent-operable authoring. -5. It implements and evaluates the design through deterministic case studies, - automated conformance tests, and a bounded manually audited agent campaign. +The current prototype emphasizes programmable authoring and inspection. +Its Python interface presents workflow and run objects instead of requiring +authors to construct network messages. This supports code-based use, but does +not establish accessibility for non-programmers. Onboarding, edit feedback, +error presentation, and the effort needed to understand a workflow remain +areas for user-experience improvement. -These contributions establish the feasibility and internal coherence of the -prototype architecture. They do not claim a new workflow algorithm or empirical -superiority over mature orchestration systems. +## Engineering question and contribution + +The engineering question is how to make a reusable workspace procedure both +operable through public interfaces and explicit enough for the runtime to +validate, execute, and record. + +The contribution connects three design choices: + +1. A typed graph makes operations, data movement, and routing decisions + inspectable rather than leaving the procedure only in a conversation. +2. Separate saved versions, environment bindings, and execution records make + editing, configuration, and operation distinct activities. +3. Programmable authoring, validation diagnostics, and run inspection expose + those distinctions to human and agent clients. + +The work integrates established workflow techniques rather than introduces a +new autonomous planning algorithm. Its design is assessed from both sides: +what an author must understand and do, and what the runtime guarantees when +the procedure executes. The comparison with related systems uses concrete +mechanisms rather than treating visual polish or graph notation as sufficient +evidence of either usability or correctness. + +## Scope of the implementation + +The prototype supports conditional routes, iteration, child workflows, and +explicit interruption/resume. Its runs have an execution-attempt limit. +General parallel fork/gather remains proposed; persisted interruption does not +mean recovery halfway through arbitrary external code. The evaluation separates +tested behavior from unmeasured usability and production-readiness claims. ## Report Outline -Section 2 frames the problem that motivates a separate execution substrate. -Section 3 positions the system against related approaches. Section 4 describes -the conceptual model of workflows, artifacts, deployments, runs, and source -bindings. Section 5 presents the system architecture and its layered boundaries. -Section 6 details the implementation of each layer. Section 7 walks through a -deterministic report-preparation case study backed by a Python source. Section 8 -evaluates the implementation against concrete evidence. Sections 9 and 10 -discuss limitations and future work. Section 11 concludes. +Section 2 derives interaction and execution requirements from the workspace +task. Section 3 compares ways to author and operate that task in related +systems. Section 4 explains the prototype's concepts through an example. +Sections 5 and 6 describe the architecture and implementation; Section 7 presents +the reproducible case study. Section 8 evaluates the available evidence. +The remaining chapters discuss limitations, future work, and conclusions. # Problem Statement And Requirements -A common pattern in agent systems lets an LLM orchestrate side effects -through sequential tool calls. ReAct-style prompting demonstrates interleaved -reasoning and action, while Toolformer-style work demonstrates learned external -API/tool use [@react-2022; @toolformer-2023]. -The problem statement here is narrower: when a tool loop is used as a reusable -workspace automation substrate, several practical platform concerns appear. +A procedure that works once is not necessarily ready for repeated operation. +For the report example, a renamed field may invalidate extraction, a different +workspace may use another notes service, or the input may omit information +needed by the final document. The author needs feedback that distinguishes these +situations, while the runtime needs rules for handling them. -- **Weak validation before execution.** A planner that assembles tool-call - sequences often lacks a typed contract describing what each step expects and - produces. Invalid plans reach the runtime and fail at execution time rather - than during authoring. Structured-output work supports the design assumption - that schema adherence can be treated as an API/runtime contract rather than - left entirely to planner inference [@openai-structured-outputs-2024]. +LLM tool-use approaches illustrate dynamic selection of actions +[@react-2022; @toolformer-2023]. This thesis does not assume that such systems +cannot use schemas or persistence. It asks which responsibilities should be +explicit in the reusable procedure rather than depend on the authoring session. -- **Poor resumability after interruption.** Raw tool-call loops do not - checkpoint their progress. If the process restarts, the agent must reconstruct - its prior state from scratch or lose work. Durable agent frameworks expose - persistence/checkpoint layers specifically because continuation, failure - recovery, and memory across interactions are runtime concerns - [@langgraph-persistence-2026]. +## Authoring and operation requirements -- **Hard-to-audit traces.** Successful tool-call chains leave logs, but the - causal structure of a multi-step procedure is not separated from the transport - or provider noise. Inspecting what happened, why a step failed, or what the - intermediate state was requires manual log parsing. Recent - agent-auditability and LLM-accountability work frames action recoverability, - lifecycle coverage, and evidence integrity as explicit requirements - [@auditable-agents-2026; @audit-trails-llm-2026]. +The following requirements describe what the interaction should support. +They are design goals, not claims that every aspect of the current experience +has been validated with users. -- **Limited reuse.** A successful tool-call procedure is embedded in a - conversation transcript or script. Extracting it into a named, versioned, - redeployable artifact is manual work the agent is not equipped to perform - reliably. +1. **Discover before connecting.** Show available operations and the inputs, + outputs, and services they require. The author should not need to inspect + server implementation code to learn how an operation can be used. +2. **Make data movement understandable.** Explain how a document path becomes + text, how text becomes structured fields, and which fields reach the result. + Distinguish a data connection from a decision about what executes next. +3. **Support revision and useful feedback.** An author should be able to revise + an unfinished procedure and locate errors in the relevant step or binding. + Diagnostics should distinguish what must change from what remains valid. +4. **Separate editing from running.** Make clear which saved version an execution + uses. Editing the next version should not silently change an earlier one. +5. **Explain execution state.** Distinguish completed, failed, and interrupted + runs, expose relevant intermediate evidence, and identify the input required + to resume an interruption. -- **Unclear boundaries between planning, execution, and provider-specific - state.** When an LLM is responsible for both deciding what to do and - managing runtime state, auth tokens, session pools, or source catalogs, the - two concerns become entangled. Provider drift, stale sessions, or auth - failures become hard to diagnose. +For machine clients, these interactions need structured responses and stable +identities. For human authors, they also need understandable terminology and +manageable amounts of information. Providing structured output satisfies an +interface requirement; it is not proof that the overall experience is usable. -The automation target for this platform is reusable workspace procedures, not -arbitrary office work end-to-end. Examples include document transformation, data -collection, tool and API calls, report preparation, and monitoring checks. -Scheduled execution is a future deployment mode, not implemented in this -prototype. The thesis frames the platform as a response to these pressures: a -typed execution substrate where persisted lifecycle records, validation, source -binding, and trace inspection are first-class platform concerns rather than -responsibilities of the planner. +## Execution requirements behind the interaction -The design requirements that follow from this problem statement are: +Those interactions require corresponding runtime contracts: -1. Typed workflow artifact, deployment, and run lifecycle with explicit schemas. -2. Source-provider boundary implemented for built-in, MCP, and Python sources, - and designed to admit future source families that can be projected into the - existing capability/source contract. -3. Server, API, and CLI surfaces intended for external-agent operation, backed - by persisted lifecycle stores. -4. Validation and inspection mechanisms intended to reduce planner - trial-and-error. -5. Next-action guidance that points an agent toward useful lifecycle operations - without replacing validation. -6. Deterministic execution for the thesis-critical evidence path. +1. **Preserve definitions and executions independently.** Save an identifiable + procedure version and keep separate records for its invocations. +2. **Validate known constraints before work starts.** Check graph structure, + declared data contracts, mappings, and required service bindings without + pretending to predict every external failure. +3. **Specify routing and state changes.** Define what selects the successor, + where outputs are written, and how repeated writes affect workflow state. +4. **Keep environment choices outside the procedure's logic.** Resolve logical + service requirements to concrete configured services and report mismatches. +5. **Bound and inspect execution.** Limit execution attempts, retain status and + trace information, and support resume at defined interruption boundaries. + +These requirements motivate the artifact, deployment, and run distinctions. +They do not prescribe those names for every competing system. + +## Costs and boundaries + +Schemas, bindings, and versioned definitions impose authoring work. A script +can be preferable for a short-lived task, especially when its author already +understands the libraries involved. The platform targets cases where reuse and +inspection justify that setup; the thesis does not establish a numerical +break-even point. + +A workflow interface must also avoid promising more than its runtime supports. +Successful validation does not guarantee a remote service will succeed, and a +saved interruption does not make arbitrary external effects reversible. +Scheduling and general fork/gather remain separate development work. # Positioning And Related Systems -The system occupies a specific position in the automation landscape. It does -not attempt to replace mature platforms in their strengths, but rather explores -a different center of gravity: typed lifecycle contracts intended to be driven -by external AI agents. The comparison below is qualitative positioning, not a -benchmark across products. +The report task provides a common lens for comparison: choose operations, pass +notes between them, add a route for incomplete information, test the procedure, +and inspect its result. The comparison follows these authoring activities into +the execution rules they expose. -## Direct LLM Tool Orchestration +The accounts below use documented mechanisms consulted on September 7, 2026. +They are not hands-on usability measurements or a complete product survey. +Differences in authoring style do not establish that one system is easier for +every audience. -Direct tool orchestration through an LLM is the most open-ended approach: the -planner can choose tools dynamically and adapt immediately. In this report's -framing, that flexibility becomes a problem when the tool loop is also expected -to provide persisted lifecycle records, validation, audit structure, and -resumability. The platform argues that reusable workspace automation benefits -from separating planning from a typed execution substrate. +## Starting with code or direct tool calls -This comparison is to the bare tool-loop pattern, not to a tool loop embedded -inside an additional workflow, tracing, persistence, or orchestration framework. +A developer can express the report procedure as function calls and use the +language's conditionals and loops. An agent can instead select successive tool +calls from the information available at each turn. Both approaches can be +combined with schemas, tests, logs, and persistence. -## Generated Scripts +Code puts the procedure close to its implementation and makes ordinary debugging +tools available. It also leaves decisions about configuration, saved versions, +and run records to the program or its surrounding infrastructure. A workflow +platform makes some of those decisions part of its public contract, at the cost +of introducing another model for the author to learn. -Generated scripts are a serious baseline. For many tasks, a script is simpler, -more maintainable, and easier to debug than a workflow graph. The platform -argument is that reusable workspace automation benefits from lifecycle -affordances that scripts do not automatically provide: typed validation, source -binding, artifact/deployment separation, run records, resumability, trace -inspection, and diagnostics with repair hints. +The prototype still uses Python for authoring. The distinction is whether that +code performs the entire procedure directly or constructs a saved workflow for +the service to execute. Neither representation is automatically better for a +one-off task. -A script can be wrapped with these affordances, but then the comparison shifts -from "script" to a custom workflow platform assembled around the script. +## n8n: connecting and inspecting data -## Workflow Automation Platforms +For an author combining report records, n8n's Merge node exposes a concrete +choice: append the incoming collections, match records by fields or position, +or produce combinations. The node's configuration and worked examples make +these operations distinguishable. Append waits for connected inputs and emits +their items in input order [@n8n-merge-2026]. -Zapier-style automation platforms are stronger today at polished -non-programmer UIs, large integration catalogs, hosted scheduling and triggers, -and operational maturity. This report uses Zapier as a representative hosted -automation platform rather than surveying the full RPA/workflow market. -Zapier's own documentation describes a hosted, stateless runtime with explicit -execution-time and payload constraints, plus published Zap limits and rate -limits [@zapier-operating-constraints; @zapier-zap-limits]. The prototype does -not claim feature parity with these products. Instead, it explores a different -trade-off: a platform exposing the full lifecycle through typed contracts -intended for external-agent operation, where local Python and MCP sources share -one workflow surface, and where artifacts, deployments, runs, and traces are -first-class inspectable records. +This connects an interface decision to an execution rule. Selecting a merge mode +does not merely change a diagram: it changes which records appear in the output. +For the report task, joining actions by owner is different from appending two +action lists. -## Agent Graph Frameworks +The lesson for the prototype is that a connection needs an understandable data +meaning. Its state reducers describe how writes update fields; they should not +be presented as interchangeable with an item join or synchronization barrier. +This comparison concerns the documented Merge mechanism, not every n8n node. -LangGraph-style durable agent graphs share the idea of typed execution -substrates for agent workflows. LangGraph's official documentation positions it -as an orchestration runtime for long-running, stateful agents, with persistence, -human-in-the-loop behavior, and durable execution [@langgraph-overview-2026; -@langgraph-persistence-2026]. This is not a claim that `lda.chat` is more -durable or more general than LangGraph. The difference claimed here is the -artifact/deployment/run lifecycle and source-provider binding model for -reusable workspace automations. +## Zapier: configuring a decision -## Model Context Protocol +An author using Zapier Paths selects fields, conditions, and values, then tests +the rules against sample data. Applied to the report task, those rules could +distinguish complete records from records needing attention. Multiple paths +can qualify, so exclusivity must follow from the rules rather than the +branching appearance alone [@zapier-paths-2026]. -MCP is a useful protocol for exposing tools, resources, and prompts. Its -official lifecycle is a client-server connection lifecycle: initialization, -operation, and shutdown [@mcp-tools-2025; @mcp-lifecycle-2025]. It is not -itself the workflow artifact, deployment, and run lifecycle. The lda.chat -platform treats MCP as one source family behind a provider boundary, not as the -product identity. This distinction is important: MCP demonstrates why -source-provider correctness matters, because a source may require persistent -sessions, auth context, catalog refresh, and prompt inventory. The platform -places this complexity behind a neutral `CapabilitySource` interface. +The same documentation describes sequential execution of qualifying paths. +Paths does not provide a shared action after all branches; common steps can be +duplicated or placed in a Sub-Zap. These constraints affect how the author +organizes a common report-rendering step [@zapier-paths-2026]. -These sources contextualize the comparison; the implementation claims in this -report remain grounded in repository evidence. +The interface therefore needs to communicate both the condition being tested +and the consequence of a match. In the prototype, an ordinary outcome selects +one successor. That is a different contract, not a claim of superior usability. -## Positioning Summary +## LangGraph: describing decisions in Python -The related approaches differ primarily in their center of gravity. The table -summarizes the comparison made in this chapter without claiming feature parity -or product superiority. +LangGraph's Graph API lets an author define state, add node functions and +routing, compile the graph, and invoke it. For the report task, a developer can +represent extracted fields in state and write a routing function that selects +what happens next [@langgraph-graph-api-2026]. -| Approach | Primary strength | Lifecycle and validation position | Relation to this work | -| --- | --- | --- | --- | -| Direct LLM tool loop | Dynamic adaptation and low authoring overhead | Durable records, validation, and replay require surrounding infrastructure | The planner remains external; reusable procedures move into a typed substrate | -| Generated script | Simplicity, debuggability, and direct access to libraries | Versioning, deployment binding, run records, and repair diagnostics are manual additions | Scripts remain a valid baseline; the prototype targets repeated managed execution | -| Hosted automation platform | Integration breadth, scheduling, UI, and operational maturity | Rich but platform-specific lifecycle and operational contracts | The prototype does not seek feature parity; it exposes a local typed lifecycle for agents | -| Durable agent graph framework | Stateful, long-running agent execution and checkpointing | Persistence and human-in-the-loop execution are first-class concerns | The distinction is the explicit artifact/deployment/run model and source binding for workspace automation | -| `lda.chat` prototype | Agent-operable lifecycle, provider-neutral sources, and structured repair surfaces | Implemented for controlled examples; production operations remain incomplete | Architectural subject of this thesis | +This authoring style exposes more behavior as code. Nodes produce state +updates, reducers determine how updates combine, and conditional routes select +subsequent execution. The documented model supports graph loops and super-step +execution; its checkpoint facilities are described separately +[@langgraph-graph-api-2026; @langgraph-persistence-2026]. -: Positioning summary for related workflow and agent-system approaches. {#tbl:positioning-summary} +The prototype shares the use of typed state and reducers, but represents +ordinary routing as a mapping from a declared outcome to a successor. +This keeps executable predicates out of ordinary edges. The cost is that some +decisions need a dedicated condition step or an additional declared outcome. +The relevant comparison is where authors express and inspect decisions, not +whether Python or a canvas is inherently the better interface. + +## Implications for this design + +The examples expose three connected design concerns: authors must understand +the data exchanged, the condition selecting work, and the meaning of execution +progress. A graph drawing alone does not answer any of them. + +| Author's question | Execution concept it exposes | +| --- | --- | +| Which result should the next step receive? | Input/output mapping | +| Will one branch run, or several? | Exclusive routing or concurrent emission | +| What happens where paths meet? | Continuation, data merge, or a barrier | +| Why did this attempt stop? | Failure versus explicit interruption | +| What does changing the workflow affect? | Saved version versus a run | + +: Interaction and execution concepts. {#tbl:positioning-summary} + +The prototype prioritizes programmable authoring, explicit mappings, and saved +execution records. Its current interface must still be evaluated for the work +required to discover operations, repair definitions, and interpret results. +Neither its typed models nor the absence of executable edge predicates proves +that it achieves those user-experience goals. + +External-tool protocols are a separate concern. Model Context Protocol (MCP) +exposes tools, resources, and prompts; it is not a competing graph model +[@mcp-tools-2025; @mcp-lifecycle-2025]. Here MCP is a source family, not the +product identity. Protocol details belong later, after the authoring and +execution concepts they support have been explained. # Conceptual Model +The report example introduces the concepts in the order an author encounters +them: choose operations, connect their data, define decisions, save a version, +and inspect an execution. The following branching example explains supported +primitives; it is not an additional measured case study. + +## Operations, data, and decisions + +Suppose report preparation must ask for missing information before rendering: + +```{.mermaid #fig:report-branch width=95% caption="Report routes."} +%%{init: {"flowchart": {"rankSpacing": 20, "nodeSpacing": 20}}}%% +flowchart LR + Read["Read
notes"] --> Extract["Extract
report"] + Extract --> Check{Complete?} + Check -->|ready| Render["Render
report"] + Check -->|needs_information| Ask["Request
information"] + Ask -->|submitted| Render + Render --> Finish([End]) +``` + +Each named operation is a **node** in the workflow. An **edge** selects the next +step after a node produces an **outcome**, such as `ready` or +`needs_information`. The node's **output** is the data it returns, such as the +extracted report fields. A missing-information outcome is a business decision; +an exception while reading a file is a runtime failure. + +The arrow to the next step does not implicitly pass all previous output into +that step. The author defines input mappings and writes relevant outputs into +workflow state. In this example, extraction writes report fields, the request +can supply missing fields on resume, and rendering reads the resulting report. +The two routes to rendering are alternatives, not concurrent branches needing +a join. + +**Schemas** declare the shapes of accepted inputs and produced results. +They help the author see which fields an operation requires and allow the +validator to detect incompatible mappings. They do not establish that an +extracted fact is true or that a remote operation will succeed. + +## Workflow state and iteration + +**State** is the workflow's working data. A **reducer** specifies how a write +changes a state field: replace the previous value, append an action item, or add +a number, for example. Reading a field and routing to another step are separate +operations. This makes data movement inspectable but requires the author to +understand the mappings. + +If the procedure processes several documents, a foreach step defines an item +body. The current item and its iteration context belong to that body; normal +item completion returns to the owning foreach. A child workflow provides a +separate invocation scope with explicit inputs and results. These boundaries +define which data is available and what completion means. + +Ordinary same-region cycles are permitted, with a run-wide step budget limiting +execution attempts. General fork/gather is not implemented. Connecting paths +does not by itself promise parallel execution, synchronization, or conflict-free +state merging. + +## Saving, configuring, and running the procedure + +While authoring, an **editable workflow** is the mutable definition being +revised. The Python client supports this object without requiring a server-side +draft workspace. Saving creates an **artifact**: an immutable workflow version +containing its graph and declared requirements. + +A **deployment** selects a saved version and connects its logical service +requirements to concrete configured services. For example, the same report +procedure could use a test notes source in one deployment and a production +notes source in another, provided both satisfy its required contracts. +Changing the bindings is different from changing how the report is assembled. + +A **run** records one execution of a deployment. Last week's successful report +and this week's interrupted attempt are different runs, even if both use the +same artifact and deployment. Inspecting a run should identify the version, +inputs, status, and available execution evidence without changing the saved +procedure. + +These distinctions explain the lifecycle from the operator's perspective: + +```{.mermaid #fig:lifecycle-records width=95% caption="Lifecycle records."} +classDiagram + direction LR + class EditableWorkflow { + mutable graph + } + class ArtifactVersion { + artifact_id + version + saved definition + } + class Deployment { + deployment_id + source bindings + } + class Run { + run_id + input and result + status and trace + } + EditableWorkflow ..> ArtifactVersion : saves + ArtifactVersion "1" <-- "0..*" Deployment : selects + Deployment "1" <-- "0..*" Run : started from +``` + +## Available operations and environment binding + +A **capability** is an operation available for use in a workflow. +A **source** groups capabilities under a configured identity. The report's +extraction operation might come from trusted Python code, while another +operation is supplied by an external tool service. + +A deployment **binding** connects a logical source requirement in the workflow +to a concrete source in the environment. Validation checks whether that source +exists and matches the saved requirements. **Source drift** means those +requirements no longer match the currently available capabilities, for example +after an input schema changes. + +Built-in sources have fixed platform identities and do not require those +deployment bindings. Configured sources remain explicit operator choices. +This provides scoped portability, not freedom from environment dependencies: +the required code, credentials, and services must still be available. + +## Inspecting failure and resuming an interruption + +A validation diagnostic concerns the definition or its dependencies before +execution. A failed run records an operational problem encountered during +execution. An interrupted run records an explicit request for input, together +with the state needed to continue. The interface should distinguish these +situations because they call for different actions. + +In the report example, `needs_information` routes to a request step. The run +then waits for a declared resume payload; resuming applies that payload through +the workflow's bindings and continues to rendering. This is a defined pause in +the procedure, not recovery halfway through an arbitrary handler. + +A **trace** records execution evidence associated with the run. It supports +questions such as which route was taken and where execution stopped, but does +not make external effects reversible. The implementation chapters explain how +these concepts become runtime records and service operations. + ## Working Glossary -The document uses these terms with specific meanings: +The core terms can now be summarized without requiring implementation vocabulary. -| Term | Meaning | Example | -| --- | ------ | ---- | -| Workflow capability | A workflow-facing callable operation exposed by a source. | `local.report.extract_report` | -| `NodeSpec` | The authoring-layer typed contract produced by decorators or source adapters. | a Python `@node` projection | -| `NodeDef` | The core-level serializable node contract: input schema, output schema, and declared outcomes. | a workflow plan node definition | -| Source | A namespace and owner of capabilities, resources, prompts, and metadata. | `local.report`, `wf.std` | -| Source family | A class of source implementations. | built-in, MCP, Python | -| Source provider | Server-side code that loads or manages sources for a source family. | Python source loading | -| Tool | A provider-native operation before projection into workflow form. | MCP tool | -| Agent-operable | A surface designed for machine clients: structured output, explicit validation, stable commands, inspectability, and bounded summaries. It does not mean independently proven agent success rates. | `wf deploy validate`, `wf run trace` | -| `RawWorkflowPlan` | A serialized workflow plan used to create an immutable artifact without first going through a mutable draft workspace. | `wf artifact create-from-plan` | -| Outcome | A control-flow label returned by a node and consumed by graph edges. | `ok`, `error`, `submitted` | -| Output | The data payload returned by a node or workflow. | `{ "report": "..." }` | -| Reducer | A pure state-merge operation selected by state schema. | `wf.std.replace`, `wf.std.append` | -| Platform source | A process-provided source with fixed identity and no deployment binding. | `wf.std`, `wf.source` | -| Deployment binding | A mapping from logical workflow source requirement to concrete source id. | `local.report=local.report`, `playwright=playwright.default` | -| Source drift | Divergence between saved workflow requirements and the currently resolved source inventory. | missing capability or changed schema | +| Term | Meaning in the report example | +| --- | --- | +| Capability | An available operation, such as extracting report fields | +| Node | One use of an operation or control step in the procedure | +| Output | Data produced by a step | +| Outcome | A declared label selecting the next step | +| State | Working data retained during execution | +| Artifact | An immutable saved version of the procedure | +| Deployment | A saved version connected to concrete services | +| Run | One execution with its own status and evidence | +| Source | A configured collection of available operations | +| Binding | A logical source requirement mapped to a concrete source | : Working glossary for the thesis terminology. {#tbl:working-glossary} -## Workflows as Typed Graphs - -A workflow is an outcome-routed typed graph. It is not presented here as a -complete general DAG engine, and it is not a free-form agent state machine. -Nodes invoke named capabilities; edges route by declared node outcomes. The -graph model is defined by four schema contracts: - -- `input_schema`: validates run input. -- `state_schema`: defines workflow memory and reducer behavior. -- `output_schema`: defines the final result shape. -- Outcome declarations: route control flow through graph edges. - -Each callable `NodeUse` step references a core `NodeDef`---a serializable -contract describing input schema, output schema, and declared outcomes. -Source families commonly produce authoring-layer `NodeSpec`s first; those are -projected into `NodeDef` contracts before the core executes a workflow. Control -steps such as conditions, foreach, joins, interrupts, subgraphs, and end steps -are separate core step variants rather than `NodeDef` calls. The validator -checks that routed outcomes are declared, that a source node does not have -duplicate edges for the same outcome, and that reachable outcome edges are -present. Reducers merge state writes according to state-field declarations. -Reducers are pure deterministic merge functions invoked by the runtime in -workflow execution order; this report does not claim CRDT semantics, arbitrary -concurrent writes, or order-independent aggregation. General fork/gather -parallelism is future work, so this report does not claim complete concurrent -graph semantics. Interrupts represent typed external input points. Subgraphs -compose workflows as nodes. - -The graph model improves inspectability by making automation structure -explicit. Node contracts, source requirements, state writes, outcomes, -validation gates, and trace records are visible before and after execution. The -platform does not guarantee safe behavior from provider code, credentials, or -external side effects, but it makes the orchestration structure inspectable. - -A key distinction in the model is between outcomes and output. Outcomes control -routing through the graph. Output carries business data. This separation allows -the same node to produce different routing signals while its data payload -follows typed schemas. - -## Lifecycle Objects - -Four distinct lifecycle objects separate concerns across the workflow lifecycle: - -1. **Draft workspace.** Mutable authoring state for agent or human iteration. - A draft captures the evolving plan, source selections, and validation - diagnostics before any commitment to an immutable artifact. - -2. **Workflow artifact.** An immutable, versioned workflow definition. An - artifact records the graph plan, input/output/state schemas, required - capabilities with schema snapshots, and a catalog version reference. Once - saved, an artifact does not change. - -3. **Deployment.** A binding contract from an artifact version to a concrete - source and runtime context. Deployments map logical source requirements to - concrete source identifiers and carry a drift policy that determines - behavior when source catalogs change. - -4. **Run.** An execution record with status, diagnostics, output, trace, and - resumable stopped or interrupted state. In this report, durability means - persisted artifact/deployment/run records and resumability from explicit - stopped or interrupted boundaries. It does not mean arbitrary mid-node crash - recovery, transactional side-effect recovery, or exactly-once execution. - -This separation ensures that authoring, versioning, environment binding, and -execution are distinct operations with distinct lifecycle affordances. - -## Source Model - -The common boundary is `CapabilitySource`. Source inventory can expose -provider-derived `NodeSpec`s, reducers, resources, and prompts, but the core -runtime ultimately executes serialized `NodeDef` contracts and handler -functions. Source-specific behavior belongs in provider packages and server -composition. - -Representative sources and source families today: - -| Source | Kind | Role | -| --- | --- | ----- | -| `wf.std` | `system` | Built-in workflow nodes and reducers | -| `wf.source` | `system` | Built-in source resource helper | -| `wf.recipes` | `system` | First-party workflow recipes | -| MCP sources | `connection` | Upstream MCP tools, resources, prompts | -| Python sources | `python` | Trusted project-local `NodeSpec` registries | - -: Source families and platform roles used by the prototype. {#tbl:source-families} - -Platform sources such as `wf.std` and `wf.source` are process-provided and do -not require deployment self-bindings. Configured sources such as MCP and Python -remain explicit server or operator choices. - -The provider seam is intentionally narrow: - -```python -class WorkflowSourceProvider(Protocol): - def load_sources(self) -> Mapping[str, CapabilitySource]: ... -``` - -This covers source families that can project configured inventory into -workflow-facing `CapabilitySource` objects. Provider-specific runtime pools, -admin hooks, auth, catalog caches, and health checks stay outside this seam -until multiple source families need the same abstraction. The narrow seam is -intentional: it prevents MCP-specific session/auth lifecycle concerns from -becoming requirements for simpler source families such as built-ins or trusted -Python sources. - -Source resolution follows a deterministic path: a logical source requirement in -a workflow is checked against platform sources first, then resolved through -deployment bindings to concrete sources. Platform source IDs have fixed runtime -identity: deployment validation rejects explicit bindings for platform sources. -The runtime then delegates to the appropriate source handler. - -## Source Resolution Path - -The resolution path for a source reference is: - -1. A workflow stores logical source references (e.g., `local.report`). -2. At runtime, platform sources such as `wf.std` resolve immediately to - fixed source IDs without deployment bindings. -3. Configured sources are resolved through the deployment's binding map, which - maps logical names to concrete source identifiers. -4. The concrete source is looked up in the server's source inventory and - delegated to the appropriate runtime handler. - -This design provides a portability mechanism across environments: the same -artifact can be deployed with different concrete source bindings, while the -workflow graph references logical names only. Portability is still scoped by -provider availability: local Python code, MCP catalogs, auth records, and source -stores can differ between environments. +Python types, provider protocols, and package boundaries implement these +concepts. Their names are introduced with their responsibilities in the +architecture and implementation chapters rather than used as prerequisites +for understanding the workflow. # System Architecture -The architecture is organized into layered boundaries, each with a distinct -responsibility. +The weekly-report example needs more than a graph executor. An author must +discover the available operations, connect them, check the resulting workflow, +and choose which saved version to run. An operator must then distinguish a +bad definition from an unavailable service or an interrupted execution. The +architecture separates these responsibilities without requiring each caller +to implement the workflow lifecycle. -## Architecture Spine +Three boundaries organize the system: authoring versus server operations, +saved definitions versus individual executions, and workflow execution versus +provider-specific calls. These boundaries are visible in the user-facing +objects as well as in the implementation. -[@fig:architecture-spine] answers: who calls whom across the user, -agent, transport, server, API, runtime, and source-provider boundaries? +## From Authoring to Server Operations -```{.mermaid #fig:architecture-spine height=80% caption="Architecture spine: external agent commands flow through the CLI/transport boundary into server-composed API operations and deterministic core execution."} +The Python client is the main programmatic authoring interface. Its `App` +object represents a connection to the workflow service. An author can inspect +a capability, use the returned object in an editable workflow, validate that +workflow, and save it. The client reconstructs server responses as Python +objects with relevant operations, rather than requiring application code to +carry response dictionaries through every step. + +Local editing does not require a server request for each graph change. +Discovery and persistence do: the service owns the available capability +inventory and stored records. Validation therefore has both a local part, +which checks the authored structure, and a server part, which checks it +against the service's contracts. + +[@fig:architecture-spine] shows this separation. The CLI is another entry +point to the service; it is not a mandatory layer beneath Python authoring. + +```{.mermaid #fig:architecture-spine caption="Shared service boundaries."} flowchart TB - subgraph Operator["Human and agent front door"] - Owner[Workflow Owner] --> Agent[External LLM Agent] - Agent --> CLI[wf CLI] + subgraph Authoring["Authoring and clients"] + Python["Python App and editable workflow"] + Client["Client port"] + CLI["Command-line interface"] + Python --> Client end - - subgraph Boundary["Transport boundary"] - CLI --> Transport[JSON-RPC / Local Adapter] - Transport --> Server[WorkflowServer] + subgraph Service["Server-composed services"] + API["Workflow API"] + Stores["Artifacts, deployments and run records"] + Inventory["Available operation contracts"] end - - subgraph ServerSide["Server-composed platform"] - Server --> API[Workflow API Surface] - Server --> Inventory[CapabilitySource Inventory] - Inventory --> API - API --> Records[Drafts / Artifacts / Deployments / Runs] - API --> Core[Workflow Core] + subgraph Transport["Transport boundary"] + RPC["HTTP JSON-RPC adapter"] end - - subgraph Providers["Source providers"] - Server --> Sources[Configured Source Providers] - Sources --> Inventory + subgraph Execution["Workflow execution"] + Core["Workflow execution core"] end - - Core --> Result[Status / Output / Trace] - Result --> CLI + subgraph Integrations["Provider integrations"] + Providers["Configured source providers"] + Handlers["Execution handlers"] + end + CLI --> RPC + Client --> RPC + RPC --> API + API --> Stores + API --> Core + Inventory --> API + Providers --> Inventory + Core --> Handlers + Providers --> Handlers ``` -[@fig:architecture-spine] shows the primary flow from workflow owner -through agent, CLI, transport, and server to the API surface, core, platform -stores, and source providers. The server composes configured sources into a -unified inventory without the core runtime being aware of provider-specific -details. +The API operation layer is independent of the wire protocol. The JSON-RPC +adapter translates requests and responses; it does not decide how a foreach +iteration returns or how a reducer applies a state update. Conversely, the +execution core does not need to know whether its caller used Python, a +command line, or another application. -## Layered Package Boundary +This distinction also limits the role of an agent. An agent can help author +or operate a workflow through these interfaces, but the runtime follows the +saved graph. It does not ask an agent to choose the next step unless the +author has explicitly included an operation that makes such a decision. -Unlike the runtime-call diagram, [@fig:package-boundary] maps architectural -responsibilities onto repository packages. It answers: which package owns each -boundary in the current implementation? +## Keeping a Definition Separate from Its Use -```{.mermaid #fig:package-boundary caption="Package boundary: repository packages form a dependency direction from CLI and transport down to API, core, artifacts, platform DTOs, and source providers."} +Saving, deploying, and running answer different questions: + +- An artifact identifies the saved workflow definition and its version. +- A deployment selects that version and supplies environment bindings. +- A run records one execution, including its input, status, and progress. + +For the report workflow, changing the extraction step produces a different +definition. Choosing the configured service that supplies extraction is an +environment decision. Processing this week's notes is an individual run. +Keeping these apart allows an operator to inspect which definition and +bindings an execution used without confusing them with the current editor +contents. + +The client exposes this progression through workflow artifact, deployment, +and run objects. A run object is a snapshot, not a live background +subscription. Refreshing it requests a new snapshot. This makes network +activity explicit, although applications must decide when to refresh and how +to present progress. + +Draft workspaces provide a separate persisted editing surface used by the +draft API and CLI. They are not a required intermediate object for every +Python authoring operation. Both routes ultimately produce a saved +definition that the deployment and execution layers can use. + +## Data Movement and Control Movement + +Within a workflow, input bindings supply a step's arguments. Output bindings +select returned values to write into workflow state. Reducers determine how +those writes combine with existing values. The returned outcome selects the +next edge. These are related operations, but none substitutes for another: +routing to a renderer does not, by itself, supply the report it needs. + +[@fig:node-execution-cycle] summarizes an ordinary callable step. Conditions, +iteration controllers, subgraphs, and interrupts have their own runtime +handlers rather than pretending to be remote capability calls. + +```{.mermaid #fig:node-execution-cycle caption="Callable step execution."} flowchart TB - CLI[wf_cli] --> Transport[wf_transport_rpc_http] - Transport --> Server[wf_server] - Server --> API[wf_api] - API --> Artifacts[wf_artifacts] - API --> Core[wf_core] - API --> Platform[wf_platform] - Server --> MCP[wf_sources_mcp] - Server --> Python[wf_sources_python] - MCP --> Platform - Python --> Platform - Artifacts --> Platform + Validation["Input validation"] --> Call["Invoke operation"] + Call --> Result["Checked result"] + subgraph Data["DATA: what becomes visible"] + Output["Output payload"] --> Bind["Output bindings"] + Bind --> Reduce["Reducers update state"] + end + subgraph Control["CONTROL: where execution goes"] + Outcome["Declared outcome"] --> Route["Select matching edge"] + Route --> Next["Next node"] + end + Result --> Output + Result --> Outcome + Reduce -.->|state available to next step| Next ``` -## Layer Responsibilities +A declared outcome such as `needs_information` is a workflow decision. A +handler exception or exhausted step budget is an execution failure. A +workflow can therefore complete with a non-success business outcome without +being a failed runtime execution. Where iteration supports collecting item +errors, that policy must be explicit; errors do not automatically become +ordinary outcome edges. -The layered architecture separates concerns as follows: +The lanes distinguish responsibilities, not concurrent tasks: the runtime +applies the result's writes before advancing along the selected route. -- **Workflow Core.** Deterministic execution semantics for graph, state, - outcomes, trace, and resume rules. The core owns no provider-specific logic. +The trace makes the sequence inspectable, but a fixed graph does not imply +identical external results. Language-model calls, remote services, and +concurrent completion order can vary between runs. The runtime's defined +routing and state-update rules should not be confused with reproducibility +of every operation it invokes. -- **Workflow API Surface.** Application operations over capabilities, drafts, - artifacts, deployments, and runs. The API surface consumes source DTOs through - a neutral `WorkflowSpecProvider` and delegates to the core for execution. - `WorkflowSpecProvider` is the API-facing reader over capability specs derived - from source inventory; it is distinct from `WorkflowSourceProvider`, which - loads source inventory into the server. +## Iteration and Child Workflow Boundaries -- **Platform Records And Policies.** Draft workspaces, workflow artifacts, - deployments, run records, source inventory snapshots, validation diagnostics, - and next-action guidance. +Suppose the report now covers two documents, `A.md` and `B.md`. The +workflow must render a report for each, then assemble the two reports. +For this first example, foreach is configured to process one document at +a time. An output binding appends each rendered report to a `reports` +state field; the assembly step reads that field. -- **Server Composition.** `WorkflowServer` assembles concrete stores, sources, - runtimes, and admin surfaces into a long-lived service. The server composes - configured providers from workflow config into a live source inventory. +The important distinction is between finishing **one document** and +finishing **the whole collection**. After A finishes, the foreach continues +with B. Only after B finishes does it follow its `done` route to assembly. +It does not begin the collection again whenever an item returns. -- **Transport.** JSON-RPC over HTTP as the current transport implementation. - The transport is protocol-neutral; the Workflow API Surface is the stable - boundary. +[@fig:foreach-region] follows one invocation from start to finish. Read +downward for time. Solid arrows request work or apply a data binding; +dashed arrows report completion. The labels identify the data and routes. -- **Source Providers.** Built-in, MCP, and Python providers project their - inventory into `CapabilitySource` objects. The boundary is designed to admit - future source families. Provider-specific behavior such as MCP session pools - or Python module loading stays within the provider package. - -## Workflow Lifecycle - -The lifecycle of a workflow through the platform follows a defined path. -[@fig:workflow-lifecycle] answers what durable record or validation gate -is created at each stage. - -```{.mermaid #fig:workflow-lifecycle height=80% caption="Workflow lifecycle: mutable drafts become immutable artifacts; deployments bind those artifacts to live sources; runs produce inspectable records and bounded traces."} -stateDiagram-v2 - direction TB - [*] --> DraftWorkspace - [*] --> RawWorkflowPlan - DraftWorkspace --> DraftValidated: validate draft - DraftValidated --> Artifact: save immutable version - RawWorkflowPlan --> Artifact: create artifact from plan - Artifact --> Deployment: bind sources - Deployment --> DeploymentValidated: validate deployment - DeploymentValidated --> Run: start run - - Run --> Completed: outcome produced - Run --> Failed: runtime error - Run --> Interrupted: explicit interrupt - Interrupted --> Run: resume payload - - Completed --> Inspectable - Failed --> Inspectable - Interrupted --> Inspectable - Inspectable --> TraceSlice: bounded trace read -``` - -Each stage is a distinct platform operation with typed inputs and outputs. -`DraftValidated` and `DeploymentValidated` in [@fig:workflow-lifecycle] are -validation gates, not separate persisted record types. Draft validation checks -schema conformance and source availability. Artifact saving captures an -immutable snapshot either from a draft save path or directly from a raw workflow -plan through `artifact create-from-plan`. Deployment validation verifies that -bound sources are currently available and compatible. Source drift is treated as -divergence between saved artifact capability requirements and the currently -resolved source inventory: missing bindings, missing or disabled sources, -missing capabilities, or changed schema contracts. Run execution produces -persisted records with trace slices and resumable stopped state. Only -interrupted or explicitly stopped runs enter the resume path; completed and -failed runs remain inspectable records. - -## Workflow Core Model - -The core model processes graph execution through typed stages. This section -separates the broad runtime loop from the ordinary callable-node path. -[@fig:core-runtime-loop] shows how the -runtime selects a frame, dispatches by step kind, records trace, and routes by -outcome. [@fig:nodeuse-execution-path] then zooms into the `NodeUse` path, -where most source-backed work occurs; it expands the `NodeUse` branch from -[@fig:core-runtime-loop]. - -```{.mermaid #fig:core-runtime-loop image-width="0.88\\linewidth" caption="Workflow core runtime loop: after workflow input validation, the runtime repeatedly selects a ready frame, dispatches by explicit step kind, records trace for routable steps, and either routes onward, stops for interrupt, or projects final output."} -flowchart TB - Start[Validate workflow input] --> Select[Select ready frame] - Select --> Dispatch{Step kind} - - Dispatch --> Node[NodeUse] - Dispatch --> Cond[Condition] - Dispatch --> Each[Foreach] - Dispatch --> Sub[Subgraph] - Dispatch --> Join[Join] - Dispatch --> Int[Interrupt] - Dispatch --> End[End] - - Node --> Trace[Append trace frame] - Cond --> Trace - Each --> Trace - Sub --> Trace - Join --> Trace - - Trace --> Route[Route by outcome edge] - Route --> Select - - Int --> Stop[Persist interrupt request] - Stop --> Resume[Resume payload and outcome] - Resume --> Route - - End --> Output[Project workflow output] -``` - -```{=latex} -\clearpage -``` - -```{.mermaid #fig:nodeuse-execution-path height=80% caption="NodeUse execution path: a callable node resolves bindings, invokes a NodeDef handler, checks the declared outcome, applies reducer-aware state writes, appends trace, and returns to outcome routing."} +```{.mermaid #fig:foreach-region width=95% caption="Two documents, one loop."} sequenceDiagram - participant Runtime as Workflow Runtime - participant Bindings as Binding Resolver - participant Node as NodeDef Handler - participant Reducers as State Reducers - participant Trace as Trace Store - - Runtime->>Runtime: validate workflow input - Runtime->>Bindings: resolve NodeUse input map - Bindings-->>Runtime: local node input - Runtime->>Node: invoke handler - Node-->>Runtime: outcome + output payload - Runtime->>Runtime: check declared outcome - Runtime->>Reducers: merge output into state - Reducers-->>Runtime: updated state - Runtime->>Trace: append trace frame - Runtime->>Runtime: route by outcome edge + participant Each as Foreach + participant Body as Render item + participant State as reports state + participant Assemble as Assemble + Note over Each,State: Start once: documents = [A.md, B.md], reports = [] + Each->>Body: loop: process A.md + Body->>State: Output binding: append reportA + Body-->>Each: ok: A is finished + Each->>Body: loop: process B.md + Body->>State: Output binding: append reportB + Body-->>Each: ok: B is finished + Note over Each: Both items finished + Each->>Assemble: done: continue after the loop + Assemble->>State: Input binding: read reports + State-->>Assemble: [reportA, reportB] ``` -Input validation gates entry. The runtime then repeatedly selects a ready frame -and executes one step. A `NodeUse` resolves input bindings from workflow input, -state, and context; invokes the handler for the selected `NodeDef`; checks that -the returned outcome is declared; builds reducer-aware state writes; records a -trace frame; and advances through the edge for that outcome. `Condition`, -`foreach`, `subgraph`, `join`, `interrupt`, and `end` steps are explicit core -model variants, not provider-specific hacks. `Join` is currently a minimal step -that returns a `"done"` outcome; it reserves a graph-level concept for future -fork/gather semantics. +The same authored render node executes twice, but the runtime must remember +which document each execution belongs to. It calls that per-item execution +record a **frame**. The foreach invocation that started the item is its +**owner**. Returning to that owner finishes the current item; arriving from +the preceding workflow step starts a new foreach invocation. -`foreach` is implemented as an explicit runtime step with frame and lineage -bookkeeping for iteration and state isolation. This report does not claim a -general parallel fork/gather model or arbitrary concurrent reducer semantics. +This explains the graph's back-edge: `render.ok -> each` is an item return, +whereas `each.done -> assemble` leaves the loop. The output binding is what +appends the report; the back-edge itself does not transport or collect data. +The graph validator rejects a body node used both inside and outside that +loop, or a nested body that returns past its immediate owner. -Failure has three visible forms. Structural and dependency failures are -reported before execution through validation diagnostics. Runtime execution -failures set the run status to `failed` and store an error string. Business -failures are modeled as ordinary declared outcomes only when the workflow -author defines and routes those outcomes. +### Calling a Child Workflow for One Document -Interrupts are first-class stop points: an `InterruptNode` builds a typed -request payload, stores an `InterruptRequest` on the run state, and marks the -run interrupted. Resume supplies a payload and resume outcome; resume bindings -write the payload back into state, and routing continues from the declared -resume outcome. This is resumability at explicit boundaries, not arbitrary -mid-handler checkpointing. +Now replace the rendering work with a call to a saved child workflow that +summarizes one document. Consider only the item for `A.md`. The parent may +know the whole collection, but the child receives only the input explicitly +mapped into its call: `{"document": "A.md"}`. -## Source Provider Boundary +The child has its own input, working state, and execution context. It cannot +read the parent's current item or `reports` field merely because the parent +called it. If it needs another value, the author must add an input binding. -The source provider boundary separates configured source families from the -workflow API surface. [@fig:source-provider-boundary] answers where -source-specific code stops and workflow-facing inventory begins. +[@fig:scope-boundaries] shows the call and its return. The item waits while +the child runs. The child's `END` completes that child invocation, not the +parent item or the whole report workflow. -```{.mermaid #fig:source-provider-boundary latex-placement="H" height=69% caption="Source provider boundary: configured provider families stop at CapabilitySource inventory consumed by the workflow API surface."} -flowchart TB - Config[Workflow Config Sources] --> Server[WorkflowServer Composition] - - Server --> Builtin[Platform Sources] - Server --> MCP[MCP Source Provider] - Server --> Python[Python Source Provider] - - Builtin --> Inventory[CapabilitySource Inventory] - MCP --> Inventory - Python --> Inventory - - Inventory --> API[Workflow API Surface] - API --> Runtime[Workflow Runtime] +```{.mermaid #fig:scope-boundaries width=95% caption="A child call returns."} +sequenceDiagram + participant Item as Parent item A + participant Child as Child workflow + participant Result as Item A reports writes + participant Each as Foreach + Item->>Child: Input binding: document = A.md + activate Child + Note over Item: Wait for child + Note over Child: Own input,
state and context + Child->>Child: Produce summaryA + Child-->>Item: END: return summaryA + deactivate Child + Item->>Result: Output binding: append summaryA to reports + Item-->>Each: ok: item A is finished + Note over Each: Continue according
to foreach mode ``` -Platform sources are always present. Configured sources are operator choices -declared in the workflow config. The server composes all sources into a unified -`CapabilitySource` inventory that the workflow API surface consumes without -provider-specific knowledge. +The implementation calls the child's isolated data environment a **scope**. +Its input binding crosses into that scope; its output binding maps the +returned result into the caller item's pending writes, or directly into +enclosing state in serial mode. The two completion points are separate: +child `END` returns to the calling node, and the calling node's route back +to foreach finishes the item. + +### When Items Run Concurrently + +In concurrent mode, A and B may be in progress together. A returning from +its child does not permit assembly while B is still running. The foreach +waits for its required item completions before following `done`. + +Each item keeps its own pending writes, rather than immediately exposing +them to its sibling. The runtime tracks that separate state history as a +**lineage**; the pending writes are its **buffer**. In the second diagram, +“Item A reports writes” is that buffer when the foreach is concurrent. In serial +mode, the binding instead updates the enclosing state so the next item can +read it. + +For the successful two-item concurrent case, the foreach combines A's and +B's writes using the declared reducers, then assembly reads the combined +state. An append reducer and a replace reducer have different effects; +neither the arrows nor the fact that both items completed chooses a merge +policy. The serial ordering shown in the first diagram is not a promise +about concurrent completion order. + +If a child or item reaches an explicit interrupt, the run must preserve +which document was active and where it was waiting. Resume continues that +saved work; it does not restart the collection or silently move the +response to another item. + +## Connecting External Operations + +Providers adapt external functionality to contracts the workflow system can +inspect and invoke. Discovery supplies schemas and declared outcomes; +execution supplies a handler for the selected binding. These are separate +responsibilities because discovering an operation does not mean its service +is currently reachable or authorized. + +The server composes providers, stores, and the API. MCP-backed operations +may need sessions, authentication, and remote catalog handling. Python-backed +operations use configured, trusted imports. These concerns remain outside +the graph scheduler. An experimental OpenAPI provider explores another +source family without making it the system's product identity or implying +that all API descriptions are interchangeable. + +This boundary reduces provider-specific logic in workflows, but cannot +erase provider differences. A schema describes a call's shape; it does not +guarantee availability, cost, side-effect safety, or semantic equivalence to +another operation with the same fields. # Implementation -## Package Structure +The implementation follows these boundaries through focused Python +packages. The important question is not the number of packages, but where a +change must be made. Improving a Python editing method should not require +changing the scheduler, and adapting a new remote service should not require +changing graph routing. -The implementation is organized into focused packages with clear boundaries: +## Following a Python Authoring Request -| Package | Responsibility | -| --- | -------- | -| `wf_core` | Deterministic workflow kernel: graph execution, state, outcomes, trace, resume | -| `wf_authoring` | Authoring primitives: `NodeSpec`, `WorkflowBuilder`, DSL, reducer authoring, recipes | -| `wf_platform` | Neutral source DTOs, source visibility, permission metadata, and policy | -| `wf_artifacts` | Artifact, deployment, and run models; file-backed stores; validation | -| `wf_api` | Application surface: capabilities, drafts, artifacts, deployments, runs | -| `wf_config` | Neutral workflow configuration models and config loading | -| `wf_server` | `WorkflowServer` composition from config, stores, and source providers | -| `wf_transport_rpc_http` | JSON-RPC over HTTP transport for CLI and future clients | -| `wf_mcp` | Legacy MCP frontend, broker/admin compatibility, and migration shims | -| `wf_sources_mcp` | MCP upstream source implementation and persistent runtime pool | -| `wf_sources_python` | Trusted in-process Python source loading and `NodeSpec`-to-`NodeDef` projection | -| `wf_openapi` | Experimental OpenAPI source provider for typed HTTP operations | -| `wf_cli` | CLI commands driving the JSON-RPC transport | +The client package, `wf_client`, connects Python objects to a narrow +`WorkflowClientPort`. `App.from_http_jsonrpc(...)` configures the connection +without making a request. A subsequent capability inspection performs I/O, +decodes the response, checks the returned identity, and constructs a +`RemoteCapability`. -: Package responsibilities in the implementation. {#tbl:package-responsibilities} +An `EditableWorkflow` subclasses the authoring layer's `WorkflowBuilder`, +reusing its graph-building methods. It adds remote validation and saving +rather than maintaining a second independent builder implementation. +Its validation method contains this early return: -(Evidence: `docs/source_architecture.md`, package boundaries in `src/`.) - -## Workflow Core - -The workflow core implements deterministic execution semantics. It processes a -typed graph definition, validates input against `input_schema`, executes the -selected node use, routes by declared outcomes, applies reducers to state -writes, and produces trace frames. The public semantics are -outcome-routed graph execution with explicit condition, foreach, subgraph, -join, interrupt, and end steps. The async runtime has internal frame and lineage -machinery for foreach admission and state isolation, but this report does not -claim a complete general fork/gather programming model. The core is -provider-agnostic; it sees `NodeDef` contracts and handler functions, not -source-specific implementations. - -Determinism here refers to core routing, state, and trace semantics for a given -workflow definition and handler results. Provider code, remote MCP calls, -resource reads, and other external side effects may still be nondeterministic. - -State writes go through reducers. The platform includes built-in `wf.std` -reducer definitions such as `replace`, `append`, `merge_object`, `add`, -`set_union`, and `max`. Reducers are pure merge functions paired with -inspectable `ReducerSpec` metadata; they are exposed in source inventory, but -they are not ordinary executable node handlers. Interrupts produce stopped run -state with a resumable checkpoint. - -(Evidence: `src/wf_core/`.) - -## Platform Domain Objects - -The platform domain defines the lifecycle objects as Pydantic models: - -- `WorkflowArtifact` captures the immutable artifact definition with required - capabilities, schema snapshots, and catalog version references. -- `WorkflowDeployment` captures source bindings with a drift policy and binding - contract. -- Run records track execution status, diagnostics, output, and trace counts. - -Source binding uses `SourceBinding` objects defined in `wf_artifacts.models` -that map logical source names to concrete source identifiers. The -`CapabilitySource` dataclass is the neutral DTO that all source providers -project into. - -The lifecycle models are stored in `wf_artifacts`; orchestration of lifecycle -operations happens one layer above, in `wf_api`. - -The API layer holds the lifecycle together rather than acting as thin CRUD over -files. `WorkflowApi` composes capability, draft, artifact, deployment, and run -sub-APIs from one `WorkflowOperationContext`. That context carries stores, -event recording, source inventory, runtime execution, and optional live-source -checks. This is why CLI, JSON-RPC, and future transports can share the same -domain operations without importing source-provider internals. - -`wf_platform` is intentionally smaller than the API layer. It owns stable -neutral source vocabulary: `CapabilitySource`, source inventory snapshots, -declarative visibility and permission metadata, source policy, source refs, -capability refs, and schema hashes. These flags describe source behavior for -inventory and validation surfaces; they are not an authorization or -policy-enforcement layer. `wf_platform` should not grow into a dumping ground -for stores, runtimes, or provider lifecycle. Those belong in `wf_api`, -`wf_server`, or the specific `wf_sources_*` package. - -The API lifecycle is deliberately centralized through one facade, per -[@fig:api-lifecycle-facade]. The facade is the application-layer mechanism that -prevents lifecycle operations from becoming disconnected CRUD calls. - -```{.mermaid #fig:api-lifecycle-facade caption="API lifecycle facade: one WorkflowOperationContext carries stores, source inventory, runtime execution, and live checks for all lifecycle sub-APIs."} -classDiagram - class WorkflowApi { - capabilities - drafts - artifacts - deployments - runs - } - class WorkflowOperationContext { - stores - source_inventory - event_recorder - runtime_runner - live_source_checker - } - class WorkflowSpecProvider - class DraftStore - class ArtifactStore - class RunStore - class WorkflowRuntimeRunner - class LiveSourceChecker - - WorkflowApi --> WorkflowOperationContext - WorkflowOperationContext --> WorkflowSpecProvider - WorkflowOperationContext --> DraftStore - WorkflowOperationContext --> ArtifactStore - WorkflowOperationContext --> RunStore - WorkflowOperationContext --> WorkflowRuntimeRunner - WorkflowOperationContext --> LiveSourceChecker +```python +local = self.validate_local() +if not local.ok: + return WorkflowValidation(local, "not_run", ()) ``` -[@fig:api-lifecycle-facade] highlights the design contribution at the -application layer: drafts, artifacts, deployments, and runs are not independent -file operations. They share source inventory, stores, event recording, runtime -execution, validation, and live-source checks through one operation context. +This excerpt from `wf_client/authoring.py` explains an observable behavior: +a structurally invalid edit produces local feedback without a server +request. Passing that check does not establish deployment readiness; the +server still validates the submitted plan against its inventory. -(Evidence: `src/wf_artifacts/models.py`, `src/wf_platform/sources.py`.) +Saving validates first, submits the plan, checks the save response, and +re-inspects the exact artifact version. The resulting artifact object is +therefore reconstructed from the stored definition, not assumed to be an +unchanged copy of the editor. Identity checks reject mismatched responses +instead of quietly attaching methods to the wrong artifact or deployment. -## Validation And Diagnostics +The same principle applies to runs. In `wf_client/runs.py`, `refresh()` +returns a decoded snapshot from `inspect_run`, checking the expected run +and deployment identities. `resume()` returns another snapshot after +submitting the response. Application code must retain the returned object; +an earlier snapshot does not mutate when the server advances. -Validation operates at multiple lifecycle points: +Serialization still exists at the service boundary. The benefit is that +callers need not manually rebuild the domain objects after every request. +The client does not eliminate the distinction between a local Python model +and a remote operation. -1. **Draft validation** checks schema conformance, source availability, and - graph structure before an artifact is saved. -2. **Deployment validation** verifies that bound sources are currently - available and that required capabilities match the source inventory. -3. **Run validation** checks input against the artifact's input schema before - execution begins. +## Executing and Bounding a Run -When validation fails, the platform produces machine-readable diagnostics with -severity, error code, logical source reference, repair hint, and the bound -source. These diagnostics are designed for machine clients: an LLM agent can -read the diagnostic and determine what to fix without blind probing. +The `wf_core` package defines the graph and execution state. Its runtime +selects ready frames and dispatches by node kind. An ordinary `NodeUse` +invokes a bound callable; `ConditionNode`, `ForeachNode`, `SubgraphNode`, +`InterruptNode`, and `EndNode` implement explicit control behavior. -For example, an invalid deployment binding can produce a diagnostic shaped like -this: +Step admission happens before dispatch. The immutable `RunLimits` policy +sets a positive maximum, with a default of 10,000 attempts. The run stores +how many attempts have been admitted. Nested execution shares that run-wide +budget, and asynchronous dispatch reserves attempts before launching work. +A failed attempted step is not free simply because it returned no useful +result. -```json -{ - "severity": "error", - "code": "binding_missing", - "logical_ref": "local.report.extract_report", - "bound_source": null, - "message": "No binding exists for logical source 'local.report'.", - "repair_hint": "Bind the logical source to a compatible concrete source." -} -``` +The budget bounds graph progress, including a cycle whose condition never +selects an exit. It is not a wall-clock timeout, a language-model token +allowance, or protection against a handler that blocks indefinitely. The +limit and consumed count persist with the run, so interrupting and resuming +does not reset the allowance. -Deployment validation also detects source drift. If a source changes -incompatibly and a deployment becomes unrunnable, the system reports the -diagnostic with a repair hint rather than silently executing against -incompatible capabilities. In this prototype, schema drift is detected through -saved required-capability schema hashes compared with current source inventory -hashes when both sides provide hashes; it does not attempt semantic -backward-compatibility analysis. +## Preserving State Across Nested Execution -(Evidence: `src/wf_artifacts/validation.py`, `tests/artifacts/test_validation.py`.) +Workflow state updates pass through reducer-aware patches. Serial iteration +must make its writes visible to subsequent serial work, while concurrent +items need separate views until their results are combined. Nesting either +mode inside the other makes write ownership more subtle than committing +every result directly to global state. -## Next-Action Guidance +The shared `commit_foreach_aware_patch` helper handles writes from ordinary +nodes, subgraph results, and interrupt responses. It walks serial owners +outward and selects the first concurrent item boundary, if present, as the +buffer destination. It continues checking ancestry before writing, so a +missing parent or cycle cannot cause a partial write merely because a +buffer destination was already found. With no concurrent boundary, the +patch commits through the enclosing serial owners. -The platform provides advisory continuation hints through `NextActions`. This -object tells a machine client whether there is an obvious next workflow-surface -tool call, what that tool is, and why. It is guidance, not authority: validation -diagnostics and runtime status remain the source of truth. +When concurrent results are combined, the patch retains their constituent +write contributions for later reducer replay. Keeping only cumulative +values would allow a surrounding iteration to replay an already-counted +prefix. This distinction matters for operations such as appending report +sections: a correct visible value at one nesting level is not necessarily +a correct contribution to the next merge. -The `NextActions` object includes `can_continue`, `can_save_now`, -`recommended_next_tool`, `reason`, `patch_examples` with concrete request -payloads, and `warnings`. This supports external-agent operation as a surface -property: a machine client can read the hint and execute the suggested -operation without reconstructing the lifecycle state. +This machinery currently supports iteration. It should not be read as a +claim that arbitrary graph fork/gather semantics are already implemented. -(Evidence: `src/wf_api/next_actions.py`.) +## Giving Expressions a Consistent Context -## Server Composition +Bindings and conditions need the same account of the current execution. +The runtime's `frame_context_view` derives structured foreach entries from +persisted frame ancestry. Entries are keyed by foreach node identity, so +nested bodies can refer to enclosing items within the same workflow scope, +rather than relying only on an innermost-item shortcut. -`WorkflowServer` is the composition boundary. It assembles concrete stores, -source providers, runtimes, and admin surfaces from workflow config. The server -does not own workflow semantics; it delegates to `WorkflowApi` for application -operations. +The walk stops at a subgraph scope boundary. An enclosing item's value must +be passed as child input if the child needs it. The reader also rejects +malformed ownership, parent cycles, and conflicting aliases; corrupt +checkpoint metadata is not treated as an innocently absent field. -The config model specifies store configuration, transport endpoints, and source -provider declarations. The current config model includes implemented source -kinds such as `mcp` and `python`; future kinds such as `openapi` would extend -the same discriminated-union pattern. +Condition evaluation receives this structured mapping, as do the input +resolution paths. This connection is necessary for validation to mean +anything: accepting a context path while evaluating conditions against a +smaller stub would let a valid-looking graph silently choose the wrong +branch. -(Evidence: `src/wf_server/config.py`.) +## Validation, Persistence, and Diagnostics -## JSON-RPC Transport +The authoring layer checks graph structure; the server checks saved plans +and environment bindings; the runtime checks actual values and execution +state. Each layer has information the earlier one lacks. Static validation +can reject an invalid foreach return, for example, but cannot prove that a +remote operation will remain available when a run reaches it. -The JSON-RPC-over-HTTP transport exposes the Workflow API Surface to CLI and -future HTTP clients. The transport is protocol-neutral; it maps JSON-RPC -method calls to `WorkflowApi` operations and returns structured JSON responses. +Diagnostics carry a code, a location, a message, and, where available, a +repair hint. These fields let a caller identify the faulty binding or node +without parsing a prose-only error. Suggested next actions are guidance, +not authorization and not evidence that a repair has succeeded. Schema +fingerprints likewise detect a changed contract representation; they do not +prove semantic compatibility. -The CLI communicates over this transport. CLI commands are designed for machine -clients as well as humans: structured output, status and inspect commands, -validation commands, compact summaries, and guarded destructive actions make -the CLI a practical surface for external agents. +The API lifecycle layer persists stopped runs and their versioned +checkpoints, including interrupted runs that may later resume. Restoration +validates the stored representation and recovers the execution state before +dispatch continues. This supports explicit pause-and-resume boundaries; it +does not promise durable recovery from every instruction inside an arbitrary +handler or exactly-once external side effects. -(Evidence: `src/wf_transport_rpc_http/`, `src/wf_cli/`, `tests/wf_cli/`.) +Separating artifact, deployment, and run storage also keeps inspection +focused. Definition inspection explains what was saved; deployment +inspection explains environment selection; run inspection and trace explain +what happened during a particular attempt. -## MCP Source Provider +## Server and Provider Responsibilities -MCP is one source family and a useful stress test for source-provider -correctness. A workflow capability call should not silently turn a stateful -external provider into a fresh one-off client call when provider state is part -of correctness. The platform contribution is the source-provider boundary and -workflow lifecycle, not an MCP wrapper. +The remaining package boundaries put these operations into a service. +`wf_api` coordinates lifecycle operations, `wf_artifacts` supplies storage +contracts and implementations, and `wf_platform` supplies shared platform +contracts. `wf_server` composes these dependencies. `wf_transport` exposes +the JSON-RPC interface, while `wf_cli` provides terminal operations. -The MCP source provider manages: +Provider implementations retain their own lifecycle requirements. MCP +support manages remote discovery and invocation through configured +connections. Python support loads trusted configured callables; it is not +a sandbox for arbitrary submitted code. OpenAPI support remains +experimental and is not evidence that every described HTTP service can +already be used without adaptation. -- Source identity and connection description. -- Auth records and catalog cache storage. -- A live `ClientSession` facade. -- A persistent session pool for stateful upstream operations. -- MCP-to-workflow converters for tools, resources, and prompts. - -The provider projects MCP tools into `NodeSpec` contracts and corresponding -core `NodeDef` contracts, making them callable from workflow graphs through the -same `CapabilitySource` boundary as Python or built-in sources. - -Evidence: - -- `src/wf_sources_mcp/` -- `tests/wf_sources_mcp/test_runtime.py` -- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` - -## Python Source Provider - -Python sources provide trusted developer extensibility. Project-local code can -become typed workflow capabilities quickly, but these are not sandboxed -non-programmer plugins. - -The loading path is: - -```text -PythonSourceConfig(path, module, registry) - -> PythonSourceProvider - -> import module - -> load NodeSpec registry - -> project specs into NodeDef contracts - -> qualify specs under source id - -> CapabilitySource(kind="python") -``` - -Python sources are static at server startup. No hot reload is implemented yet. -The provider imports the configured module, reads the named registry attribute, -projects each `NodeSpec` into the workflow capability inventory, and makes the -corresponding `NodeDef` contract available to workflow plans. This keeps the -core provider-agnostic. - -(Evidence: `src/wf_sources_python/`, `tests/wf_sources_python/test_loader.py`.) - -## Experimental OpenAPI Source Provider - -The repository also contains an experimental `wf_openapi` source provider. It -parses OpenAPI documents, projects HTTP operations into typed `NodeSpec` -contracts, and executes calls through HTTP request/response validation. This -shows the provider boundary can extend beyond MCP and trusted Python sources, -but it is not used by the thesis case study or agent challenge evaluation. - -(Evidence: `src/wf_openapi/`, `tests/openapi/`.) +These seams make additional interfaces possible, but an interface still +needs its own interaction design. The availability of API operations and +typed client objects does not, by itself, establish that workflow authoring, +diagnosis, or recovery is easy for a new user. The case study and evaluation +therefore need to distinguish demonstrated operations from broader +usability claims. # Case Study: Deterministic Report Workflow -The thesis case study is a document/report preparation workflow backed by local -fixtures and trusted Python sources. It demonstrates the full lifecycle: -config validation, server startup, capability discovery, draft creation, -artifact saving, deployment validation, run execution, run inspection, and -trace viewing. The case study is deterministic and does not require an LLM -call, remote OAuth, or provider quota. +Consider an author who receives weekly notes and needs two deliverables: a +structured report for further processing and Markdown for a reader. The +example workflow makes this small procedure reusable: -This case study evaluates lifecycle integration rather than graph -expressiveness. Graph features such as interrupts, foreach, subgraphs, joins, -and reducer behavior are covered by targeted tests and code evidence in the -evaluation section. +The three operations read notes, extract a report, and render Markdown. -The thesis-critical automated report-workflow run executes the full deterministic -report pipeline through the artifact, deployment, and run lifecycle: -`read_notes -> extract_report -> render_markdown_report`. This keeps the case -study small enough to audit while still exercising source discovery, multi-node -dataflow, state mapping, artifact saving, deployment binding, run output, and -trace inspection. +The input is deliberately constrained. Notes contain named sections and +action lines with owner, task, and due-date fields. Extraction parses that +format; it is not language-model summarization of arbitrary documents. With +fixed input and local Python operations, the result can be checked without +remote credentials, service quotas, or variation in generated text. -## Case Study Components +The bundle at +[`examples/report_workflow/`](../../examples/report_workflow/) supplies the +operations, fixture notes, server configuration, and a saved raw-plan example. +The walkthrough below expresses the same procedure through the current +Python client. The example README retains a command-line route for operators +who need it; that route is not a prerequisite for using this interface. -The example bundle lives at -[`examples/report_workflow/`](../../examples/report_workflow/) and contains: +## Starting with Available Operations -- `ops.py` --- a Python source exposing `read_notes`, `extract_report`, and - `render_markdown_report` as typed `NodeSpec` capabilities. -- `input.md` --- fixture Markdown notes with summary, actions, risks, and - followups sections. -- `cap-input.json` --- a capability-call payload generated from the fixture. -- `run-input.json` --- a workflow-run payload pointing at the fixture. -- `workflow.plan.json` --- the three-node raw workflow plan used for artifact - creation. -- `wf.config.json` --- a local server and client config using the - `local.report` Python source. +The example configuration registers three trusted Python operations under +`local.report`. Their Pydantic models describe the input and output +contracts. The author consumes those operations from the service inventory; +the client does not import their implementations to execute them locally. -(Evidence: `examples/report_workflow/README.md`, `examples/report_workflow/ops.py`.) - -## Python Source Definition - -The Python source defines three capabilities with Pydantic input/output -schemas: +The following blocks form one asynchronous Python session. They assume a +server using `examples/report_workflow/wf.config.json`, reachable at its +configured address, and a fresh artifact name or unused version. The fixture +read assumes the client is running from the repository root. ```python -@node(name="read_notes") -def read_notes(payload: ReadInput) -> ReadOutput: - return ReadOutput(text=Path(payload.path).read_text(encoding="utf-8")) +from pathlib import Path +from pydantic import BaseModel -@node(name="extract_report") -def extract_report(payload: ExtractInput) -> ReportOutput: - # Parses Markdown sections into structured report fields - ... +from examples.report_workflow.ops import ReportOutput +from wf_authoring import input_from, input_path, output_to, state_path +from wf_client import App - -@node(name="render_markdown_report") -def render_markdown_report(payload: MarkdownInput) -> MarkdownOutput: - # Renders structured report as Markdown - ... +app = App.from_http_jsonrpc("http://127.0.0.1:8771/rpc") +read_notes = await app.capability("local.report.read_notes") +extract_report = await app.capability("local.report.extract_report") +render_report = await app.capability("local.report.render_markdown_report") ``` -Each function is decorated with `@node`, which produces a `NodeSpec` with typed -input and output schemas. The registry is a plain list of decorated functions: +Each lookup returns a capability object with its schemas and declared +outcomes. Importing `ReportOutput` above only reuses the fixture's data model +for authoring; the three capability objects still refer to server-side +operations. An application without that shared model could use the inspected +JSON Schemas instead. + +This discovery step exposes a practical requirement: authors need to know +which operations exist and what data they accept before connecting them. +A name alone is not enough to establish a compatible pipeline. + +## Describing the Workflow's Data + +The workflow has one public input, intermediate state, and two public +outputs. They are declared separately so that intermediate notes do not +accidentally become part of the result contract. ```python -registry = [read_notes, extract_report, render_markdown_report] +class NotesInput(BaseModel): + text: str + + +class ReportState(BaseModel): + notes: str = "" + report: ReportOutput | None = None + markdown: str = "" + + +class ReportResult(BaseModel): + report: ReportOutput + markdown: str + + +graph = app.new_workflow( + "report_python_showcase", + input_schema=NotesInput, + state_schema=ReportState, + output_schema=ReportResult, +) ``` -The `wf.config.json` configures the source as: +The models export schemas for the workflow contract. They do not make the +saved workflow dependent on a live Python class instance. The initially +absent report belongs to intermediate state; the public result requires a +report because a completed successful pipeline should have produced one. -```json -{ - "kind": "python", - "id": "local.report", - "path": ".", - "module": "ops", - "registry": "registry" -} +## Connecting Data and Decisions + +The author now creates three node uses. Each use selects an operation and +declares its data bindings: + +```python +read = graph.use( + read_notes, + id="read", + input=[input_from(input_path("text"), "text")], + output=[output_to("text", state_path("notes"))], +) +extract = graph.use( + extract_report, + id="extract", + input=[input_from(state_path("notes"), "text")], + output=[output_to((), state_path("report"))], +) +render = graph.use( + render_report, + id="render", + input=[input_from(state_path("report"), "report")], + output=[output_to("markdown", state_path("markdown"))], +) + +end = graph.end("ok", id="finished") +graph.set_entry_point(read) +graph.connect(read, "ok", extract) +graph.connect(extract, "ok", render) +graph.connect(render, "ok", end) +graph.set_output([ + input_from(state_path("report"), "report"), + input_from(state_path("markdown"), "markdown"), +]) ``` -This tells the Python source provider to import `ops.py`, read the `registry` -attribute, and project each function into the workflow capability inventory -under the `local.report` namespace. +The empty tuple in `output_to((), ...)` selects the extraction step's +whole output object. The other output bindings select individual fields. +The `connect` calls then specify execution order for the `ok` outcome; +they do not implicitly carry those objects between steps. -(Evidence: `examples/report_workflow/ops.py`, `examples/report_workflow/wf.config.json`.) +This explicitness is both a benefit and an authoring cost. The mapping is +inspectable, and changing a route does not silently change a data source. +However, even a linear three-step procedure requires contracts, bindings, +and routes. Typed helpers reduce raw serialization work without removing +the need to understand these distinctions. -## Lifecycle Runbook +## Diagnosing and Repairing a Binding -The case study exercises the full lifecycle through the same CLI/API surface -that external agents use. The main body summarizes the state transitions; the -appendix gives the complete repository-root command transcript. +An editable graph can temporarily be invalid. Suppose a final output binding +names a state field that does not exist: -First, config validation preflights the static Python source before server -startup. This catches malformed source config or import failures before the -workflow server is asked to compose source inventory. Starting the configured -server then creates a `WorkflowServer` with stores, transport, platform sources, -and the `local.report` Python source loaded into capability inventory. +```python +graph.set_output([ + input_from(state_path("missing_report"), "report"), + input_from(state_path("markdown"), "markdown"), +]) +broken = await graph.validate() +assert not broken.ok +assert broken.remote_status == "not_run" +for issue in broken.local.errors: + print(issue.code, issue.path, issue.message) -Capability discovery shows the available report operations, and a direct -capability call to `local.report.extract_report` verifies the typed source -contract independently of the workflow lifecycle. This is useful because an -agent can inspect or smoke-test a source before saving a workflow artifact. - -The draft path demonstrates agent-oriented authoring. A draft workspace can be -seeded from one capability's input and output schemas: - -```powershell -wf draft create report_ws --capability local.report.extract_report +graph.set_output([ + input_from(state_path("report"), "report"), + input_from(state_path("markdown"), "markdown"), +]) +(await graph.validate()).raise_for_errors() ``` -That command is intentionally a best-effort bootstrap, not a complete workflow -synthesizer. Focused edit commands such as `wf draft set-name`, -`wf draft set-input`, `wf draft set-output`, `wf draft bind`, -`wf draft add-step`, `wf draft branch`, `wf draft handle`, and -`wf draft set-workflow-output` cover common schema, mapping, step, and routing -edits without forcing an agent to write RFC 6902 JSON Patch by hand. Raw -`wf draft patch` remains the escape hatch for structural edits that focused -commands do not yet cover. The raw-plan import path is the alternative route -when the author already has a complete plan: it bypasses the draft workspace -and creates the artifact directly. +The local report identifies the invalid source path in the workflow's output +projection. No server validation request is made for that invalid graph. +The repair changes the projection, not the renderer or its outgoing edge. +This illustrates why data bindings and control routes need separate feedback. +The next section saves only the repaired definition. -The tested thesis path imports the complete three-node plan as an immutable -artifact: +## Saving a Version and Choosing Its Environment -```powershell -wf artifact create-from-plan workflow.plan.json ` - --artifact report_case_study --version 1 ` - --title "Report Case Study" --outcome ok +Before saving, the author can request validation and inspect its diagnostics. +The example stops on errors: + +```python +validation = await graph.validate() +validation.raise_for_errors() +artifact = await graph.save(version=1) + +deployment = await artifact.deploy( + "report_python_showcase.local", + bindings={"local.report": "local.report"}, +) +readiness = await deployment.validate() +if not readiness.runnable: + raise RuntimeError(readiness.diagnostics) ``` -Artifact creation captures the workflow graph, required capability snapshots, -declared outcome, and logical source requirements. Deployment saving then binds -the logical source `local.report` to the concrete configured source -`local.report`, for example with -`wf deploy save report_case_study.default --artifact report_case_study --version 1 --binding local.report=local.report`. -Deployment validation checks that the bound source exists and still satisfies -the artifact's saved requirements before execution. +The artifact is the saved version of the authored procedure. The deployment +binds its logical source requirement to the configured source. Both happen +to be called `local.report` here; the mapping still records an environment +choice rather than a new graph edge. -Run execution starts from the deployment, validates input, executes the -three-node pipeline, records trace frames, and stores a completed run record -with output and diagnostics. `run inspect`, `run trace`, and `run list` then -provide the inspection surface used by both humans and agents. +The sequence below summarizes the public operations; it omits internal +validation and re-inspection calls made by individual client methods. -Evidence: +```{.mermaid #fig:python-lifecycle caption="Authoring through the service."} +sequenceDiagram + actor Author + participant Client as Python client + participant API as Workflow service + participant Provider as Bound operation + Author->>Client: Build and revise graph + Client->>Client: Check structure locally + Client->>API: Validate and save definition + API-->>Client: Saved artifact version + Client->>API: Bind deployment to version + API-->>Client: Deployment and readiness + Client->>API: Run with input + API->>Provider: Invoke graph steps + Provider-->>API: Outputs and outcomes + API-->>Client: Stopped run snapshot + Client->>API: Inspect run and bounded trace + API-->>Client: Stored execution evidence + Client-->>Author: Result or diagnostic +``` -- `examples/report_workflow/README.md` -- `tests/examples/test_report_workflow_example.py` +An edit to the workflow would be saved as another version, not applied +retroactively to the artifact used by this deployment. Conversely, selecting +a different source environment is a deployment concern. A readiness check +can reject a missing or incompatible binding before a run is attempted. -## Expected Output +## Running and Inspecting the Result -The case study produces a structured report with: +The client reads the notes and sends their contents. The server therefore +does not need access to the client's file path. -- Title: "Weekly Project Update" -- Three action items with owner, task, and due date -- Risks mentioning Google Drive MCP quota -- Followups for Markdown rendering and baseline comparison -- Rendered Markdown beginning with `# Weekly Project Update` +```python +notes = Path("examples/report_workflow/input.md").read_text(encoding="utf-8") +run = await deployment.run({"text": notes}, max_steps=100) +if run.status != "completed": + raise RuntimeError((run.status, run.diagnostics)) -The workflow output includes both the typed `ReportOutput` object and a -Markdown rendering produced by the final node, making validation deterministic. +result = ReportResult.model_validate(run.output) +assert result.report.title == "Weekly Project Update" +assert len(result.report.action_items) == 3 +assert result.markdown.startswith("# Weekly Project Update") -## Automated Test Evidence +run = await run.refresh() +trace = await run.trace(start=0, limit=10) +``` -The case study is backed by automated tests that exercise the same lifecycle -programmatically: +The run exposes a status, output, and diagnostics independently of the +editable graph. The output crosses the API as data; the explicit +`model_validate` call reconstructs the example's Pydantic result model. +Refreshing obtains the latest stored snapshot, while the bounded trace +provides step-level evidence when output alone is insufficient. -1. **Capability load and call.** A test loads the config, builds the server, - lists capabilities under `local.report`, and calls `extract_report` with - fixture input. The test asserts the outcome is `ok`, the title matches, and - the action items and risks contain expected values. +For this fixed fixture, the result includes three action items, the +recorded risks and followups, and a Markdown report headed +“Weekly Project Update.” Checking these fields establishes that the +example's data reached the intended outputs. It does not establish that +the report is useful for every reader or that extraction works on +unstructured notes. -2. **Artifact/deployment/run path.** A test loads `workflow.plan.json`, which - runs `read_notes -> extract_report -> render_markdown_report`, saves the - artifact, creates a deployment with source bindings, starts a run, and - asserts that the run completes with both structured report output and - rendered Markdown output. +## What This Case Demonstrates -The thesis-critical run path therefore demonstrates the full lifecycle using a -deterministic three-node pipeline. A supplemental browser-click example remains -supporting evidence for human-interaction-style workflows and before/after -snapshot outputs. +The existing tests in +[`test_report_workflow_example.py`](../../tests/examples/test_report_workflow_example.py) +check the source's input rules, rendering and extraction, capability +discovery and invocation, and the artifact/deployment/run lifecycle using +the raw-plan fixture. Those tests are evidence for the report operations +and lifecycle. They are not a user study of the Python walkthrough. -(Evidence: `tests/examples/test_report_workflow_example.py`.) +The Python presentation makes the current authoring experience concrete: +inspect operations, declare contracts, connect data and outcomes, save, +select bindings, and inspect an execution. The CLI and draft surface offer +another way to perform related lifecycle operations; they are not required +steps in this Python walkthrough. + +A direct Python script would be shorter for these three local functions. +The workflow system earns its additional structure when the definition +must be saved, bound to an environment, validated independently, and +inspected through a shared service. This example demonstrates that +integration, not a performance advantage over function calls. + +Nor does a linear pipeline exercise all graph semantics. It has no +conditional branch, foreach body, child workflow, or interrupt. Targeted +runtime tests provide evidence for those mechanisms; they should not be +credited to a case that never executes them. Ease of authoring and diagnosis +also requires evidence beyond a successful fixture run. # Evaluation -The evaluation uses concrete evidence: automated tests, live smoke tests, and -the deterministic case study. The evidence claim is that the prototype -demonstrates the architecture and workflow lifecycle under controlled examples. +Evaluation distinguishes three questions: whether the runtime follows its +contract, whether the public lifecycle composes correctly, and whether an +author can use that lifecycle effectively. The current evidence addresses +the first two through controlled tests and an executable walkthrough. +Evaluation of the intended shell-backed authoring experience remains pending. ## Prototype Conformance Criteria -The evaluation is organized around prototype conformance criteria derived from -the research question. These criteria test whether the implemented substrate has -the intended lifecycle, validation, source, and inspection behavior under -controlled examples; they do not constitute a broad reliability or user study. -The later Agent Instruction Layer section explains why CLI/API conformance is -necessary but not sufficient for broad agent-success claims. +The implementation should preserve definition and execution identity, reject +invalid structures and bindings, apply declared state updates, and expose +stopped runs for inspection or explicit resume. It should also allow +configured source families to supply operations without making the graph +scheduler specific to one remote protocol. -| Criterion | Question | Evidence Type | -| --- | ---- | --- | -| Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests | -| Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests | -| Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests | -| Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow and browser-click case studies | -| Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests | -| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests | -| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness | +These are conformance criteria, not claims of general reliability. +The evidence index gives the source and test paths behind each row below. +A referenced suite identifies where a behavior is tested; it does not imply +that every suite was rerun during this document revision. -: Prototype conformance criteria used for evaluation. {#tbl:prototype-conformance} - -This is a prototype system evaluation, not a broad user study or reliability -benchmark. - -## Qualitative Comparison - -| Capability | Direct LLM tool loop | Generated script | Representative hosted automation platform | `lda.chat` prototype | -| --- | --- | --- | --- | --- | -| Versioned workflow artifact | Not inherent | Manual | Often yes | Prototype support | -| Deployment/source binding | Not inherent | Manual config | Platform-specific | Prototype support | -| Typed validation before run | Tool-schema dependent | Custom | Varies | Controlled-test support | -| Persisted prototype run record | Not inherent | Custom | Often yes | Explicit stopped/interrupted boundaries only | -| Source drift diagnostics | Not inherent | Custom | Varies | Schema-hash controlled examples | -| Agent-operable repair hints | Not inherent | Custom | Usually human UI | Prototype support | -| Scheduling | Depends on agent | External scheduler | Yes | Future work | - -: Qualitative comparison against direct tool loops, scripts, and mature automation products. {#tbl:qualitative-comparison} - -The comparison positions the architecture; it is not a quantitative claim that -the prototype outperforms mature automation products. "Not inherent" means the -feature can be added by surrounding infrastructure, but is not provided by the -bare strategy alone. "Mature automation platform" summarizes representative -hosted automation products discussed in the Related Work chapter; it is not a -market-wide survey. - -## Formative Agent-Trial Findings - -Before the checked 36-trial campaign, exploratory agent runs were used as -design feedback. Prompts, product behavior, workspace isolation, and enabled -tools changed during this period, so these runs are not pooled into the outcome, -duration, or token statistics. They instead provide process-tracing evidence: -recurring agent failures exposed public-surface gaps, and subsequent slices -addressed those gaps. - -| Formative observation | Product or harness response | Engineering interpretation | +| Claim | Evidence | Boundary | | --- | --- | --- | -| Agents could build raw plans through the Python API but could not import them through the public CLI/RPC lifecycle | Added JSON-RPC and `wf artifact create-from-plan` support | A working internal API is insufficient when the agent-facing front door omits it | -| Agents inspected source and tests to infer raw-plan and component shapes | Added the compact and verbose `wf schema` catalog and expanded workflow skills | Public schema discovery is part of the product contract | -| Local CLI mode silently omitted configured Python sources | Routed local CLI composition through the configuration-aware server builder | Equivalent CLI targets must compose equivalent source inventories | -| Output bindings failed when destination schemas or referenced `$defs` were absent | Added capability-aware schema projection, generalized `wf draft bind`, and workflow-output editing | Binding helpers must propagate known schemas rather than force agents to reproduce JSON Schema internals | -| Forward routes failed while the target step had not yet been added | Preserved invalid intermediate drafts and returned direct route-repair guidance | Mutable authoring state must tolerate repairable incompleteness | -| Draft bootstrap bound optional inputs that were absent at run time | Changed capability bootstrap to bind required inputs only and report optional inputs as notes | Best-effort synthesis should avoid inventing runtime requirements | -| Agents misreported source reads or returned reports only as files | Added tool-evidence policy checks, explicit instruction profiles, inline-report requirements, and authoritative manual audit | Agent self-reports are evidence inputs, not final evaluation truth | +| Separate versions and runs | Lifecycle; E1 | No universal portability | +| Python lifecycle objects | Walkthrough; E2 | Not a usability study | +| Invalid bindings diagnosed | Validation; E3 | Not business correctness | +| Nested execution ownership | Runtime; E4 | No general fork/gather | +| Persisted step limits | Budget; E4 | Not a handler timeout | +| Provider-supplied operations | Sources; E5 | Unequal provider features | +| Explicit interrupt resume | Resume; E1 | Not arbitrary crash replay | -: Formative agent-trial observations that shaped product and harness changes. {#tbl:formative-agent-findings} +: Claims, evidence, and boundaries. {#tbl:prototype-conformance} -These findings support the design of the operation, repair, and instruction -surfaces. They do not estimate how frequently a new agent or model will encounter -the same failures. +## Current Walkthrough Check -## Evidence Package +During this revision on September 7, 2026, the Python blocks in the report +walkthrough were executed in order against the example server configuration +with an isolated temporary store. An in-process client port substituted for +the HTTP connection. The check exercised discovery, graph construction, +validation, saving, deployment, execution, refresh, and trace inspection. -The evidence supporting the thesis claims is summarized below. +The fixed fixture produced the expected title, three action items, and +Markdown heading. A deliberate invalid output binding was rejected before +saving; restoring the projection allowed validation and execution to proceed. +This adds a repair example without claiming that a new user would find the +diagnostic sufficient. -- **Deployment validation catches source drift.** Evidence: - `test_validation.py`. It asserts that missing, disabled, or changed - capabilities produce diagnostics. Result: pass in the focused test suite. -- **Interrupted runs resume at explicit boundaries.** Evidence: - `test_run_api.py` and resume-concurrency tests. They assert that stopped run - state is persisted and resumed through the run API. Result: pass in the - focused test suite. -- **Python source lifecycle works.** Evidence: - `test_report_workflow_example.py`. It asserts that a Python capability can be - loaded, saved as an artifact, deployed, and executed. Result: pass in the - focused test suite. -- **Serial multi-node workflow works.** Evidence: - `test_browser_click_workflow_example.py`. It asserts that `open_click_page`, - `wait_for_click`, and `collect_snapshots` complete with before/after evidence. - Result: pass in the focused test suite. -- **Bounded agent-operability campaign is checkable.** Evidence: - `agent-challenge-cohort.json`, generated results and figures, local report - hashes, and Appendix C. It asserts that two challenges, two models, three - instruction profiles, and three audited repetitions per cell are explicitly - recorded. Result: 36 audited trials: 27 pass, 8 invalid, 1 fail. -- **CLI and JSON-RPC share the API surface.** Evidence: - `tests/wf_transport_rpc_http/` and `tests/wf_cli/`. They assert that transport - and CLI operations delegate to the same workflow API surface. Result: pass in - the focused test suite. - -The bullet list summarizes repository evidence verified at the recorded commit. - -::: {#include-agent-challenge-results} -::: - -## Verification Snapshot - -This report records one focused verification snapshot to make the evidence -claims auditable from the text. - -| Field | Value | -| --- | --------- | -| Date run | 2026-06-16 | -| Baseline commit | `e24f2892` before subsequent document-polish edits | -| Result | `72 passed in 9.22s` | -| Environment | Local Windows development environment, Python via `uv` | -| Scope | Documentation links, report workflow, browser-click workflow, challenge harness, deployment validation, and run API tests | - -: Focused verification snapshot recorded during document preparation. {#tbl:verification-snapshot} - -Command: +The check does not exercise network startup or HTTP transport, and is not a +performance benchmark. The documentation and report-example test files are +also run together as a focused verification target: ```powershell -uv run pytest tests/docs tests/examples/test_report_workflow_example.py ` - tests/examples/test_browser_click_workflow_example.py ` - tests/examples/test_opencode_browser_click_challenge.py ` - tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q +$suites = @( + "tests/docs/test_big_doc_links.py", + "tests/examples/test_report_workflow_example.py" +) +uv run pytest @suites -q -n 0 ``` -## Implemented Scope Matrix +Those tests cover document integration and the existing raw-plan fixture. +They do not independently execute every Python block in the thesis; the +walkthrough execution is a separate editorial smoke check. -| Area | Implemented evidence | Not claimed | Future work | -| --- | ---- | ---- | ---- | -| Workflow lifecycle | Draft, artifact, deployment, run, trace, and list/inspect/resume surfaces | Exactly-once execution or arbitrary mid-node crash recovery | Transactional stores and richer run debugging | -| Source providers | Built-in, MCP, and Python source families | Symmetric feature depth across all providers | Provider add/update/remove/reload lifecycle | -| Execution model | Outcome-routed graph with node, condition, foreach, subgraph, join, interrupt, and end steps | General fork/gather programming model | Parallel fork/gather and aggregation | -| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output, and a bounded 36-trial campaign | Broad model generalization, controlled profile effects, or token reduction | Broader challenge suite and controlled comparative evaluation | -| Auth/security | Auth record plumbing and source diagnostics | Production security, encrypted-at-rest secrets, RBAC, sandboxing | Secret-manager integration and policy enforcement | +## Separating Design Comparison from Evaluation -: Implemented scope, explicit non-claims, and future work. {#tbl:implemented-scope} +The earlier comparison of n8n, Zapier, and LangGraph explains different +authoring and execution choices. The same task has not been measured across +those systems under matched conditions. This report therefore cannot rank +their usability, reliability, or performance against the prototype. -### Architecture And Code Walkthrough +Execution tests ask whether contracts, routing, state updates, and persistence +behave as specified. Interaction evaluation asks whether an author can +discover operations, express a procedure, understand errors, and recover +without inspecting implementation code. Passing one kind of test does not +answer the other. -The four-layer architecture (core, API surface, server composition, transport) -is implemented in separate packages with clear boundaries. The Workflow API -Surface is protocol-neutral; JSON-RPC and CLI are transport implementations -that delegate to the same `WorkflowApi` facade. +The report walkthrough exposes costs such as explicit bindings and deployment +selection, but does not measure whether those costs are acceptable to new +users. Similarly, structured diagnostics and inspection objects may help an +agent avoid trial and error, but reduced retries, token use, and repair time +remain hypotheses rather than measured outcomes. -### Workflow Lifecycle Tests +## Current Evaluation Boundaries -Automated tests cover artifact creation, deployment validation, run execution, -run inspection, and trace retrieval. These tests exercise the full lifecycle -from plan to completed run. +A new agent evaluation should target the actual application-facing tools, +including the intended shell-backed workflow interface once integrated. +Its allowed operations, environment, task fixtures, and success criteria +must be fixed before collecting results. An agent's self-report should be +checked against saved artifacts, deployment identity, run output, and +recorded interactions. -Evidence: - -- `tests/wf_api/test_artifact_api.py` -- `tests/wf_api/test_run_api.py` - -### Validation And Diagnostics Tests - -Tests verify that draft validation catches schema violations, deployment -validation detects source drift, and diagnostics include repair hints. The -validation tests demonstrate that failed states are machine-readable and include -repair guidance. - -Evidence: - -- `tests/artifacts/test_validation.py` -- `tests/wf_api/test_source_admin_api.py` - -### Source Provider Tests - -MCP source provider tests cover tool discovery, resource listing, prompt -inventory, stateful session reuse, and auth binding. Python source provider -tests cover module import, `NodeSpec` projection, and capability calling. The -tests exercise the source-provider boundary across different source families. - -Evidence: - -- `tests/wf_sources_mcp/test_runtime.py` -- `tests/wf_sources_python/test_loader.py` -- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` - -### Stateful MCP Session Tests - -MCP-backed server tests verify that stateful sessions are reused across -workflow calls rather than creating fresh one-off clients. This demonstrates -source-provider correctness for providers whose behavior depends on session -state. - -Evidence: - -- `tests/wf_sources_mcp/test_runtime.py` -- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` - -### Python Source Case Study - -The report workflow example demonstrates the source abstraction is not -MCP-only. A Python source with three typed capabilities is loaded and exposed -through the source inventory; the automated lifecycle test runs the -deterministic three-node report pipeline through artifact, deployment, and run -records. The browser-click example complements this with a serial three-node -Python workflow. - -Evidence: - -- `examples/report_workflow/` -- `examples/browser_click_workflow/` -- `tests/examples/test_report_workflow_example.py` -- `tests/examples/test_browser_click_workflow_example.py` - -### CLI And Transport Tests - -CLI and transport tests verify that the surface intended for external-agent -operation is exposed through JSON-RPC. Structured output, validation commands, -and inspect commands produce machine-readable responses. - -Evidence: - -- `tests/wf_cli/` -- `tests/wf_transport_rpc_http/` - -### Config Validation - -Config validation catches import and path errors before server startup. This -prevents the server from starting with broken source configurations and -provides earlier, structured failure feedback. - -(Evidence: `src/wf_config/`.) - -## Planner-Efficiency Design Hypothesis, Not Measured Outcome - -The platform targets planner efficiency and operational clarity rather than -runtime throughput. The design hypothesis is that typed contracts, validation, -diagnostics, compact outputs, and traces are intended to reduce blind retries: - -- Validation calls return structured diagnostics with repair hints. -- Source catalogs let agents discover available capabilities without probing. -- Compact JSON output is intended to reduce token usage compared to raw - provider payloads. -- Next-action guidance provides a suggested next step without the agent having - to reconstruct lifecycle state. - -A before/after comparison is illustrative: in early ad-hoc agent/tool -interaction, an agent might spend multiple attempts discovering a valid tool -sequence through trial and error. With the typed lifecycle, the agent validates -a draft, reads the diagnostic, fixes the specific issue, and proceeds. This -report evaluates whether the diagnostic and lifecycle surfaces exist and are -actionable; it does not measure retry reduction, token savings, or -convergence rates across agents. - -The tradeoff is that this lifecycle can require more authoring turns up front: -an agent may discover capabilities, create or patch a draft, validate, save an -artifact, bind a deployment, and validate again before the first production -run. The intended exchange is higher authoring overhead for more deterministic, -inspectable, and reusable runtime execution. - -Threat to validity: the audited agent campaign tests product operability, not -planner efficiency. It changed alongside the prototype and prompt rules and has -no direct-tool-loop baseline. Claims regarding convergence, retry reduction, or -token savings should therefore be interpreted as design hypotheses rather than -experimentally validated results. - -## Agent Instruction Layer - -The product-facing CLI and JSON-RPC surfaces are not sufficient by themselves -for agent operability. External agents also need an instruction layer: skills, -runbooks, and prompt templates that explain the lifecycle, valid command paths, -plan shapes, validation workflow, and failure rules without requiring the agent -to inspect implementation code. - -This became visible in early browser-click challenge trials. When the prompt -or skills did not clearly explain the raw-plan and draft-authoring paths, -agents sometimes looked at tests, source files, prior trial artifacts, or -existing example stores to infer the correct shape. That behavior may still -produce a successful workflow run, but it weakens the evaluation because the -trial no longer measures whether the public product surface and instruction -layer were sufficient. - -For this reason, the challenge report schema tracks read-behavior flags such as -skills, docs, product code, adjacent attempts, prior stores, and existing -solutions. These flags are not moral judgments about an agent; they are audit -metadata. They distinguish product-surface success from success that depended -on reverse-engineering implementation details or reading nearby answers. - -The design implication is that agent-facing infrastructure has three layers: -the operation surface (`wf` and JSON-RPC), the repair surface (validation -diagnostics, traces, compact output, and next actions), and the instruction -surface (skills and runbooks). The bounded campaign measures their combined -operability but does not causally isolate any one layer. +The earlier CLI evaluation is retained separately in +[repository history](../historical/thesis/2026-09-07-retired-agent-evaluation.md). +It is not included as evidence for the current authoring experience, and +there are no replacement campaign results in this thesis. A demonstration +of the surrounding application will not by itself establish authoring +success across tasks or users. ## Falsifiability Criteria -The design would fail its own criteria if: +The implementation would fail its stated contracts if, for example: -- source providers routinely required changes to `wf_core`; -- deployments could not detect missing or drifted source requirements before - execution; -- run records could not be inspected or resumed at explicit interruption - boundaries; -- external agents had to import implementation internals rather than using the - public CLI/API lifecycle for ordinary authoring and execution. +- a saved run could not identify the definition and bindings it used; +- an invalid foreach boundary were accepted and executed as another region; +- nested state writes were lost or counted twice; +- resume reset the run-wide budget or resumed the wrong item; +- condition evaluation used a different context model from validation; +- ordinary source invocation required provider-specific graph routing. -## Evaluation Questions - -The implementation addresses these evaluation questions: - -1. Can a source capability be discovered, called, saved into a workflow, - deployed, and run? --- Demonstrated in controlled tests by the Python source - case study and its automated tests. - -2. Can an interrupted run persist at an explicit interruption boundary and - resume? --- Demonstrated in controlled tests by run persistence and resume - tests. - -3. Can the same server be used through CLI and JSON-RPC transport? --- - Demonstrated in controlled tests: the CLI and transport tests exercise both - surfaces against the same server composition. - -4. Can a new source family be added without changing `wf_core`? --- - Demonstrated for the implemented built-in, MCP, and Python split: the - source-provider boundary is in `wf_platform` and `wf_server`, not in the - core. Future source families should fit this pattern if they can be - projected into the same capability/source contract. - -5. Are large raw provider payloads bounded in CLI output? --- Partially. Source - inventory previews are bounded by `SOURCE_PREVIEW_LIMIT`; `wf cap call` - offers compact/text rendering with `--max-output-chars`. Raw JSON output - remains intentionally lossless. - -6. Can platform sources such as `wf.std` be used without self-bindings? --- - Demonstrated in validation tests: platform sources have - `binding_required: False` in their source policy, and deployment validation - rejects unnecessary platform source bindings. - -7. Can source resources be referenced by logical source and dereferenced - through a bounded helper? --- Demonstrated for `wf.source.read_resource`, - which resolves logical source refs through runtime context with bounded - output policy. - -8. Does the structured surface reduce failed attempts before success? --- Not - measured in this report. The validation diagnostics, compact output, and - next-action guidance are designed for this purpose, but retry reduction - remains future evaluation work. - -9. Do validation and deployment validation catch source drift? --- Demonstrated - in controlled validation tests: deployment validation reports unrunnable - state with diagnostics instead of silently executing against incompatible - capabilities. +These cases support focused regression tests. Broader architectural claims, +such as accommodating future source families without changing the runtime, +remain design expectations to assess as those integrations are built. # Limitations -The following limitations are stated explicitly to maintain credibility and -motivate future work: +The prototype demonstrates a workflow lifecycle under controlled conditions. +Its main limitations concern how much authors must understand, which +execution guarantees are provided, and how far the available evidence can +be generalized. -## Threat Model And Non-Goals +## Authoring and Diagnosis Still Require Technical Knowledge -The prototype assumes trusted operators, trusted local Python sources, and -non-production credential handling. It does not attempt sandboxing, -least-privilege execution, multi-tenant isolation, human approval gates, -role-based authorization, or secret-manager-backed auth. These are product and -deployment concerns beyond the controlled system-design evidence in this report. +The Python client reduces manual serialization and provides editable graphs +and inspectable objects. It does not remove the need to understand schemas, +state bindings, outcomes, and deployment selection. The report example makes +this cost visible: a short procedure requires more declarations than direct +function calls. -- **Python sources are trusted in-process code.** No sandbox is implemented. - A Python source can execute arbitrary code within the server process. +Diagnostics identify many invalid structures and bindings, but an accurate +message is not necessarily an understandable repair instruction. Authors +still need to distinguish a graph error from an environment problem or a +failed external operation. The current evidence does not establish that new +users can make these distinctions without assistance. -- **Python sources are static at server startup.** No hot reload is - implemented. Changing a Python source requires restarting the server. +Inspection also requires judgment. A trace shows recorded execution, not +whether a report is factually correct or a remote side effect was desirable. +The system remains in development, and its authoring and operational +experience needs evaluation and refinement. -- **Source provider lifecycle is early.** The provider seam covers static - inventory loading. Admin, apply, auth, and live health checks are not part - of the current provider protocol, especially for non-MCP mutable sources. +## Execution Guarantees Have Defined Boundaries -- **Workflow portability is scoped.** Local Python code, MCP catalogs, auth - records, and source stores can differ between environments. An artifact that - is runnable in one environment may require different bindings in another. +Foreach iteration, nested workflow scopes, structured context, and run-wide +step budgets are implemented foundations. They do not yet provide general +fork/gather control for arbitrary branches. In particular, concurrent +iteration should not be presented as a solution to correlating branches +that split, loop, and later meet at different gather points. -- **No broad external evaluation.** The prototype has not been evaluated against - a large external provider catalog or a broad user study. The evidence claim - is limited to controlled examples. +Persisted interrupted runs can resume at explicit boundaries. This is not +arbitrary mid-handler crash recovery, replay of every external call, or an +exactly-once side-effect guarantee. A run's step limit bounds admitted graph +steps; it does not bound a handler's execution time or the cost of its +external requests. -- **File-backed stores.** The current implementation uses filesystem stores as - proof for durable lifecycle and to keep serialized records inspectable during - prototype development. Durability itself should not be framed as - filesystem-specific; SQL or transactional stores are future work. +Schema validation checks declared structure, not business truth. A returned +report may satisfy its schema while containing incorrect information. +Similarly, a fixed graph specifies routing but does not make remote +responses or concurrent completion order reproducible. -- **Prototype auth.** Auth records and admin surfaces exist as plumbing for - source readiness. End-to-end production credential handling, encrypted-at-rest - storage, and secret-manager integration are not verified as core thesis - claims. +## Deployment and Trust Assumptions -- **No run deletion.** Run records cannot be deleted through the current API. +The controlled examples assume trusted operators and trusted Python sources. +Python operations execute in the server process without a sandbox. The +thesis does not establish multi-tenant isolation, role-based authorization, +or production-grade credential management. An explicit workflow interrupt +can request a response, but is not by itself an authenticated approval or +access-control mechanism. -- **No MCP widget/resource proxying.** Upstream interactive widgets are not - carried through the durable workflow path. +Source bindings make environment choices inspectable rather than making +workflows universally portable. A destination environment still needs +compatible operations, credentials, and dependencies. Python sources are +loaded at startup; changing their code requires a server restart. The shared +provider interface does not yet unify every provider's administration, +authentication, and live health behavior. -- **Crash recovery at stopped boundaries.** Recovery is available at - stopped/interrupted run boundaries, not at arbitrary mid-node checkpoints. +Filesystem-backed stores support the demonstrated persistence paths. Their +use does not establish production performance, cross-process contention +behavior at scale, or disaster recovery. A future database implementation +would still need to preserve the lifecycle's transaction and ownership +contracts; changing the storage engine alone would not prove those +properties. -- **No offline scheduling.** Scheduled execution of deployments is not - implemented. +Scheduled deployment execution is not implemented. Nor does exposing a +provider's callable operations imply that its interactive widgets or entire +user experience are reproduced through the workflow API. -- **No visual workflow editor.** The platform is driven through CLI and API - surfaces; no graphical editor exists yet. +## Limits of the Evidence -- **External planner boundary.** The platform serves external agents through - public workflow operations; an integrated autonomous planning layer is not - part of the current prototype. +The deterministic report fixture demonstrates lifecycle integration, not +broad document understanding or graph expressiveness. Targeted tests cover +additional execution mechanisms, but their passing results apply to the +cases and revisions tested. -- **No general fork/gather.** Fork and gather workflow control is future work. +The current walkthrough check verifies documented calls against an isolated +service API, not network deployment or usability for an independent author. +An evaluation of the intended shell-backed application remains pending. -- **No approval, roles, policy, or multi-user review.** There is no - role-based access control or review workflow. +There is no matched cross-system experiment or broad human user study. +Consequently, this report cannot claim that the prototype is easier to use, +more reliable, or faster than the systems discussed earlier. The comparison +explains design choices; the implementation evidence tests this system's +own behavior. # Future Work -The remaining work is prioritized by whether it strengthens the prototype's -operational foundation or expands its feature scope. +The next work should strengthen execution semantics without losing sight +of the author who must understand them. The live roadmap records engineering +order; the priorities below explain why that work matters to this design. -## Near-Term Engineering Priorities +## Establish General Fork and Gather Semantics -- **Provider lifecycle.** Add, update, remove, apply, and reload operations - for multiple source families. Extend the provider protocol beyond static - inventory loading. +The immediate runtime direction is to consolidate identity resolution and +then establish explicit fork/gather behavior. Existing frames, scopes, +iteration activations, and state lineages provide foundations, but their +relationships must remain coherent through nested execution and resume. -- **Python development reload.** Hot reload for Python sources during - development, without requiring server restart. +A fork creates concurrent execution branches; a gather must determine +which arriving branches belong together before combining their state. +Loops, partial gathers, and repeated visits make this more than waiting +for a fixed number of arrivals. The design must also preserve contribution +identity so that a write already included in one merge is not applied +again in a later merge. -- **Production auth and secret stores.** Encrypted-at-rest credential storage, - secret-manager integration, and production-grade auth flows. +This work remains planned, with reference-model verification preceding +production implementation. Pressure cases should become executable tests +for correlation, merge behavior, and recovery. The authoring contract also +needs to make clear which graphs are rejected before execution and which +decisions remain the author's responsibility. -- **SQL and transactional stores.** Replace file-backed stores with - transactional storage for production durability. +## Evaluate the Authoring and Recovery Experience -- **Richer run debugging.** Time-travel debugging, run rewind, and - mid-execution inspection beyond stopped and interrupted resume. +A focused usability study should ask participants to discover an operation, +build a small workflow, change its contract, diagnose a broken binding, and +inspect a failed or interrupted run. Useful measures include task completion, +time to a correct repair, unnecessary retries, and reliance on source-code +inspection. -## Longer-Term Capability Expansion +Execution and interaction should be evaluated together without conflating +them. For example, a validation rule may correctly reject a graph while its +diagnostic fails to explain the ownership boundary that was crossed. +Conversely, a convenient editing operation must not hide a change to the +workflow's execution meaning. -- **OpenAPI or fetch-style source provider stabilization.** The repository has - an experimental OpenAPI source family; future work is hardening, operator - documentation, auth integration, and broader HTTP coverage rather than the - first proof of concept. +The Python client and CLI should receive evidence appropriate to their own +interaction styles. Broader agent trials can vary tasks and instruction +profiles, while human evaluation can test whether the lifecycle vocabulary +and data-binding model are understandable without implementation knowledge. -- **LLM nodes as typed source capabilities.** LLM calls exposed as - `NodeSpec` contracts, allowing planners to compose LLM steps into workflows - without making the core runtime model-aware. +## Scheduling and the Surrounding Application -- **Agent interface and planner loop.** Add a surrounding layer that combines a - chat or web interface, a planner graph, and `wf` operations exposed as tools. - This layer can drive the implemented workflow lifecycle without moving - planning logic into the core runtime. +Scheduled deployment execution is a required product direction, not an +implemented capability. It introduces trigger identity, overlap policy, +and recovery decisions in addition to time-expression parsing. It should +build on the same run lifecycle rather than create a separate execution +model. Suspending an already-running workflow until a time or event is a +related but distinct design question; a wait node is not specified here. -- **Scheduler and daemon operations.** Offline scheduling for deployments, - cron-triggered runs, and server daemon lifecycle. +The surrounding application is intended to combine assistant-backed chat +with workflow administration. A shell can let an assistant retain Python +objects across interactions, while typed client objects can support +dedicated views of artifacts, deployments, and runs. Specialized display +payloads, potentially using MIME types, are a presentation option to +investigate rather than an established public contract. -- **Fork and gather workflow control.** General parallel execution and result - aggregation within workflow graphs. +This direction does not establish a completed workflow-agent integration. +It needs a concrete interaction design, demonstrated public-client use, +and its own evaluation before contributing success claims to the thesis. -- **UI and admin dashboard.** First-party workflow UI for listing, inspecting, - and editing workflows. +## Extend Operations When Concrete Use Requires Them -- **Richer evaluation.** Larger source catalogs, real-world workflow - benchmarks, and broader agent evaluation with more attempts and failure - categories. +Provider expansion and operational hardening should similarly follow actual +requirements. Examples include broader OpenAPI coverage, source reload, +secret-manager integration, and an alternative storage backend. Each needs +its own compatibility, failure, and deployment evidence; none follows +automatically from the existence of a provider or store interface. + +Richer debugging should clarify what can safely be resumed or repeated, +especially around external side effects. Showing more trace information is +different from promising that an earlier action can be undone. # Conclusion -External LLM agents can be used to author and operate workflows, but reusable -workflow lifecycle records should live in a typed platform substrate. This -report described the design and implementation of `lda.chat`, a prototype -platform that separates planning from execution across controlled built-in, -MCP, and Python source examples. +This report examined how a useful procedure can become a reusable workflow +that an author can define and an operator can inspect. The implemented +system separates the saved definition, its environment bindings, and each +execution into artifacts, deployments, and runs. Those distinctions give +workflow use a record beyond the lifetime of an editing session or a single +script invocation. -The implementation supports five bounded claims: +The graph model separates data movement from control movement. Contracts +and bindings describe what a step receives and writes; outcomes choose +transitions; explicit runtime constructs govern iteration, child scopes, +and interruption. The Python client exposes this model through authoring +and inspection objects, while the API and provider boundaries connect it +to configured operations. -1. A typed artifact, deployment, and run lifecycle provides persisted workflow - records and resumability at explicit stopped/interrupted boundaries. -2. The source-provider boundary lets built-in, MCP, and Python sources share - one workflow surface, and is designed to admit future source families that - can be projected into the existing capability/source contract without - core-runtime changes. -3. Validation and diagnostics produce machine-readable failure states with - repair hints intended to support planner repair loops. -4. The CLI and JSON-RPC transport provide a surface designed for external LLM - agents to drive without direct runtime access. -5. The deterministic report-workflow case study demonstrates the full lifecycle - from config validation through run execution and trace inspection. +The report case demonstrates that a small typed procedure can be saved, +deployed, executed, and inspected. Targeted tests support specific +validation, state, and persistence behaviors. These results support the +feasibility of the design under the tested conditions, +not a claim of production readiness or superior usability. -The remaining work is clear and bounded: provider lifecycle, production auth, -scheduling, fork/gather, richer debugging, and broader evaluation. The prototype -demonstrates the architecture; the thesis contribution is the platform design -and evidence that the design can work across multiple source families under -controlled conditions. The implemented contribution is therefore the durable, -typed workflow substrate required by an agent-facing automation system; the -agent interface and autonomous planning loop can be layered over it as future -work. +The central trade-off remains visible: explicit contracts and lifecycle +boundaries improve inspectability but ask authors to understand more than +a sequence of function calls. The next stage must therefore test both the +correctness of richer execution semantics and the clarity of the experience +used to author and diagnose them. A workflow system is useful only when its +execution rules are dependable and its users can understand what they have +asked it to do. # References {#sec:refs .unnumbered} @@ -1747,357 +1569,71 @@ work. \appendix -# Case Study Command Transcript - -The following commands demonstrate the full lifecycle of the report workflow -case study. All commands assume execution from the repository root. - -## Config Validation - -```powershell -uv run wf config validate examples/report_workflow/wf.config.json -``` - -## Server Startup - -```powershell -uv run wf-rpc-server --config examples/report_workflow/wf.config.json -``` - -## Status Check - -```powershell -uv run wf --config examples/report_workflow/wf.config.json status -``` - -## Capability Discovery - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` - cap list --source local.report -``` - -## Capability Call - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -cap call local.report.extract_report ` ---input-file examples/report_workflow/cap-input.json --format compact -``` - -## Draft Bootstrap And Focused Edits - -`wf draft create --capability` is a best-effort bootstrap. It creates a one-step -draft from the selected capability's wrapper hints. Focused commands then cover -common edits without requiring the agent to write RFC 6902 patches by hand. - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -draft create report_ws --capability local.report.extract_report ` - --name report_case_study --title "Report Case Study" - -uv run wf --config examples/report_workflow/wf.config.json ` -draft set-name report_ws --revision 1 --name report_case_study - -uv run wf --config examples/report_workflow/wf.config.json ` -draft set-input report_ws --revision 2 --step call ` ---map input.text=text - -uv run wf --config examples/report_workflow/wf.config.json ` -draft set-output report_ws --revision 3 --step call ` ---map title=state.title --map summary=state.summary -``` - -For structural growth, prefer focused helpers such as `draft add-step`, -`draft branch`, `draft handle`, and `draft bind` when they cover the intended -edit. Use `draft patch` only as the low-level fallback, or import a complete -raw plan when the full graph is already available. - -## Draft Validation - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -draft validate report_ws -``` - -## Artifact Saving - -The tested case-study artifact imports the complete three-node plan: - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -artifact create-from-plan examples/report_workflow/workflow.plan.json ` ---artifact report_case_study --version 1 ` ---title "Report Case Study" --outcome ok ` ---binding local.report=local.report -``` - -## Deployment Saving - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -deploy save report_case_study.default --artifact report_case_study ` ---version 1 --binding local.report=local.report -``` - -## Deployment Validation - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -deploy validate report_case_study.default -``` - -## Run Execution - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -run start report_case_study.default ` ---input-file examples/report_workflow/run-input.json ` ---trace-from 0 --trace-limit 5 -``` - -## Run Inspection - -```powershell -uv run wf --config examples/report_workflow/wf.config.json run list --limit 5 -uv run wf --config examples/report_workflow/wf.config.json run inspect -``` - -## Run Trace - -```powershell -uv run wf --config examples/report_workflow/wf.config.json ` -run trace --from 0 --limit 5 -``` - # Evidence Index -This appendix maps thesis claims to implementation evidence. It is a guardrail -against unsupported claims and complements the focused verification snapshot in -the Evaluation section. +This appendix maps the evaluation's evidence identifiers to implementation +and tests. Paths identify inspectable evidence; they are not a claim that +all listed suites passed in one newly recorded full-system run. ## Core Workflow Lifecycle -Claim: The platform separates mutable drafts, immutable artifacts, deployments, -runs, and traces. +E1: artifacts, deployments, stopped runs, and explicit resume boundaries. -Evidence: - -- `src/wf_artifacts/models.py`: artifact/deployment models. -- `src/wf_artifacts/runs/`: run records and run store. -- `src/wf_api/service.py`: facade for workflow lifecycle operations. +- `src/wf_artifacts/models.py` +- `src/wf_artifacts/runs/` +- `src/wf_api/run_lifecycle.py` - `tests/wf_api/test_artifact_api.py` - `tests/wf_api/test_run_api.py` +## Python Authoring and Inspection + +E2: reconstructed client objects, editable workflows, and the report fixture. + +- `src/wf_client/` +- `tests/wf_client/test_authoring.py` +- `tests/wf_client/test_deployments.py` +- `tests/wf_client/test_runs.py` +- `examples/report_workflow/` +- `tests/examples/test_report_workflow_example.py` + +The example's README retains the command-line route for operators who need +it. That alternative interface is not an additional evaluated case in the +current thesis. + +## Validation and Diagnostics + +E3: structural validation, source compatibility, and repair information. + +- `src/wf_core/validation/` +- `src/wf_artifacts/validation.py` +- `tests/artifacts/test_validation.py` +- `tests/core/test_structured_context_validation.py` +- `tests/core/test_foreach_control_regions.py` + +## Execution Ownership and Budgets + +E4: nesting, structured context, concurrent iteration, and persisted limits. + +- `src/wf_core/runtime/` +- `tests/core/test_foreach_back_edges.py` +- `tests/core/test_concurrent_foreach_interrupts.py` +- `tests/core/test_structured_runtime_context.py` +- `tests/core/test_run_step_budget.py` +- `tests/core/test_run_step_budget_codec.py` +- `tests/core/test_run_step_budget_async.py` + ## Source Provider Boundary -Claim: Workflow execution consumes source-provided capabilities without making -the core runtime MCP-specific. +E5: source contracts and provider-specific execution behind server composition. -Evidence: - -- `src/wf_platform/sources.py`: neutral source DTOs and source policy. -- `src/wf_server/config.py`: server composition for configured sources. -- `src/wf_sources_mcp/`: MCP source family. -- `src/wf_sources_python/`: Python source family. -- `docs/source_architecture.md` - -## Agent-Operable Surface - -Claim: The workflow lifecycle is designed to be operated by external agents -through stable CLI/API surfaces. - -Evidence: - -- `src/wf_cli/` -- `src/wf_transport_rpc_http/` -- `tests/wf_cli/` -- `tests/wf_transport_rpc_http/` -- `docs/wf_cli.md` -- `examples/agent_challenges/browser_click_challenge/`: challenge harness for - CLI-operability trials. -- `docs/thesis/agent-challenge-cohort.json`: explicit 36-trial audited cohort. - -## Validation And Diagnostics - -Claim: Validation and diagnostics make failed workflow states machine-readable -and include repair hints. - -Evidence: - -- `src/wf_artifacts/validation.py` -- `src/wf_api/next_actions.py` -- `src/wf_api/source_admin.py` -- `tests/artifacts/test_validation.py` -- `tests/wf_api/test_source_admin_api.py` - -## Stateful MCP Source Correctness - -Claim: MCP-backed sources can preserve stateful sessions across workflow calls. - -Evidence: - -- `src/wf_sources_mcp/runtime/` -- `src/wf_sources_mcp/client/` +- `src/wf_platform/sources.py` +- `src/wf_server/config.py` +- `src/wf_sources_python/` +- `src/wf_sources_mcp/` +- `tests/wf_sources_python/test_loader.py` - `tests/wf_sources_mcp/test_runtime.py` - `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py` -## Python Source Case Study - -Claim: The source-provider model is not MCP-only. - -Evidence: - -- `examples/report_workflow/` -- `src/wf_sources_python/` -- `tests/examples/test_report_workflow_example.py` -- `tests/wf_sources_python/test_loader.py` -- `examples/browser_click_workflow/` -- `tests/examples/test_browser_click_workflow_example.py` -- `examples/agent_challenges/browser_click_challenge/` -- `tests/examples/test_opencode_browser_click_challenge.py` - -## Agent Challenge Evaluation Protocol - -Claim: The project has a repeatable protocol and a bounded audited campaign for -evaluating whether external agents can use the product-facing CLI lifecycle. - -Evidence: - -- `examples/agent_challenges/browser_click_challenge/challenge.yaml`: - browser-click challenge manifest with success assertions. -- `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`: - task-specific prompt for the browser-click challenge. -- `examples/agent_challenges/report_workflow_challenge/challenge.yaml`: - report-workflow challenge manifest with success assertions. -- `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`: - task-specific prompt for the report-workflow challenge. -- `examples/agent_challenges/run_trials.py`: - central trial runner accepting any challenge manifest and instruction profile. -- `examples/agent_challenges/manifests.py`: - generic manifest loading and path resolution. -- `examples/agent_challenges/prompts.py`: - prompt composition with base, profile, and challenge fragments. -- `tests/examples/test_opencode_browser_click_challenge.py` -- `tests/examples/test_report_workflow_challenge.py` -- `tests/examples/test_agent_challenge_harness_v2.py` - -Two data-driven challenges exist (browser-click and report-workflow), both -supporting `none`, `skills`, and `all` instruction profiles. The checked cohort -contains three manually audited repetitions for each challenge/model/profile -cell. Its explicit manifest and generated figures appear in the Evaluation -chapter. Because repository snapshots and prompt rules changed across waves, -the results are longitudinal engineering evidence rather than a controlled -model leaderboard. - -## Limitations - -Claim: This is a prototype platform substrate, not a finished automation -product. - -Evidence: - -- `docs/historical/thesis/thesis-outline.md` -- `docs/current_roadmap.md` -- Absence of scheduler, visual-editor, and secret-manager production packages - in the current source tree. - -# Agent Challenge Harness - -## Shared Challenge Protocol - -The agent-challenge harness is an evaluation instrument for the CLI surface -intended for external-agent operation. It deliberately evaluates the -product-facing lifecycle rather than general Python programmability. A valid -solution uses `uv run wf ...` commands for artifact creation, deployment saving, -and run execution. Importing `WorkflowApi`, building `WorkflowServer` directly, -calling source functions directly, or solving the task as a standalone script is -treated as a bypass even if the visible output is correct. - -Both checked challenges accept two product-facing authoring paths: - -1. **Draft path.** Create a draft from one capability, apply focused draft edits - or an RFC 6902 patch, validate, save, deploy, and run. -2. **Raw-plan path.** Write a `RawWorkflowPlan` and load it with - `wf artifact create-from-plan`, then deploy and run. - -The challenge report is an inline YAML self-report with fields for product-path -use, helper-script use, workflow file, deployment id, run id, read-behavior -flags, attempt counts, missed requirements, and challenge-specific assertions. -The harness uses that block for automatic convenience classification, but the -official outcome is manually reviewed. - -## Browser-Click Challenge - -The browser-click challenge asks an external agent to build and successfully run -a workflow that opens a local page with a visible button. The workflow records a -before-click snapshot, performs or waits for a click, records an after-click -snapshot, and returns both snapshots from a deployed workflow run. Its success -contract requires `before_clicked: false`, `after_clicked: true`, no failed run, -and no leftover browser or HTTP-server process. - -## Report-Workflow Challenge - -The report-workflow challenge asks an external agent to build and successfully -run a three-step workflow over a local Python source: `read_notes`, -`extract_report`, and `render_markdown_report`. Its success contract requires a -deployed workflow run, a title matching the expected report title, rendered -Markdown output, and no helper-script or direct-API bypass. - -## Manual Audit Rubric - -Manual review checks the command transcript, the workflow file, the deployment -id, the run id, the run output or trace, and whether the agent read product -source code, adjacent attempts, prior stores, or existing solutions. The -decision precedence is: - -| Condition | Official outcome | Reason | -| --- | --- | --- | -| Product path completed and the audit trail has no disqualifying reads or bypasses | Pass | Supports product-surface operability | -| Product path completed but the agent used a disqualifying source, prior artifact, adjacent attempt, or hidden answer | Invalid | The output exists but cannot support clean evaluation | -| No product-path artifact, deployment, and run evidence | Fail | The task contract was not established | -| Product path exists, but a helper script or direct API bypass materially contributed | Invalid | Output exists, but the trial is contaminated | -| No product-path artifact, deployment, and run evidence; task solved through a helper script or direct API | Fail | The product-facing challenge contract was not established | - -: Manual audit decision rules for agent-challenge trials. {#tbl:agent-challenge-audit-rubric} - -This distinction is intentional. Agent benchmark literature and practice show -that automated scores and self-reports can be misleading when an agent can -inspect hidden answers, prior artifacts, source code, or evaluator state -[@nist-agent-cheating-2025; @openai-swebench-audit-2026]. The harness therefore -records possible invalidation flags such as helper-script bypass, -adjacent-attempt leakage, prior-store reuse, product-code dependency, false YAML -claims, timeouts, parse failures, and missing run evidence. - -## Cohort Manifest And Reproducibility - -The harness, both challenge workflows, and the 36-trial checked cohort are -implemented and manually audited. The Evaluation chapter reports official -outcomes, automatic/manual disagreement, duration, and recorded token totals. -It does not claim controlled model superiority, normalized throughput, or retry -reduction because the product, prompts, and hosted service conditions were not -held constant across waves. - -```{=latex} -\clearpage -``` - -Evidence: - -- `examples/browser_click_workflow/` -- `examples/agent_challenges/browser_click_challenge/` -- `examples/agent_challenges/report_workflow_challenge/` -- `docs/thesis/agent-challenge-cohort.json` -- `docs/thesis/agent-challenge-results.md` -- `tests/examples/test_browser_click_workflow_example.py` -- `tests/examples/test_opencode_browser_click_challenge.py` -- `tests/examples/test_report_workflow_challenge.py` - -```{.mermaid #fig:agent-challenge-audit width=50% latex-placement="H" caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."} -flowchart TB - Transcript[Agent transcript and files] --> YAML[YAML self-report] - YAML --> Classifier[Automatic convenience classification] - Transcript --> Audit[Manual audit] - Classifier --> Audit - Audit --> Outcome[Official outcome] -``` +The existing providers demonstrate this separation for their implemented +operations. They do not establish equal lifecycle features across providers +or prove compatibility with every future source family. diff --git a/tests/docs/test_big_doc_links.py b/tests/docs/test_big_doc_links.py index 5b88c11d..4d6e289e 100644 --- a/tests/docs/test_big_doc_links.py +++ b/tests/docs/test_big_doc_links.py @@ -25,9 +25,7 @@ def test_big_doc_links_case_study_and_embeds_evidence_index() -> None: assert any(link.startswith("../../examples/report_workflow") for link in links) assert re.search(r"^# Evidence Index$", doc, flags=re.MULTILINE) assert re.search(r"^## Core Workflow Lifecycle$", doc, flags=re.MULTILINE) - assert re.search( - r"^## Agent Challenge Evaluation Protocol$", doc, flags=re.MULTILINE - ) + assert re.search(r"^## Current Evaluation Boundaries$", doc, flags=re.MULTILINE) def test_project_map_links_big_doc() -> None: @@ -60,22 +58,19 @@ def test_big_doc_keeps_mcp_as_source_family() -> None: assert "product identity" in doc -def test_thesis_has_no_placeholder_author_and_keeps_appendix_evidence_together() -> ( - None -): +def test_thesis_retires_campaign_without_removing_historical_evidence() -> None: doc = (ROOT / "docs" / "thesis" / "system-design-implementation.md").read_text( encoding="utf-8" ) - appendix = doc.split("# Agent Challenge Harness", maxsplit=1)[1] - figure = appendix.index("#fig:agent-challenge-audit") - evidence = appendix.index("Evidence:") - page_break = appendix.index("\\clearpage") - assert 'author: "draft"' not in doc assert "This draft includes" not in doc - assert page_break < evidence < figure - assert figure > evidence - assert 'latex-placement="H"' in appendix[figure : figure + 300] + assert "# Agent Challenge Harness" not in doc + assert "36-trial" not in doc + assert "36 audited trials" not in doc + archive = ROOT / "docs/historical/thesis/2026-09-07-retired-agent-evaluation.md" + historical = archive.read_text(encoding="utf-8") + assert "# Agent Challenge Harness" in historical + assert "36" in historical def test_thesis_bundle_has_reproducible_agent_evaluation_assets() -> None: @@ -94,7 +89,7 @@ def test_thesis_bundle_has_reproducible_agent_evaluation_assets() -> None: assert (thesis / "agent-challenge-cohort.json").is_file() assert (thesis / "agent-challenge-results.md").is_file() - assert "include-agent-challenge-results" in doc + assert "include-agent-challenge-results" not in doc assert "include-markdown.lua" in generate_script assert "figure-format.lua" in generate_script assert "thesisFigureFormat" in generate_script