|
|
|
@@ -0,0 +1,755 @@
|
|
|
|
|
# Retired thesis evaluation and appendices
|
|
|
|
|
|
|
|
|
|
<!-- markdownlint-disable MD013 MD024 MD025 -->
|
|
|
|
|
|
|
|
|
|
Archived on September 7, 2026, before removing the earlier CLI campaign
|
|
|
|
|
from the active thesis. This is historical evidence, not an evaluation of
|
|
|
|
|
the current Python client or a future shell-backed application.
|
|
|
|
|
|
|
|
|
|
The sections below preserve the previous wording, including claims and
|
|
|
|
|
paths that were valid only for their recorded revisions. Bare repository
|
|
|
|
|
paths are rooted at the repository, not this archive directory.
|
|
|
|
|
|
|
|
|
|
Supporting artifacts remain at their existing locations to preserve the
|
|
|
|
|
reproducible evaluation tooling:
|
|
|
|
|
|
|
|
|
|
- [Generated results](../../thesis/agent-challenge-results.md)
|
|
|
|
|
- [Cohort manifest](../../thesis/agent-challenge-cohort.json)
|
|
|
|
|
- [Evaluation generator](../../thesis/generate_agent_challenge_evaluation.py)
|
|
|
|
|
- [Challenge harness](../../../examples/agent_challenges/)
|
|
|
|
|
- [Bibliography for historical citation keys](../../thesis/references.bib)
|
|
|
|
|
|
|
|
|
|
# Evaluation
|
|
|
|
|
|
|
|
|
|
The evaluation uses concrete evidence: automated tests, live smoke tests, and
|
|
|
|
|
the deterministic case study. The evidence claim is that the prototype
|
|
|
|
|
demonstrates the architecture and workflow lifecycle under controlled examples.
|
|
|
|
|
|
|
|
|
|
The current design account is being revised against the September 7, 2026
|
|
|
|
|
implementation. The recorded campaign and verification snapshots below concern
|
|
|
|
|
earlier revisions and retain their original dates and interfaces. In particular,
|
|
|
|
|
the CLI-based campaign does not evaluate the newer Python client. Later
|
|
|
|
|
capabilities require their own evidence.
|
|
|
|
|
|
|
|
|
|
## Prototype Conformance Criteria
|
|
|
|
|
|
|
|
|
|
The evaluation is organized around prototype conformance criteria derived from
|
|
|
|
|
the research question. These criteria test whether the implemented substrate has
|
|
|
|
|
the intended lifecycle, validation, source, and inspection behavior under
|
|
|
|
|
controlled examples; they do not constitute a broad reliability or user study.
|
|
|
|
|
The later Agent Instruction Layer section explains why CLI/API conformance is
|
|
|
|
|
necessary but not sufficient for broad agent-success claims.
|
|
|
|
|
|
|
|
|
|
| Criterion | Question | Evidence Type |
|
|
|
|
|
| --- | ---- | --- |
|
|
|
|
|
| Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests |
|
|
|
|
|
| Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests |
|
|
|
|
|
| Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests |
|
|
|
|
|
| Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow and browser-click case studies |
|
|
|
|
|
| Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests |
|
|
|
|
|
| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests |
|
|
|
|
|
| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness |
|
|
|
|
|
|
|
|
|
|
: Prototype conformance criteria used for evaluation. {#tbl:prototype-conformance}
|
|
|
|
|
|
|
|
|
|
This is a prototype system evaluation, not a broad user study or reliability
|
|
|
|
|
benchmark.
|
|
|
|
|
|
|
|
|
|
## Separating Design Comparison from Evaluation
|
|
|
|
|
|
|
|
|
|
The earlier comparison of n8n, Zapier, and LangGraph explains different
|
|
|
|
|
authoring and execution choices. It is not a comparative experiment. The
|
|
|
|
|
same task has not been implemented and measured across those systems under
|
|
|
|
|
matched conditions, so the evidence here cannot rank their usability,
|
|
|
|
|
reliability, or performance against this prototype.
|
|
|
|
|
|
|
|
|
|
There are also two distinct questions within the prototype. Execution tests
|
|
|
|
|
ask whether contracts, routing, state updates, and persistence behave as
|
|
|
|
|
specified. Interaction evaluation asks whether a person or agent can discover
|
|
|
|
|
operations, express a procedure, understand errors, and recover without
|
|
|
|
|
inspecting implementation code. Passing the first kind of test does not
|
|
|
|
|
answer the second.
|
|
|
|
|
|
|
|
|
|
The report walkthrough exposes concrete interaction costs, such as explicit
|
|
|
|
|
bindings and deployment selection, but does not measure whether those costs
|
|
|
|
|
are acceptable to new users. The recorded agent campaign supplies narrower
|
|
|
|
|
interaction evidence for its own tasks, interfaces, and instruction profiles.
|
|
|
|
|
Neither is a substitute for a comparative user study.
|
|
|
|
|
|
|
|
|
|
## Current Walkthrough Check
|
|
|
|
|
|
|
|
|
|
During this revision on September 7, 2026, all five Python blocks in the
|
|
|
|
|
report walkthrough were extracted from the document and executed in order.
|
|
|
|
|
The check substituted an in-process client port for the HTTP connection and
|
|
|
|
|
used the example server configuration with an isolated temporary store.
|
|
|
|
|
The resulting run completed, its report assertions passed, and its trace
|
|
|
|
|
contained four steps, including the explicit end node.
|
|
|
|
|
|
|
|
|
|
This checks the documented Python calls against the current service API. It
|
|
|
|
|
does not test network startup, HTTP transport, or an independent author
|
|
|
|
|
following the instructions. The check was an editorial smoke test, not a new
|
|
|
|
|
agent-campaign trial or a permanent regression test. The documentation and
|
|
|
|
|
report-example test files were also rerun together: 15 tests passed.
|
|
|
|
|
|
|
|
|
|
The historical campaign results and verification snapshot below are retained
|
|
|
|
|
with their original scope. These current checks do not update their sample
|
|
|
|
|
sizes or retroactively evaluate the Python client.
|
|
|
|
|
|
|
|
|
|
## Formative Agent-Trial Findings
|
|
|
|
|
|
|
|
|
|
Before the checked 36-trial campaign, exploratory agent runs were used as
|
|
|
|
|
design feedback. Prompts, product behavior, workspace isolation, and enabled
|
|
|
|
|
tools changed during this period, so these runs are not pooled into the outcome,
|
|
|
|
|
duration, or token statistics. They instead provide process-tracing evidence:
|
|
|
|
|
recurring agent failures exposed public-surface gaps, and subsequent slices
|
|
|
|
|
addressed those gaps.
|
|
|
|
|
|
|
|
|
|
| Formative observation | Product or harness response | Engineering interpretation |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| Agents could build raw plans through the Python API but could not import them through the public CLI/RPC lifecycle | Added JSON-RPC and `wf artifact create-from-plan` support | A working internal API is insufficient when the agent-facing front door omits it |
|
|
|
|
|
| Agents inspected source and tests to infer raw-plan and component shapes | Added the compact and verbose `wf schema` catalog and expanded workflow skills | Public schema discovery is part of the product contract |
|
|
|
|
|
| Local CLI mode silently omitted configured Python sources | Routed local CLI composition through the configuration-aware server builder | Equivalent CLI targets must compose equivalent source inventories |
|
|
|
|
|
| Output bindings failed when destination schemas or referenced `$defs` were absent | Added capability-aware schema projection, generalized `wf draft bind`, and workflow-output editing | Binding helpers must propagate known schemas rather than force agents to reproduce JSON Schema internals |
|
|
|
|
|
| Forward routes failed while the target step had not yet been added | Preserved invalid intermediate drafts and returned direct route-repair guidance | Mutable authoring state must tolerate repairable incompleteness |
|
|
|
|
|
| Draft bootstrap bound optional inputs that were absent at run time | Changed capability bootstrap to bind required inputs only and report optional inputs as notes | Best-effort synthesis should avoid inventing runtime requirements |
|
|
|
|
|
| Agents misreported source reads or returned reports only as files | Added tool-evidence policy checks, explicit instruction profiles, inline-report requirements, and authoritative manual audit | Agent self-reports are evidence inputs, not final evaluation truth |
|
|
|
|
|
|
|
|
|
|
: Formative agent-trial observations that shaped product and harness changes. {#tbl:formative-agent-findings}
|
|
|
|
|
|
|
|
|
|
These findings support the design of the operation, repair, and instruction
|
|
|
|
|
surfaces. They do not estimate how frequently a new agent or model will encounter
|
|
|
|
|
the same failures.
|
|
|
|
|
|
|
|
|
|
## Evidence Package
|
|
|
|
|
|
|
|
|
|
The evidence supporting the thesis claims is summarized below.
|
|
|
|
|
|
|
|
|
|
- **Deployment validation catches source drift.** Evidence:
|
|
|
|
|
`test_validation.py`. It asserts that missing, disabled, or changed
|
|
|
|
|
capabilities produce diagnostics. Result: pass in the focused test suite.
|
|
|
|
|
- **Interrupted runs resume at explicit boundaries.** Evidence:
|
|
|
|
|
`test_run_api.py` and resume-concurrency tests. They assert that stopped run
|
|
|
|
|
state is persisted and resumed through the run API. Result: pass in the
|
|
|
|
|
focused test suite.
|
|
|
|
|
- **Python source lifecycle works.** Evidence:
|
|
|
|
|
`test_report_workflow_example.py`. It asserts that a Python capability can be
|
|
|
|
|
loaded, saved as an artifact, deployed, and executed. Result: pass in the
|
|
|
|
|
focused test suite.
|
|
|
|
|
- **Serial multi-node workflow works.** Evidence:
|
|
|
|
|
`test_browser_click_workflow_example.py`. It asserts that `open_click_page`,
|
|
|
|
|
`wait_for_click`, and `collect_snapshots` complete with before/after evidence.
|
|
|
|
|
Result: pass in the focused test suite.
|
|
|
|
|
- **Bounded agent-operability campaign is checkable.** Evidence:
|
|
|
|
|
`agent-challenge-cohort.json`, generated results and figures, local report
|
|
|
|
|
hashes, and Appendix C. It asserts that two challenges, two models, three
|
|
|
|
|
instruction profiles, and three audited repetitions per cell are explicitly
|
|
|
|
|
recorded. Result: 36 audited trials: 27 pass, 8 invalid, 1 fail.
|
|
|
|
|
- **CLI and JSON-RPC share the API surface.** Evidence:
|
|
|
|
|
`tests/wf_transport_rpc_http/` and `tests/wf_cli/`. They assert that transport
|
|
|
|
|
and CLI operations delegate to the same workflow API surface. Result: pass in
|
|
|
|
|
the focused test suite.
|
|
|
|
|
|
|
|
|
|
The bullet list summarizes repository evidence verified at the recorded commit.
|
|
|
|
|
|
|
|
|
|
Historical generated results: see the linked results document above.
|
|
|
|
|
|
|
|
|
|
## Verification Snapshot
|
|
|
|
|
|
|
|
|
|
This report records one focused verification snapshot to make the evidence
|
|
|
|
|
claims auditable from the text.
|
|
|
|
|
|
|
|
|
|
| Field | Value |
|
|
|
|
|
| --- | --------- |
|
|
|
|
|
| Date run | 2026-06-16 |
|
|
|
|
|
| Baseline commit | `e24f2892` before subsequent document-polish edits |
|
|
|
|
|
| Result | `72 passed in 9.22s` |
|
|
|
|
|
| Environment | Local Windows development environment, Python via `uv` |
|
|
|
|
|
| Scope | Documentation links, report workflow, browser-click workflow, challenge harness, deployment validation, and run API tests |
|
|
|
|
|
|
|
|
|
|
: Focused verification snapshot recorded during document preparation. {#tbl:verification-snapshot}
|
|
|
|
|
|
|
|
|
|
Command:
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run pytest tests/docs tests/examples/test_report_workflow_example.py `
|
|
|
|
|
tests/examples/test_browser_click_workflow_example.py `
|
|
|
|
|
tests/examples/test_opencode_browser_click_challenge.py `
|
|
|
|
|
tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Implemented Scope Matrix
|
|
|
|
|
|
|
|
|
|
| Area | Implemented evidence | Not claimed | Future work |
|
|
|
|
|
| --- | ---- | ---- | ---- |
|
|
|
|
|
| Workflow lifecycle | Draft, artifact, deployment, run, trace, and list/inspect/resume surfaces | Exactly-once execution or arbitrary mid-node crash recovery | Transactional stores and richer run debugging |
|
|
|
|
|
| Source providers | Built-in, MCP, and Python source families | Symmetric feature depth across all providers | Provider add/update/remove/reload lifecycle |
|
|
|
|
|
| Execution model | Outcome-routed graph with node, condition, foreach, subgraph, join, interrupt, and end steps | General fork/gather programming model | Parallel fork/gather and aggregation |
|
|
|
|
|
| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output, and a bounded 36-trial campaign | Broad model generalization, controlled profile effects, or token reduction | Broader challenge suite and controlled comparative evaluation |
|
|
|
|
|
| Auth/security | Auth record plumbing and source diagnostics | Production security, encrypted-at-rest secrets, RBAC, sandboxing | Secret-manager integration and policy enforcement |
|
|
|
|
|
|
|
|
|
|
: Implemented scope, explicit non-claims, and future work. {#tbl:implemented-scope}
|
|
|
|
|
|
|
|
|
|
### Architecture And Code Walkthrough
|
|
|
|
|
|
|
|
|
|
The four-layer architecture (core, API surface, server composition, transport)
|
|
|
|
|
is implemented in separate packages with clear boundaries. The Workflow API
|
|
|
|
|
Surface is protocol-neutral; JSON-RPC and CLI are transport implementations
|
|
|
|
|
that delegate to the same `WorkflowApi` facade.
|
|
|
|
|
|
|
|
|
|
### Workflow Lifecycle Tests
|
|
|
|
|
|
|
|
|
|
Automated tests cover artifact creation, deployment validation, run execution,
|
|
|
|
|
run inspection, and trace retrieval. These tests exercise the full lifecycle
|
|
|
|
|
from plan to completed run.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `tests/wf_api/test_artifact_api.py`
|
|
|
|
|
- `tests/wf_api/test_run_api.py`
|
|
|
|
|
|
|
|
|
|
### Validation And Diagnostics Tests
|
|
|
|
|
|
|
|
|
|
Tests verify that draft validation catches schema violations, deployment
|
|
|
|
|
validation detects source drift, and diagnostics include repair hints. The
|
|
|
|
|
validation tests demonstrate that failed states are machine-readable and include
|
|
|
|
|
repair guidance.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `tests/artifacts/test_validation.py`
|
|
|
|
|
- `tests/wf_api/test_source_admin_api.py`
|
|
|
|
|
|
|
|
|
|
### Source Provider Tests
|
|
|
|
|
|
|
|
|
|
MCP source provider tests cover tool discovery, resource listing, prompt
|
|
|
|
|
inventory, stateful session reuse, and auth binding. Python source provider
|
|
|
|
|
tests cover module import, `NodeSpec` projection, and capability calling. The
|
|
|
|
|
tests exercise the source-provider boundary across different source families.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `tests/wf_sources_mcp/test_runtime.py`
|
|
|
|
|
- `tests/wf_sources_python/test_loader.py`
|
|
|
|
|
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
|
|
|
|
|
|
|
|
|
### Stateful MCP Session Tests
|
|
|
|
|
|
|
|
|
|
MCP-backed server tests verify that stateful sessions are reused across
|
|
|
|
|
workflow calls rather than creating fresh one-off clients. This demonstrates
|
|
|
|
|
source-provider correctness for providers whose behavior depends on session
|
|
|
|
|
state.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `tests/wf_sources_mcp/test_runtime.py`
|
|
|
|
|
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
|
|
|
|
|
|
|
|
|
### Python Source Case Study
|
|
|
|
|
|
|
|
|
|
The report workflow example demonstrates the source abstraction is not
|
|
|
|
|
MCP-only. A Python source with three typed capabilities is loaded and exposed
|
|
|
|
|
through the source inventory; the automated lifecycle test runs the
|
|
|
|
|
deterministic three-node report pipeline through artifact, deployment, and run
|
|
|
|
|
records. The browser-click example complements this with a serial three-node
|
|
|
|
|
Python workflow.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `examples/report_workflow/`
|
|
|
|
|
- `examples/browser_click_workflow/`
|
|
|
|
|
- `tests/examples/test_report_workflow_example.py`
|
|
|
|
|
- `tests/examples/test_browser_click_workflow_example.py`
|
|
|
|
|
|
|
|
|
|
### CLI And Transport Tests
|
|
|
|
|
|
|
|
|
|
CLI and transport tests verify that the surface intended for external-agent
|
|
|
|
|
operation is exposed through JSON-RPC. Structured output, validation commands,
|
|
|
|
|
and inspect commands produce machine-readable responses.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `tests/wf_cli/`
|
|
|
|
|
- `tests/wf_transport_rpc_http/`
|
|
|
|
|
|
|
|
|
|
### Config Validation
|
|
|
|
|
|
|
|
|
|
Config validation catches import and path errors before server startup. This
|
|
|
|
|
prevents the server from starting with broken source configurations and
|
|
|
|
|
provides earlier, structured failure feedback.
|
|
|
|
|
|
|
|
|
|
(Evidence: `src/wf_config/`.)
|
|
|
|
|
|
|
|
|
|
## Planner-Efficiency Design Hypothesis, Not Measured Outcome
|
|
|
|
|
|
|
|
|
|
The platform targets planner efficiency and operational clarity rather than
|
|
|
|
|
runtime throughput. The design hypothesis is that typed contracts, validation,
|
|
|
|
|
diagnostics, compact outputs, and traces are intended to reduce blind retries:
|
|
|
|
|
|
|
|
|
|
- Validation calls return structured diagnostics with repair hints.
|
|
|
|
|
- Source catalogs let agents discover available capabilities without probing.
|
|
|
|
|
- Compact JSON output is intended to reduce token usage compared to raw
|
|
|
|
|
provider payloads.
|
|
|
|
|
- Next-action guidance provides a suggested next step without the agent having
|
|
|
|
|
to reconstruct lifecycle state.
|
|
|
|
|
|
|
|
|
|
A before/after comparison is illustrative: in early ad-hoc agent/tool
|
|
|
|
|
interaction, an agent might spend multiple attempts discovering a valid tool
|
|
|
|
|
sequence through trial and error. With the typed lifecycle, the agent validates
|
|
|
|
|
a draft, reads the diagnostic, fixes the specific issue, and proceeds. This
|
|
|
|
|
report evaluates whether the diagnostic and lifecycle surfaces exist and are
|
|
|
|
|
actionable; it does not measure retry reduction, token savings, or
|
|
|
|
|
convergence rates across agents.
|
|
|
|
|
|
|
|
|
|
The tradeoff is that this lifecycle can require more authoring turns up front:
|
|
|
|
|
an agent may discover capabilities, create or patch a draft, validate, save an
|
|
|
|
|
artifact, bind a deployment, and validate again before the first production
|
|
|
|
|
run. The intended exchange is higher authoring overhead for more deterministic,
|
|
|
|
|
inspectable, and reusable runtime execution.
|
|
|
|
|
|
|
|
|
|
Threat to validity: the audited agent campaign tests product operability, not
|
|
|
|
|
planner efficiency. It changed alongside the prototype and prompt rules and has
|
|
|
|
|
no direct-tool-loop baseline. Claims regarding convergence, retry reduction, or
|
|
|
|
|
token savings should therefore be interpreted as design hypotheses rather than
|
|
|
|
|
experimentally validated results.
|
|
|
|
|
|
|
|
|
|
## Agent Instruction Layer
|
|
|
|
|
|
|
|
|
|
The product-facing CLI and JSON-RPC surfaces are not sufficient by themselves
|
|
|
|
|
for agent operability. External agents also need an instruction layer: skills,
|
|
|
|
|
runbooks, and prompt templates that explain the lifecycle, valid command paths,
|
|
|
|
|
plan shapes, validation workflow, and failure rules without requiring the agent
|
|
|
|
|
to inspect implementation code.
|
|
|
|
|
|
|
|
|
|
This became visible in early browser-click challenge trials. When the prompt
|
|
|
|
|
or skills did not clearly explain the raw-plan and draft-authoring paths,
|
|
|
|
|
agents sometimes looked at tests, source files, prior trial artifacts, or
|
|
|
|
|
existing example stores to infer the correct shape. That behavior may still
|
|
|
|
|
produce a successful workflow run, but it weakens the evaluation because the
|
|
|
|
|
trial no longer measures whether the public product surface and instruction
|
|
|
|
|
layer were sufficient.
|
|
|
|
|
|
|
|
|
|
For this reason, the challenge report schema tracks read-behavior flags such as
|
|
|
|
|
skills, docs, product code, adjacent attempts, prior stores, and existing
|
|
|
|
|
solutions. These flags are not moral judgments about an agent; they are audit
|
|
|
|
|
metadata. They distinguish product-surface success from success that depended
|
|
|
|
|
on reverse-engineering implementation details or reading nearby answers.
|
|
|
|
|
|
|
|
|
|
The design implication is that agent-facing infrastructure has three layers:
|
|
|
|
|
the operation surface (`wf` and JSON-RPC), the repair surface (validation
|
|
|
|
|
diagnostics, traces, compact output, and next actions), and the instruction
|
|
|
|
|
surface (skills and runbooks). The bounded campaign measures their combined
|
|
|
|
|
operability but does not causally isolate any one layer.
|
|
|
|
|
|
|
|
|
|
## Falsifiability Criteria
|
|
|
|
|
|
|
|
|
|
The design would fail its own criteria if:
|
|
|
|
|
|
|
|
|
|
- source providers routinely required changes to `wf_core`;
|
|
|
|
|
- deployments could not detect missing or drifted source requirements before
|
|
|
|
|
execution;
|
|
|
|
|
- run records could not be inspected or resumed at explicit interruption
|
|
|
|
|
boundaries;
|
|
|
|
|
- external agents had to import implementation internals rather than using the
|
|
|
|
|
public CLI/API lifecycle for ordinary authoring and execution.
|
|
|
|
|
|
|
|
|
|
## Evaluation Questions
|
|
|
|
|
|
|
|
|
|
The implementation addresses these evaluation questions:
|
|
|
|
|
|
|
|
|
|
1. Can a source capability be discovered, called, saved into a workflow,
|
|
|
|
|
deployed, and run? --- Demonstrated in controlled tests by the Python source
|
|
|
|
|
case study and its automated tests.
|
|
|
|
|
|
|
|
|
|
2. Can an interrupted run persist at an explicit interruption boundary and
|
|
|
|
|
resume? --- Demonstrated in controlled tests by run persistence and resume
|
|
|
|
|
tests.
|
|
|
|
|
|
|
|
|
|
3. Can the same server be used through CLI and JSON-RPC transport? ---
|
|
|
|
|
Demonstrated in controlled tests: the CLI and transport tests exercise both
|
|
|
|
|
surfaces against the same server composition.
|
|
|
|
|
|
|
|
|
|
4. Can a new source family be added without changing `wf_core`? ---
|
|
|
|
|
Demonstrated for the implemented built-in, MCP, and Python split: the
|
|
|
|
|
source-provider boundary is in `wf_platform` and `wf_server`, not in the
|
|
|
|
|
core. Future source families should fit this pattern if they can be
|
|
|
|
|
projected into the same capability/source contract.
|
|
|
|
|
|
|
|
|
|
5. Are large raw provider payloads bounded in CLI output? --- Partially. Source
|
|
|
|
|
inventory previews are bounded by `SOURCE_PREVIEW_LIMIT`; `wf cap call`
|
|
|
|
|
offers compact/text rendering with `--max-output-chars`. Raw JSON output
|
|
|
|
|
remains intentionally lossless.
|
|
|
|
|
|
|
|
|
|
6. Can platform sources such as `wf.std` be used without self-bindings? ---
|
|
|
|
|
Demonstrated in validation tests: platform sources have
|
|
|
|
|
`binding_required: False` in their source policy, and deployment validation
|
|
|
|
|
rejects unnecessary platform source bindings.
|
|
|
|
|
|
|
|
|
|
7. Can source resources be referenced by logical source and dereferenced
|
|
|
|
|
through a bounded helper? --- Demonstrated for `wf.source.read_resource`,
|
|
|
|
|
which resolves logical source refs through runtime context with bounded
|
|
|
|
|
output policy.
|
|
|
|
|
|
|
|
|
|
8. Does the structured surface reduce failed attempts before success? --- Not
|
|
|
|
|
measured in this report. The validation diagnostics, compact output, and
|
|
|
|
|
next-action guidance are designed for this purpose, but retry reduction
|
|
|
|
|
remains future evaluation work.
|
|
|
|
|
|
|
|
|
|
9. Do validation and deployment validation catch source drift? --- Demonstrated
|
|
|
|
|
in controlled validation tests: deployment validation reports unrunnable
|
|
|
|
|
state with diagnostics instead of silently executing against incompatible
|
|
|
|
|
capabilities.
|
|
|
|
|
|
|
|
|
|
# Case Study Command Transcript
|
|
|
|
|
|
|
|
|
|
The following commands demonstrate the full lifecycle of the report workflow
|
|
|
|
|
case study. All commands assume execution from the repository root.
|
|
|
|
|
|
|
|
|
|
## Config Validation
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf config validate examples/report_workflow/wf.config.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Server Startup
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf-rpc-server --config examples/report_workflow/wf.config.json
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Status Check
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json status
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Capability Discovery
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
cap list --source local.report
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Capability Call
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
cap call local.report.extract_report `
|
|
|
|
|
--input-file examples/report_workflow/cap-input.json --format compact
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Draft Bootstrap And Focused Edits
|
|
|
|
|
|
|
|
|
|
`wf draft create --capability` is a best-effort bootstrap. It creates a one-step
|
|
|
|
|
draft from the selected capability's wrapper hints. Focused commands then cover
|
|
|
|
|
common edits without requiring the agent to write RFC 6902 patches by hand.
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
draft create report_ws --capability local.report.extract_report `
|
|
|
|
|
--name report_case_study --title "Report Case Study"
|
|
|
|
|
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
draft set-name report_ws --revision 1 --name report_case_study
|
|
|
|
|
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
draft set-input report_ws --revision 2 --step call `
|
|
|
|
|
--map input.text=text
|
|
|
|
|
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
draft set-output report_ws --revision 3 --step call `
|
|
|
|
|
--map title=state.title --map summary=state.summary
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
For structural growth, prefer focused helpers such as `draft add-step`,
|
|
|
|
|
`draft branch`, `draft handle`, and `draft bind` when they cover the intended
|
|
|
|
|
edit. Use `draft patch` only as the low-level fallback, or import a complete
|
|
|
|
|
raw plan when the full graph is already available.
|
|
|
|
|
|
|
|
|
|
## Draft Validation
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
draft validate report_ws
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Artifact Saving
|
|
|
|
|
|
|
|
|
|
The tested case-study artifact imports the complete three-node plan:
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
artifact create-from-plan examples/report_workflow/workflow.plan.json `
|
|
|
|
|
--artifact report_case_study --version 1 `
|
|
|
|
|
--title "Report Case Study" --outcome ok `
|
|
|
|
|
--binding local.report=local.report
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Deployment Saving
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
deploy save report_case_study.default --artifact report_case_study `
|
|
|
|
|
--version 1 --binding local.report=local.report
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Deployment Validation
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
deploy validate report_case_study.default
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Run Execution
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
run start report_case_study.default `
|
|
|
|
|
--input-file examples/report_workflow/run-input.json `
|
|
|
|
|
--trace-from 0 --trace-limit 5
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Run Inspection
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json run list --limit 5
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json run inspect <run_id>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Run Trace
|
|
|
|
|
|
|
|
|
|
```powershell
|
|
|
|
|
uv run wf --config examples/report_workflow/wf.config.json `
|
|
|
|
|
run trace <run_id> --from 0 --limit 5
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
# Evidence Index
|
|
|
|
|
|
|
|
|
|
This appendix maps thesis claims to implementation evidence. It is a guardrail
|
|
|
|
|
against unsupported claims and complements the focused verification snapshot in
|
|
|
|
|
the Evaluation section.
|
|
|
|
|
|
|
|
|
|
## Core Workflow Lifecycle
|
|
|
|
|
|
|
|
|
|
Claim: The platform separates mutable drafts, immutable artifacts, deployments,
|
|
|
|
|
runs, and traces.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `src/wf_artifacts/models.py`: artifact/deployment models.
|
|
|
|
|
- `src/wf_artifacts/runs/`: run records and run store.
|
|
|
|
|
- `src/wf_api/service.py`: facade for workflow lifecycle operations.
|
|
|
|
|
- `tests/wf_api/test_artifact_api.py`
|
|
|
|
|
- `tests/wf_api/test_run_api.py`
|
|
|
|
|
|
|
|
|
|
## Source Provider Boundary
|
|
|
|
|
|
|
|
|
|
Claim: Workflow execution consumes source-provided capabilities without making
|
|
|
|
|
the core runtime MCP-specific.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `src/wf_platform/sources.py`: neutral source DTOs and source policy.
|
|
|
|
|
- `src/wf_server/config.py`: server composition for configured sources.
|
|
|
|
|
- `src/wf_sources_mcp/`: MCP source family.
|
|
|
|
|
- `src/wf_sources_python/`: Python source family.
|
|
|
|
|
- `docs/source_architecture.md`
|
|
|
|
|
|
|
|
|
|
## Agent-Operable Surface
|
|
|
|
|
|
|
|
|
|
Claim: The workflow lifecycle is designed to be operated by external agents
|
|
|
|
|
through stable CLI/API surfaces.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `src/wf_cli/`
|
|
|
|
|
- `src/wf_transport_rpc_http/`
|
|
|
|
|
- `tests/wf_cli/`
|
|
|
|
|
- `tests/wf_transport_rpc_http/`
|
|
|
|
|
- `docs/wf_cli.md`
|
|
|
|
|
- `examples/agent_challenges/browser_click_challenge/`: challenge harness for
|
|
|
|
|
CLI-operability trials.
|
|
|
|
|
- `docs/thesis/agent-challenge-cohort.json`: explicit 36-trial audited cohort.
|
|
|
|
|
|
|
|
|
|
## Validation And Diagnostics
|
|
|
|
|
|
|
|
|
|
Claim: Validation and diagnostics make failed workflow states machine-readable
|
|
|
|
|
and include repair hints.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `src/wf_artifacts/validation.py`
|
|
|
|
|
- `src/wf_api/next_actions.py`
|
|
|
|
|
- `src/wf_api/source_admin.py`
|
|
|
|
|
- `tests/artifacts/test_validation.py`
|
|
|
|
|
- `tests/wf_api/test_source_admin_api.py`
|
|
|
|
|
|
|
|
|
|
## Stateful MCP Source Correctness
|
|
|
|
|
|
|
|
|
|
Claim: MCP-backed sources can preserve stateful sessions across workflow calls.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `src/wf_sources_mcp/runtime/`
|
|
|
|
|
- `src/wf_sources_mcp/client/`
|
|
|
|
|
- `tests/wf_sources_mcp/test_runtime.py`
|
|
|
|
|
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
|
|
|
|
|
|
|
|
|
## Python Source Case Study
|
|
|
|
|
|
|
|
|
|
Claim: The source-provider model is not MCP-only.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `examples/report_workflow/`
|
|
|
|
|
- `src/wf_sources_python/`
|
|
|
|
|
- `tests/examples/test_report_workflow_example.py`
|
|
|
|
|
- `tests/wf_sources_python/test_loader.py`
|
|
|
|
|
- `examples/browser_click_workflow/`
|
|
|
|
|
- `tests/examples/test_browser_click_workflow_example.py`
|
|
|
|
|
- `examples/agent_challenges/browser_click_challenge/`
|
|
|
|
|
- `tests/examples/test_opencode_browser_click_challenge.py`
|
|
|
|
|
|
|
|
|
|
## Agent Challenge Evaluation Protocol
|
|
|
|
|
|
|
|
|
|
Claim: The project has a repeatable protocol and a bounded audited campaign for
|
|
|
|
|
evaluating whether external agents can use the product-facing CLI lifecycle.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `examples/agent_challenges/browser_click_challenge/challenge.yaml`:
|
|
|
|
|
browser-click challenge manifest with success assertions.
|
|
|
|
|
- `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`:
|
|
|
|
|
task-specific prompt for the browser-click challenge.
|
|
|
|
|
- `examples/agent_challenges/report_workflow_challenge/challenge.yaml`:
|
|
|
|
|
report-workflow challenge manifest with success assertions.
|
|
|
|
|
- `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`:
|
|
|
|
|
task-specific prompt for the report-workflow challenge.
|
|
|
|
|
- `examples/agent_challenges/run_trials.py`:
|
|
|
|
|
central trial runner accepting any challenge manifest and instruction profile.
|
|
|
|
|
- `examples/agent_challenges/manifests.py`:
|
|
|
|
|
generic manifest loading and path resolution.
|
|
|
|
|
- `examples/agent_challenges/prompts.py`:
|
|
|
|
|
prompt composition with base, profile, and challenge fragments.
|
|
|
|
|
- `tests/examples/test_opencode_browser_click_challenge.py`
|
|
|
|
|
- `tests/examples/test_report_workflow_challenge.py`
|
|
|
|
|
- `tests/examples/test_agent_challenge_harness_v2.py`
|
|
|
|
|
|
|
|
|
|
Two data-driven challenges exist (browser-click and report-workflow), both
|
|
|
|
|
supporting `none`, `skills`, and `all` instruction profiles. The checked cohort
|
|
|
|
|
contains three manually audited repetitions for each challenge/model/profile
|
|
|
|
|
cell. Its explicit manifest and generated figures appear in the Evaluation
|
|
|
|
|
chapter. Because repository snapshots and prompt rules changed across waves,
|
|
|
|
|
the results are longitudinal engineering evidence rather than a controlled
|
|
|
|
|
model leaderboard.
|
|
|
|
|
|
|
|
|
|
## Limitations
|
|
|
|
|
|
|
|
|
|
Claim: This is a prototype platform substrate, not a finished automation
|
|
|
|
|
product.
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `docs/historical/thesis/thesis-outline.md`
|
|
|
|
|
- `docs/current_roadmap.md`
|
|
|
|
|
- Absence of scheduler, visual-editor, and secret-manager production packages
|
|
|
|
|
in the current source tree.
|
|
|
|
|
|
|
|
|
|
# Agent Challenge Harness
|
|
|
|
|
|
|
|
|
|
## Shared Challenge Protocol
|
|
|
|
|
|
|
|
|
|
The agent-challenge harness is an evaluation instrument for the CLI surface
|
|
|
|
|
intended for external-agent operation. It deliberately evaluates the
|
|
|
|
|
product-facing lifecycle rather than general Python programmability. A valid
|
|
|
|
|
solution uses `uv run wf ...` commands for artifact creation, deployment saving,
|
|
|
|
|
and run execution. Importing `WorkflowApi`, building `WorkflowServer` directly,
|
|
|
|
|
calling source functions directly, or solving the task as a standalone script is
|
|
|
|
|
treated as a bypass even if the visible output is correct.
|
|
|
|
|
|
|
|
|
|
Both checked challenges accept two product-facing authoring paths:
|
|
|
|
|
|
|
|
|
|
1. **Draft path.** Create a draft from one capability, apply focused draft edits
|
|
|
|
|
or an RFC 6902 patch, validate, save, deploy, and run.
|
|
|
|
|
2. **Raw-plan path.** Write a `RawWorkflowPlan` and load it with
|
|
|
|
|
`wf artifact create-from-plan`, then deploy and run.
|
|
|
|
|
|
|
|
|
|
The challenge report is an inline YAML self-report with fields for product-path
|
|
|
|
|
use, helper-script use, workflow file, deployment id, run id, read-behavior
|
|
|
|
|
flags, attempt counts, missed requirements, and challenge-specific assertions.
|
|
|
|
|
The harness uses that block for automatic convenience classification, but the
|
|
|
|
|
official outcome is manually reviewed.
|
|
|
|
|
|
|
|
|
|
## Browser-Click Challenge
|
|
|
|
|
|
|
|
|
|
The browser-click challenge asks an external agent to build and successfully run
|
|
|
|
|
a workflow that opens a local page with a visible button. The workflow records a
|
|
|
|
|
before-click snapshot, performs or waits for a click, records an after-click
|
|
|
|
|
snapshot, and returns both snapshots from a deployed workflow run. Its success
|
|
|
|
|
contract requires `before_clicked: false`, `after_clicked: true`, no failed run,
|
|
|
|
|
and no leftover browser or HTTP-server process.
|
|
|
|
|
|
|
|
|
|
## Report-Workflow Challenge
|
|
|
|
|
|
|
|
|
|
The report-workflow challenge asks an external agent to build and successfully
|
|
|
|
|
run a three-step workflow over a local Python source: `read_notes`,
|
|
|
|
|
`extract_report`, and `render_markdown_report`. Its success contract requires a
|
|
|
|
|
deployed workflow run, a title matching the expected report title, rendered
|
|
|
|
|
Markdown output, and no helper-script or direct-API bypass.
|
|
|
|
|
|
|
|
|
|
## Manual Audit Rubric
|
|
|
|
|
|
|
|
|
|
Manual review checks the command transcript, the workflow file, the deployment
|
|
|
|
|
id, the run id, the run output or trace, and whether the agent read product
|
|
|
|
|
source code, adjacent attempts, prior stores, or existing solutions. The
|
|
|
|
|
decision precedence is:
|
|
|
|
|
|
|
|
|
|
| Condition | Official outcome | Reason |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| Product path completed and the audit trail has no disqualifying reads or bypasses | Pass | Supports product-surface operability |
|
|
|
|
|
| Product path completed but the agent used a disqualifying source, prior artifact, adjacent attempt, or hidden answer | Invalid | The output exists but cannot support clean evaluation |
|
|
|
|
|
| No product-path artifact, deployment, and run evidence | Fail | The task contract was not established |
|
|
|
|
|
| Product path exists, but a helper script or direct API bypass materially contributed | Invalid | Output exists, but the trial is contaminated |
|
|
|
|
|
| No product-path artifact, deployment, and run evidence; task solved through a helper script or direct API | Fail | The product-facing challenge contract was not established |
|
|
|
|
|
|
|
|
|
|
: Manual audit decision rules for agent-challenge trials. {#tbl:agent-challenge-audit-rubric}
|
|
|
|
|
|
|
|
|
|
This distinction is intentional. Agent benchmark literature and practice show
|
|
|
|
|
that automated scores and self-reports can be misleading when an agent can
|
|
|
|
|
inspect hidden answers, prior artifacts, source code, or evaluator state
|
|
|
|
|
[@nist-agent-cheating-2025; @openai-swebench-audit-2026]. The harness therefore
|
|
|
|
|
records possible invalidation flags such as helper-script bypass,
|
|
|
|
|
adjacent-attempt leakage, prior-store reuse, product-code dependency, false YAML
|
|
|
|
|
claims, timeouts, parse failures, and missing run evidence.
|
|
|
|
|
|
|
|
|
|
## Cohort Manifest And Reproducibility
|
|
|
|
|
|
|
|
|
|
The harness, both challenge workflows, and the 36-trial checked cohort are
|
|
|
|
|
implemented and manually audited. The Evaluation chapter reports official
|
|
|
|
|
outcomes, automatic/manual disagreement, duration, and recorded token totals.
|
|
|
|
|
It does not claim controlled model superiority, normalized throughput, or retry
|
|
|
|
|
reduction because the product, prompts, and hosted service conditions were not
|
|
|
|
|
held constant across waves.
|
|
|
|
|
|
|
|
|
|
```{=latex}
|
|
|
|
|
\clearpage
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
- `examples/browser_click_workflow/`
|
|
|
|
|
- `examples/agent_challenges/browser_click_challenge/`
|
|
|
|
|
- `examples/agent_challenges/report_workflow_challenge/`
|
|
|
|
|
- `docs/thesis/agent-challenge-cohort.json`
|
|
|
|
|
- `docs/thesis/agent-challenge-results.md`
|
|
|
|
|
- `tests/examples/test_browser_click_workflow_example.py`
|
|
|
|
|
- `tests/examples/test_opencode_browser_click_challenge.py`
|
|
|
|
|
- `tests/examples/test_report_workflow_challenge.py`
|
|
|
|
|
|
|
|
|
|
```{.mermaid #fig:agent-challenge-audit width=50% latex-placement="H" caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."}
|
|
|
|
|
flowchart TB
|
|
|
|
|
Transcript[Agent transcript and files] --> YAML[YAML self-report]
|
|
|
|
|
YAML --> Classifier[Automatic convenience classification]
|
|
|
|
|
Transcript --> Audit[Manual audit]
|
|
|
|
|
Classifier --> Audit
|
|
|
|
|
Audit --> Outcome[Official outcome]
|
|
|
|
|
```
|
