docs: revise thesis narrative, diagrams, and build layout

docs/historical/thesis/2026-09-07-retired-agent-evaluation.md

@@ -0,0 +1,755 @@
+# Retired thesis evaluation and appendices
+
+<!-- markdownlint-disable MD013 MD024 MD025 -->
+
+Archived on September 7, 2026, before removing the earlier CLI campaign
+from the active thesis. This is historical evidence, not an evaluation of
+the current Python client or a future shell-backed application.
+
+The sections below preserve the previous wording, including claims and
+paths that were valid only for their recorded revisions. Bare repository
+paths are rooted at the repository, not this archive directory.
+
+Supporting artifacts remain at their existing locations to preserve the
+reproducible evaluation tooling:
+
+- [Generated results](../../thesis/agent-challenge-results.md)
+- [Cohort manifest](../../thesis/agent-challenge-cohort.json)
+- [Evaluation generator](../../thesis/generate_agent_challenge_evaluation.py)
+- [Challenge harness](../../../examples/agent_challenges/)
+- [Bibliography for historical citation keys](../../thesis/references.bib)
+
+# Evaluation
+
+The evaluation uses concrete evidence: automated tests, live smoke tests, and
+the deterministic case study. The evidence claim is that the prototype
+demonstrates the architecture and workflow lifecycle under controlled examples.
+
+The current design account is being revised against the September 7, 2026
+implementation. The recorded campaign and verification snapshots below concern
+earlier revisions and retain their original dates and interfaces. In particular,
+the CLI-based campaign does not evaluate the newer Python client. Later
+capabilities require their own evidence.
+
+## Prototype Conformance Criteria
+
+The evaluation is organized around prototype conformance criteria derived from
+the research question. These criteria test whether the implemented substrate has
+the intended lifecycle, validation, source, and inspection behavior under
+controlled examples; they do not constitute a broad reliability or user study.
+The later Agent Instruction Layer section explains why CLI/API conformance is
+necessary but not sufficient for broad agent-success claims.
+
+| Criterion | Question | Evidence Type |
+| --- | ---- | --- |
+| Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests |
+| Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests |
+| Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests |
+| Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow and browser-click case studies |
+| Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests |
+| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests |
+| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness |
+
+: Prototype conformance criteria used for evaluation. {#tbl:prototype-conformance}
+
+This is a prototype system evaluation, not a broad user study or reliability
+benchmark.
+
+## Separating Design Comparison from Evaluation
+
+The earlier comparison of n8n, Zapier, and LangGraph explains different
+authoring and execution choices. It is not a comparative experiment. The
+same task has not been implemented and measured across those systems under
+matched conditions, so the evidence here cannot rank their usability,
+reliability, or performance against this prototype.
+
+There are also two distinct questions within the prototype. Execution tests
+ask whether contracts, routing, state updates, and persistence behave as
+specified. Interaction evaluation asks whether a person or agent can discover
+operations, express a procedure, understand errors, and recover without
+inspecting implementation code. Passing the first kind of test does not
+answer the second.
+
+The report walkthrough exposes concrete interaction costs, such as explicit
+bindings and deployment selection, but does not measure whether those costs
+are acceptable to new users. The recorded agent campaign supplies narrower
+interaction evidence for its own tasks, interfaces, and instruction profiles.
+Neither is a substitute for a comparative user study.
+
+## Current Walkthrough Check
+
+During this revision on September 7, 2026, all five Python blocks in the
+report walkthrough were extracted from the document and executed in order.
+The check substituted an in-process client port for the HTTP connection and
+used the example server configuration with an isolated temporary store.
+The resulting run completed, its report assertions passed, and its trace
+contained four steps, including the explicit end node.
+
+This checks the documented Python calls against the current service API. It
+does not test network startup, HTTP transport, or an independent author
+following the instructions. The check was an editorial smoke test, not a new
+agent-campaign trial or a permanent regression test. The documentation and
+report-example test files were also rerun together: 15 tests passed.
+
+The historical campaign results and verification snapshot below are retained
+with their original scope. These current checks do not update their sample
+sizes or retroactively evaluate the Python client.
+
+## Formative Agent-Trial Findings
+
+Before the checked 36-trial campaign, exploratory agent runs were used as
+design feedback. Prompts, product behavior, workspace isolation, and enabled
+tools changed during this period, so these runs are not pooled into the outcome,
+duration, or token statistics. They instead provide process-tracing evidence:
+recurring agent failures exposed public-surface gaps, and subsequent slices
+addressed those gaps.
+
+| Formative observation | Product or harness response | Engineering interpretation |
+| --- | --- | --- |
+| Agents could build raw plans through the Python API but could not import them through the public CLI/RPC lifecycle | Added JSON-RPC and `wf artifact create-from-plan` support | A working internal API is insufficient when the agent-facing front door omits it |
+| Agents inspected source and tests to infer raw-plan and component shapes | Added the compact and verbose `wf schema` catalog and expanded workflow skills | Public schema discovery is part of the product contract |
+| Local CLI mode silently omitted configured Python sources | Routed local CLI composition through the configuration-aware server builder | Equivalent CLI targets must compose equivalent source inventories |
+| Output bindings failed when destination schemas or referenced `$defs` were absent | Added capability-aware schema projection, generalized `wf draft bind`, and workflow-output editing | Binding helpers must propagate known schemas rather than force agents to reproduce JSON Schema internals |
+| Forward routes failed while the target step had not yet been added | Preserved invalid intermediate drafts and returned direct route-repair guidance | Mutable authoring state must tolerate repairable incompleteness |
+| Draft bootstrap bound optional inputs that were absent at run time | Changed capability bootstrap to bind required inputs only and report optional inputs as notes | Best-effort synthesis should avoid inventing runtime requirements |
+| Agents misreported source reads or returned reports only as files | Added tool-evidence policy checks, explicit instruction profiles, inline-report requirements, and authoritative manual audit | Agent self-reports are evidence inputs, not final evaluation truth |
+
+: Formative agent-trial observations that shaped product and harness changes. {#tbl:formative-agent-findings}
+
+These findings support the design of the operation, repair, and instruction
+surfaces. They do not estimate how frequently a new agent or model will encounter
+the same failures.
+
+## Evidence Package
+
+The evidence supporting the thesis claims is summarized below.
+
+- **Deployment validation catches source drift.** Evidence:
+  `test_validation.py`. It asserts that missing, disabled, or changed
+  capabilities produce diagnostics. Result: pass in the focused test suite.
+- **Interrupted runs resume at explicit boundaries.** Evidence:
+  `test_run_api.py` and resume-concurrency tests. They assert that stopped run
+  state is persisted and resumed through the run API. Result: pass in the
+  focused test suite.
+- **Python source lifecycle works.** Evidence:
+  `test_report_workflow_example.py`. It asserts that a Python capability can be
+  loaded, saved as an artifact, deployed, and executed. Result: pass in the
+  focused test suite.
+- **Serial multi-node workflow works.** Evidence:
+  `test_browser_click_workflow_example.py`. It asserts that `open_click_page`,
+  `wait_for_click`, and `collect_snapshots` complete with before/after evidence.
+  Result: pass in the focused test suite.
+- **Bounded agent-operability campaign is checkable.** Evidence:
+  `agent-challenge-cohort.json`, generated results and figures, local report
+  hashes, and Appendix C. It asserts that two challenges, two models, three
+  instruction profiles, and three audited repetitions per cell are explicitly
+  recorded. Result: 36 audited trials: 27 pass, 8 invalid, 1 fail.
+- **CLI and JSON-RPC share the API surface.** Evidence:
+  `tests/wf_transport_rpc_http/` and `tests/wf_cli/`. They assert that transport
+  and CLI operations delegate to the same workflow API surface. Result: pass in
+  the focused test suite.
+
+The bullet list summarizes repository evidence verified at the recorded commit.
+
+Historical generated results: see the linked results document above.
+
+## Verification Snapshot
+
+This report records one focused verification snapshot to make the evidence
+claims auditable from the text.
+
+| Field | Value |
+| --- | --------- |
+| Date run | 2026-06-16 |
+| Baseline commit | `e24f2892` before subsequent document-polish edits |
+| Result | `72 passed in 9.22s` |
+| Environment | Local Windows development environment, Python via `uv` |
+| Scope | Documentation links, report workflow, browser-click workflow, challenge harness, deployment validation, and run API tests |
+
+: Focused verification snapshot recorded during document preparation. {#tbl:verification-snapshot}
+
+Command:
+
+```powershell
+uv run pytest tests/docs tests/examples/test_report_workflow_example.py `
+  tests/examples/test_browser_click_workflow_example.py `
+  tests/examples/test_opencode_browser_click_challenge.py `
+  tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q
+```
+
+## Implemented Scope Matrix
+
+| Area | Implemented evidence | Not claimed | Future work |
+| --- | ---- | ---- | ---- |
+| Workflow lifecycle | Draft, artifact, deployment, run, trace, and list/inspect/resume surfaces | Exactly-once execution or arbitrary mid-node crash recovery | Transactional stores and richer run debugging |
+| Source providers | Built-in, MCP, and Python source families | Symmetric feature depth across all providers | Provider add/update/remove/reload lifecycle |
+| Execution model | Outcome-routed graph with node, condition, foreach, subgraph, join, interrupt, and end steps | General fork/gather programming model | Parallel fork/gather and aggregation |
+| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output, and a bounded 36-trial campaign | Broad model generalization, controlled profile effects, or token reduction | Broader challenge suite and controlled comparative evaluation |
+| Auth/security | Auth record plumbing and source diagnostics | Production security, encrypted-at-rest secrets, RBAC, sandboxing | Secret-manager integration and policy enforcement |
+
+: Implemented scope, explicit non-claims, and future work. {#tbl:implemented-scope}
+
+### Architecture And Code Walkthrough
+
+The four-layer architecture (core, API surface, server composition, transport)
+is implemented in separate packages with clear boundaries. The Workflow API
+Surface is protocol-neutral; JSON-RPC and CLI are transport implementations
+that delegate to the same `WorkflowApi` facade.
+
+### Workflow Lifecycle Tests
+
+Automated tests cover artifact creation, deployment validation, run execution,
+run inspection, and trace retrieval. These tests exercise the full lifecycle
+from plan to completed run.
+
+Evidence:
+
+- `tests/wf_api/test_artifact_api.py`
+- `tests/wf_api/test_run_api.py`
+
+### Validation And Diagnostics Tests
+
+Tests verify that draft validation catches schema violations, deployment
+validation detects source drift, and diagnostics include repair hints. The
+validation tests demonstrate that failed states are machine-readable and include
+repair guidance.
+
+Evidence:
+
+- `tests/artifacts/test_validation.py`
+- `tests/wf_api/test_source_admin_api.py`
+
+### Source Provider Tests
+
+MCP source provider tests cover tool discovery, resource listing, prompt
+inventory, stateful session reuse, and auth binding. Python source provider
+tests cover module import, `NodeSpec` projection, and capability calling. The
+tests exercise the source-provider boundary across different source families.
+
+Evidence:
+
+- `tests/wf_sources_mcp/test_runtime.py`
+- `tests/wf_sources_python/test_loader.py`
+- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
+
+### Stateful MCP Session Tests
+
+MCP-backed server tests verify that stateful sessions are reused across
+workflow calls rather than creating fresh one-off clients. This demonstrates
+source-provider correctness for providers whose behavior depends on session
+state.
+
+Evidence:
+
+- `tests/wf_sources_mcp/test_runtime.py`
+- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
+
+### Python Source Case Study
+
+The report workflow example demonstrates the source abstraction is not
+MCP-only. A Python source with three typed capabilities is loaded and exposed
+through the source inventory; the automated lifecycle test runs the
+deterministic three-node report pipeline through artifact, deployment, and run
+records. The browser-click example complements this with a serial three-node
+Python workflow.
+
+Evidence:
+
+- `examples/report_workflow/`
+- `examples/browser_click_workflow/`
+- `tests/examples/test_report_workflow_example.py`
+- `tests/examples/test_browser_click_workflow_example.py`
+
+### CLI And Transport Tests
+
+CLI and transport tests verify that the surface intended for external-agent
+operation is exposed through JSON-RPC. Structured output, validation commands,
+and inspect commands produce machine-readable responses.
+
+Evidence:
+
+- `tests/wf_cli/`
+- `tests/wf_transport_rpc_http/`
+
+### Config Validation
+
+Config validation catches import and path errors before server startup. This
+prevents the server from starting with broken source configurations and
+provides earlier, structured failure feedback.
+
+(Evidence: `src/wf_config/`.)
+
+## Planner-Efficiency Design Hypothesis, Not Measured Outcome
+
+The platform targets planner efficiency and operational clarity rather than
+runtime throughput. The design hypothesis is that typed contracts, validation,
+diagnostics, compact outputs, and traces are intended to reduce blind retries:
+
+- Validation calls return structured diagnostics with repair hints.
+- Source catalogs let agents discover available capabilities without probing.
+- Compact JSON output is intended to reduce token usage compared to raw
+  provider payloads.
+- Next-action guidance provides a suggested next step without the agent having
+  to reconstruct lifecycle state.
+
+A before/after comparison is illustrative: in early ad-hoc agent/tool
+interaction, an agent might spend multiple attempts discovering a valid tool
+sequence through trial and error. With the typed lifecycle, the agent validates
+a draft, reads the diagnostic, fixes the specific issue, and proceeds. This
+report evaluates whether the diagnostic and lifecycle surfaces exist and are
+actionable; it does not measure retry reduction, token savings, or
+convergence rates across agents.
+
+The tradeoff is that this lifecycle can require more authoring turns up front:
+an agent may discover capabilities, create or patch a draft, validate, save an
+artifact, bind a deployment, and validate again before the first production
+run. The intended exchange is higher authoring overhead for more deterministic,
+inspectable, and reusable runtime execution.
+
+Threat to validity: the audited agent campaign tests product operability, not
+planner efficiency. It changed alongside the prototype and prompt rules and has
+no direct-tool-loop baseline. Claims regarding convergence, retry reduction, or
+token savings should therefore be interpreted as design hypotheses rather than
+experimentally validated results.
+
+## Agent Instruction Layer
+
+The product-facing CLI and JSON-RPC surfaces are not sufficient by themselves
+for agent operability. External agents also need an instruction layer: skills,
+runbooks, and prompt templates that explain the lifecycle, valid command paths,
+plan shapes, validation workflow, and failure rules without requiring the agent
+to inspect implementation code.
+
+This became visible in early browser-click challenge trials. When the prompt
+or skills did not clearly explain the raw-plan and draft-authoring paths,
+agents sometimes looked at tests, source files, prior trial artifacts, or
+existing example stores to infer the correct shape. That behavior may still
+produce a successful workflow run, but it weakens the evaluation because the
+trial no longer measures whether the public product surface and instruction
+layer were sufficient.
+
+For this reason, the challenge report schema tracks read-behavior flags such as
+skills, docs, product code, adjacent attempts, prior stores, and existing
+solutions. These flags are not moral judgments about an agent; they are audit
+metadata. They distinguish product-surface success from success that depended
+on reverse-engineering implementation details or reading nearby answers.
+
+The design implication is that agent-facing infrastructure has three layers:
+the operation surface (`wf` and JSON-RPC), the repair surface (validation
+diagnostics, traces, compact output, and next actions), and the instruction
+surface (skills and runbooks). The bounded campaign measures their combined
+operability but does not causally isolate any one layer.
+
+## Falsifiability Criteria
+
+The design would fail its own criteria if:
+
+- source providers routinely required changes to `wf_core`;
+- deployments could not detect missing or drifted source requirements before
+  execution;
+- run records could not be inspected or resumed at explicit interruption
+  boundaries;
+- external agents had to import implementation internals rather than using the
+  public CLI/API lifecycle for ordinary authoring and execution.
+
+## Evaluation Questions
+
+The implementation addresses these evaluation questions:
+
+1. Can a source capability be discovered, called, saved into a workflow,
+   deployed, and run? --- Demonstrated in controlled tests by the Python source
+   case study and its automated tests.
+
+2. Can an interrupted run persist at an explicit interruption boundary and
+   resume? --- Demonstrated in controlled tests by run persistence and resume
+   tests.
+
+3. Can the same server be used through CLI and JSON-RPC transport? ---
+   Demonstrated in controlled tests: the CLI and transport tests exercise both
+   surfaces against the same server composition.
+
+4. Can a new source family be added without changing `wf_core`? ---
+   Demonstrated for the implemented built-in, MCP, and Python split: the
+   source-provider boundary is in `wf_platform` and `wf_server`, not in the
+   core. Future source families should fit this pattern if they can be
+   projected into the same capability/source contract.
+
+5. Are large raw provider payloads bounded in CLI output? --- Partially. Source
+   inventory previews are bounded by `SOURCE_PREVIEW_LIMIT`; `wf cap call`
+   offers compact/text rendering with `--max-output-chars`. Raw JSON output
+   remains intentionally lossless.
+
+6. Can platform sources such as `wf.std` be used without self-bindings? ---
+   Demonstrated in validation tests: platform sources have
+   `binding_required: False` in their source policy, and deployment validation
+   rejects unnecessary platform source bindings.
+
+7. Can source resources be referenced by logical source and dereferenced
+   through a bounded helper? --- Demonstrated for `wf.source.read_resource`,
+   which resolves logical source refs through runtime context with bounded
+   output policy.
+
+8. Does the structured surface reduce failed attempts before success? --- Not
+   measured in this report. The validation diagnostics, compact output, and
+   next-action guidance are designed for this purpose, but retry reduction
+   remains future evaluation work.
+
+9. Do validation and deployment validation catch source drift? --- Demonstrated
+   in controlled validation tests: deployment validation reports unrunnable
+   state with diagnostics instead of silently executing against incompatible
+   capabilities.
+
+# Case Study Command Transcript
+
+The following commands demonstrate the full lifecycle of the report workflow
+case study. All commands assume execution from the repository root.
+
+## Config Validation
+
+```powershell
+uv run wf config validate examples/report_workflow/wf.config.json
+```
+
+## Server Startup
+
+```powershell
+uv run wf-rpc-server --config examples/report_workflow/wf.config.json
+```
+
+## Status Check
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json status
+```
+
+## Capability Discovery
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+ cap list --source local.report
+```
+
+## Capability Call
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+cap call local.report.extract_report `
+--input-file examples/report_workflow/cap-input.json --format compact
+```
+
+## Draft Bootstrap And Focused Edits
+
+`wf draft create --capability` is a best-effort bootstrap. It creates a one-step
+draft from the selected capability's wrapper hints. Focused commands then cover
+common edits without requiring the agent to write RFC 6902 patches by hand.
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+draft create report_ws --capability local.report.extract_report `
+ --name report_case_study --title "Report Case Study"
+
+uv run wf --config examples/report_workflow/wf.config.json `
+draft set-name report_ws --revision 1 --name report_case_study
+
+uv run wf --config examples/report_workflow/wf.config.json `
+draft set-input report_ws --revision 2 --step call `
+--map input.text=text
+
+uv run wf --config examples/report_workflow/wf.config.json `
+draft set-output report_ws --revision 3 --step call `
+--map title=state.title --map summary=state.summary
+```
+
+For structural growth, prefer focused helpers such as `draft add-step`,
+`draft branch`, `draft handle`, and `draft bind` when they cover the intended
+edit. Use `draft patch` only as the low-level fallback, or import a complete
+raw plan when the full graph is already available.
+
+## Draft Validation
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+draft validate report_ws
+```
+
+## Artifact Saving
+
+The tested case-study artifact imports the complete three-node plan:
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+artifact create-from-plan examples/report_workflow/workflow.plan.json `
+--artifact report_case_study --version 1 `
+--title "Report Case Study" --outcome ok `
+--binding local.report=local.report
+```
+
+## Deployment Saving
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+deploy save report_case_study.default --artifact report_case_study `
+--version 1 --binding local.report=local.report
+```
+
+## Deployment Validation
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+deploy validate report_case_study.default
+```
+
+## Run Execution
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+run start report_case_study.default `
+--input-file examples/report_workflow/run-input.json `
+--trace-from 0 --trace-limit 5
+```
+
+## Run Inspection
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json run list --limit 5
+uv run wf --config examples/report_workflow/wf.config.json run inspect <run_id>
+```
+
+## Run Trace
+
+```powershell
+uv run wf --config examples/report_workflow/wf.config.json `
+run trace <run_id> --from 0 --limit 5
+```
+
+# Evidence Index
+
+This appendix maps thesis claims to implementation evidence. It is a guardrail
+against unsupported claims and complements the focused verification snapshot in
+the Evaluation section.
+
+## Core Workflow Lifecycle
+
+Claim: The platform separates mutable drafts, immutable artifacts, deployments,
+runs, and traces.
+
+Evidence:
+
+- `src/wf_artifacts/models.py`: artifact/deployment models.
+- `src/wf_artifacts/runs/`: run records and run store.
+- `src/wf_api/service.py`: facade for workflow lifecycle operations.
+- `tests/wf_api/test_artifact_api.py`
+- `tests/wf_api/test_run_api.py`
+
+## Source Provider Boundary
+
+Claim: Workflow execution consumes source-provided capabilities without making
+the core runtime MCP-specific.
+
+Evidence:
+
+- `src/wf_platform/sources.py`: neutral source DTOs and source policy.
+- `src/wf_server/config.py`: server composition for configured sources.
+- `src/wf_sources_mcp/`: MCP source family.
+- `src/wf_sources_python/`: Python source family.
+- `docs/source_architecture.md`
+
+## Agent-Operable Surface
+
+Claim: The workflow lifecycle is designed to be operated by external agents
+through stable CLI/API surfaces.
+
+Evidence:
+
+- `src/wf_cli/`
+- `src/wf_transport_rpc_http/`
+- `tests/wf_cli/`
+- `tests/wf_transport_rpc_http/`
+- `docs/wf_cli.md`
+- `examples/agent_challenges/browser_click_challenge/`: challenge harness for
+  CLI-operability trials.
+- `docs/thesis/agent-challenge-cohort.json`: explicit 36-trial audited cohort.
+
+## Validation And Diagnostics
+
+Claim: Validation and diagnostics make failed workflow states machine-readable
+and include repair hints.
+
+Evidence:
+
+- `src/wf_artifacts/validation.py`
+- `src/wf_api/next_actions.py`
+- `src/wf_api/source_admin.py`
+- `tests/artifacts/test_validation.py`
+- `tests/wf_api/test_source_admin_api.py`
+
+## Stateful MCP Source Correctness
+
+Claim: MCP-backed sources can preserve stateful sessions across workflow calls.
+
+Evidence:
+
+- `src/wf_sources_mcp/runtime/`
+- `src/wf_sources_mcp/client/`
+- `tests/wf_sources_mcp/test_runtime.py`
+- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
+
+## Python Source Case Study
+
+Claim: The source-provider model is not MCP-only.
+
+Evidence:
+
+- `examples/report_workflow/`
+- `src/wf_sources_python/`
+- `tests/examples/test_report_workflow_example.py`
+- `tests/wf_sources_python/test_loader.py`
+- `examples/browser_click_workflow/`
+- `tests/examples/test_browser_click_workflow_example.py`
+- `examples/agent_challenges/browser_click_challenge/`
+- `tests/examples/test_opencode_browser_click_challenge.py`
+
+## Agent Challenge Evaluation Protocol
+
+Claim: The project has a repeatable protocol and a bounded audited campaign for
+evaluating whether external agents can use the product-facing CLI lifecycle.
+
+Evidence:
+
+- `examples/agent_challenges/browser_click_challenge/challenge.yaml`:
+  browser-click challenge manifest with success assertions.
+- `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`:
+  task-specific prompt for the browser-click challenge.
+- `examples/agent_challenges/report_workflow_challenge/challenge.yaml`:
+  report-workflow challenge manifest with success assertions.
+- `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`:
+  task-specific prompt for the report-workflow challenge.
+- `examples/agent_challenges/run_trials.py`:
+  central trial runner accepting any challenge manifest and instruction profile.
+- `examples/agent_challenges/manifests.py`:
+  generic manifest loading and path resolution.
+- `examples/agent_challenges/prompts.py`:
+  prompt composition with base, profile, and challenge fragments.
+- `tests/examples/test_opencode_browser_click_challenge.py`
+- `tests/examples/test_report_workflow_challenge.py`
+- `tests/examples/test_agent_challenge_harness_v2.py`
+
+Two data-driven challenges exist (browser-click and report-workflow), both
+supporting `none`, `skills`, and `all` instruction profiles. The checked cohort
+contains three manually audited repetitions for each challenge/model/profile
+cell. Its explicit manifest and generated figures appear in the Evaluation
+chapter. Because repository snapshots and prompt rules changed across waves,
+the results are longitudinal engineering evidence rather than a controlled
+model leaderboard.
+
+## Limitations
+
+Claim: This is a prototype platform substrate, not a finished automation
+product.
+
+Evidence:
+
+- `docs/historical/thesis/thesis-outline.md`
+- `docs/current_roadmap.md`
+- Absence of scheduler, visual-editor, and secret-manager production packages
+  in the current source tree.
+
+# Agent Challenge Harness
+
+## Shared Challenge Protocol
+
+The agent-challenge harness is an evaluation instrument for the CLI surface
+intended for external-agent operation. It deliberately evaluates the
+product-facing lifecycle rather than general Python programmability. A valid
+solution uses `uv run wf ...` commands for artifact creation, deployment saving,
+and run execution. Importing `WorkflowApi`, building `WorkflowServer` directly,
+calling source functions directly, or solving the task as a standalone script is
+treated as a bypass even if the visible output is correct.
+
+Both checked challenges accept two product-facing authoring paths:
+
+1. **Draft path.** Create a draft from one capability, apply focused draft edits
+   or an RFC 6902 patch, validate, save, deploy, and run.
+2. **Raw-plan path.** Write a `RawWorkflowPlan` and load it with
+   `wf artifact create-from-plan`, then deploy and run.
+
+The challenge report is an inline YAML self-report with fields for product-path
+use, helper-script use, workflow file, deployment id, run id, read-behavior
+flags, attempt counts, missed requirements, and challenge-specific assertions.
+The harness uses that block for automatic convenience classification, but the
+official outcome is manually reviewed.
+
+## Browser-Click Challenge
+
+The browser-click challenge asks an external agent to build and successfully run
+a workflow that opens a local page with a visible button. The workflow records a
+before-click snapshot, performs or waits for a click, records an after-click
+snapshot, and returns both snapshots from a deployed workflow run. Its success
+contract requires `before_clicked: false`, `after_clicked: true`, no failed run,
+and no leftover browser or HTTP-server process.
+
+## Report-Workflow Challenge
+
+The report-workflow challenge asks an external agent to build and successfully
+run a three-step workflow over a local Python source: `read_notes`,
+`extract_report`, and `render_markdown_report`. Its success contract requires a
+deployed workflow run, a title matching the expected report title, rendered
+Markdown output, and no helper-script or direct-API bypass.
+
+## Manual Audit Rubric
+
+Manual review checks the command transcript, the workflow file, the deployment
+id, the run id, the run output or trace, and whether the agent read product
+source code, adjacent attempts, prior stores, or existing solutions. The
+decision precedence is:
+
+| Condition | Official outcome | Reason |
+| --- | --- | --- |
+| Product path completed and the audit trail has no disqualifying reads or bypasses | Pass | Supports product-surface operability |
+| Product path completed but the agent used a disqualifying source, prior artifact, adjacent attempt, or hidden answer | Invalid | The output exists but cannot support clean evaluation |
+| No product-path artifact, deployment, and run evidence | Fail | The task contract was not established |
+| Product path exists, but a helper script or direct API bypass materially contributed | Invalid | Output exists, but the trial is contaminated |
+| No product-path artifact, deployment, and run evidence; task solved through a helper script or direct API | Fail | The product-facing challenge contract was not established |
+
+: Manual audit decision rules for agent-challenge trials. {#tbl:agent-challenge-audit-rubric}
+
+This distinction is intentional. Agent benchmark literature and practice show
+that automated scores and self-reports can be misleading when an agent can
+inspect hidden answers, prior artifacts, source code, or evaluator state
+[@nist-agent-cheating-2025; @openai-swebench-audit-2026]. The harness therefore
+records possible invalidation flags such as helper-script bypass,
+adjacent-attempt leakage, prior-store reuse, product-code dependency, false YAML
+claims, timeouts, parse failures, and missing run evidence.
+
+## Cohort Manifest And Reproducibility
+
+The harness, both challenge workflows, and the 36-trial checked cohort are
+implemented and manually audited. The Evaluation chapter reports official
+outcomes, automatic/manual disagreement, duration, and recorded token totals.
+It does not claim controlled model superiority, normalized throughput, or retry
+reduction because the product, prompts, and hosted service conditions were not
+held constant across waves.
+
+```{=latex}
+\clearpage
+```
+
+Evidence:
+
+- `examples/browser_click_workflow/`
+- `examples/agent_challenges/browser_click_challenge/`
+- `examples/agent_challenges/report_workflow_challenge/`
+- `docs/thesis/agent-challenge-cohort.json`
+- `docs/thesis/agent-challenge-results.md`
+- `tests/examples/test_browser_click_workflow_example.py`
+- `tests/examples/test_opencode_browser_click_challenge.py`
+- `tests/examples/test_report_workflow_challenge.py`
+
+```{.mermaid #fig:agent-challenge-audit width=50% latex-placement="H" caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."}
+flowchart TB
+  Transcript[Agent transcript and files] --> YAML[YAML self-report]
+  YAML --> Classifier[Automatic convenience classification]
+  Transcript --> Audit[Manual audit]
+  Classifier --> Audit
+  Audit --> Outcome[Official outcome]
+```

docs/thesis/.gitignore

@@ -4,3 +4,5 @@
 *.typ
 !figures/*.svg
 !title-pages.tex
+!header-includes.tex
+!header-includes.html

docs/thesis/evidence-index.md

@@ -1,14 +1,19 @@
 # Thesis Evidence Index
 
 The claim-to-evidence map now lives inline in
-[`system-design-implementation.md`](system-design-implementation.md), Appendix B:
+[`system-design-implementation.md`](system-design-implementation.md), Appendix A:
 Evidence Index.
 
 This file remains as a stable pointer for older roadmap and project-map links.
 
-For the external-agent challenge evaluation workflow, including trial profiles,
+The earlier CLI campaign is retired from the active thesis. Its preserved
-manual audits, and report interpretation, see
+text and supporting data are indexed in the
+[historical evaluation](../historical/thesis/2026-09-07-retired-agent-evaluation.md).
+No replacement shell-backed agent results are claimed yet.
+
+For the retained challenge harness, including trial profiles, manual audits,
+and report interpretation, see
 [`../runbooks/agent-challenge-evaluation.md`](../runbooks/agent-challenge-evaluation.md).
-The primary checked cohort and generated aggregate are
+The historical cohort and generated aggregate remain at
 [`agent-challenge-cohort.json`](agent-challenge-cohort.json) and
 [`agent-challenge-results.md`](agent-challenge-results.md).

docs/thesis/generate.ps1

@@ -2,8 +2,16 @@
 # use pandoc -M to change the key: diagram:engine:mermaid:outputFormat to svg or pdf if output is html or pdf.
 # use the script at stuff/pandoc-diagram.ps1 to set the env vars and pass the filter to pandoc.
 param(
+    [Parameter(Mandatory = $true, HelpMessage = "The output type to generate. Must be either 'html' or 'pdf'.")]
     [string]$type,
-    [Parameter(ValueFromRemainingArguments = $true)]
+    [Parameter(Mandatory = $true, HelpMessage = "The input Markdown file to process.")]
+    [string]$InputFile = (Join-Path $PSScriptRoot "system-design-implementation.md"),
+    [string]$OutputFile = (Join-Path $PSScriptRoot "system-design-implementation.$type"),
+    [Parameter(ValueFromRemainingArguments = $true, HelpMessage = "Additional arguments to pass to pandoc.
+    
+    Notable use: 
+    - passing --resource-path to specify the directory where images and other resources are located,
+    - passing --pdf-engine=xelatex to specify the PDF engine for PDF output.")]
     [string[]]$RemainingArgs
 )
 function New-PandocDiagramMetadata([string] $outputFormat) {
@@ -100,24 +108,61 @@ if ($type -eq "pdf") {
     $title_pages_args = @("--include-in-header", $title_pages_header)
 }
 
+$tex_preamble_header = Join-Path $PSScriptRoot "header-includes.tex"
+if (-not (Test-Path $tex_preamble_header)) {
+    Write-Error "header-includes.tex is missing. Make sure the thesis preamble header exists."
+    exit 1
+}
+$html_preamble_header = Join-Path $PSScriptRoot "header-includes.html"
+if (-not (Test-Path $html_preamble_header)) {
+    Write-Error "header-includes.html is missing. Make sure the thesis preamble header exists."
+    exit 1
+}
+
+$preamble_args = @()
+if ($type -eq "pdf") {
+    $preamble_args += @("--include-in-header", $tex_preamble_header)
+}
+elseif ($type -eq "html") {
+    $preamble_args += @("--include-in-header", $html_preamble_header)
+}
+
+$info_md = Join-Path $PSScriptRoot "info.md"
+if (-not (Test-Path $info_md)) {
+    Write-Error "info.md is missing. Make sure the thesis info.md file exists."
+    exit 1
+}
+
+$input_files = @($info_md, $InputFile)
 
 $metatempfile = New-TemporaryFile
 $pandocExitCode = 0
 
+$pandoc_args = $(
+    @(
+        "--lua-filter", $include_markdown_filter,
+        "--lua-filter", $diagram_filter,
+        "--lua-filter", $figure_format_filter,
+        "--filter=pandoc-crossref",
+        "--pdf-engine=xelatex"
+    ) + 
+    $title_pages_args + 
+    $preamble_args + 
+    @(
+        "--metadata", "thesisFigureFormat=$outputFormat",
+        "--metadata", "thesisAgentResults=$agent_results",
+        "--metadata-file", $metatempfile,
+        "--embed-resources", "--standalone", "--citeproc"
+    ) +
+    $input_files + 
+    @("--output", $OutputFile) + 
+    $RemainingArgs
+).Where({ [string]::IsNullOrWhiteSpace($_) -eq $false })
+
 try {
     Set-Content -Path $metatempfile -Value $metadata
-    pandoc `
+    Write-Host "Running pandoc for file $($InputFile | Split-Path -Leaf) to generate $($OutputFile | Split-Path -Leaf)..."
-        --lua-filter $include_markdown_filter `
+    pandoc @pandoc_args
-        --lua-filter $diagram_filter `
-        --lua-filter $figure_format_filter `
-        --filter=pandoc-crossref `
-        --pdf-engine=xelatex `
-        @title_pages_args `
-        --metadata thesisFigureFormat=$outputFormat `
-        --metadata thesisAgentResults=$agent_results `
-        --metadata-file=$metatempfile `
-        --embed-resources --standalone --citeproc `
-        @RemainingArgs
     $pandocExitCode = $LASTEXITCODE
 }
 finally {

docs/thesis/gengen.ps1

@@ -1,6 +1,6 @@
 # Generate both HTML and PDF outputs for one Markdown file.
 param(
-    [string]$file = $(throw "File is required."),
+    [string]$file = (Join-Path $PSScriptRoot "system-design-implementation.md"),
     [Parameter(ValueFromRemainingArguments = $true)]
     [string[]]$RemainingArgs = @()
 )
@@ -37,6 +37,6 @@ function Get-OutputFilenames([string] $file, [string] $type) {
 Write-Host "Generating HTML and PDF for $file..."
 Write-Debug "Remaining args: $($RemainingArgs | Format-List)"
 
-& $PSScriptRoot\generate.ps1 -type html -- -i $file -o (Get-OutputFilenames $file "html") --resource-path $resourcePath @RemainingArgs
+& $PSScriptRoot\generate.ps1 -type html -i $file -o (Get-OutputFilenames $file "html") -- --resource-path $resourcePath @RemainingArgs
 
-& $PSScriptRoot\generate.ps1 -type pdf -- -i $file -o (Get-OutputFilenames $file "pdf") --resource-path $resourcePath @RemainingArgs
+& $PSScriptRoot\generate.ps1 -type pdf -i $file -o (Get-OutputFilenames $file "pdf") -- --resource-path $resourcePath @RemainingArgs

docs/thesis/header-includes.html

@@ -0,0 +1,6 @@
+<style>
+    code {
+        white-space: pre-wrap;
+        word-break: break-word;
+    }
+</style>

docs/thesis/header-includes.tex

@@ -0,0 +1,29 @@
+\usepackage{graphicx}
+% \usepackage{booktabs} % exists in the template
+\usepackage{tabulary}
+\usepackage{hyperref}
+\usepackage{hyperxmp}
+\usepackage[dvipsnames]{xcolor}
+\usepackage{fancyhdr}
+\usepackage{float}
+\pagestyle{fancy}
+\usepackage{seqsplit}
+
+% Pandoc emits inline code as \texttt{...}. This blunt wrapper keeps long
+% paths and commands from overflowing PDF table cells.
+\let\origtexttt\texttt
+\renewcommand{\texttt}[1]{{\origtexttt{\seqsplit{#1}}}}
+
+\usepackage{fvextra}
+\fvset{breaklines=true, breaknonspaceingroup=true, breakanywhere=true}
+\fancyhead[L]{\small lda.chat}
+\fancyhead[R]{\small\leftmark}
+\fancyfoot[C]{\thepage}
+\setlength{\parskip}{0.6em}
+\setlength{\parindent}{0pt}
+\setkeys{Gin}{width=\linewidth,height=0.55\textheight,keepaspectratio}
+\renewcommand{\arraystretch}{1.3}
+
+% \hypersetup{pdfauthor={lda.chat}, pdftitle={Design and Implementation of lda.chat}}
+\usepackage{newfloat}
+\DeclareFloatingEnvironment{none}

docs/thesis/info.md

@@ -0,0 +1,15 @@
+---
+# this file supplies info, and it is prepend(!) into the other file.
+author: "Lương Đức Anh"
+student-id: "23BI14021"
+---
+
+# Acknowledgements {.unnumbered}
+
+I would like to express my sincere gratitude to **Eng. Trần Văn Trường** at **Dicom Technology Co. Ltd.** for the trust, autonomy, and practical space to pursue this project and develop its technical direction independently. His thoughtful input, critical perspective, and support throughout the development process helped shape the project into a concrete and technically grounded engineering project.
+
+I am also grateful to **Dr. Nghiêm Thị Phương** at the **University of Science and Technology of Hanoi** for coordinating the university--company requirements of the thesis, providing academic guidance on the submission process, and helping ensure that the final report met the university's formal academic requirements.
+
+I would also like to thank the **University of Science and Technology of Hanoi**, the **Department of Information and Communication Technology**, and **Dicom Technology Co. Ltd.** for providing the academic and professional environment in which this thesis could be carried out.
+
+Finally, I am deeply thankful to my family and friends for their encouragement, patience, and support during the development and writing of this thesis.

docs/thesis/references.bib

@@ -1,3 +1,27 @@
+@online{n8n-merge-2026,
+  title   = {Merge},
+  author  = {{n8n}},
+  year    = {2026},
+  url     = {https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.merge/},
+  urldate = {2026-09-07}
+}
+
+@online{zapier-paths-2026,
+  title   = {Add branching logic to Zap workflows with Paths},
+  author  = {{Zapier}},
+  year    = {2026},
+  url     = {https://help.zapier.com/hc/en-us/articles/8496288555917-Add-branching-logic-to-Zap-workflows-with-Paths},
+  urldate = {2026-09-07}
+}
+
+@online{langgraph-graph-api-2026,
+  title   = {Graph API overview},
+  author  = {{LangChain}},
+  year    = {2026},
+  url     = {https://docs.langchain.com/oss/python/langgraph/graph-api},
+  urldate = {2026-09-07}
+}
+
 @online{mcp-tools-2025,
   title   = {Tools},
   author  = {{Model Context Protocol}},

tests/docs/test_big_doc_links.py

@@ -25,9 +25,7 @@ def test_big_doc_links_case_study_and_embeds_evidence_index() -> None:
     assert any(link.startswith("../../examples/report_workflow") for link in links)
     assert re.search(r"^# Evidence Index$", doc, flags=re.MULTILINE)
     assert re.search(r"^## Core Workflow Lifecycle$", doc, flags=re.MULTILINE)
-    assert re.search(
+    assert re.search(r"^## Current Evaluation Boundaries$", doc, flags=re.MULTILINE)
-        r"^## Agent Challenge Evaluation Protocol$", doc, flags=re.MULTILINE
-    )
 
 
 def test_project_map_links_big_doc() -> None:
@@ -60,22 +58,19 @@ def test_big_doc_keeps_mcp_as_source_family() -> None:
     assert "product identity" in doc
 
 
-def test_thesis_has_no_placeholder_author_and_keeps_appendix_evidence_together() -> (
+def test_thesis_retires_campaign_without_removing_historical_evidence() -> None:
-    None
-):
     doc = (ROOT / "docs" / "thesis" / "system-design-implementation.md").read_text(
         encoding="utf-8"
     )
-    appendix = doc.split("# Agent Challenge Harness", maxsplit=1)[1]
-    figure = appendix.index("#fig:agent-challenge-audit")
-    evidence = appendix.index("Evidence:")
-    page_break = appendix.index("\\clearpage")
-
     assert 'author: "draft"' not in doc
     assert "This draft includes" not in doc
-    assert page_break < evidence < figure
+    assert "# Agent Challenge Harness" not in doc
-    assert figure > evidence
+    assert "36-trial" not in doc
-    assert 'latex-placement="H"' in appendix[figure : figure + 300]
+    assert "36 audited trials" not in doc
+    archive = ROOT / "docs/historical/thesis/2026-09-07-retired-agent-evaluation.md"
+    historical = archive.read_text(encoding="utf-8")
+    assert "# Agent Challenge Harness" in historical
+    assert "36" in historical
 
 
 def test_thesis_bundle_has_reproducible_agent_evaluation_assets() -> None:
@@ -94,7 +89,7 @@ def test_thesis_bundle_has_reproducible_agent_evaluation_assets() -> None:
 
     assert (thesis / "agent-challenge-cohort.json").is_file()
     assert (thesis / "agent-challenge-results.md").is_file()
-    assert "include-agent-challenge-results" in doc
+    assert "include-agent-challenge-results" not in doc
     assert "include-markdown.lua" in generate_script
     assert "figure-format.lua" in generate_script
     assert "thesisFigureFormat" in generate_script