docs: add agent challenge evaluation runbook

docs/add/evidence-index.md

@@ -5,3 +5,7 @@ The claim-to-evidence map now lives inline in
 Evidence Index.
 
 This file remains as a stable pointer for older roadmap and project-map links.
+
+For the external-agent challenge evaluation workflow, including trial profiles,
+manual audits, and report interpretation, see
+[`../runbooks/agent-challenge-evaluation.md`](../runbooks/agent-challenge-evaluation.md).

docs/current_roadmap.md

@@ -225,6 +225,10 @@ stable.
   every V2 trial and regenerate both after audit without mutating raw evidence.
   Implementation:
   [`report projections`](historical/superpowers/plans/2026-06-23-agent-challenge-report-projections.md).
+- Completed: shared agent challenge evaluation runbook documents trial
+  execution, instruction profiles, manual audit, and the distinction between
+  evaluation validity and policy coverage:
+  [`agent challenge evaluation`](runbooks/agent-challenge-evaluation.md).
 
 ## Historical References
 

docs/project_map.md

@@ -11,6 +11,8 @@ For source provider setup examples, see
 
 For a presentation-oriented summary of the current product path and demo flow,
 see [`workflow platform presentation`](add/2026-06-workflow-platform-presentation.md).
+For running and auditing external-agent workflow challenges, see
+[`agent challenge evaluation`](runbooks/agent-challenge-evaluation.md).
 
 ## Packages
 
@@ -71,6 +73,8 @@ see [`workflow platform presentation`](add/2026-06-workflow-platform-presentatio
   `--keep-temp` to preserve the generated config/store on failure.
 - `examples/browser_click_workflow/` is a serial browser-click workflow
   example with bounded before/after snapshots and full lifecycle tests.
+- `examples/agent_challenges/` contains reusable opencode challenge harnesses
+  for evaluating whether agents can use the public workflow CLI/server path.
 
 ## Documentation
 
@@ -78,6 +82,9 @@ see [`workflow platform presentation`](add/2026-06-workflow-platform-presentatio
   — formal thesis/system-design draft.
 - [`docs/add/evidence-index.md`](add/evidence-index.md) — claim-to-evidence
   map for the thesis draft.
+- [`docs/runbooks/agent-challenge-evaluation.md`](runbooks/agent-challenge-evaluation.md)
+  — operator runbook for challenge trials, manual audits, and report
+  interpretation.
 
 ## Tests
 

docs/runbooks/agent-challenge-evaluation.md

@@ -0,0 +1,184 @@
+# Agent Challenge Evaluation Runbook
+
+This runbook is for running and auditing external-agent trials against the
+workflow product surface. It covers the shared harness in
+`examples/agent_challenges/`, not a specific challenge implementation.
+
+The goal is to measure whether an agent can use public workflow commands and
+instructions to build, deploy, and run a workflow. The harness is also a UX
+instrument: failed or contaminated trials often point to missing docs, confusing
+commands, or product gaps.
+
+## What The Harness Produces
+
+Each trial writes three kinds of evidence:
+
+- Raw result JSON in the challenge `results/` directory. This is immutable raw
+  evidence from the runner.
+- Machine report JSON beside the raw result, named `*.report.json`. This is the
+  bounded projection for analysis.
+- Human report Markdown inside the trial workspace, named `final-report.md`.
+  This is the file to read first during manual review.
+
+Manual audits add one more file:
+
+- `manual-audit.yaml` inside the trial workspace. Re-running the audit command
+  regenerates the human and machine report projections without mutating the raw
+  result JSON.
+
+## Run One Trial
+
+Run from the repository root. Use `--attach` when an opencode server is already
+running.
+
+```powershell
+uv run python examples/agent_challenges/run_trials.py `
+  --challenge examples/agent_challenges/browser_click_challenge/challenge.yaml `
+  --instruction-profile skills `
+  --model opencode/deepseek-v4-flash-free `
+  --variant high `
+  --trials 1 `
+  --attach http://127.0.0.1:4096
+```
+
+For the report workflow challenge:
+
+```powershell
+uv run python examples/agent_challenges/run_trials.py `
+  --challenge examples/agent_challenges/report_workflow_challenge/challenge.yaml `
+  --instruction-profile skills `
+  --model opencode/deepseek-v4-flash-free `
+  --variant high `
+  --trials 1 `
+  --attach http://127.0.0.1:4096
+```
+
+The runner prints summary JSON with the trial classification, result path, and
+report paths. Read the corresponding `final-report.md` before trusting the
+classification.
+
+## Instruction Profiles
+
+Use profiles to separate product usability from instruction quality:
+
+| Profile | Meaning | Typical Use |
+| --- | --- | --- |
+| `none` | Base prompt plus challenge prompt only. | Tests discoverability with almost no agent instructions. |
+| `skills` | Adds the workflow CLI skill bundle. | Tests the intended public agent instruction layer. |
+| `all` | Allows broader docs/code exploration. | Tests whether the repository contains enough information to solve the task, but results are less clean. |
+
+The same model/challenge should be run across profiles when comparing the value
+of the instruction layer.
+
+## Suggested Matrix
+
+Start small and grow only after the harness output is stable:
+
+```text
+challenge in [browser_click, report_workflow]
+profile in [none, skills, all]
+model in [deepseek-v4-flash-free, mimo-v2.5-free, nemotron-3-ultra-free]
+trials per cell = 3 to 5 while iterating, more for claims
+```
+
+Do not treat one successful run as a model-quality result. One run can be useful
+as a product UX finding, but not as an aggregate benchmark.
+
+## Manual Review Checklist
+
+Open the trial workspace `final-report.md` and compare it to the raw transcript
+when needed.
+
+Check these items:
+
+- Did the agent use the product path: `wf artifact create-from-plan` or draft
+  commands, `wf deploy save` or `wf deploy create`, and `wf run start`?
+- Did the run actually complete with the required output fields?
+- Did the agent write a helper script that directly drives `WorkflowApi` or
+  bypasses the CLI/server path?
+- Did the agent read implementation code under `src/` or tests under `tests/`?
+- Did the agent read a ready-made solution, prior store, adjacent attempt, or
+  generated workspace from another trial?
+- Did the agent self-report those reads honestly in the YAML block?
+- Did opaque shell commands hide important behavior that needs manual review?
+- Did the report include a real `run_id`, deployment id, and workflow file path?
+
+## Validity And Coverage
+
+The report separates policy validity from policy coverage.
+
+`evaluation_validity` answers whether the automatic evidence found a rule
+violation:
+
+- `clean`: no observed disallowed reads or policy violations.
+- `contaminated`: observed disallowed evidence, such as reading a ready-made
+  solution or forbidden prior result.
+- `unauditable`: reserved for missing or corrupt raw evidence.
+
+`policy_coverage` answers how much of the evidence the automatic pass could
+inspect:
+
+- `complete`: automatic policy checks could inspect the recorded tool evidence.
+- `partial`: some behavior happened through opaque shell commands. This does not
+  automatically invalidate the trial, but it requires manual review.
+
+Manual audit is authoritative for final interpretation. A technically successful
+workflow can still be invalid as evaluation evidence if the agent copied from an
+existing solution or bypassed the public product path.
+
+## Save A Manual Audit
+
+Use manual audit after reading the report and raw transcript.
+
+```powershell
+uv run python examples/agent_challenges/save_manual_audit.py `
+  --from-result examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-001.json `
+  --manual-classification invalid `
+  --auditor codex `
+  --set-read product_code=true `
+  --set-read existing_solution=true `
+  --notes "Technical workflow run succeeded, but the trial is invalid because the agent inspected an existing solution."
+```
+
+Use `--manual-classification valid` when the run satisfies the challenge and no
+disqualifying evidence is found. Use `invalid` when the workflow ran but the
+evaluation is contaminated or bypassed. Use another classification only when the
+challenge manifest or report schema explicitly defines it.
+
+## Common Invalid Patterns
+
+Mark the trial invalid or at least contaminated when any of these happen:
+
+- The agent reads a complete existing solution, such as a fixture workflow plan
+  for the same challenge.
+- The agent writes a one-off Python runner that calls `WorkflowApi` directly
+  instead of using `wf` or the JSON-RPC server path.
+- The agent solves the browser/report task outside the workflow runtime.
+- The agent uses prior trial artifacts or adjacent generated workspaces.
+- The agent reports `product_code: false` after reading `src/`, `tests/`, or
+  example implementation files needed to infer the answer.
+
+Reading docs and skills is allowed unless the chosen profile says otherwise.
+Reading implementation code is not automatically a product failure, but it must
+be reported and usually makes the trial less useful as public-surface evidence.
+
+## What To Claim
+
+Safe claims from small samples:
+
+- A specific model run did or did not complete the challenge.
+- A specific UX problem appeared, such as a confusing command name or missing
+  schema documentation.
+- The instruction layer helped or failed in a specific observed case.
+
+Avoid stronger claims until the matrix has enough audited trials:
+
+- Model A is better than Model B.
+- The system generally reduces token usage.
+- Agents can reliably author workflows without code reads.
+- The benchmark is statistically meaningful.
+
+Use the challenge evidence as product-design feedback first. Treat aggregate
+model benchmarking as a later result once trial counts and audit rules are
+stable.
+

examples/agent_challenges/browser_click_challenge/README.md

@@ -4,6 +4,9 @@ This challenge tests whether an agent can discover and use the workflow product
 path to build and run a browser-click workflow. It is evidence tooling, not
 product runtime code.
 
+For the shared operator workflow, audit commands, and report interpretation, see
+[`docs/runbooks/agent-challenge-evaluation.md`](../../../docs/runbooks/agent-challenge-evaluation.md).
+
 The deterministic workflow example is:
 
 ```text

examples/agent_challenges/report_workflow_challenge/README.md

@@ -4,6 +4,9 @@ This challenge tests whether an agent can discover and use the workflow product
 path to build and run a three-node report pipeline. It is evidence tooling, not
 product runtime code.
 
+For the shared operator workflow, audit commands, and report interpretation, see
+[`docs/runbooks/agent-challenge-evaluation.md`](../../../docs/runbooks/agent-challenge-evaluation.md).
+
 The deterministic workflow steps are: `read_notes -> extract_report ->
 render_markdown_report`.
 

tests/docs/test_docs_links.py

@@ -27,3 +27,25 @@ def test_source_provider_guide_links_python_runbook() -> None:
 
     assert "runbooks/python-source.md" in text
     assert "wf source diagnose" in text
+
+
+def test_project_map_links_agent_challenge_runbook() -> None:
+    text = _read_doc("docs/project_map.md")
+
+    assert "runbooks/agent-challenge-evaluation.md" in text
+
+
+def test_agent_challenge_readmes_link_shared_runbook() -> None:
+    browser = _read_doc("examples/agent_challenges/browser_click_challenge/README.md")
+    report = _read_doc("examples/agent_challenges/report_workflow_challenge/README.md")
+
+    assert "docs/runbooks/agent-challenge-evaluation.md" in browser
+    assert "docs/runbooks/agent-challenge-evaluation.md" in report
+
+
+def test_agent_challenge_runbook_defines_validity_and_coverage() -> None:
+    text = _read_doc("docs/runbooks/agent-challenge-evaluation.md")
+
+    assert "evaluation_validity" in text
+    assert "policy_coverage" in text
+    assert "Manual audit is authoritative" in text