docs: publish thesis evaluation bundle
This commit is contained in:
+1
-1
@@ -10,7 +10,7 @@ implementation plans are kept for context, not as active instructions.
|
||||
and verification commands.
|
||||
- [`add/2026-06-workflow-platform-presentation.md`](add/2026-06-workflow-platform-presentation.md):
|
||||
concise presentation narrative for the current product shape and demo flow.
|
||||
- [`add/thesis-outline.md`](add/thesis-outline.md): thesis/report scaffold,
|
||||
- [`thesis/thesis-outline.md`](thesis/thesis-outline.md): thesis/report scaffold,
|
||||
argument structure, evidence checklist, limitations, and future work.
|
||||
- [`current_roadmap.md`](current_roadmap.md): active next-work list after the
|
||||
core type-shape cleanup.
|
||||
|
||||
+13
-14
@@ -244,9 +244,9 @@ stable.
|
||||
- Completed thesis case-study evidence bundle: `examples/report_workflow/`
|
||||
provides a deterministic report workflow with Python source, fixture input,
|
||||
config, runbook, and tests.
|
||||
- Completed thesis system-design draft: `docs/add/system-design-implementation.md`
|
||||
- Completed thesis system-design draft: `docs/thesis/system-design-implementation.md`
|
||||
now frames the platform as a formal system design/implementation report backed
|
||||
by `docs/add/evidence-index.md` and the report-workflow case study.
|
||||
by `docs/thesis/evidence-index.md` and the report-workflow case study.
|
||||
- Completed supplemental browser-click workflow example with serial multi-node lifecycle evidence.
|
||||
- Completed: an opencode browser-click challenge harness captures external
|
||||
agent trials against the deterministic browser-click workflow example without
|
||||
@@ -308,25 +308,24 @@ stable.
|
||||
- Completed: repeated idempotent `wf draft bind input/state -> local` behavior
|
||||
is covered by regression tests.
|
||||
|
||||
Agent evaluation cohort policy and collection plan:
|
||||
Agent evaluation cohort status and policy:
|
||||
|
||||
- Treat trials collected while product code, prompts, fixtures, harness logic,
|
||||
or workspace isolation were changing as formative evaluation. Preserve them
|
||||
as qualitative evidence linking observed agent failures to product/harness
|
||||
fixes, but do not pool their timing, token, or success metrics with a frozen
|
||||
cohort.
|
||||
- The first frozen two-model matrix used product baseline `c926eb5`, two
|
||||
challenges, `none|skills|all` profiles, one run per cell, concurrency 4, and
|
||||
the same OpenCode/MCP tool environment. It produced 12/12 technical workflow
|
||||
successes and 11/12 manually valid benchmark results; one run was invalidated
|
||||
for using an existing solution.
|
||||
- Collect two additional runs per matrix cell next (`--trials 2`) to reach
|
||||
N=3 per cell / 36 total frozen-matrix trials. Review and audit that cohort
|
||||
before deciding whether to add two more runs per cell for N=5 / 60 total.
|
||||
- Completed: the primary longitudinal campaign now has N=3 per cell / 36
|
||||
manually audited trials across two challenges, two models, and
|
||||
`none|skills|all` profiles. The explicit cohort manifest, aggregate Markdown,
|
||||
and SVG/PDF figures live in `docs/thesis/`.
|
||||
- The 36 trials span repository snapshots and a base-prompt change before the
|
||||
third wave. Treat the aggregate as longitudinal product/prompt engineering
|
||||
evidence, not as a frozen model comparison or causal profile experiment.
|
||||
- Keep product code, challenge prompts, supplied skill bundle, model variants,
|
||||
timeout, concurrency, fixtures, and enabled tool set fixed during a cohort.
|
||||
Roadmap/evaluation notes may evolve, but record the product baseline and
|
||||
rendered prompt hashes with every result.
|
||||
timeout, concurrency, fixtures, and enabled tool set fixed if a future
|
||||
controlled cohort is collected. Record the product baseline and rendered
|
||||
prompt hashes with every result.
|
||||
- Keep manual audit authoritative for final pass/fail/invalid interpretation.
|
||||
Automatic policy findings remain review inputs, not bespoke exceptions or
|
||||
final benchmark outcomes.
|
||||
|
||||
+2
-2
@@ -80,9 +80,9 @@ For verified Python 3.14 dependency constraints and their removal criteria, see
|
||||
|
||||
## Documentation
|
||||
|
||||
- [`docs/add/system-design-implementation.md`](add/system-design-implementation.md)
|
||||
- [`docs/thesis/system-design-implementation.md`](thesis/system-design-implementation.md)
|
||||
— formal thesis/system-design draft.
|
||||
- [`docs/add/evidence-index.md`](add/evidence-index.md) — claim-to-evidence
|
||||
- [`docs/thesis/evidence-index.md`](thesis/evidence-index.md) — claim-to-evidence
|
||||
map for the thesis draft.
|
||||
- [`docs/runbooks/agent-challenge-evaluation.md`](runbooks/agent-challenge-evaluation.md)
|
||||
— operator runbook for challenge trials, manual audits, and report
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
*.html
|
||||
*.pdf
|
||||
*.tex
|
||||
*.typ
|
||||
!figures/*.svg
|
||||
@@ -0,0 +1,625 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"cohort_id": "agent-operability-n3-2026-06-30",
|
||||
"title": "Agent-operability longitudinal n=3 campaign",
|
||||
"selection_rule": "The latest three completed, manually audited trials per challenge, model, and instruction profile as of 2026-06-30.",
|
||||
"limitations": [
|
||||
"The three waves span repository snapshots; they are longitudinal engineering evidence, not a controlled model comparison.",
|
||||
"The base prompt changed before wave 3 to require the challenge report inline.",
|
||||
"The models were free hosted OpenCode endpoints, so service load and latency were not controlled."
|
||||
],
|
||||
"runs": [
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-034.report.json",
|
||||
"report_sha256": "337cc7db84c1c95d58e8ab85525ed2739f8fab58acc69fcd6f74f041f7b463eb",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 34,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 305.975,
|
||||
"tokens_total": 1267986,
|
||||
"audited_at": "2026-06-29T21:19:49Z",
|
||||
"audit_notes": "Valid product-path success. Draft authoring completed after repaired bindings and a deployment binding retry; no disqualifying reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-035.report.json",
|
||||
"report_sha256": "0ba1212eae4303d066577f170a40e0a5892b9945eeb80005be9873772e562348",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 35,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 255.076,
|
||||
"tokens_total": 1198504,
|
||||
"audited_at": "2026-06-29T21:19:49Z",
|
||||
"audit_notes": "Valid product-path success. Used supplied skills and trial workspace files only; no disqualifying reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-036.report.json",
|
||||
"report_sha256": "c523a041223ac83d68d4c72ec8b6d7668ee62dc3f8b84482d84858815a803dff",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 36,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 223.147,
|
||||
"tokens_total": 1041287,
|
||||
"audited_at": "2026-06-29T21:19:50Z",
|
||||
"audit_notes": "Valid product-path success. All-profile reads were docs/skills/challenge context, not existing solution or implementation code."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-027.report.json",
|
||||
"report_sha256": "b287a707f1f34abf1e3ef3110f2815b576e75f86f3c130d8913ff152d9a39a00",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 27,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "failed",
|
||||
"duration_seconds": 1103.134,
|
||||
"tokens_total": 1073828,
|
||||
"audited_at": "2026-06-30T09:39:44Z",
|
||||
"audit_notes": "Manual outcome pass with resume evidence. Automatic contamination came from broad repo/workspace reads, but sampled transcript did not show product-code, adjacent-solution, prior-store, or helper-script bypass."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-028.report.json",
|
||||
"report_sha256": "ea400f221a58fd13dc29432b5b1857eabdc730e314777d09a07e0b265621759e",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 28,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 164.709,
|
||||
"tokens_total": 564051,
|
||||
"audited_at": "2026-06-29T21:20:17Z",
|
||||
"audit_notes": "Valid product-path success. Used supplied skill docs and raw-plan path; no existing solution, product code, prior store, or adjacent attempt reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-029.report.json",
|
||||
"report_sha256": "a55e2658b71ef1dc17174a76a7a2304b4818becd7e8cef499616b8af6b8af3df",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 29,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 197.079,
|
||||
"tokens_total": 641558,
|
||||
"audited_at": "2026-06-29T21:20:18Z",
|
||||
"audit_notes": "Valid product-path success. All-profile docs/skills reads did not include existing solution or implementation code."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-024.report.json",
|
||||
"report_sha256": "5e40f75599c291808e1c378962989e749f80f4daeeeabce08777dee8c08b1173",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 24,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "failed",
|
||||
"duration_seconds": 934.753,
|
||||
"tokens_total": 326439,
|
||||
"audited_at": "2026-06-30T09:38:52Z",
|
||||
"audit_notes": "Manual outcome pass with resume evidence. Reads were limited to trial workspace fixtures and public CLI capability inspection in sampled report."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-025.report.json",
|
||||
"report_sha256": "d7c5be60ec253c42667154299127d960e2afce23914d6b0e3aac4d13932e389e",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 25,
|
||||
"repository_commit": "30ad99fca1e382d13fc1daacf9772ce5e709013f",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "failed",
|
||||
"duration_seconds": 854.959,
|
||||
"tokens_total": 56915,
|
||||
"audited_at": "2026-06-30T09:38:53Z",
|
||||
"audit_notes": "Manual outcome pass with resume evidence. Automatic contamination came from supplied skills/example-directory reads; sampled evidence did not show reading an existing solution or product implementation code."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-026.report.json",
|
||||
"report_sha256": "e08248b55ca9af2a6cf4163176cb29f22254261c1bbbe7257ea244956c53702d",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 26,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 249.318,
|
||||
"tokens_total": 1346114,
|
||||
"audited_at": "2026-06-30T11:33:50Z",
|
||||
"audit_notes": "Invalid as clean benchmark. Product path completed, but the run used existing solution/example implementation as reference."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-024.report.json",
|
||||
"report_sha256": "1fb02a19d0a3e0c7baec01e2d15df11f2016224d56e872edc910b796e224d4be",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 24,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 187.106,
|
||||
"tokens_total": 612379,
|
||||
"audited_at": "2026-06-30T11:33:49Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used none profile, workspace files only, raw-plan path, no helper script or disallowed reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-023.report.json",
|
||||
"report_sha256": "a38ae6ced8eae200b713c23a4cf6ebd07ec3cf72f1991490cad52436a8df3cf2",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 23,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 253.113,
|
||||
"tokens_total": 823612,
|
||||
"audited_at": "2026-06-30T11:33:49Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used supplied skills and workspace files only, completed through public wf commands, no helper script or disallowed reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 1,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-025.report.json",
|
||||
"report_sha256": "08ca446fca8e708c89ee426d5e174949f398953b708e4212684337258fe3d68c",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 25,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 196.772,
|
||||
"tokens_total": 690458,
|
||||
"audited_at": "2026-06-30T11:33:49Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used all profile resources without reading product code, existing solution, prior store, or adjacent attempts; raw-plan path succeeded on first attempt."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-037.report.json",
|
||||
"report_sha256": "52f429902bfd68b1b8893b5be03fa050d302968b2b6bb8906613e9ac5c6bde32",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 37,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 257.747,
|
||||
"tokens_total": 1355428,
|
||||
"audited_at": "2026-06-30T11:32:57Z",
|
||||
"audit_notes": "Clean browser-click pass. Agent used draft path, public wf commands, no helper script, no product-code or existing-solution reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-038.report.json",
|
||||
"report_sha256": "dc643e38440d6c644c4149be49b400deb021dfe2eb68f5ff605c54ef8491a26a",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 38,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 210.851,
|
||||
"tokens_total": 1009244,
|
||||
"audited_at": "2026-06-30T11:32:57Z",
|
||||
"audit_notes": "Manual outcome pass with policy note. Useful evidence that skills profile solved browser via draft path."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-039.report.json",
|
||||
"report_sha256": "ab4cda9f59c9e101d551cbd3a3399e8f3b600b31bfa9f6a7e8321dbe1947ad88",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 39,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 240.817,
|
||||
"tokens_total": 1271971,
|
||||
"audited_at": "2026-06-30T11:32:58Z",
|
||||
"audit_notes": "Clean browser-click pass. Agent used all-profile skills/workspace resources without disallowed reads."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-030.report.json",
|
||||
"report_sha256": "01e19f100066e1e69609268086924dfeed9d9a1f396e28283c0e5b571796eb8b",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 30,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "fail",
|
||||
"task_outcome": "failed",
|
||||
"duration_seconds": 198.976,
|
||||
"tokens_total": 575081,
|
||||
"audited_at": "2026-06-30T11:32:58Z",
|
||||
"audit_notes": "Manual outcome fail for challenge contract. Treat as useful UX/process-cleanup evidence; could be false-positive Chrome/process detection, but no contrary evidence was captured."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-031.report.json",
|
||||
"report_sha256": "2de73f2b87182ae822a1a1cb0065c9f4fd243248c0293b7b7a0bacc8b523bcde",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 31,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "failed",
|
||||
"duration_seconds": 373.645,
|
||||
"tokens_total": 1659079,
|
||||
"audited_at": "2026-06-30T11:32:58Z",
|
||||
"audit_notes": "Invalid benchmark sample. Useful prompt finding: final self-report must be inline; prompt was updated after this run."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-032.report.json",
|
||||
"report_sha256": "8c758fb289a1dd41b82ab181dda33bc52e7163f48002b641913666ddc0f2cf7f",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 32,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 283.632,
|
||||
"tokens_total": 1152542,
|
||||
"audited_at": "2026-06-30T11:32:59Z",
|
||||
"audit_notes": "Clean browser-click pass. Agent used raw-plan path, completed with before=false and after=true, no leftover processes."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-027.report.json",
|
||||
"report_sha256": "6e0fce88c196713a23fde3558df204ba5b8047b46c48f5e7b27249cd79d20a34",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 27,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 314.767,
|
||||
"tokens_total": 1349666,
|
||||
"audited_at": "2026-06-30T10:42:07Z",
|
||||
"audit_notes": "Clean pass for report workflow. Agent used raw-plan path, public wf commands, no helper script, no product-code or existing-solution reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-028.report.json",
|
||||
"report_sha256": "7d1db2a8b2974d2815856faffd8e4d2898eb90d663e670024e1e52eb54928852",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 28,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 244.242,
|
||||
"tokens_total": 1087828,
|
||||
"audited_at": "2026-06-30T10:42:07Z",
|
||||
"audit_notes": "Manual outcome pass. Keep a policy note: not as clean as mimo skills because the transcript read challenge directory/prompt artifacts, but it did not appear to copy a ready-made answer."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-029.report.json",
|
||||
"report_sha256": "55c484089b471462132398fcb955afab8b36ea00723cd5ada19855ac6f7d0661",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 29,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 187.826,
|
||||
"tokens_total": 749971,
|
||||
"audited_at": "2026-06-30T10:42:08Z",
|
||||
"audit_notes": "Invalid as clean benchmark. Product path completed, but the run used an existing solution/example implementation as reference."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-026.report.json",
|
||||
"report_sha256": "6423d2f6fb918e33fca2f826532ee1b34dd8a208cbb29cdb988e26fcda191a8b",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 26,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 224.583,
|
||||
"tokens_total": 757043,
|
||||
"audited_at": "2026-06-30T10:42:08Z",
|
||||
"audit_notes": "Invalid as strict none-profile benchmark. Useful UX evidence: missed source binding, then missing end edge, then succeeded."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-027.report.json",
|
||||
"report_sha256": "97c9365feeb5401e91d0a0bd92d7a75e87745c08b1dbdaadd639e2a8a777136c",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 27,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 258.463,
|
||||
"tokens_total": 1043878,
|
||||
"audited_at": "2026-06-30T10:42:08Z",
|
||||
"audit_notes": "Clean pass. Agent used supplied skills and workspace files only, completed through public wf commands, no helper script or disallowed reads observed."
|
||||
},
|
||||
{
|
||||
"wave": 2,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-028.report.json",
|
||||
"report_sha256": "5305c841ea80b697b0f72eeedc693bb2fcb87623ceabfe88100606e46b65b79e",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 28,
|
||||
"repository_commit": "04b1cf30f93f5946bfb7f0b001036319365aa237",
|
||||
"base_prompt_hash": "ae06f961d3c64d4d53da259cc0f9492290a3ccb6c6984433fb0986a4ad5643fb",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 223.048,
|
||||
"tokens_total": 899893,
|
||||
"audited_at": "2026-06-30T10:42:09Z",
|
||||
"audit_notes": "Clean pass. Agent used all profile resources without reading product code, existing solution, prior store, or adjacent attempt; raw-plan path succeeded on first attempt."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-040.report.json",
|
||||
"report_sha256": "fc0a9d519bc32bc9721b394bb2f4c3b3d9773996f89aed77a7932351040e51a2",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 40,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 261.703,
|
||||
"tokens_total": 1211918,
|
||||
"audited_at": "2026-06-30T11:43:50Z",
|
||||
"audit_notes": "Invalid for clean benchmark. Product path completed through draft authoring and run output showed before.clicked=false and after.clicked=true, but there was a disallowed repository-root read."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-041.report.json",
|
||||
"report_sha256": "e4906b96c644040cf740485043b6e816b6d08975458ecb63cb33d20fdfe41abf",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 41,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 160.444,
|
||||
"tokens_total": 658537,
|
||||
"audited_at": "2026-06-30T11:44:09Z",
|
||||
"audit_notes": "Clean browser-click pass. Agent used supplied skills and workspace files, completed through public wf CLI draft/deploy/run path, returned inline report, and evidence shows before.clicked=false and after.clicked=true with no leftover process claim."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_deepseek-v4-flash-free-trial-042.report.json",
|
||||
"report_sha256": "706c3b226ee197649a90524919c4b62f447c15fba6396dcf86ac8f6ef61a2d52",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 42,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 1663.355,
|
||||
"tokens_total": 1088332,
|
||||
"audited_at": "2026-06-30T12:06:41Z",
|
||||
"audit_notes": "Invalid for clean benchmark. Product path completed through draft authoring and run output showed before.clicked=false and after.clicked=true, but there was a repository-root read and a very long recovery path."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-033.report.json",
|
||||
"report_sha256": "ef4feb1fc5f6fe0324e4e74fd1ee19281832c11c1c1087888af4b8b858b3d54b",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 33,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 211.796,
|
||||
"tokens_total": 759769,
|
||||
"audited_at": "2026-06-30T11:45:06Z",
|
||||
"audit_notes": "Browser-click pass with caveat. Product path completed via raw-plan import, deployment, and run; evidence shows before.clicked=false and after.clicked=true. Manifest read is recorded but not treated as answer leakage."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-034.report.json",
|
||||
"report_sha256": "d8e81e671df9cc2cf421118ccdb9139d6bfee9dc307cb312ecb4b50f5a911675",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 34,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 223.292,
|
||||
"tokens_total": 742181,
|
||||
"audited_at": "2026-06-30T11:47:29Z",
|
||||
"audit_notes": "Clean browser-click pass. Agent used supplied skills and workspace files, completed through public wf CLI draft/deploy/run path, and evidence shows before.clicked=false and after.clicked=true."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/browser_click_challenge/results/opencode_mimo-v2.5-free-trial-035.report.json",
|
||||
"report_sha256": "fd52890062cc309f82204e3b022d37a8494ec0a6bde880ad49b12f24bfbcf678",
|
||||
"challenge": "browser_click",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 35,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 199.298,
|
||||
"tokens_total": 674210,
|
||||
"audited_at": "2026-06-30T11:47:45Z",
|
||||
"audit_notes": "Browser-click pass with caveat. Product path completed via raw-plan import, deployment, and run; evidence shows before.clicked=false and after.clicked=true. Challenge README read is recorded as docs/example context, not answer leakage."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-030.report.json",
|
||||
"report_sha256": "fd5f2c4f62c3997e749e233f15c70c1de48c4312ffaffd312e36666028824270",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "none",
|
||||
"trial_index": 30,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 402.334,
|
||||
"tokens_total": 2824556,
|
||||
"audited_at": "2026-06-30T12:39:21Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used workspace files only, completed through public wf CLI draft/deploy/run path, and output markdown matched the expected title."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-031.report.json",
|
||||
"report_sha256": "b44fbd5c6cbbc3031a40384fe3ae7b861a363a21f1a69326f0b55a01b8898029",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 31,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 341.685,
|
||||
"tokens_total": 1650195,
|
||||
"audited_at": "2026-06-30T12:43:17Z",
|
||||
"audit_notes": "Invalid for clean benchmark. Product path completed and output was correct, but the agent read the source example directory outside the supplied trial workspace."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_deepseek-v4-flash-free-trial-032.report.json",
|
||||
"report_sha256": "a14f4467e4173b454ce139e3bf3d09d656e90f0ffc31ca66e83a1cc4015e6e8c",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/deepseek-v4-flash-free",
|
||||
"profile": "all",
|
||||
"trial_index": 32,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 217.551,
|
||||
"tokens_total": 844578,
|
||||
"audited_at": "2026-06-30T12:40:00Z",
|
||||
"audit_notes": "Report-workflow pass with caveat. Agent used raw-plan import and completed through public wf CLI. Repo-index read is recorded but not treated as answer leakage."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-029.report.json",
|
||||
"report_sha256": "ce2a905b9b9caca531cc93201c48bd4a85ceb5454321f4dc68da231d937af4b2",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "none",
|
||||
"trial_index": 29,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "invalid",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 470.566,
|
||||
"tokens_total": 2030961,
|
||||
"audited_at": "2026-06-30T12:40:16Z",
|
||||
"audit_notes": "Invalid for clean benchmark. Product path completed, but store/repo reads make the clean evidence unreliable."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-030.report.json",
|
||||
"report_sha256": "b116ef6ba104c2ad0451e7a4412c0b70c121c8ce1e170f596189a6cd66710e47",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "skills",
|
||||
"trial_index": 30,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 160.917,
|
||||
"tokens_total": 521020,
|
||||
"audited_at": "2026-06-30T12:40:35Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used supplied skills and workspace files, completed through raw-plan import/deploy/run, and output markdown matched the expected title."
|
||||
},
|
||||
{
|
||||
"wave": 3,
|
||||
"report": "examples/agent_challenges/report_workflow_challenge/results/opencode_mimo-v2.5-free-trial-031.report.json",
|
||||
"report_sha256": "de86f4fec7577f2bfe3d99db8b8b82076d055ee66b13cb4ae8a419b6478c4001",
|
||||
"challenge": "report_workflow",
|
||||
"model": "opencode/mimo-v2.5-free",
|
||||
"profile": "all",
|
||||
"trial_index": 31,
|
||||
"repository_commit": "5e9b76eb946d94838f6d933e0fe99c8d6030072f",
|
||||
"base_prompt_hash": "b94659167706e5e5edc13429e1dd6c4f59a4f226bd9f926a4a4757c91becf160",
|
||||
"manual_outcome": "pass",
|
||||
"task_outcome": "success",
|
||||
"duration_seconds": 144.907,
|
||||
"tokens_total": 459046,
|
||||
"audited_at": "2026-06-30T12:40:53Z",
|
||||
"audit_notes": "Clean report-workflow pass. Agent used supplied skills/docs plus workspace files, completed through raw-plan import/deploy/run, and output markdown matched the expected title."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
## Audited Agent Challenge Campaign
|
||||
|
||||
The primary campaign contains 36 audited trials: 27 passes, 8 invalid samples, and 1 failure.
|
||||
|
||||
The campaign crosses two challenges, two hosted models, three instruction profiles (`none`, `skills`, and `all`), and three repetitions per cell. The checked cohort snapshot records report hashes, prompt hashes, the repository commit, automatic metrics, and manual-audit outcomes; local raw report files are verified against those hashes when present.
|
||||
|
||||
Because repository snapshots and one prompt rule changed between waves, this is longitudinal engineering evidence, not a controlled model comparison.
|
||||
|
||||
Selection rule: The latest three completed, manually audited trials per challenge, model, and instruction profile as of 2026-06-30.
|
||||
|
||||
| Challenge / model / profile | Pass | Invalid | Fail |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| browser / deepseek / none | 2 | 1 | 0 |
|
||||
| browser / deepseek / skills | 3 | 0 | 0 |
|
||||
| browser / deepseek / all | 2 | 1 | 0 |
|
||||
| browser / mimo / none | 2 | 0 | 1 |
|
||||
| browser / mimo / skills | 2 | 1 | 0 |
|
||||
| browser / mimo / all | 3 | 0 | 0 |
|
||||
| report / deepseek / none | 3 | 0 | 0 |
|
||||
| report / deepseek / skills | 2 | 1 | 0 |
|
||||
| report / deepseek / all | 1 | 2 | 0 |
|
||||
| report / mimo / none | 1 | 2 | 0 |
|
||||
| report / mimo / skills | 3 | 0 | 0 |
|
||||
| report / mimo / all | 3 | 0 | 0 |
|
||||
|
||||
: Audited outcomes by challenge, model, and instruction profile. {#tbl:agent-challenge-outcomes}
|
||||
|
||||
A manual `pass` requires both successful product-path evidence and an acceptable audit trail. `Invalid` means the sample cannot support the clean benchmark claim, commonly because the agent read repository or example material outside its supplied workspace. `Fail` means the challenge contract itself was not established.
|
||||
|
||||
{#fig:agent-challenge-audited-outcomes-by-cell width=95%}
|
||||
|
||||
[@fig:agent-challenge-audited-outcomes-by-cell] reports all three repetitions rather than hiding invalid samples. The profile labels are descriptive; this campaign does not isolate instruction-profile effects.
|
||||
|
||||
{#fig:agent-challenge-automatic-vs-manual-outcomes width=75%}
|
||||
|
||||
[@fig:agent-challenge-automatic-vs-manual-outcomes] shows why the manual layer matters. Seven automatically successful trials were invalid as clean evidence, while three automatically failed reports were accepted after their saved run evidence and report artifacts were manually audited.
|
||||
|
||||
{#fig:agent-challenge-longitudinal-outcomes width=75%}
|
||||
|
||||
The waves in [@fig:agent-challenge-longitudinal-outcomes] are not an improvement curve: product commits, prompt wording, and enforcement changed. They preserve the chronology needed to study those changes.
|
||||
|
||||
{#fig:agent-challenge-duration-and-tokens width=95%}
|
||||
|
||||
[@fig:agent-challenge-duration-and-tokens] separates each challenge and metric into its own panel. Circle and square markers redundantly identify the models without relying on color. Wall-clock duration includes hosted-service latency, and OpenCode token totals include cache-read accounting, so neither axis is a normalized model-efficiency metric.
|
||||
|
||||
### Campaign Limitations
|
||||
|
||||
- The three waves span repository snapshots; they are longitudinal engineering evidence, not a controlled model comparison.
|
||||
- The base prompt changed before wave 3 to require the challenge report inline.
|
||||
- The models were free hosted OpenCode endpoints, so service load and latency were not controlled.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 275 KiB |
@@ -9,3 +9,6 @@ This file remains as a stable pointer for older roadmap and project-map links.
|
||||
For the external-agent challenge evaluation workflow, including trial profiles,
|
||||
manual audits, and report interpretation, see
|
||||
[`../runbooks/agent-challenge-evaluation.md`](../runbooks/agent-challenge-evaluation.md).
|
||||
The primary checked cohort and generated aggregate are
|
||||
[`agent-challenge-cohort.json`](agent-challenge-cohort.json) and
|
||||
[`agent-challenge-results.md`](agent-challenge-results.md).
|
||||
@@ -0,0 +1,23 @@
|
||||
-- Keep one Markdown image reference while selecting print-native PDFs for XeLaTeX.
|
||||
local thesis_figure_format = "svg"
|
||||
|
||||
local function read_metadata(meta)
|
||||
if meta.thesisFigureFormat then
|
||||
thesis_figure_format = pandoc.utils.stringify(meta.thesisFigureFormat)
|
||||
end
|
||||
return meta
|
||||
end
|
||||
|
||||
local function select_figure_format(image)
|
||||
if thesis_figure_format == "pdf"
|
||||
and image.src:match("^figures/.*%.svg$") then
|
||||
image.src = image.src:gsub("%.svg$", ".pdf")
|
||||
end
|
||||
return image
|
||||
end
|
||||
|
||||
-- Separate passes guarantee metadata is available before image traversal.
|
||||
return {
|
||||
{ Meta = read_metadata },
|
||||
{ Image = select_figure_format },
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 77 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 40 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 112 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 40 KiB |
@@ -49,19 +49,58 @@ if (-not (Test-Path $diagram_filter)) {
|
||||
exit 1
|
||||
}
|
||||
|
||||
$figure_format_filter = Join-Path $PSScriptRoot "figure-format.lua"
|
||||
if (-not (Test-Path $figure_format_filter)) {
|
||||
Write-Error "figure-format.lua filter not found at $figure_format_filter. Make sure it exists, then rerun this command."
|
||||
exit 1
|
||||
}
|
||||
|
||||
$include_markdown_filter = Join-Path $PSScriptRoot "include-markdown.lua"
|
||||
if (-not (Test-Path $include_markdown_filter)) {
|
||||
Write-Error "include-markdown.lua filter not found at $include_markdown_filter. Make sure it exists, then rerun this command."
|
||||
exit 1
|
||||
}
|
||||
|
||||
$agent_results = Join-Path $PSScriptRoot "agent-challenge-results.md"
|
||||
if (-not (Test-Path $agent_results)) {
|
||||
Write-Error "agent-challenge-results.md is missing. Run generate_agent_challenge_evaluation.py first."
|
||||
exit 1
|
||||
}
|
||||
|
||||
$title_pages_header = Join-Path $PSScriptRoot "title-pages.tex"
|
||||
if (-not (Test-Path $title_pages_header)) {
|
||||
Write-Error "title-pages.tex is missing. Make sure the thesis front matter header exists."
|
||||
exit 1
|
||||
}
|
||||
$title_pages_args = @()
|
||||
if ($type -eq "pdf") {
|
||||
$title_pages_args = @("--include-in-header", $title_pages_header)
|
||||
}
|
||||
|
||||
|
||||
$metatempfile = New-TemporaryFile
|
||||
$pandocExitCode = 0
|
||||
|
||||
try {
|
||||
Set-Content -Path $metatempfile -Value $metadata
|
||||
pandoc `
|
||||
--lua-filter $include_markdown_filter `
|
||||
--lua-filter $diagram_filter `
|
||||
--lua-filter $figure_format_filter `
|
||||
--filter=pandoc-crossref `
|
||||
--pdf-engine=xelatex `
|
||||
@title_pages_args `
|
||||
--metadata thesisFigureFormat=$outputFormat `
|
||||
--metadata thesisAgentResults=$agent_results `
|
||||
--metadata-file=$metatempfile `
|
||||
--embed-resources --standalone --citeproc `
|
||||
@RemainingArgs
|
||||
$pandocExitCode = $LASTEXITCODE
|
||||
}
|
||||
finally {
|
||||
Remove-Item $metatempfile
|
||||
}
|
||||
|
||||
if ($pandocExitCode -ne 0) {
|
||||
throw "pandoc failed with exit code $pandocExitCode"
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
THESIS_DIR = Path(__file__).resolve().parent
|
||||
ROOT = THESIS_DIR.parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from examples.agent_challenges.evaluation import ( # noqa: E402
|
||||
load_evaluation_cohort,
|
||||
render_evaluation_figures,
|
||||
render_evaluation_markdown,
|
||||
)
|
||||
|
||||
|
||||
def _write_text_atomic(path: Path, content: str) -> None:
|
||||
"""Replace generated Markdown without exposing a partially written file."""
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(content, encoding="utf-8", newline="\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def generate(*, manifest_path: Path, output_dir: Path) -> tuple[Path, ...]:
|
||||
"""Generate the audited Markdown rollup plus SVG/PDF figure pairs."""
|
||||
cohort = load_evaluation_cohort(manifest_path, repository_root=ROOT)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
markdown_path = output_dir / "agent-challenge-results.md"
|
||||
_write_text_atomic(markdown_path, render_evaluation_markdown(cohort))
|
||||
figure_paths = render_evaluation_figures(cohort, output_dir / "figures")
|
||||
return (markdown_path, *figure_paths)
|
||||
|
||||
|
||||
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Regenerate thesis agent-challenge results and figures."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--manifest",
|
||||
type=Path,
|
||||
default=THESIS_DIR / "agent-challenge-cohort.json",
|
||||
help="Explicit audited cohort manifest.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=Path,
|
||||
default=THESIS_DIR,
|
||||
help="Directory receiving Markdown and figures/ outputs.",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(argv)
|
||||
for path in generate(
|
||||
manifest_path=args.manifest.resolve(), output_dir=args.output_dir.resolve()
|
||||
):
|
||||
print(path.relative_to(ROOT))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -5,6 +5,7 @@ param(
|
||||
[string[]]$RemainingArgs = @()
|
||||
)
|
||||
$DebugPreference = "Continue"
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
# file exists?
|
||||
if (-not (Test-Path $file)) {
|
||||
@@ -12,6 +13,15 @@ if (-not (Test-Path $file)) {
|
||||
exit 1
|
||||
}
|
||||
|
||||
$file = (Resolve-Path $file).Path
|
||||
$resourcePath = [System.IO.Path]::GetDirectoryName($file)
|
||||
|
||||
$evaluationGenerator = Join-Path $PSScriptRoot "generate_agent_challenge_evaluation.py"
|
||||
& uv run python $evaluationGenerator
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "agent challenge evaluation generation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
|
||||
# name without extension
|
||||
function Get-OutputFilenames([string] $file, [string] $type) {
|
||||
$parentdir = [System.IO.Path]::GetDirectoryName($file)
|
||||
@@ -24,6 +34,6 @@ function Get-OutputFilenames([string] $file, [string] $type) {
|
||||
Write-Host "Generating HTML and PDF for $file..."
|
||||
Write-Debug "Remaining args: $($RemainingArgs | Format-List)"
|
||||
|
||||
& $PSScriptRoot\generate.ps1 -type html -- -i $file -o (Get-OutputFilenames $file "html") @RemainingArgs
|
||||
& $PSScriptRoot\generate.ps1 -type html -- -i $file -o (Get-OutputFilenames $file "html") --resource-path $resourcePath @RemainingArgs
|
||||
|
||||
& $PSScriptRoot\generate.ps1 -type pdf -- -i $file -o (Get-OutputFilenames $file "pdf") @RemainingArgs
|
||||
& $PSScriptRoot\generate.ps1 -type pdf -- -i $file -o (Get-OutputFilenames $file "pdf") --resource-path $resourcePath @RemainingArgs
|
||||
@@ -0,0 +1,32 @@
|
||||
-- Replace explicit include markers with parsed Markdown before cross-reference processing.
|
||||
local agent_results_path = nil
|
||||
|
||||
local function read_metadata(meta)
|
||||
if meta.thesisAgentResults then
|
||||
agent_results_path = pandoc.utils.stringify(meta.thesisAgentResults)
|
||||
end
|
||||
return meta
|
||||
end
|
||||
|
||||
local function include_results(div)
|
||||
if div.identifier ~= "include-agent-challenge-results" then
|
||||
return nil
|
||||
end
|
||||
if not agent_results_path or agent_results_path == "" then
|
||||
error("thesisAgentResults metadata is required for the evaluation include")
|
||||
end
|
||||
|
||||
local file, open_error = io.open(agent_results_path, "r")
|
||||
if not file then
|
||||
error("cannot open agent challenge results: " .. tostring(open_error))
|
||||
end
|
||||
local content = file:read("*a")
|
||||
file:close()
|
||||
return pandoc.read(content, "markdown").blocks
|
||||
end
|
||||
|
||||
-- Separate passes guarantee metadata is available before block traversal.
|
||||
return {
|
||||
{ Meta = read_metadata },
|
||||
{ Div = include_results },
|
||||
}
|
||||
+265
-81
@@ -1,14 +1,16 @@
|
||||
---
|
||||
title: "Design and Implementation of lda.chat"
|
||||
subtitle: "Infrastructure for AI Agents to Author and Execute Workspace Workflows"
|
||||
author: "draft"
|
||||
date: "2026-06-14"
|
||||
author: "lda"
|
||||
date: "2026-06-30"
|
||||
lang: "en-US"
|
||||
documentclass: report
|
||||
papersize: a4
|
||||
fontsize: 10pt
|
||||
toc: true
|
||||
toc-depth: 2
|
||||
lof: true
|
||||
lot: true
|
||||
numbersections: true
|
||||
bibliography: references.bib
|
||||
link-citations: true
|
||||
@@ -76,6 +78,62 @@ diagram:
|
||||
theme: neutral
|
||||
---
|
||||
|
||||
# Acknowledgements {.unnumbered}
|
||||
|
||||
I thank **[External Supervisor Name]** at **[Company Name]** for providing the
|
||||
trust, autonomy, and practical space to pursue this project and to develop its
|
||||
technical direction independently.
|
||||
|
||||
I also thank **[Internal Supervisor Name]** at **[University Name]** for academic
|
||||
guidance, critical feedback, and support in shaping the work into a formal
|
||||
thesis.
|
||||
|
||||
# List of Abbreviations {.unnumbered}
|
||||
|
||||
| Abbreviation | Meaning |
|
||||
| --- | --- |
|
||||
| API | Application Programming Interface |
|
||||
| CLI | Command-Line Interface |
|
||||
| DAG | Directed Acyclic Graph |
|
||||
| JSON-RPC | JavaScript Object Notation Remote Procedure Call |
|
||||
| LLM | Large Language Model |
|
||||
| MCP | Model Context Protocol |
|
||||
| RPC | Remote Procedure Call |
|
||||
| USTH | University of Science and Technology of Hanoi |
|
||||
|
||||
: Abbreviations used in the thesis. {#tbl:abbreviations}
|
||||
|
||||
# Abstract {.unnumbered}
|
||||
|
||||
External large language model agents can assemble sequences of tool calls, but
|
||||
reusable workspace automation also requires lifecycle state, validation,
|
||||
deployment binding, persistence, and inspectable execution. This thesis presents
|
||||
the design and implementation of `lda.chat`, a prototype workflow substrate that
|
||||
separates agent planning from typed runtime execution. The system represents
|
||||
workflows as outcome-routed graphs and manages them through a
|
||||
Draft--Artifact--Deployment--Run lifecycle. A neutral source-provider boundary
|
||||
projects built-in, Model Context Protocol, and Python capabilities into the same
|
||||
workflow surface, while structured diagnostics and repair guidance support
|
||||
agent-operated authoring through CLI and JSON-RPC interfaces.
|
||||
|
||||
The implementation is evaluated through automated conformance tests, a
|
||||
deterministic three-node report workflow, a browser-interaction workflow, and a
|
||||
manually audited external-agent campaign. The campaign contains 36 trials across
|
||||
two challenges, two hosted models, three instruction profiles, and three
|
||||
longitudinal waves. Manual audit classified 27 trials as passes, eight as invalid
|
||||
evaluation samples, and one as a failure. The disagreement between automatic
|
||||
completion and manual outcomes demonstrates why successful execution alone is
|
||||
insufficient evidence when agents can inspect implementation files, prior
|
||||
artifacts, or evaluator state.
|
||||
|
||||
The contribution is architectural rather than algorithmic: a typed workflow
|
||||
lifecycle, a provider-neutral capability boundary, and an agent-operable
|
||||
validation and inspection surface implemented as a working prototype. The study
|
||||
does not establish production security, broad model generalization, or reduced
|
||||
token use; the agent campaign records evolving product and prompt snapshots and
|
||||
is therefore longitudinal engineering evidence rather than a controlled model
|
||||
comparison.
|
||||
|
||||
# Introduction
|
||||
|
||||
This report assumes a setting in which external LLM agents are used as workflow
|
||||
@@ -99,12 +157,33 @@ providers expose capabilities; stores preserve persisted lifecycle records. The
|
||||
implementation demonstrates this model across controlled built-in, MCP, and
|
||||
Python source examples.
|
||||
|
||||
**Scope of claims.** This report does not claim production security, broad
|
||||
external-agent evaluation, arbitrary mid-node crash recovery, scheduling,
|
||||
role-based access control, general workflow parallelism, or a bundled autonomous
|
||||
agent brain. Claims about planner efficiency are design hypotheses supported by
|
||||
structured diagnostics and controlled examples, not measured retry-reduction
|
||||
results.
|
||||
**Scope of claims.** This report does not claim production security, broad or
|
||||
representative external-agent evaluation, arbitrary mid-node crash recovery,
|
||||
scheduling, role-based access control, general workflow parallelism, or a
|
||||
bundled autonomous agent brain. It reports a bounded, manually audited
|
||||
36-trial agent-operability campaign. Claims about planner efficiency remain
|
||||
design hypotheses: the campaign was not a controlled retry-reduction or token
|
||||
efficiency experiment.
|
||||
|
||||
## Contributions
|
||||
|
||||
This work makes five architectural and systems-engineering contributions:
|
||||
|
||||
1. It defines a typed Draft--Artifact--Deployment--Run lifecycle for workflows
|
||||
authored and operated by external agents.
|
||||
2. It separates planner decisions from runtime execution, persisted state,
|
||||
validation, and trace collection.
|
||||
3. It defines a provider-neutral capability boundary through which built-in,
|
||||
MCP, and Python sources share one workflow model without provider logic in
|
||||
the core runtime.
|
||||
4. It exposes structured validation diagnostics, repair hints, next-action
|
||||
guidance, and inspection surfaces intended for agent-operated authoring.
|
||||
5. It implements and evaluates the design through deterministic case studies,
|
||||
automated conformance tests, and a bounded manually audited agent campaign.
|
||||
|
||||
These contributions establish the feasibility and internal coherence of the
|
||||
prototype architecture. They do not claim a new workflow algorithm or empirical
|
||||
superiority over mature orchestration systems.
|
||||
|
||||
## Report Outline
|
||||
|
||||
@@ -255,6 +334,22 @@ places this complexity behind a neutral `CapabilitySource` interface.
|
||||
These sources contextualize the comparison; the implementation claims in this
|
||||
report remain grounded in repository evidence.
|
||||
|
||||
## Positioning Summary
|
||||
|
||||
The related approaches differ primarily in their center of gravity. The table
|
||||
summarizes the comparison made in this chapter without claiming feature parity
|
||||
or product superiority.
|
||||
|
||||
| Approach | Primary strength | Lifecycle and validation position | Relation to this work |
|
||||
| --- | --- | --- | --- |
|
||||
| Direct LLM tool loop | Dynamic adaptation and low authoring overhead | Durable records, validation, and replay require surrounding infrastructure | The planner remains external; reusable procedures move into a typed substrate |
|
||||
| Generated script | Simplicity, debuggability, and direct access to libraries | Versioning, deployment binding, run records, and repair diagnostics are manual additions | Scripts remain a valid baseline; the prototype targets repeated managed execution |
|
||||
| Hosted automation platform | Integration breadth, scheduling, UI, and operational maturity | Rich but platform-specific lifecycle and operational contracts | The prototype does not seek feature parity; it exposes a local typed lifecycle for agents |
|
||||
| Durable agent graph framework | Stateful, long-running agent execution and checkpointing | Persistence and human-in-the-loop execution are first-class concerns | The distinction is the explicit artifact/deployment/run model and source binding for workspace automation |
|
||||
| `lda.chat` prototype | Agent-operable lifecycle, provider-neutral sources, and structured repair surfaces | Implemented for controlled examples; production operations remain incomplete | Architectural subject of this thesis |
|
||||
|
||||
: Positioning summary for related workflow and agent-system approaches. {#tbl:positioning-summary}
|
||||
|
||||
# Conceptual Model
|
||||
|
||||
## Working Glossary
|
||||
@@ -278,6 +373,8 @@ The document uses these terms with specific meanings:
|
||||
| Deployment binding | A mapping from logical workflow source requirement to concrete source id. | `local.report=local.report`, `playwright=playwright.default` |
|
||||
| Source drift | Divergence between saved workflow requirements and the currently resolved source inventory. | missing capability or changed schema |
|
||||
|
||||
: Working glossary for the thesis terminology. {#tbl:working-glossary}
|
||||
|
||||
## Workflows as Typed Graphs
|
||||
|
||||
A workflow is an outcome-routed typed graph. It is not presented here as a
|
||||
@@ -362,6 +459,8 @@ Representative sources and source families today:
|
||||
| MCP sources | `connection` | Upstream MCP tools, resources, prompts |
|
||||
| Python sources | `python` | Trusted project-local `NodeSpec` registries |
|
||||
|
||||
: Source families and platform roles used by the prototype. {#tbl:source-families}
|
||||
|
||||
Platform sources such as `wf.std` and `wf.source` are process-provided and do
|
||||
not require deployment self-bindings. Configured sources such as MCP and Python
|
||||
remain explicit server or operator choices.
|
||||
@@ -676,6 +775,8 @@ The implementation is organized into focused packages with clear boundaries:
|
||||
| `wf_sources_python` | Trusted in-process Python source loading and `NodeSpec`-to-`NodeDef` projection |
|
||||
| `wf_cli` | CLI commands driving the JSON-RPC transport |
|
||||
|
||||
: Package responsibilities in the implementation. {#tbl:package-responsibilities}
|
||||
|
||||
(Evidence: `docs/source_architecture.md`, package boundaries in `src/`.)
|
||||
|
||||
## Workflow Core
|
||||
@@ -883,8 +984,11 @@ The provider projects MCP tools into `NodeSpec` contracts and corresponding
|
||||
core `NodeDef` contracts, making them callable from workflow graphs through the
|
||||
same `CapabilitySource` boundary as Python or built-in sources.
|
||||
|
||||
(Evidence: `src/wf_sources_mcp/`, `tests/wf_sources_mcp/test_runtime.py`,
|
||||
`tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`.)
|
||||
Evidence:
|
||||
|
||||
- `src/wf_sources_mcp/`
|
||||
- `tests/wf_sources_mcp/test_runtime.py`
|
||||
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
||||
|
||||
## Python Source Provider
|
||||
|
||||
@@ -1035,7 +1139,9 @@ The tested thesis path imports the complete three-node plan as an immutable
|
||||
artifact:
|
||||
|
||||
```powershell
|
||||
wf artifact create-from-plan workflow.plan.json --artifact report_case_study --version 1 --title "Report Case Study" --outcome ok
|
||||
wf artifact create-from-plan workflow.plan.json `
|
||||
--artifact report_case_study --version 1 `
|
||||
--title "Report Case Study" --outcome ok
|
||||
```
|
||||
|
||||
Artifact creation captures the workflow graph, required capability snapshots,
|
||||
@@ -1051,7 +1157,10 @@ three-node pipeline, records trace frames, and stores a completed run record
|
||||
with output and diagnostics. `run inspect`, `run trace`, and `run list` then
|
||||
provide the inspection surface used by both humans and agents.
|
||||
|
||||
(Evidence: `examples/report_workflow/README.md`, `tests/examples/test_report_workflow_example.py`.)
|
||||
Evidence:
|
||||
|
||||
- `examples/report_workflow/README.md`
|
||||
- `tests/examples/test_report_workflow_example.py`
|
||||
|
||||
## Expected Output
|
||||
|
||||
@@ -1102,7 +1211,7 @@ the research question. These criteria test whether the implemented substrate has
|
||||
the intended lifecycle, validation, source, and inspection behavior under
|
||||
controlled examples; they do not constitute a broad reliability or user study.
|
||||
The later Agent Instruction Layer section explains why CLI/API conformance is
|
||||
necessary but not sufficient for aggregate agent-success claims.
|
||||
necessary but not sufficient for broad agent-success claims.
|
||||
|
||||
| Criterion | Question | Evidence Type |
|
||||
| --- | ---- | --- |
|
||||
@@ -1114,6 +1223,8 @@ necessary but not sufficient for aggregate agent-success claims.
|
||||
| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests |
|
||||
| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness |
|
||||
|
||||
: Prototype conformance criteria used for evaluation. {#tbl:prototype-conformance}
|
||||
|
||||
This is a prototype system evaluation, not a broad user study or reliability
|
||||
benchmark.
|
||||
|
||||
@@ -1129,11 +1240,38 @@ benchmark.
|
||||
| Agent-operable repair hints | Not inherent | Custom | Usually human UI | Prototype support |
|
||||
| Scheduling | Depends on agent | External scheduler | Yes | Future work |
|
||||
|
||||
: Qualitative comparison against direct tool loops, scripts, and mature automation products. {#tbl:qualitative-comparison}
|
||||
|
||||
The comparison positions the architecture; it is not a quantitative claim that
|
||||
the prototype outperforms mature automation products. "Not inherent" means the
|
||||
feature can be added by surrounding infrastructure, but is not provided by the
|
||||
bare strategy alone.
|
||||
|
||||
## Formative Agent-Trial Findings
|
||||
|
||||
Before the checked 36-trial campaign, exploratory agent runs were used as
|
||||
design feedback. Prompts, product behavior, workspace isolation, and enabled
|
||||
tools changed during this period, so these runs are not pooled into the outcome,
|
||||
duration, or token statistics. They instead provide process-tracing evidence:
|
||||
recurring agent failures exposed public-surface gaps, and subsequent slices
|
||||
addressed those gaps.
|
||||
|
||||
| Formative observation | Product or harness response | Engineering interpretation |
|
||||
| --- | --- | --- |
|
||||
| Agents could build raw plans through the Python API but could not import them through the public CLI/RPC lifecycle | Added JSON-RPC and `wf artifact create-from-plan` support | A working internal API is insufficient when the agent-facing front door omits it |
|
||||
| Agents inspected source and tests to infer raw-plan and component shapes | Added the compact and verbose `wf schema` catalog and expanded workflow skills | Public schema discovery is part of the product contract |
|
||||
| Local CLI mode silently omitted configured Python sources | Routed local CLI composition through the configuration-aware server builder | Equivalent CLI targets must compose equivalent source inventories |
|
||||
| Output bindings failed when destination schemas or referenced `$defs` were absent | Added capability-aware schema projection, generalized `wf draft bind`, and workflow-output editing | Binding helpers must propagate known schemas rather than force agents to reproduce JSON Schema internals |
|
||||
| Forward routes failed while the target step had not yet been added | Preserved invalid intermediate drafts and returned direct route-repair guidance | Mutable authoring state must tolerate repairable incompleteness |
|
||||
| Draft bootstrap bound optional inputs that were absent at run time | Changed capability bootstrap to bind required inputs only and report optional inputs as notes | Best-effort synthesis should avoid inventing runtime requirements |
|
||||
| Agents misreported source reads or returned reports only as files | Added tool-evidence policy checks, explicit instruction profiles, inline-report requirements, and authoritative manual audit | Agent self-reports are evidence inputs, not final evaluation truth |
|
||||
|
||||
: Formative agent-trial observations that shaped product and harness changes. {#tbl:formative-agent-findings}
|
||||
|
||||
These findings support the design of the operation, repair, and instruction
|
||||
surfaces. They do not estimate how frequently a new agent or model will encounter
|
||||
the same failures.
|
||||
|
||||
## Evidence Package
|
||||
|
||||
The evidence supporting the thesis claims includes:
|
||||
@@ -1144,15 +1282,20 @@ The evidence supporting the thesis claims includes:
|
||||
| Interrupted runs resume at explicit boundaries | `tests/wf_api/test_run_api.py` and resume-concurrency tests | Stopped run state is persisted and resumed through the run API | Pass in focused test suite |
|
||||
| Python source lifecycle works | `tests/examples/test_report_workflow_example.py` | Python capability -> artifact -> deployment -> run completes | Pass in focused test suite |
|
||||
| Serial multi-node workflow works | `tests/examples/test_browser_click_workflow_example.py` | `open_click_page` -> `wait_for_click` -> `collect_snapshots` completes with before/after evidence | Pass in focused test suite |
|
||||
| Agent challenge harness implementation exists; no aggregate agent-performance claim | `examples/agent_challenges/browser_click_challenge/`, `examples/agent_challenges/report_workflow_challenge/`, and Appendix C | The shared harness can prompt, classify, collect tool/token evidence, and support manual audit across browser-click and report-workflow trials | Harness tests pass; aggregate model results pending |
|
||||
| Bounded agent-operability campaign is checkable | `docs/thesis/agent-challenge-cohort.json`, generated results/figures, local report hashes, and Appendix C | Two challenges, two models, three instruction profiles, and three audited repetitions per cell | 36 audited trials: 27 pass, 8 invalid, 1 fail |
|
||||
| CLI and JSON-RPC share the API surface | `tests/wf_transport_rpc_http/` and `tests/wf_cli/` | Transport and CLI delegate to the same workflow operations | Pass in focused test suite |
|
||||
|
||||
: Claim-to-evidence summary for the thesis evaluation. {#tbl:evidence-package}
|
||||
|
||||
The table summarizes repository evidence; it is not a substitute for rerunning
|
||||
the verification commands before final submission.
|
||||
|
||||
::: {#include-agent-challenge-results}
|
||||
:::
|
||||
|
||||
## Verification Snapshot
|
||||
|
||||
This draft includes one focused verification snapshot to make the evidence
|
||||
This report records one focused verification snapshot to make the evidence
|
||||
claims auditable from the text. A final submission should regenerate this table
|
||||
from the exact submitted commit.
|
||||
|
||||
@@ -1164,10 +1307,15 @@ from the exact submitted commit.
|
||||
| Environment | Local Windows development environment, Python via `uv` |
|
||||
| Scope | Documentation links, report workflow, browser-click workflow, challenge harness, deployment validation, and run API tests |
|
||||
|
||||
: Focused verification snapshot recorded during document preparation. {#tbl:verification-snapshot}
|
||||
|
||||
Command:
|
||||
|
||||
```powershell
|
||||
uv run pytest tests/docs tests/examples/test_report_workflow_example.py tests/examples/test_browser_click_workflow_example.py tests/examples/test_opencode_browser_click_challenge.py tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q
|
||||
uv run pytest tests/docs tests/examples/test_report_workflow_example.py `
|
||||
tests/examples/test_browser_click_workflow_example.py `
|
||||
tests/examples/test_opencode_browser_click_challenge.py `
|
||||
tests/artifacts/test_validation.py tests/wf_api/test_run_api.py -q
|
||||
```
|
||||
|
||||
## Implemented Scope Matrix
|
||||
@@ -1177,9 +1325,11 @@ uv run pytest tests/docs tests/examples/test_report_workflow_example.py tests/ex
|
||||
| Workflow lifecycle | Draft, artifact, deployment, run, trace, and list/inspect/resume surfaces | Exactly-once execution or arbitrary mid-node crash recovery | Transactional stores and richer run debugging |
|
||||
| Source providers | Built-in, MCP, and Python source families | Symmetric feature depth across all providers | Provider add/update/remove/reload lifecycle |
|
||||
| Execution model | Outcome-routed graph with node, condition, foreach, subgraph, join, interrupt, and end steps | General fork/gather programming model | Parallel fork/gather and aggregation |
|
||||
| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output | Measured agent success or token reduction | Broader agent challenge suite and aggregate evaluation |
|
||||
| Agent-operable surface | CLI, JSON-RPC, validation diagnostics, next-action hints, compact output, and a bounded 36-trial campaign | Broad model generalization, controlled profile effects, or token reduction | Broader challenge suite and controlled comparative evaluation |
|
||||
| Auth/security | Auth record plumbing and source diagnostics | Production security, encrypted-at-rest secrets, RBAC, sandboxing | Secret-manager integration and policy enforcement |
|
||||
|
||||
: Implemented scope, explicit non-claims, and future work. {#tbl:implemented-scope}
|
||||
|
||||
### Architecture And Code Walkthrough
|
||||
|
||||
The four-layer architecture (core, API surface, server composition, transport)
|
||||
@@ -1193,7 +1343,10 @@ Automated tests cover artifact creation, deployment validation, run execution,
|
||||
run inspection, and trace retrieval. These tests exercise the full lifecycle
|
||||
from plan to completed run.
|
||||
|
||||
(Evidence: `tests/wf_api/test_artifact_api.py`, `tests/wf_api/test_run_api.py`.)
|
||||
Evidence:
|
||||
|
||||
- `tests/wf_api/test_artifact_api.py`
|
||||
- `tests/wf_api/test_run_api.py`
|
||||
|
||||
### Validation And Diagnostics Tests
|
||||
|
||||
@@ -1202,7 +1355,10 @@ validation detects source drift, and diagnostics include repair hints. The
|
||||
validation tests demonstrate that failed states are machine-readable and include
|
||||
repair guidance.
|
||||
|
||||
(Evidence: `tests/artifacts/test_validation.py`, `tests/wf_api/test_source_admin_api.py`.)
|
||||
Evidence:
|
||||
|
||||
- `tests/artifacts/test_validation.py`
|
||||
- `tests/wf_api/test_source_admin_api.py`
|
||||
|
||||
### Source Provider Tests
|
||||
|
||||
@@ -1211,9 +1367,11 @@ inventory, stateful session reuse, and auth binding. Python source provider
|
||||
tests cover module import, `NodeSpec` projection, and capability calling. The
|
||||
tests exercise the source-provider boundary across different source families.
|
||||
|
||||
(Evidence: `tests/wf_sources_mcp/test_runtime.py`,
|
||||
`tests/wf_sources_python/test_loader.py`,
|
||||
`tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`.)
|
||||
Evidence:
|
||||
|
||||
- `tests/wf_sources_mcp/test_runtime.py`
|
||||
- `tests/wf_sources_python/test_loader.py`
|
||||
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
||||
|
||||
### Stateful MCP Session Tests
|
||||
|
||||
@@ -1222,8 +1380,10 @@ workflow calls rather than creating fresh one-off clients. This demonstrates
|
||||
source-provider correctness for providers whose behavior depends on session
|
||||
state.
|
||||
|
||||
(Evidence: `tests/wf_sources_mcp/test_runtime.py`,
|
||||
`tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`.)
|
||||
Evidence:
|
||||
|
||||
- `tests/wf_sources_mcp/test_runtime.py`
|
||||
- `tests/wf_transport_rpc_http/test_mcp_backed_server_rpc.py`
|
||||
|
||||
### Python Source Case Study
|
||||
|
||||
@@ -1234,10 +1394,12 @@ deterministic three-node report pipeline through artifact, deployment, and run
|
||||
records. The browser-click example complements this with a serial three-node
|
||||
Python workflow.
|
||||
|
||||
(Evidence: `examples/report_workflow/`,
|
||||
`examples/browser_click_workflow/`,
|
||||
`tests/examples/test_report_workflow_example.py`,
|
||||
`tests/examples/test_browser_click_workflow_example.py`.)
|
||||
Evidence:
|
||||
|
||||
- `examples/report_workflow/`
|
||||
- `examples/browser_click_workflow/`
|
||||
- `tests/examples/test_report_workflow_example.py`
|
||||
- `tests/examples/test_browser_click_workflow_example.py`
|
||||
|
||||
### CLI And Transport Tests
|
||||
|
||||
@@ -1245,7 +1407,10 @@ CLI and transport tests verify that the surface intended for external-agent
|
||||
operation is exposed through JSON-RPC. Structured output, validation commands,
|
||||
and inspect commands produce machine-readable responses.
|
||||
|
||||
(Evidence: `tests/wf_cli/`, `tests/wf_transport_rpc_http/`.)
|
||||
Evidence:
|
||||
|
||||
- `tests/wf_cli/`
|
||||
- `tests/wf_transport_rpc_http/`
|
||||
|
||||
### Config Validation
|
||||
|
||||
@@ -1273,7 +1438,7 @@ interaction, an agent might spend multiple attempts discovering a valid tool
|
||||
sequence through trial and error. With the typed lifecycle, the agent validates
|
||||
a draft, reads the diagnostic, fixes the specific issue, and proceeds. This
|
||||
report evaluates whether the diagnostic and lifecycle surfaces exist and are
|
||||
actionable; it does not yet measure retry reduction, token savings, or
|
||||
actionable; it does not measure retry reduction, token savings, or
|
||||
convergence rates across agents.
|
||||
|
||||
The tradeoff is that this lifecycle can require more authoring turns up front:
|
||||
@@ -1282,9 +1447,11 @@ artifact, bind a deployment, and validate again before the first production
|
||||
run. The intended exchange is higher authoring overhead for more deterministic,
|
||||
inspectable, and reusable runtime execution.
|
||||
|
||||
Threat to validity: no controlled agent study was conducted. Claims regarding
|
||||
agent efficiency, convergence, retry reduction, or token savings should be
|
||||
interpreted as design hypotheses rather than experimentally validated results.
|
||||
Threat to validity: the audited agent campaign tests product operability, not
|
||||
planner efficiency. It changed alongside the prototype and prompt rules and has
|
||||
no direct-tool-loop baseline. Claims regarding convergence, retry reduction, or
|
||||
token savings should therefore be interpreted as design hypotheses rather than
|
||||
experimentally validated results.
|
||||
|
||||
## Agent Instruction Layer
|
||||
|
||||
@@ -1311,17 +1478,20 @@ on reverse-engineering implementation details or reading nearby answers.
|
||||
The design implication is that agent-facing infrastructure has three layers:
|
||||
the operation surface (`wf` and JSON-RPC), the repair surface (validation
|
||||
diagnostics, traces, compact output, and next actions), and the instruction
|
||||
surface (skills and runbooks). A future aggregate evaluation should measure all
|
||||
three together.
|
||||
surface (skills and runbooks). The bounded campaign measures their combined
|
||||
operability but does not causally isolate any one layer.
|
||||
|
||||
## Falsifiability Criteria
|
||||
|
||||
The design would fail its own criteria if source providers routinely required
|
||||
changes to `wf_core`, if deployments could not detect missing or drifted source
|
||||
requirements before execution, if run records could not be inspected or resumed
|
||||
at explicit interruption boundaries, or if external agents had to import
|
||||
implementation internals rather than using the public CLI/API lifecycle for
|
||||
ordinary authoring and execution.
|
||||
The design would fail its own criteria if:
|
||||
|
||||
- source providers routinely required changes to `wf_core`;
|
||||
- deployments could not detect missing or drifted source requirements before
|
||||
execution;
|
||||
- run records could not be inspected or resumed at explicit interruption
|
||||
boundaries;
|
||||
- external agents had to import implementation internals rather than using the
|
||||
public CLI/API lifecycle for ordinary authoring and execution.
|
||||
|
||||
## Evaluation Questions
|
||||
|
||||
@@ -1435,37 +1605,42 @@ deployment concerns beyond the controlled system-design evidence in this report.
|
||||
|
||||
# Future Work
|
||||
|
||||
The following areas are identified as likely future work:
|
||||
The remaining work is prioritized by whether it strengthens the prototype's
|
||||
operational foundation or expands its feature scope.
|
||||
|
||||
## Near-Term Engineering Priorities
|
||||
|
||||
- **Provider lifecycle.** Add, update, remove, apply, and reload operations
|
||||
for multiple source families. Extend the provider protocol beyond static
|
||||
inventory loading.
|
||||
|
||||
- **OpenAPI or fetch-style source provider.** Broader HTTP integration through
|
||||
a new source family, complementing MCP and Python sources.
|
||||
|
||||
- **Python development reload.** Hot reload for Python sources during
|
||||
development, without requiring server restart.
|
||||
|
||||
- **LLM nodes as typed source capabilities.** LLM calls exposed as
|
||||
`NodeSpec` contracts, allowing planners to compose LLM steps into workflows
|
||||
without making the core runtime model-aware.
|
||||
|
||||
- **Production auth and secret stores.** Encrypted-at-rest credential storage,
|
||||
secret-manager integration, and production-grade auth flows.
|
||||
|
||||
- **SQL and transactional stores.** Replace file-backed stores with
|
||||
transactional storage for production durability.
|
||||
|
||||
- **Richer run debugging.** Time-travel debugging, run rewind, and
|
||||
mid-execution inspection beyond stopped and interrupted resume.
|
||||
|
||||
## Longer-Term Capability Expansion
|
||||
|
||||
- **OpenAPI or fetch-style source provider.** Broader HTTP integration through
|
||||
a new source family, complementing MCP and Python sources.
|
||||
|
||||
- **LLM nodes as typed source capabilities.** LLM calls exposed as
|
||||
`NodeSpec` contracts, allowing planners to compose LLM steps into workflows
|
||||
without making the core runtime model-aware.
|
||||
|
||||
- **Scheduler and daemon operations.** Offline scheduling for deployments,
|
||||
cron-triggered runs, and server daemon lifecycle.
|
||||
|
||||
- **Fork and gather workflow control.** General parallel execution and result
|
||||
aggregation within workflow graphs.
|
||||
|
||||
- **Richer run debugging.** Time-travel debugging, run rewind, and
|
||||
mid-execution inspection beyond stopped and interrupted resume.
|
||||
|
||||
- **UI and admin dashboard.** First-party workflow UI for listing, inspecting,
|
||||
and editing workflows.
|
||||
|
||||
@@ -1676,7 +1851,8 @@ Evidence:
|
||||
- `tests/wf_transport_rpc_http/`
|
||||
- `docs/wf_cli.md`
|
||||
- `examples/agent_challenges/browser_click_challenge/`: challenge harness for
|
||||
CLI-operability trials; aggregate model results require manual audit.
|
||||
CLI-operability trials.
|
||||
- `docs/thesis/agent-challenge-cohort.json`: explicit 36-trial audited cohort.
|
||||
|
||||
## Validation And Diagnostics
|
||||
|
||||
@@ -1719,9 +1895,8 @@ Evidence:
|
||||
|
||||
## Agent Challenge Evaluation Protocol
|
||||
|
||||
Claim: The project has a repeatable protocol for evaluating whether external
|
||||
agents can use the product-facing CLI lifecycle, but aggregate model results are
|
||||
not yet claimed.
|
||||
Claim: The project has a repeatable protocol and a bounded audited campaign for
|
||||
evaluating whether external agents can use the product-facing CLI lifecycle.
|
||||
|
||||
Evidence:
|
||||
|
||||
@@ -1743,10 +1918,13 @@ Evidence:
|
||||
- `tests/examples/test_report_workflow_challenge.py`
|
||||
- `tests/examples/test_agent_challenge_harness_v2.py`
|
||||
|
||||
Two data-driven challenges now exist (browser-click and report-workflow), both
|
||||
supporting `none`, `skills`, and `all` instruction profiles. Normalized metrics
|
||||
and manual audit are implemented. Repeated audited model results are still
|
||||
pending; no aggregate success claim is made.
|
||||
Two data-driven challenges exist (browser-click and report-workflow), both
|
||||
supporting `none`, `skills`, and `all` instruction profiles. The checked cohort
|
||||
contains three manually audited repetitions for each challenge/model/profile
|
||||
cell. Its explicit manifest and generated figures appear in the Evaluation
|
||||
chapter. Because repository snapshots and prompt rules changed across waves,
|
||||
the results are longitudinal engineering evidence rather than a controlled
|
||||
model leaderboard.
|
||||
|
||||
## Limitations
|
||||
|
||||
@@ -1755,7 +1933,7 @@ product.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `docs/add/thesis-outline.md`
|
||||
- `docs/thesis/thesis-outline.md`
|
||||
- `docs/current_roadmap.md`
|
||||
- Absence of scheduler, visual-editor, and secret-manager production packages
|
||||
in the current source tree.
|
||||
@@ -1765,9 +1943,9 @@ Evidence:
|
||||
The browser-click challenge harness is an evaluation instrument for the CLI
|
||||
surface intended for external-agent operation. It asks an external agent to
|
||||
build and successfully run a workflow that opens a local page with a visible
|
||||
button, records a
|
||||
before-click snapshot, performs or waits for a click, records an after-click
|
||||
snapshot, and returns both snapshots from a deployed workflow run.
|
||||
button. The workflow records a before-click snapshot, performs or waits for a
|
||||
click, records an after-click snapshot, and returns both snapshots from a
|
||||
deployed workflow run.
|
||||
|
||||
The harness deliberately evaluates the product-facing lifecycle rather than
|
||||
general Python programmability. A valid solution uses `uv run wf ...` commands
|
||||
@@ -1776,27 +1954,18 @@ for artifact creation, deployment saving, and run execution. Importing
|
||||
directly, or solving the task as a standalone browser script is treated as a
|
||||
bypass even if the visible output is correct.
|
||||
|
||||
The current browser-click prompt allows two product-facing authoring paths:
|
||||
The browser-click challenge accepts two product-facing authoring paths:
|
||||
|
||||
1. **Draft path.** Create a draft from one capability, apply focused draft edits
|
||||
or an RFC 6902 patch, validate, save, deploy, and run.
|
||||
2. **Raw-plan path.** Write a raw workflow plan and load it with
|
||||
`wf artifact create-from-plan`, then deploy and run.
|
||||
|
||||
```{.mermaid #fig:agent-challenge-audit width=50% caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."}
|
||||
flowchart TB
|
||||
Transcript[Agent transcript and files] --> YAML[YAML self-report]
|
||||
YAML --> Classifier[Automatic convenience classification]
|
||||
Transcript --> Audit[Manual audit]
|
||||
Classifier --> Audit
|
||||
Audit --> Outcome[Official outcome]
|
||||
```
|
||||
|
||||
The challenge report is a YAML self-report with fields for product-path use,
|
||||
helper-script use, workflow file, deployment id, run id, before/after booleans,
|
||||
read-behavior flags, attempt counts, missed requirements, and notes. The harness
|
||||
uses that block for automatic convenience classification, but the official
|
||||
outcome is manual-reviewed. Manual review checks the command transcript, the
|
||||
outcome is manually reviewed. Manual review checks the command transcript, the
|
||||
workflow file, the run id, the run output or trace, and whether the agent read
|
||||
product source code, adjacent attempts, prior stores, or existing solutions.
|
||||
|
||||
@@ -1808,18 +1977,33 @@ records possible invalidation flags such as helper-script bypass,
|
||||
adjacent-attempt leakage, prior-store reuse, product-code dependency, false YAML
|
||||
claims, timeouts, parse failures, and missing run evidence.
|
||||
|
||||
At the time of this report, the harness and browser-click workflow are
|
||||
implemented and unit-tested, and informal preliminary trials have informed CLI
|
||||
and prompt improvements. The report does not claim aggregate model success
|
||||
rates, timeout distributions, command counts, or retry-reduction results yet.
|
||||
Those require repeated, clean-workspace trials across the selected free opencode
|
||||
models and manual audit of saved trial reports.
|
||||
The harness, both challenge workflows, and the 36-trial checked cohort are
|
||||
implemented and manually audited. The Evaluation
|
||||
chapter reports official outcomes, automatic/manual disagreement, duration,
|
||||
and recorded token totals. It does not claim controlled model superiority,
|
||||
normalized throughput, or retry reduction because the product, prompts, and
|
||||
hosted service conditions were not held constant across waves.
|
||||
|
||||
```{=latex}
|
||||
\clearpage
|
||||
```
|
||||
|
||||
Evidence:
|
||||
|
||||
- `examples/browser_click_workflow/`
|
||||
- `examples/agent_challenges/browser_click_challenge/`
|
||||
- `examples/agent_challenges/report_workflow_challenge/`
|
||||
- `docs/thesis/agent-challenge-cohort.json`
|
||||
- `docs/thesis/agent-challenge-results.md`
|
||||
- `tests/examples/test_browser_click_workflow_example.py`
|
||||
- `tests/examples/test_opencode_browser_click_challenge.py`
|
||||
- `tests/examples/test_report_workflow_challenge.py`
|
||||
|
||||
```{.mermaid #fig:agent-challenge-audit width=50% latex-placement="H" caption="Agent challenge audit flow: automatic YAML classification is only a convenience input to manual audit, which determines the official outcome."}
|
||||
flowchart TB
|
||||
Transcript[Agent transcript and files] --> YAML[YAML self-report]
|
||||
YAML --> Classifier[Automatic convenience classification]
|
||||
Transcript --> Audit[Manual audit]
|
||||
Classifier --> Audit
|
||||
Audit --> Outcome[Official outcome]
|
||||
```
|
||||
Reference in New Issue
Block a user