lda-wf/docs/superpowers/specs/2026-06-23-agent-challenge-report-projections-design.md

Agent Challenge Report Projections Design

Status

Approved for implementation planning on 2026-06-23.

Purpose

Agent challenge trials currently preserve raw result JSON, prompt text, and normalized metrics, but the central V2 runner does not automatically create a bounded report for review. Existing report helpers are also not wired into the V2 run or manual-audit paths.

This slice adds one normalized report model with two generated projections:

• human-readable Markdown in the trial workspace;
• machine-readable JSON beside the raw result.

The raw trial result remains immutable evidence. Manual audit is authoritative and regenerates both projections without rewriting raw evidence.

Goals

1. Make every completed, failed, timed-out, or partially parsed trial produce a bounded report automatically.
2. Let a reviewer understand a trial without opening a large raw result file.
3. Preserve a machine-readable report suitable for later aggregation.
4. Show ordered commands and tool calls without copying full tool outputs.
5. Keep automatic outcome, policy validity, agent self-report, and manual grade distinct.
6. Regenerate both projections after manual audit.

Non-Goals

• Aggregate model scoring or statistical claims.
• Replacing the immutable raw result JSON.
• Automatically deciding whether reading an example constituted copying a ready-made solution when the tool trace is ambiguous.
• Adding the future branching/foreach challenge.
• Writing the broader operator/evaluation runbook; that follows this slice.

Artifact Layout

For a trial slug such as opencode_deepseek-v4-flash-free-trial-001:

results/
  opencode_deepseek-v4-flash-free-trial-001.json
  opencode_deepseek-v4-flash-free-trial-001.report.json

workspaces/opencode_deepseek-v4-flash-free-trial-001/
  rendered-prompt.md
  metrics.json
  final-report.md
  manual-audit.yaml              # created after human review

The existing result JSON is raw evidence and is never modified by report or audit commands. The .report.json and final-report.md files are disposable projections that can be regenerated.

Normalized Report Model

Add a versioned Pydantic TrialReport DTO. Both output formats must be derived from the same instance so their meaning cannot drift.

The model contains these sections:

1. Identity and provenance
   • report schema version;
   • challenge id;
   • model, variant, profile, and trial index;
   • repository commit and dirty flag;
   • prompt hashes;
   • raw result path and workspace path.
2. Outcome
   • automatic task outcome;
   • automatic evaluation validity;
   • duration and return code;
   • assertion or parser failures.
3. Agent self-report
   • extracted challenge_report, when present;
   • final agent answer, bounded for human rendering.
4. Command and tool brief
   • ordinal, tool, status, title;
   • shell command or read/search path when available;
   • failure marker, output character count, and output hash;
   • no full output payload;
   • bounded preview only when useful.
5. Automatic evidence
   • token, cost, step, tool-call, and failure counts;
   • observed read categories;
   • product-code escalation;
   • disallowed reads and opaque commands.
6. Discrepancies
   • machine-observable conflicts between self-report and policy evidence;
   • for example, read.product_code=false while observed reads include source, tests, docs, or implementation examples;
   • ambiguous cases remain manual-review pointers rather than automatic facts.
7. Manual audit
   • pending when no audit exists;
   • official outcome, auditor, timestamp, corrections, evidence overrides, and notes after audit.
8. Follow-up pointers
   • concise warnings such as missing report fields, parse failures, opaque commands, or required manual checks.

Markdown Projection

final-report.md uses this stable order:

# Trial Report
## Outcome
## Agent Self-Report
## Commands And Tool Calls
## Automatic Evidence
## Policy Findings
## Self-Report Discrepancies
## Manual Audit
## Follow-Up Notes

The Markdown must remain bounded. It may include complete commands because commands are audit evidence, but it must not include full tool outputs or raw JSONL. Long commands and previews receive explicit length limits and retain a hash/character count in the machine report.

Machine Projection

<trial>.report.json is TrialReport.model_dump(mode="json") written with stable indentation and key ordering. It contains the normalized command/tool brief and audit information, not full stdout/stderr.

This report is the future aggregation input. Aggregators must not need to parse Markdown or the agent-authored YAML block.

Runner Flow

The V2 runner must:

1. execute the trial and retain available stdout/stderr;
2. normalize metrics, policy, parsed text, and challenge report;
3. write the raw result JSON first;
4. build a TrialReport from the raw result with audit status pending;
5. write <trial>.report.json beside the raw result;
6. write final-report.md in the workspace;
7. print or return both generated paths.

Report-generation failure must not destroy raw evidence. The runner should retain the raw result and surface a concise report-generation error to the caller.

The raw result gains explicit challenge_id, workspace_path, and result_path fields. Report code must not infer the workspace from an agent-authored workflow_file path.

Manual Audit Flow

The existing save_manual_audit.py remains the review entrypoint.

It must:

1. load the immutable raw result;
2. resolve the workspace from the explicit raw-result field;
3. write manual-audit.yaml in the workspace;
4. rebuild TrialReport from raw evidence plus the audit;
5. overwrite both generated projections;
6. print the audit, Markdown, and machine-report paths.

The command keeps current correction flags such as --set-read, --set-evidence, --correction, and --notes. A reviewer should be able to correct a trial in one command without opening the raw result.

Prompt Clarification

Put the following shared rule in the base challenge prompt so every challenge uses the same definition:

Files under tests/ and examples/ may contain complete or partial solutions. If you inspect them, report read.product_code: true; also report read.existing_solution: true when they provide a ready-made solution, or read.adjacent_attempts: true when they contain prior trial outputs.

Challenge-specific prompts may explain additional evidence fields but must not weaken this rule.

Error Handling

• Missing or malformed challenge reports remain visible as automatic failures or follow-up pointers.
• Missing manual audit means pending, not failure.
• Unknown tool event types remain counted in automatic evidence.
• Report generation accepts timeout and parse-error results with partial data.
• Invalid manual audit input fails without mutating either generated report.
• Projection writes use temporary files followed by replacement so reviewers do not observe half-written reports.

Testing

Add focused tests proving:

1. one TrialReport generates equivalent Markdown and JSON meaning;
2. commands are ordered and full tool output is absent;
3. the V2 runner writes raw result, machine report, and Markdown report;
4. timeout and parse-error trials still generate reports;
5. explicit workspace/result paths are used instead of self-report inference;
6. automatic discrepancies catch observable self-report conflicts;
7. ambiguous existing-solution reads remain manual pointers;
8. manual audit writes YAML and regenerates both projections;
9. failed audit validation leaves prior projections unchanged;
10. shared prompt text includes the tests/ and examples/ disclosure rule.

Follow-Up

After this slice:

1. write the generic challenge operator/evaluation runbook;
2. run repeated browser-click and report-workflow trials;
3. design a third challenge using branching and foreach to test graph reasoning, routing, iteration isolation, reducers, and recovery from validation failures.