lda-wf/docs/historical/superpowers/plans/2026-06-23-agent-challenge-report-projections.md

Agent Challenge Report Projections Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Generate equivalent bounded Markdown and JSON reports for every V2 trial and regenerate both after audit without mutating raw evidence.

Architecture: Add strict Pydantic report DTOs, project one TrialReport to Markdown and JSON, wire projection generation after raw result persistence, and make V2 audit rebuild both. Preserve referenced V1 behavior.

Tech Stack: Python 3.14, Pydantic v2, PyYAML, pathlib, argparse, pytest, Ruff, basedpyright.

---

File Structure

• Create examples/agent_challenges/report_models.py.
• Modify examples/agent_challenges/reports.py, runner.py, run_trials.py, audit.py, and base-prompt.md.
• Create tests/examples/test_agent_challenge_reports.py.
• Modify tests/examples/test_agent_challenge_harness_v2.py.
• Update docs/current_roadmap.md and archive this plan.

Task 1: Define The Normalized Report

Files:

• Create: examples/agent_challenges/report_models.py

• Modify: examples/agent_challenges/reports.py

• Create: tests/examples/test_agent_challenge_reports.py

• Step 1: Write a failing bounded-report test

Build a _raw_result(tmp_path) fixture matching the current V2 result shape, including explicit paths, identity, challenge report, parsed text, metrics, policy, tool calls, and deliberately large stdout/output previews.

def test_trial_report_is_bounded_machine_projection(tmp_path: Path) -> None:
    payload = build_trial_report(_raw_result(tmp_path), audit=None).model_dump(
        mode="json"
    )
    assert payload["schema_version"] == 1
    assert payload["identity"]["challenge_id"] == "fixture"
    assert payload["outcome"]["task_outcome"] == "success"
    assert payload["commands_and_tools"][0]["detail"].endswith(
        "workflow.plan.json"
    )
    serialized = json.dumps(payload)
    assert "large raw stream" not in serialized
    assert "full tool output" not in serialized
    assert payload["manual_audit"]["status"] == "pending"

• Step 2: Run and verify import failure

uv run pytest tests/examples/test_agent_challenge_reports.py::test_trial_report_is_bounded_machine_projection -q

• Step 3: Implement strict DTOs

Create TrialIdentity, TrialOutcome, CommandToolBrief, TokenSummary, AutomaticEvidence, ManualAuditSummary, and TrialReport using ConfigDict(extra="forbid").

class StrictReportModel(BaseModel):
    model_config = ConfigDict(extra="forbid")


class TrialIdentity(StrictReportModel):
    challenge_id: str
    model: str
    variant: str
    instruction_profile: str
    trial_index: int
    repository_commit: str | None = None
    repository_dirty: bool | None = None
    prompt_hashes: dict[str, str] = Field(default_factory=dict)
    raw_result_path: str
    workspace_path: str


class TrialOutcome(StrictReportModel):
    task_outcome: str
    evaluation_validity: str
    duration_seconds: float
    returncode: int | None
    assertion_failures: list[str] = Field(default_factory=list)
    parse_errors: dict[str, dict[str, str]] = Field(default_factory=dict)


class CommandToolBrief(StrictReportModel):
    ordinal: int
    tool: str
    status: str
    title: str
    detail: str | None = None
    failed: bool
    output_chars: int
    output_sha256: str


class TokenSummary(StrictReportModel):
    total: int = 0
    input: int = 0
    output: int = 0
    reasoning: int = 0
    cache_read: int = 0
    cache_write: int = 0


class AutomaticEvidence(StrictReportModel):
    step_count: int = 0
    tool_call_count: int = 0
    failed_tool_call_count: int = 0
    tool_counts: dict[str, int] = Field(default_factory=dict)
    tokens: TokenSummary = Field(default_factory=TokenSummary)
    cost: float = 0.0
    unknown_event_count: int = 0
    reads_by_category: dict[str, list[str]] = Field(default_factory=dict)
    escalated_to_product_code: bool = False
    disallowed_reads: list[str] = Field(default_factory=list)
    opaque_shell_commands: list[str] = Field(default_factory=list)


class ManualAuditSummary(StrictReportModel):
    status: Literal["pending", "complete"] = "pending"
    official_outcome: str | None = None
    auditor: str | None = None
    audited_at: str | None = None
    corrections: list[str] = Field(default_factory=list)
    notes: str = ""
    read_flags: dict[str, bool] = Field(default_factory=dict)
    evidence: dict[str, Any] = Field(default_factory=dict)


class TrialReport(StrictReportModel):
    schema_version: Literal[1] = 1
    identity: TrialIdentity
    outcome: TrialOutcome
    agent_self_report: dict[str, Any] | None = None
    final_agent_answer: str | None = None
    commands_and_tools: list[CommandToolBrief] = Field(default_factory=list)
    automatic_evidence: AutomaticEvidence
    policy_findings: list[str] = Field(default_factory=list)
    self_report_discrepancies: list[str] = Field(default_factory=list)
    manual_audit: ManualAuditSummary = Field(default_factory=ManualAuditSummary)
    follow_up_notes: list[str] = Field(default_factory=list)

Identity holds challenge/model/variant/profile/index, repository provenance, prompt hashes, raw path, and workspace path. Outcome holds task outcome, validity, duration, return code, assertions, and parser errors. Evidence holds bounded metrics, read categories, disallowed reads, and opaque commands.

• Step 4: Implement the builder

def build_trial_report(
    result: dict[str, object],
    *,
    audit: dict[str, object] | None,
) -> TrialReport:
    """Project immutable raw evidence and optional audit into one report."""

Use focused helpers. Require explicit paths; bound final text to 8,000 chars and command detail to 1,000; retain only tool ordinal/name/status/title/detail, failure, output size/hash; exclude stdout/stderr/previews/metadata/full input. Flag observable self-report conflicts. Treat example reads plus existing_solution=false as a manual follow-up, not automatic guilt.

• Step 5: Verify and commit

uv run pytest tests/examples/test_agent_challenge_reports.py -q
git add examples/agent_challenges/report_models.py examples/agent_challenges/reports.py tests/examples/test_agent_challenge_reports.py
git commit -m "feat: add normalized agent trial report model"

Task 2: Render And Write Both Projections

Files:

• Modify: examples/agent_challenges/reports.py

• Modify: tests/examples/test_agent_challenge_reports.py

• Step 1: Add failing projection tests

Assert writes to workspace/final-report.md and results/trial.report.json, stable heading order, bounded commands, no raw outputs, pending audit, valid JSON, and no temporary-file residue.

• Step 2: Run and verify missing APIs

uv run pytest tests/examples/test_agent_challenge_reports.py -k projection -q

• Step 3: Implement Markdown renderer

Use exactly these headings:

# Trial Report
## Outcome
## Agent Self-Report
## Commands And Tool Calls
## Automatic Evidence
## Policy Findings
## Self-Report Discrepancies
## Manual Audit
## Follow-Up Notes

Render empty sections explicitly and commands as ordered bounded entries.

• Step 4: Implement atomic writes

@dataclass(frozen=True, slots=True)
class TrialReportPaths:
    markdown: Path
    machine: Path


def _atomic_write_text(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary = path.with_name(f".{path.name}.tmp")
    temporary.write_text(text, encoding="utf-8")
    temporary.replace(path)


def write_trial_report_projections(
    report: TrialReport,
    *,
    markdown_path: Path,
    machine_path: Path,
) -> TrialReportPaths:
    machine = json.dumps(
        report.model_dump(mode="json"), indent=2, sort_keys=True
    ) + "\n"
    markdown = render_trial_report_markdown(report).rstrip() + "\n"
    _atomic_write_text(machine_path, machine)
    _atomic_write_text(markdown_path, markdown)
    return TrialReportPaths(markdown=markdown_path, machine=machine_path)

• Step 5: Verify and commit

uv run pytest tests/examples/test_agent_challenge_reports.py -q
git add examples/agent_challenges/reports.py tests/examples/test_agent_challenge_reports.py
git commit -m "feat: write human and machine trial reports"

Task 3: Generate Reports From The Runner

Files:

• Modify: examples/agent_challenges/runner.py

• Modify: examples/agent_challenges/run_trials.py

• Modify: tests/examples/test_agent_challenge_harness_v2.py

• Step 1: Extend runner integration tests

For success and timeout, assert explicit workspace_path, result_path, and report_paths; raw, Markdown, and machine files exist; machine challenge id is correct; Markdown contains the final answer.

• Step 2: Run and verify failure

uv run pytest tests/examples/test_agent_challenge_harness_v2.py -k "runner_to_report or timeout" -q

• Step 3: Add explicit paths before the single raw write

"challenge_id": challenge.manifest.id,
"workspace_path": str(workspace.root.resolve()),
"result_path": str(result_path.resolve()),
"report_paths": {
    "markdown": str((workspace.root / "final-report.md").resolve()),
    "machine": str(result_path.with_suffix(".report.json").resolve()),
},

Include reads_by_category in policy. Write raw result once before projection generation; never rewrite it.

• Step 4: Generate pending-audit projections

Call build_trial_report(result, audit=None) and write_trial_report_projections. On failure preserve raw evidence and return a concise report_generation_error.

• Step 5: Print report paths

Add result_path and report_paths to each central-runner summary.

• Step 6: Verify and commit

uv run pytest tests/examples/test_agent_challenge_harness_v2.py -k "runner or timeout" -q
git add examples/agent_challenges/runner.py examples/agent_challenges/run_trials.py tests/examples/test_agent_challenge_harness_v2.py
git commit -m "feat: generate reports after agent trials"

Task 4: Regenerate Reports After Manual Audit

Files:

• Modify: examples/agent_challenges/audit.py

• Modify: examples/agent_challenges/save_manual_audit.py

• Modify: tests/examples/test_agent_challenge_reports.py

• Step 1: Add failing regeneration test

paths = save_v2_manual_audit(
    result_path,
    official_outcome="pass",
    auditor="reviewer",
    audited_at="2026-06-23T00:00:00Z",
    read_overrides={"existing_solution": True},
    corrections=["Agent inspected a ready-made workflow plan."],
    notes="Technical run passed; self-report corrected.",
)

Assert YAML, JSON, and Markdown contain the official grade/corrections.

• Step 2: Add invalid-audit preservation test

Use outcome maybe, expect ValueError, and assert previous projections remain byte-for-byte unchanged.

• Step 3: Run and verify missing API

uv run pytest tests/examples/test_agent_challenge_reports.py -k "manual_audit or invalid" -q

• Step 4: Implement V2 audit writer

Add:

@dataclass(frozen=True, slots=True)
class V2AuditPaths:
    audit: Path
    markdown: Path
    machine: Path


def save_v2_manual_audit(
    result_path: Path,
    *,
    official_outcome: str,
    auditor: str = "human",
    audited_at: str | None = None,
    read_overrides: dict[str, bool] | None = None,
    evidence_overrides: dict[str, object] | None = None,
    corrections: list[str] | None = None,
    notes: str = "",
) -> V2AuditPaths:
    """Write authoritative audit data and regenerate both report projections."""

Allow only pass, fail, invalid; require V2; use explicit raw paths; validate before writes; atomically write audit; rebuild both projections; never rewrite raw result.

• Step 5: Route existing CLI by harness version

For V2, interpret --manual-classification as official outcome, reject --from-report, and print JSON containing all three paths. Preserve referenced V1 behavior.

• Step 6: Verify and commit

uv run pytest tests/examples/test_agent_challenge_reports.py -q
uv run python examples/agent_challenges/save_manual_audit.py --help
git add examples/agent_challenges/audit.py examples/agent_challenges/save_manual_audit.py tests/examples/test_agent_challenge_reports.py
git commit -m "feat: regenerate trial reports after audit"

Task 5: Clarify Shared Self-Reporting Rules

Files:

• Modify: examples/agent_challenges/base-prompt.md

• Modify: tests/examples/test_agent_challenge_harness_v2.py

• Step 1: Add failing assertions

Assert the prompt mentions tests/, examples/, read.product_code: true, read.existing_solution: true, and read.adjacent_attempts: true.

• Step 2: Add approved paragraph

Files under `tests/` and `examples/` may contain complete or partial solutions.
If you inspect them, report `read.product_code: true`; also report
`read.existing_solution: true` when they provide a ready-made solution, or
`read.adjacent_attempts: true` when they contain prior trial outputs.

• Step 3: Verify and commit

uv run pytest tests/examples/test_agent_challenge_harness_v2.py -k base_prompt -q
git add examples/agent_challenges/base-prompt.md tests/examples/test_agent_challenge_harness_v2.py
git commit -m "docs: clarify challenge self-report rules"

Task 6: Final Verification And Documentation

Files:

• Modify: docs/current_roadmap.md

• Move this plan to docs/historical/superpowers/plans/.

• Step 1: Run verification

uv run pytest tests/examples/test_agent_challenge_reports.py tests/examples/test_agent_challenge_harness_v2.py tests/examples/test_agent_challenge_skill_bundle.py tests/examples/test_opencode_browser_click_challenge.py tests/examples/test_report_workflow_challenge.py -q
uv run ruff check examples/agent_challenges tests/examples/test_agent_challenge_reports.py tests/examples/test_agent_challenge_harness_v2.py
uv run ruff format --check examples/agent_challenges tests/examples/test_agent_challenge_reports.py tests/examples/test_agent_challenge_harness_v2.py
uv run basedpyright --level error examples/agent_challenges tests/examples/test_agent_challenge_reports.py tests/examples/test_agent_challenge_harness_v2.py
git diff --check

• Step 2: Smoke CLI help

uv run python examples/agent_challenges/run_trials.py --help
uv run python examples/agent_challenges/save_manual_audit.py --help

• Step 3: Update roadmap and archive

Record completion, then move this plan to docs/historical/superpowers/plans/2026-06-23-agent-challenge-report-projections.md.

• Step 4: Review boundaries

Confirm raw result is written once, machine report excludes raw outputs, Markdown headings are stable, explicit paths are used, audit regenerates both, V1 remains intact, and no runbook/branching challenge leaked into this slice.

• Step 5: Commit completion docs

git add docs/current_roadmap.md docs/superpowers/plans/2026-06-23-agent-challenge-report-projections.md docs/historical/superpowers/plans/2026-06-23-agent-challenge-report-projections.md
git commit -m "docs: record agent challenge report projections"