feat: migrate agent challenges to manifests

2026-06-23 01:00:31 +07:00
parent 893872f821
commit bc851ea99c
24 changed files with 663 additions and 913 deletions
@@ -0,0 +1,457 @@
+# Agent Challenge Migration And Expansion Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Migrate browser-click to the generic data-only harness, remove challenge-local executables, and add a deterministic report-workflow challenge using the same central runner and evidence format.
+
+**Architecture:** Each challenge is a manifest, invariant task prompt, README, workspace template, and ignored result/workspace directories. Browser/report success assertions are declarative and provisional; the central generic classifier/report/audit path owns all execution. No compatibility wrappers remain because they have no production callers.
+
+**Tech Stack:** YAML manifests, Markdown prompts, generic Python challenge harness, pytest.
+
+---
+
+## File Structure
+
+- Create `examples/agent_challenges/browser_click_challenge/challenge.yaml`.
+- Move/rewrite `workspace_template/prompt.md` as
+  `browser_click_challenge/challenge-prompt.md`.
+- Modify `browser_click_challenge/README.md` for central commands/profiles.
+- Delete all browser-click `.py` wrappers and package marker.
+- Create `examples/agent_challenges/report_workflow_challenge/challenge.yaml`.
+- Create `report_workflow_challenge/challenge-prompt.md`.
+- Replace/rename `report_workflow_challenge/readme.md` as `README.md`, preserving
+  useful user-authored challenge ideas.
+- Create `report_workflow_challenge/workspace_template/.gitignore`.
+- Create `report_workflow_challenge/results/.gitignore`.
+- Create `report_workflow_challenge/workspaces/.gitignore`.
+- Rewrite `tests/examples/test_opencode_browser_click_challenge.py` around the
+  generic harness and browser manifest.
+- Create `tests/examples/test_report_workflow_challenge.py`.
+- Modify thesis/evidence docs only after both data-driven challenges are tested.
+
+### Task 1: Convert Browser Click To A Manifest
+
+**Files:**
+- Create: `examples/agent_challenges/browser_click_challenge/challenge.yaml`
+- Create: `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`
+- Modify: `tests/examples/test_opencode_browser_click_challenge.py`
+
+- [ ] **Step 1: Replace wrapper-based manifest tests**
+
+In `tests/examples/test_opencode_browser_click_challenge.py`, replace imports of
+`browser_click_challenge.challenge` with:
+
+```python
+from examples.agent_challenges.manifests import load_challenge_manifest
+from examples.agent_challenges.models import InstructionProfile
+from examples.agent_challenges.workspace import prepare_v2_trial_workspace
+
+
+ROOT = Path(__file__).resolve().parents[2]
+BROWSER_CHALLENGE = (
+    ROOT
+    / "examples"
+    / "agent_challenges"
+    / "browser_click_challenge"
+    / "challenge.yaml"
+)
+```
+
+Add:
+
+```python
+def test_browser_click_manifest_declares_task_success_contract() -> None:
+    loaded = load_challenge_manifest(BROWSER_CHALLENGE)
+
+    assert loaded.manifest.id == "browser_click"
+    assert loaded.manifest.source.id == "local.browser_click"
+    assert loaded.manifest.report.success_assertions == {
+        "before_clicked": False,
+        "after_clicked": True,
+        "run_failed": False,
+        "leftover_processes": False,
+    }
+
+
+def test_browser_click_workspace_uses_generic_profile_copy(tmp_path: Path) -> None:
+    loaded = load_challenge_manifest(BROWSER_CHALLENGE)
+    bundle = ROOT / "examples/agent_challenges/instruction_bundles/workflow_cli.yaml"
+
+    workspace = prepare_v2_trial_workspace(
+        loaded,
+        profile=InstructionProfile.SKILLS,
+        model="opencode/test",
+        index=1,
+        workspaces_dir=tmp_path,
+        instruction_bundle=bundle,
+    )
+
+    assert workspace.config_path.is_file()
+    assert (workspace.root / ".agent/skills/wf-cli/SKILL.md").is_file()
+```
+
+- [ ] **Step 2: Run and verify the manifest is missing**
+
+```powershell
+uv run pytest tests/examples/test_opencode_browser_click_challenge.py -k "manifest or generic_profile" -q
+```
+
+Expected: fail because `challenge.yaml` is missing.
+
+- [ ] **Step 3: Create browser-click manifest**
+
+```yaml
+version: 1
+id: browser_click
+prompt: challenge-prompt.md
+workspace_template: workspace_template
+source:
+  id: local.browser_click
+  root: ../../browser_click_workflow
+  module: ops
+  registry: registry
+store_root: .wf_browser_click_store
+server:
+  config: ../../browser_click_workflow/wf.config.json
+  default_port: 8772
+report:
+  required_fields:
+    - used_product_path
+    - used_helper_script
+    - workflow_file
+    - deployment_id
+    - run_id
+    - before_clicked
+    - after_clicked
+    - run_failed
+    - leftover_processes
+    - read
+    - attempts
+    - missed_requirements
+    - notes
+  success_assertions:
+    before_clicked: false
+    after_clicked: true
+    run_failed: false
+    leftover_processes: false
+```
+
+- [ ] **Step 4: Split task-specific prompt from harness policy**
+
+Create `challenge-prompt.md` by retaining only:
+
+- browser-click task requirements;
+- deterministic source location/identity;
+- allowed draft/raw authoring paths specific to this task;
+- task-specific evidence requirements;
+- task-specific YAML fields and reporting meanings.
+
+Remove generic content now owned by base/profile prompts:
+
+- product-path/helper-script policy;
+- workspace path placeholder;
+- code-read/profile policy;
+- prior-attempt contamination policy;
+- generic manual-audit explanation.
+
+The prompt must still require `before_clicked`, `after_clicked`,
+`leftover_processes`, command/run evidence, and the final YAML block.
+
+- [ ] **Step 5: Run manifest/workspace tests and commit**
+
+```powershell
+uv run pytest tests/examples/test_opencode_browser_click_challenge.py -k "manifest or generic_profile" -q
+git add examples/agent_challenges/browser_click_challenge/challenge.yaml examples/agent_challenges/browser_click_challenge/challenge-prompt.md tests/examples/test_opencode_browser_click_challenge.py
+git commit -m "feat: define browser click challenge manifest"
+```
+
+### Task 2: Remove Browser-Local Executables
+
+**Files:**
+- Delete: `examples/agent_challenges/browser_click_challenge/__init__.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/challenge.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/classification.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/opencode_io.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/reports.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/save_trial_report.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/save_manual_audit.py`
+- Delete: `examples/agent_challenges/browser_click_challenge/workspace_template/prompt.md`
+- Modify: `examples/agent_challenges/browser_click_challenge/README.md`
+- Modify: `tests/examples/test_opencode_browser_click_challenge.py`
+
+- [ ] **Step 1: Replace direct-execution wrapper tests**
+
+Delete tests that invoke challenge-local scripts. Add central command tests:
+
+```python
+def test_central_runner_accepts_browser_challenge() -> None:
+    result = subprocess.run(
+        [
+            sys.executable,
+            "examples/agent_challenges/run_trials.py",
+            "--help",
+        ],
+        cwd=ROOT,
+        text=True,
+        capture_output=True,
+        check=False,
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "--challenge" in result.stdout
+    assert "--instruction-profile" in result.stdout
+```
+
+Keep generic classification/report/audit tests in
+`test_agent_challenge_harness_v2.py`; remove duplicate wrapper identity tests.
+
+- [ ] **Step 2: Rewrite README commands**
+
+Document:
+
+```powershell
+uv run python examples/agent_challenges/run_trials.py `
+  --challenge examples/agent_challenges/browser_click_challenge/challenge.yaml `
+  --instruction-profile skills `
+  --model opencode/mimo-v2.5-free `
+  --variant high `
+  --trials 1 `
+  --attach http://127.0.0.1:4096
+```
+
+Also show `none` and `all` as separate invocations and explain the 3,600-second
+hard ceiling.
+
+- [ ] **Step 3: Delete challenge-local Python wrappers**
+
+Use `git rm` for every listed `.py` wrapper and the old copied prompt. Do not
+leave compatibility re-exports; there are no production callers.
+
+- [ ] **Step 4: Search for stale wrapper paths**
+
+```powershell
+rg -n "browser_click_challenge/(run_opencode_trials|save_trial_report|save_manual_audit)|browser_click_challenge\\.(challenge|classification|reports|opencode_io)" . -g '!docs/historical/**'
+```
+
+Expected: no live source/test/docs references.
+
+- [ ] **Step 5: Run browser and generic harness tests**
+
+```powershell
+uv run pytest tests/examples/test_opencode_browser_click_challenge.py tests/examples/test_agent_challenge_harness_v2.py -q
+```
+
+Expected: pass.
+
+- [ ] **Step 6: Commit wrapper removal**
+
+```powershell
+git add examples/agent_challenges/browser_click_challenge tests/examples/test_opencode_browser_click_challenge.py
+git commit -m "refactor: centralize browser challenge execution"
+```
+
+### Task 3: Add The Report Workflow Challenge
+
+**Files:**
+- Create: `examples/agent_challenges/report_workflow_challenge/challenge.yaml`
+- Create: `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`
+- Create/rename: `examples/agent_challenges/report_workflow_challenge/README.md`
+- Create: `examples/agent_challenges/report_workflow_challenge/workspace_template/.gitignore`
+- Create: `examples/agent_challenges/report_workflow_challenge/results/.gitignore`
+- Create: `examples/agent_challenges/report_workflow_challenge/workspaces/.gitignore`
+- Create: `tests/examples/test_report_workflow_challenge.py`
+
+- [ ] **Step 1: Preserve and normalize the user-authored challenge notes**
+
+Read the existing untracked
+`examples/agent_challenges/report_workflow_challenge/readme.md`. Preserve its
+intended complex report/email/save challenge as future challenge-design notes in
+the canonical `README.md`; do not silently delete it. The first executable
+challenge remains the deterministic three-node report pipeline so results are
+auditable.
+
+- [ ] **Step 2: Write failing report manifest tests**
+
+Create `tests/examples/test_report_workflow_challenge.py`:
+
+```python
+from __future__ import annotations
+
+from pathlib import Path
+
+from examples.agent_challenges.manifests import load_challenge_manifest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+REPORT_CHALLENGE = (
+    ROOT
+    / "examples"
+    / "agent_challenges"
+    / "report_workflow_challenge"
+    / "challenge.yaml"
+)
+
+
+def test_report_workflow_manifest_uses_report_source() -> None:
+    loaded = load_challenge_manifest(REPORT_CHALLENGE)
+
+    assert loaded.manifest.id == "report_workflow"
+    assert loaded.manifest.source.id == "local.report"
+    assert loaded.manifest.report.success_assertions == {
+        "title_matches": True,
+        "markdown_rendered": True,
+        "run_failed": False,
+    }
+    assert loaded.prompt_path.is_file()
+    assert loaded.workspace_template.is_dir()
+
+
+def test_report_challenge_prompt_requires_full_product_lifecycle() -> None:
+    loaded = load_challenge_manifest(REPORT_CHALLENGE)
+    prompt = loaded.prompt_path.read_text(encoding="utf-8")
+
+    assert "read_notes" in prompt
+    assert "extract_report" in prompt
+    assert "render_markdown_report" in prompt
+    assert "deployment" in prompt.lower()
+    assert "run_id" in prompt
+```
+
+- [ ] **Step 3: Run and verify the report manifest is missing**
+
+```powershell
+uv run pytest tests/examples/test_report_workflow_challenge.py -q
+```
+
+Expected: fail because `challenge.yaml` is missing.
+
+- [ ] **Step 4: Create report challenge manifest**
+
+```yaml
+version: 1
+id: report_workflow
+prompt: challenge-prompt.md
+workspace_template: workspace_template
+source:
+  id: local.report
+  root: ../../report_workflow
+  module: ops
+  registry: registry
+store_root: .wf_report_challenge_store
+server:
+  config: ../../report_workflow/wf.config.json
+  default_port: 8773
+report:
+  required_fields:
+    - used_product_path
+    - used_helper_script
+    - workflow_file
+    - deployment_id
+    - run_id
+    - title_matches
+    - markdown_rendered
+    - run_failed
+    - read
+    - attempts
+    - missed_requirements
+    - notes
+  success_assertions:
+    title_matches: true
+    markdown_rendered: true
+    run_failed: false
+```
+
+- [ ] **Step 5: Create invariant report challenge prompt**
+
+Require the agent to:
+
+1. discover `local.report` capabilities through public `wf` commands;
+2. author a workflow that executes
+   `read_notes -> extract_report -> render_markdown_report`;
+3. create an artifact and deployment through `wf`;
+4. run it against the supplied deterministic input;
+5. report the run id and evidence that title equals `Weekly Project Update` and
+   Markdown begins with `# Weekly Project Update`;
+6. avoid helper scripts that directly drive workflow internals;
+7. finish with the manifest-required YAML fields.
+
+Do not include profile policy or implementation-code instructions; the base and
+profile prompts own those rules.
+
+- [ ] **Step 6: Create workspace/result directories and README**
+
+Each `.gitignore` should contain:
+
+```gitignore
+*
+!.gitignore
+```
+
+README must show one central runner command per profile and explain that
+automatic success is provisional until manual audit.
+
+- [ ] **Step 7: Run report challenge tests and commit**
+
+```powershell
+uv run pytest tests/examples/test_report_workflow_challenge.py -q
+git add examples/agent_challenges/report_workflow_challenge tests/examples/test_report_workflow_challenge.py
+git commit -m "feat: add report workflow agent challenge"
+```
+
+### Task 4: Prove Both Challenges Use The Same Harness
+
+**Files:**
+- Modify: `tests/examples/test_agent_challenge_harness_v2.py`
+- Modify: `docs/add/evidence-index.md`
+- Modify: `docs/add/system-design-implementation.md`
+- Modify: `docs/current_roadmap.md`
+- Move after completion: `docs/superpowers/plans/2026-06-22-agent-challenge-migrations.md` to `docs/historical/superpowers/plans/2026-06-22-agent-challenge-migrations.md`
+
+- [ ] **Step 1: Add parameterized workspace preparation test**
+
+Parameterize browser/report manifest paths and assert both:
+
+- load through the same `load_challenge_manifest`;
+- prepare under each profile;
+- write a valid local config;
+- render profile/challenge prompts;
+- produce different challenge hashes but the same base/profile hashes.
+
+- [ ] **Step 2: Run all challenge tests**
+
+```powershell
+uv run pytest tests/examples/test_agent_challenge_harness_v2.py tests/examples/test_opencode_browser_click_challenge.py tests/examples/test_report_workflow_challenge.py tests/examples/test_agent_challenge_skill_bundle.py -q
+```
+
+Expected: pass.
+
+- [ ] **Step 3: Run static checks**
+
+```powershell
+uv run ruff check examples/agent_challenges tests/examples
+uv run ruff format --check examples/agent_challenges tests/examples
+uv run basedpyright --level error examples/agent_challenges tests/examples
+git diff --check
+```
+
+Expected: clean except accepted Windows CRLF warnings.
+
+- [ ] **Step 4: Update evidence docs without claiming results**
+
+Document that:
+
+- two data-driven challenges now exist;
+- both support `none`, `skills`, and `all` conditions;
+- normalized metrics and manual audit are implemented;
+- repeated audited model results are still pending and no aggregate success
+  claim is made.
+
+- [ ] **Step 5: Archive and commit the plan**
+
+```powershell
+New-Item -ItemType Directory -Force docs/historical/superpowers/plans | Out-Null
+Move-Item docs/superpowers/plans/2026-06-22-agent-challenge-migrations.md docs/historical/superpowers/plans/2026-06-22-agent-challenge-migrations.md
+git add tests/examples docs/add/evidence-index.md docs/add/system-design-implementation.md docs/current_roadmap.md docs/superpowers/plans/2026-06-22-agent-challenge-migrations.md docs/historical/superpowers/plans/2026-06-22-agent-challenge-migrations.md
+git commit -m "docs: record data-driven agent challenges"
+```