lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: migrate agent challenges to manifests

Browse Source
This commit is contained in:
lda
2026-06-23 01:00:31 +07:00 Verified
parent 893872f821
commit bc851ea99c
24 changed files with 663 additions and 913 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/add/system-design-implementation.md
+23 -8
View File
@@ -1723,15 +1723,28 @@ not yet claimed.
Evidence:
- `examples/agent_challenges/browser_click_challenge/workspace_template/prompt.md`:
challenge prompt and self-report schema.
- `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`:
trial runner.
- `examples/agent_challenges/browser_click_challenge/classification.py`:
convenience classifier for returned reports.
- `examples/agent_challenges/browser_click_challenge/reports.py`: report
extraction and saving.
- `examples/agent_challenges/browser_click_challenge/challenge.yaml`:
browser-click challenge manifest with success assertions.
- `examples/agent_challenges/browser_click_challenge/challenge-prompt.md`:
task-specific prompt for the browser-click challenge.
- `examples/agent_challenges/report_workflow_challenge/challenge.yaml`:
report-workflow challenge manifest with success assertions.
- `examples/agent_challenges/report_workflow_challenge/challenge-prompt.md`:
task-specific prompt for the report-workflow challenge.
- `examples/agent_challenges/run_trials.py`:
central trial runner accepting any challenge manifest and instruction profile.
- `examples/agent_challenges/manifests.py`:
generic manifest loading and path resolution.
- `examples/agent_challenges/prompts.py`:
prompt composition with base, profile, and challenge fragments.
- `tests/examples/test_opencode_browser_click_challenge.py`
- `tests/examples/test_report_workflow_challenge.py`
- `tests/examples/test_agent_challenge_harness_v2.py`
Two data-driven challenges now exist (browser-click and report-workflow), both
supporting `none`, `skills`, and `all` instruction profiles. Normalized metrics
and manual audit are implemented. Repeated audited model results are still
pending; no aggregate success claim is made.
## Limitations
@@ -1804,5 +1817,7 @@ Evidence:
- `examples/browser_click_workflow/`
- `examples/agent_challenges/browser_click_challenge/`
- `examples/agent_challenges/report_workflow_challenge/`
- `tests/examples/test_browser_click_workflow_example.py`
- `tests/examples/test_opencode_browser_click_challenge.py`
- `tests/examples/test_report_workflow_challenge.py`
docs/current_roadmap.md
+3 -1
View File
@@ -218,7 +218,9 @@ stable.
- Completed: the generic agent challenge harness now supports data-driven
manifests, layered prompts, explicit `none|skills|all` profiles, one-hour hard
ceilings, normalized OpenCode tool/token evidence, policy findings, and
manual-audited reports.
manual-audited reports. Two data-driven challenges exist: browser-click and
report-workflow. The central `run_trials.py` runner accepts any challenge
manifest.
## Historical References
docs/superpowers/plans/2026-06-22-agent-challenge-migrations.md → docs/historical/superpowers/plans/2026-06-22-agent-challenge-migrations.md
View File
examples/agent_challenges/browser_click_challenge/README.md
+42 -147
View File
@@ -1,12 +1,8 @@
# Opencode Browser Click Challenge Harness
# Browser Click Challenge
This harness runs agent trials against the browser-click workflow challenge.
It is evidence tooling, not product runtime code.
The harness mechanics (workspace preparation, opencode I/O, report handling,
trial runner) live in the shared modules under
[`examples/agent_challenges/`](../). The browser-click challenge provides
challenge-specific metadata, prompt template, classification, and defaults.
This challenge tests whether an agent can discover and use the workflow product
path to build and run a browser-click workflow. It is evidence tooling, not
product runtime code.
The deterministic workflow example is:
@@ -14,142 +10,51 @@ The deterministic workflow example is:
examples/browser_click_workflow/
```
## Default Behavior
## Running Trials
By default the harness does not start a `wf-rpc-server`. It prompts agents to
use a per-trial configured local CLI path:
Use the central runner from the repository root:
```powershell
uv run wf --config examples/agent_challenges/browser_click_challenge/workspaces/<trial>/wf.config.json --local
```
For each local-mode trial, the harness copies `workspace_template/` into
`workspaces/<model>-trial-<n>/`, generates a config whose Python source path is
relative to that copied config, and injects the config path into the prompt. This
builds the configured workflow server in the CLI process for each command and
uses the copied workspace's durable store. It does not reuse in-memory source
sessions across CLI invocations.
Use `--workspace-template` and `--source-root` to run a same-shape challenge with
a different prompt template or Python source root. Both default to the bundled
browser-click example settings.
Use `--start-server` when the trial should exercise the JSON-RPC server path.
With `--start-server`, the harness starts:
```powershell
uv run wf-rpc-server --config examples/browser_click_workflow/wf.config.json --host 127.0.0.1 --port 8772
```
It waits until `uv run wf --url http://127.0.0.1:8772/rpc status` passes,
injects that URL into the prompt, runs opencode, then stops the server. Use
`--server-url` to target an already-running server.
## One Trial
From the repository root:
```powershell
uv run python examples/agent_challenges/browser_click_challenge/run_opencode_trials.py `
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/browser_click_challenge/challenge.yaml `
--instruction-profile skills `
--model opencode/mimo-v2.5-free `
--variant high `
--trials 1
```
Variant challenge example:
```powershell
uv run python examples/agent_challenges/browser_click_challenge/run_opencode_trials.py `
--workspace-template examples/agent_challenges/browser_click_challenge/workspace_template `
--source-root examples/browser_click_workflow `
--workspaces-dir examples/agent_challenges/browser_click_challenge/workspaces_alt `
--results-dir examples/agent_challenges/browser_click_challenge/results_alt
```
Results are written to:
```text
examples/agent_challenges/browser_click_challenge/results/
```
## Manual Authoring Workspace
For manual authoring trials, use:
```text
examples/agent_challenges/browser_click_challenge/workspace_template/
```
It contains the prompt template and local store ignore rules without exposing a
generated draft patch answer file. The harness copies it automatically for
normal local-mode trials and writes `wf.config.json` into the copied workspace.
For manual experiments, copy it under:
```text
examples/agent_challenges/browser_click_challenge/workspaces/
```
`workspaces/` is ignored by Git so trial-created patches, plans, and scratch
files can be graded by hand without polluting the repository. The template's
store directory is also ignored.
## Saving Trial Reports
To save an agent's final answer from a harness result into its trial workspace:
```powershell
uv run python examples/agent_challenges/browser_click_challenge/save_trial_report.py `
--from-result examples/agent_challenges/browser_click_challenge/results/<trial>.json
```
The script infers the workspace from the result file and only writes
`<trial>/final-report.md`. Add evaluator commentary by editing that file after it
is saved. For manually copied reports, pass an explicit workspace and
`--input-file final-answer.md`.
## Saving Manual Audits
Keep `final-report.md` as captured agent output. If manual review finds that
the agent's YAML self-report was wrong or incomplete, write a sidecar audit:
```powershell
uv run python examples/agent_challenges/browser_click_challenge/save_manual_audit.py `
--from-result examples/agent_challenges/browser_click_challenge/results/<trial>.json `
--manual-classification success_code_assisted `
--set-read product_code=true `
--set-evidence trace_count=3 `
--correction "read.product_code: agent reported false, audited true" `
--notes "Valid product run, but raw plan shape came from product code/tests."
```
The script infers the workspace, run id, deployment id, automatic
classification, read flags, attempts, and notes from the result file. It writes
`<trial>/manual-audit.yaml`. Later benchmark summaries should treat the sidecar
as the human override, not mutate the captured agent report.
## Optional Opencode Server Attachment
`--attach` is opencode's server attach flag. It connects this non-interactive
run to an already-running opencode server, for example:
```powershell
--trials 1 `
--attach http://127.0.0.1:4096
```
It is not a direct MCP server URL. If that opencode server is configured with
Playwright MCP tools, then the attached run can use those tools through
opencode. One possible MCP server command for such an opencode setup is:
Profiles `none` and `all` are separate invocations:
```json
{
"command": "npx",
"args": ["-y", "@playwright/mcp@latest"]
}
```powershell
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/browser_click_challenge/challenge.yaml `
--instruction-profile none `
--model opencode/mimo-v2.5-free `
--trials 1
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/browser_click_challenge/challenge.yaml `
--instruction-profile all `
--model opencode/mimo-v2.5-free `
--trials 1
```
The baseline challenge does not require Playwright MCP. The score is based on
whether the agent used the workflow product path and produced the expected
workflow output.
The hard timeout ceiling is 3,600 seconds per trial.
## Default Behavior
By default the harness does not start a `wf-rpc-server`. It prompts agents to
use a per-trial configured local CLI path. Use `--start-server` when the trial
should exercise the JSON-RPC server path.
## Workspace Layout
- `workspace_template/` contains files copied into each isolated trial workspace.
- `workspaces/` holds per-trial workspaces (gitignored).
- `results/` holds per-trial result JSON files (gitignored).
- `challenge.yaml` declares the manifest, source, server, and report schema.
- `challenge-prompt.md` contains the task-specific prompt.
## Required Agent Report
@@ -198,22 +103,12 @@ Each trial is classified as one of:
- `parse_error`: the harness could not read opencode JSON/JSONL output.
- `unknown`: no clear success or failure signal was found.
Automatic success assertions are provisional until manual audit.
## Shared Harness Modules
The generic modules in `examples/agent_challenges/` provide reusable harness
logic:
| Module | Purpose |
|--------|---------|
| `workspace.py` | `ChallengeDef`, `TrialConfig`, `TrialWorkspace`, `prepare_trial_workspace`, `write_trial_config`, `starting_trial_index`, `wf_command_prefix_for_config`, `render_prompt`, `server_command` |
| `runner.py` | `ManagedServer`, `start_server`, `run_trial`, `main` — generic trial runner parameterized by `ChallengeDef` and a classification function |
| `opencode_io.py` | `build_opencode_command`, `parse_opencode_output`, `result_text` |
| `reports.py` | `save_report`, `report_from_result`, `save_report_from_result_payload` |
| `classification.py` | `extract_challenge_report` (generic YAML extraction), `_contains_bool_marker` |
The browser-click challenge's `run_opencode_trials.py` and `save_trial_report.py`
are thin wrappers that pass `BROWSER_CLICK_DEF` and the browser-click
classification function to the generic runner. The same pattern can be used to
add new challenges without duplicating the harness.
logic. See `workspace.py`, `runner.py`, `opencode_io.py`, `reports.py`,
`classification.py`, and `manifests.py`.
Committed tests cover harness logic only. They do not invoke opencode.
examples/agent_challenges/browser_click_challenge/__init__.py
View File
examples/agent_challenges/browser_click_challenge/workspace_template/prompt.md → examples/agent_challenges/browser_click_challenge/challenge-prompt.md
+9 -30
View File
@@ -7,30 +7,11 @@ Build and successfully run a workflow that:
3. Captures a before snapshot and an after snapshot.
4. Returns both snapshots as workflow output.
Use this command prefix for product-facing operations:
Discover the `local.browser_click` source capabilities through `wf cap list`,
`wf cap inspect`, and `wf schema`. Repository implementation inspection is
profile-controlled; do not read source files unless your profile permits it.
```powershell
{{wf_command_prefix}}
```
{{server_context}}
Use this repository's workflow product path. Start by checking the relevant
agent skills under `skills/`, especially the workflow/CLI skill references, and
use broad docs/code search only after those pointers are not enough. You should
use the `wf` CLI with the command prefix above, create or reuse a workflow
deployment, and run the deployment through the workflow API. Do not solve the
challenge with only a standalone Playwright/Python script.
Do not create a new helper script whose only job is to drive `WorkflowApi`
directly. The challenge is about whether the product-facing CLI/server workflow
can be discovered and used.
If you need to write a workflow definition, write a declarative JSON/YAML file
and then apply/run it through the product-facing workflow tools. Do not hide the
workflow construction inside a Python script.
Write all files you create for this attempt inside the designated workspace.
## Workflow Authoring Paths
Two product-facing authoring paths are acceptable:
@@ -49,6 +30,8 @@ alias.
Do not use a pre-existing generated patch or raw-plan answer file. If you find
one, ignore it and author your own workflow definition.
## Disallowed Approaches
These do not satisfy the challenge, even if they produce the right output:
- importing `WorkflowApi`, `WorkflowServer`, or source functions directly;
@@ -61,13 +44,7 @@ These do not satisfy the challenge, even if they produce the right output:
- reusing artifacts, deployments, stores, workflow files, or run outputs created
by earlier trials.
The repository already includes a deterministic source example at:
```text
examples/browser_click_workflow/
```
You may inspect and use it.
## Evidence Requirements
Your final answer should include a short human-readable report with:
@@ -79,6 +56,8 @@ Your final answer should include a short human-readable report with:
- whether any server/browser process remains running,
- important failed attempts and how you fixed them.
## Required YAML Report
End your answer with exactly one fenced YAML block using this shape:
```yaml
examples/agent_challenges/browser_click_challenge/challenge.py
-76
View File
@@ -1,76 +0,0 @@
from __future__ import annotations
from pathlib import Path
from typing import Literal
from examples.agent_challenges.workspace import ( # noqa: F401 - re-exported for backward compat
ChallengeDef,
TrialConfig,
TrialWorkspace,
render_prompt,
rpc_url_for_port,
server_command,
)
Classification = Literal[
"success",
"workflow_script",
"workflow_not_used",
"run_failed",
"timeout",
"parse_error",
"unknown",
]
CHALLENGE_REPORT_REQUIRED_FIELDS = {
"used_product_path",
"used_helper_script",
"workflow_file",
"deployment_id",
"run_id",
"before_clicked",
"after_clicked",
"run_failed",
"leftover_processes",
"read",
"attempts",
"missed_requirements",
"notes",
}
CHALLENGE_REPORT_READ_FIELDS = {
"skills",
"docs",
"product_code",
"adjacent_attempts",
"prior_store",
"existing_solution",
}
CHALLENGE_REPORT_ATTEMPT_FIELDS = {"total", "failed"}
ROOT = Path(__file__).resolve().parents[3]
CHALLENGE_DIR = Path(__file__).resolve().parent
BROWSER_CLICK_DEF = ChallengeDef(
name="browser_click",
source_root=ROOT / "examples" / "browser_click_workflow",
source_id="local.browser_click",
source_module="ops",
source_registry="registry",
store_root=".wf_browser_click_store",
default_workspace_template=CHALLENGE_DIR / "workspace_template",
default_workspaces_dir=CHALLENGE_DIR / "workspaces",
default_results_dir=CHALLENGE_DIR / "results",
default_prompt=CHALLENGE_DIR / "workspace_template" / "prompt.md",
default_server_port=8772,
server_config_arg="examples/browser_click_workflow/wf.config.json",
)
EXAMPLE_CONFIG_ARG = BROWSER_CLICK_DEF.server_config_arg
EXAMPLE_CONFIG = ROOT / EXAMPLE_CONFIG_ARG
EXAMPLE_SOURCE_ROOT = BROWSER_CLICK_DEF.source_root
LOCAL_WF_COMMAND_PREFIX = f"uv run wf --config {EXAMPLE_CONFIG_ARG} --local"
DEFAULT_PROMPT = BROWSER_CLICK_DEF.default_prompt
DEFAULT_RESULTS_DIR = BROWSER_CLICK_DEF.default_results_dir
DEFAULT_WORKSPACES_DIR = BROWSER_CLICK_DEF.default_workspaces_dir
DEFAULT_WORKSPACE_TEMPLATE = BROWSER_CLICK_DEF.default_workspace_template
DEFAULT_SERVER_PORT = BROWSER_CLICK_DEF.default_server_port
examples/agent_challenges/browser_click_challenge/challenge.yaml
+33
View File
@@ -0,0 +1,33 @@
version: 1
id: browser_click
prompt: challenge-prompt.md
workspace_template: workspace_template
source:
id: local.browser_click
root: ../../browser_click_workflow
module: ops
registry: registry
store_root: .wf_browser_click_store
server:
config: ../../browser_click_workflow/wf.config.json
default_port: 8772
report:
required_fields:
- used_product_path
- used_helper_script
- workflow_file
- deployment_id
- run_id
- before_clicked
- after_clicked
- run_failed
- leftover_processes
- read
- attempts
- missed_requirements
- notes
success_assertions:
before_clicked: false
after_clicked: true
run_failed: false
leftover_processes: false
examples/agent_challenges/browser_click_challenge/classification.py
-158
View File
@@ -1,158 +0,0 @@
from __future__ import annotations
from typing import Any
from examples.agent_challenges.browser_click_challenge.challenge import (
CHALLENGE_REPORT_ATTEMPT_FIELDS,
CHALLENGE_REPORT_READ_FIELDS,
CHALLENGE_REPORT_REQUIRED_FIELDS,
Classification,
)
from examples.agent_challenges.classification import (
_contains_bool_marker,
extract_challenge_report,
)
def classify_output(text: str) -> Classification:
report = extract_challenge_report(text)
if report is not None:
return classify_challenge_report(report)
lowered = text.lower()
product_command_markers = [
"wf ",
"wf-rpc-server",
]
workflow_evidence_markers = [
"deployment",
"run id",
"run_",
]
used_product_command = any(marker in lowered for marker in product_command_markers)
has_workflow_evidence = any(
marker in lowered for marker in workflow_evidence_markers
)
used_helper_script = (
"uv run python" in lowered
or "python examples/" in lowered
or "run_workflow.py" in lowered
)
failed = any(
marker in lowered
for marker in [
"error:",
"failed",
"traceback",
"exception",
"validation failed",
]
)
before_false = _contains_bool_marker(lowered, "before.clicked", "false") or (
'"before"' in lowered and '"clicked": false' in lowered
)
after_true = _contains_bool_marker(lowered, "after.clicked", "true") or (
'"after"' in lowered and '"clicked": true' in lowered
)
if used_product_command and before_false and after_true and not failed:
return "success"
if has_workflow_evidence and used_helper_script and before_false and after_true:
return "workflow_script"
if (used_product_command or has_workflow_evidence) and failed:
return "run_failed"
if not has_workflow_evidence and (
before_false or after_true or "playwright" in lowered
):
return "workflow_not_used"
return "unknown"
def classify_challenge_report(report: dict[str, Any]) -> Classification:
if challenge_report_schema_errors(report):
return "unknown"
used_product_path = report.get("used_product_path") is True
used_helper_script = report.get("used_helper_script") is True
workflow_file = report.get("workflow_file")
deployment_id = report.get("deployment_id")
run_id = report.get("run_id")
before_clicked = report.get("before_clicked")
after_clicked = report.get("after_clicked")
failed = report.get("run_failed") is True
if failed:
return "run_failed"
if used_helper_script:
return "workflow_script"
if (
used_product_path
and isinstance(workflow_file, str)
and bool(workflow_file)
and isinstance(deployment_id, str)
and bool(deployment_id)
and isinstance(run_id, str)
and bool(run_id)
and before_clicked is False
and after_clicked is True
):
return "success"
if not used_product_path and (
before_clicked is not None or after_clicked is not None
):
return "workflow_not_used"
return "unknown"
def challenge_report_schema_errors(report: dict[str, Any]) -> list[str]:
errors: list[str] = []
missing = sorted(CHALLENGE_REPORT_REQUIRED_FIELDS.difference(report))
errors.extend(f"missing challenge_report.{field}" for field in missing)
for field in (
"used_product_path",
"used_helper_script",
"before_clicked",
"after_clicked",
"run_failed",
"leftover_processes",
):
if field in report and not isinstance(report[field], bool):
errors.append(f"challenge_report.{field} must be boolean")
for field in ("workflow_file", "deployment_id", "run_id", "notes"):
if field in report and not isinstance(report[field], str):
errors.append(f"challenge_report.{field} must be string")
read = report.get("read")
if isinstance(read, dict):
missing_read = sorted(CHALLENGE_REPORT_READ_FIELDS.difference(read))
errors.extend(
f"missing challenge_report.read.{field}" for field in missing_read
)
for field in CHALLENGE_REPORT_READ_FIELDS.intersection(read):
if not isinstance(read[field], bool):
errors.append(f"challenge_report.read.{field} must be boolean")
elif "read" in report:
errors.append("challenge_report.read must be object")
attempts = report.get("attempts")
if isinstance(attempts, dict):
missing_attempts = sorted(CHALLENGE_REPORT_ATTEMPT_FIELDS.difference(attempts))
errors.extend(
f"missing challenge_report.attempts.{field}" for field in missing_attempts
)
for field in CHALLENGE_REPORT_ATTEMPT_FIELDS.intersection(attempts):
value = attempts[field]
if not isinstance(value, int) or isinstance(value, bool) or value < 0:
errors.append(f"challenge_report.attempts.{field} must be >= 0 integer")
elif "attempts" in report:
errors.append("challenge_report.attempts must be object")
missed = report.get("missed_requirements")
if "missed_requirements" in report and (
not isinstance(missed, list)
or any(not isinstance(item, str) for item in missed)
):
errors.append("challenge_report.missed_requirements must be list of strings")
return errors
examples/agent_challenges/browser_click_challenge/opencode_io.py
-9
View File
@@ -1,9 +0,0 @@
from __future__ import annotations
from examples.agent_challenges.opencode_io import ( # noqa: F401
_event_text,
_parse_jsonl_tail,
build_opencode_command,
parse_opencode_output,
result_text,
)
examples/agent_challenges/browser_click_challenge/reports.py
-8
View File
@@ -1,8 +0,0 @@
from __future__ import annotations
from examples.agent_challenges.reports import ( # noqa: F401
main,
report_from_result,
save_report,
save_report_from_result_payload,
)
examples/agent_challenges/browser_click_challenge/run_opencode_trials.py
-179
View File
@@ -1,179 +0,0 @@
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
from typing import Any
# Support direct execution as `python examples/.../run_opencode_trials.py`.
# ruff: noqa: I001 - imports must stay after sys.path.insert
ROOT = Path(__file__).resolve().parents[3]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from examples.agent_challenges.browser_click_challenge.challenge import ( # noqa: E402
BROWSER_CLICK_DEF,
CHALLENGE_DIR,
CHALLENGE_REPORT_ATTEMPT_FIELDS,
CHALLENGE_REPORT_READ_FIELDS,
CHALLENGE_REPORT_REQUIRED_FIELDS,
Classification,
DEFAULT_PROMPT,
DEFAULT_RESULTS_DIR,
DEFAULT_SERVER_PORT,
DEFAULT_WORKSPACE_TEMPLATE,
DEFAULT_WORKSPACES_DIR,
EXAMPLE_CONFIG,
EXAMPLE_CONFIG_ARG,
EXAMPLE_SOURCE_ROOT,
LOCAL_WF_COMMAND_PREFIX,
)
from examples.agent_challenges.browser_click_challenge.classification import ( # noqa: E402
_contains_bool_marker,
challenge_report_schema_errors,
classify_challenge_report,
classify_output,
extract_challenge_report,
)
from examples.agent_challenges.opencode_io import ( # noqa: E402
_event_text,
_parse_jsonl_tail,
build_opencode_command,
parse_opencode_output,
result_text,
)
from examples.agent_challenges.reports import ( # noqa: E402
save_report_from_result_payload,
)
from examples.agent_challenges.runner import ( # noqa: E402
ManagedServer,
main as _generic_main,
run_trial as _generic_run_trial,
start_server as _generic_start_server,
stop_server as _generic_stop_server,
)
from examples.agent_challenges.workspace import ( # noqa: E402
TrialConfig,
TrialWorkspace,
prepare_trial_workspace as _generic_prepare,
render_prompt,
rpc_url_for_port,
server_command,
starting_trial_index as _generic_starting_index,
trial_output_path as _generic_trial_output_path,
wf_command_prefix_for_config as _generic_wf_prefix,
)
__all__ = [
"BROWSER_CLICK_DEF",
"CHALLENGE_DIR",
"CHALLENGE_REPORT_ATTEMPT_FIELDS",
"CHALLENGE_REPORT_READ_FIELDS",
"CHALLENGE_REPORT_REQUIRED_FIELDS",
"Classification",
"DEFAULT_PROMPT",
"DEFAULT_RESULTS_DIR",
"DEFAULT_SERVER_PORT",
"DEFAULT_WORKSPACE_TEMPLATE",
"DEFAULT_WORKSPACES_DIR",
"EXAMPLE_CONFIG",
"EXAMPLE_CONFIG_ARG",
"EXAMPLE_SOURCE_ROOT",
"LOCAL_WF_COMMAND_PREFIX",
"ManagedServer",
"ROOT",
"TrialConfig",
"TrialWorkspace",
"_contains_bool_marker",
"_event_text",
"_parse_jsonl_tail",
"build_opencode_command",
"challenge_report_schema_errors",
"classify_challenge_report",
"classify_output",
"extract_challenge_report",
"main",
"parse_opencode_output",
"prepare_trial_workspace",
"render_prompt",
"result_text",
"rpc_url_for_port",
"run_trial",
"save_report_from_result_payload",
"server_command",
"start_server",
"starting_trial_index",
"stop_server",
"trial_output_path",
"wf_command_prefix_for_config",
]
def main(argv: list[str] | None = None) -> int:
return _generic_main(BROWSER_CLICK_DEF, classify_output, argv)
def prepare_trial_workspace(
*,
model: str,
index: int,
workspaces_dir: Path = DEFAULT_WORKSPACES_DIR,
template_dir: Path = DEFAULT_WORKSPACE_TEMPLATE,
source_root: Path = EXAMPLE_SOURCE_ROOT,
) -> TrialWorkspace:
return _generic_prepare(
BROWSER_CLICK_DEF,
model=model,
index=index,
workspaces_dir=workspaces_dir,
template_dir=template_dir,
source_root=source_root,
)
def run_trial(
config: TrialConfig,
*,
index: int,
results_dir: Path,
) -> dict[str, Any]:
return _generic_run_trial(
config, index=index, results_dir=results_dir, classify_fn=classify_output
)
def start_server(
*,
port: int,
timeout_seconds: int = 30,
) -> ManagedServer:
return _generic_start_server(
BROWSER_CLICK_DEF, port=port, timeout_seconds=timeout_seconds
)
def stop_server(process: subprocess.Popen[str]) -> None:
return _generic_stop_server(process)
def starting_trial_index(
*,
model: str,
results_dir: Path,
workspaces_dir: Path,
) -> int:
return _generic_starting_index(
model=model, results_dir=results_dir, workspaces_dir=workspaces_dir
)
def trial_output_path(results_dir: Path, *, model: str, index: int) -> Path:
return _generic_trial_output_path(results_dir, model=model, index=index)
def wf_command_prefix_for_config(config_path: Path) -> str:
return _generic_wf_prefix(config_path)
if __name__ == "__main__":
raise SystemExit(main())
examples/agent_challenges/browser_click_challenge/save_manual_audit.py
-21
View File
@@ -1,21 +0,0 @@
from __future__ import annotations
import sys
from pathlib import Path
# Support direct execution as `python examples/.../save_manual_audit.py`.
ROOT = Path(__file__).resolve().parents[3]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from examples.agent_challenges.audit import ( # noqa: E402
audit_from_result,
main,
save_manual_audit,
)
__all__ = ["audit_from_result", "main", "save_manual_audit"]
if __name__ == "__main__":
raise SystemExit(main())
examples/agent_challenges/browser_click_challenge/save_trial_report.py
-21
View File
@@ -1,21 +0,0 @@
from __future__ import annotations
import sys
from pathlib import Path
# Support direct execution as `python examples/.../save_trial_report.py`.
ROOT = Path(__file__).resolve().parents[3]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from examples.agent_challenges.reports import ( # noqa: E402
main,
report_from_result,
save_report,
)
__all__ = ["main", "report_from_result", "save_report"]
if __name__ == "__main__":
raise SystemExit(main())
examples/agent_challenges/report_workflow_challenge/README.md
+92
View File
@@ -0,0 +1,92 @@
# Report Workflow Challenge
This challenge tests whether an agent can discover and use the workflow product
path to build and run a three-node report pipeline. It is evidence tooling, not
product runtime code.
The deterministic workflow steps are: `read_notes -> extract_report ->
render_markdown_report`.
## Running Trials
Use the central runner from the repository root:
```powershell
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/report_workflow_challenge/challenge.yaml `
--instruction-profile skills `
--model opencode/mimo-v2.5-free `
--trials 1
```
Profiles `none` and `all` are separate invocations:
```powershell
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/report_workflow_challenge/challenge.yaml `
--instruction-profile none `
--model opencode/mimo-v2.5-free `
--trials 1
uv run python examples/agent_challenges/run_trials.py `
--challenge examples/agent_challenges/report_workflow_challenge/challenge.yaml `
--instruction-profile all `
--model opencode/mimo-v2.5-free `
--trials 1
```
The hard timeout ceiling is 3,600 seconds per trial.
## Workspace Layout
- `workspace_template/` holds local store ignore rules (gitignored contents).
- `workspaces/` holds per-trial workspaces (gitignored).
- `results/` holds per-trial result JSON files (gitignored).
- `challenge.yaml` declares the manifest, source, server, and report schema.
- `challenge-prompt.md` contains the task-specific prompt.
## Required Agent Report
The prompt asks the agent to end with one fenced YAML block:
```yaml
challenge_report:
used_product_path: true
used_helper_script: false
workflow_file: "path/to/workflow.json-or-yaml"
deployment_id: "report_workflow.default"
run_id: "run_..."
title_matches: true
markdown_rendered: true
run_failed: false
read:
skills: true
docs: true
product_code: false
adjacent_attempts: false
prior_store: false
existing_solution: false
attempts:
total: 1
failed: 0
missed_requirements:
- "none"
notes: "short explanation"
```
Automatic success assertions are provisional until manual audit.
## Future Design Notes
> Ideas preserved from the original user-authored challenge notes.
The initial challenge is the deterministic three-node report pipeline so results
are auditable. Future expansions could include:
- docs collection -> foreach summarize by person/point
- render markdown -> send to email
- foreach (collect by person -> render markdown -> lookup name -> send to email)
- save somewhere
This exercises many workflow capabilities. To make it easier, the docs could be
very structured: each doc by topic, points by person, or some better system.
examples/agent_challenges/report_workflow_challenge/challenge-prompt.md
+120
View File
@@ -0,0 +1,120 @@
# Report Workflow Challenge
Build and successfully run a workflow that processes structured notes into a
rendered markdown report.
The workflow must execute these three steps in sequence:
1. `read_notes` -- read structured input notes
2. `extract_report` -- extract report content from the notes
3. `render_markdown_report` -- render the extracted content as a markdown report
## Workflow Authoring Paths
Two product-facing authoring paths are acceptable:
- draft path: create a draft from one capability, use focused draft edit
commands or your own RFC 6902 JSON Patch, then validate/save/deploy/run it;
- raw-plan path: write your own complete raw workflow plan file and use
`wf artifact create-from-plan` before deploy/run.
Do not mix the formats. Drafts use `steps`, `routes`, and step field `use`.
Raw plans use `nodes`, `edges`, and node field `node`. Do not pass draft JSON to
`wf artifact create-from-plan`.
The deployment command is `wf deploy save`; `wf deploy create` is accepted as an
alias.
Do not use a pre-existing generated patch or raw-plan answer file. If you find
one, ignore it and author your own workflow definition.
## Disallowed Approaches
These do not satisfy the challenge, even if they produce the right output:
- importing `WorkflowApi`, `WorkflowServer`, or source functions directly;
- writing a Python script that calls internal APIs to create artifacts,
deployments, or runs;
- calling the source functions directly instead of running a deployed workflow;
- solving it as a standalone script with no `wf artifact`, `wf deploy`, and
`wf run` lifecycle;
- reusing artifacts, deployments, stores, workflow files, or run outputs created
by earlier trials.
## Evidence Requirements
Your final answer should include a short human-readable report with:
- the commands you ran,
- the deployment id,
- the run id if one was produced,
- evidence that the report title equals `Weekly Project Update`,
- evidence that the rendered markdown begins with `# Weekly Project Update`,
- important failed attempts and how you fixed them.
## Required YAML Report
End your answer with exactly one fenced YAML block using this shape:
```yaml
challenge_report:
used_product_path: true
used_helper_script: false
workflow_file: "path/to/workflow.json-or-yaml"
deployment_id: "report_workflow.default"
run_id: "run_..."
title_matches: true
markdown_rendered: true
run_failed: false
read:
skills: true
docs: true
product_code: false
adjacent_attempts: false
prior_store: false
existing_solution: false
attempts:
total: 1
failed: 0
missed_requirements:
- "none"
notes: "short explanation"
```
Reporting rules:
- The YAML block is a self-report only. It will be manually audited against your
commands, files, and run output.
- Set `used_product_path: true` only if you used `uv run wf ...` commands for
the artifact/deployment/run lifecycle, either in local same-process mode or
through `wf-rpc-server`.
- Set `used_helper_script: true` if you wrote or ran any script whose main
purpose was to drive the workflow API, JSON-RPC API, server internals, or
source functions outside `wf`.
- Set `title_matches: true` if the rendered report title equals
`Weekly Project Update`.
- Set `markdown_rendered: true` if the output is valid markdown beginning with
`# Weekly Project Update`.
- Set `read.product_code: true` if you or a spawned subagent grepped, searched,
or read source files under `src/`, `tests/`, or implementation examples to
determine plan shape or product behavior.
- Set `read.docs: true` if you read files under `docs/`.
- Set `read.skills: true` if you read files under `skills/`.
- Set `read.adjacent_attempts: true` if you read files under other trial
workspaces, prior result files, generated reports, or previous attempt
artifacts.
- Set `read.prior_store: true` if you inspected or reused `.wf_*` stores, saved
artifacts, deployments, or runs from outside your current trial workspace.
- Set `read.existing_solution: true` if you copied or inspected a ready-made
solution plan/workflow for this same challenge.
- `attempts.total` should count distinct product-lifecycle attempts, including
failed artifact creation, failed deployment validation, failed run starts, and
abandoned workflow plans.
- `attempts.failed` should count attempts that failed validation, failed to run,
produced wrong output, or were abandoned.
Spawned subagents count as you. If a subagent reads product code, set
`read.product_code: true`. If a subagent reads prior attempts, set
`read.adjacent_attempts: true`.
If something fails, report the exact command and error instead of hiding it.
examples/agent_challenges/report_workflow_challenge/challenge.yaml
+31
View File
@@ -0,0 +1,31 @@
version: 1
id: report_workflow
prompt: challenge-prompt.md
workspace_template: workspace_template
source:
id: local.report
root: ../../report_workflow
module: ops
registry: registry
store_root: .wf_report_challenge_store
server:
config: ../../report_workflow/wf.config.json
default_port: 8773
report:
required_fields:
- used_product_path
- used_helper_script
- workflow_file
- deployment_id
- run_id
- title_matches
- markdown_rendered
- run_failed
- read
- attempts
- missed_requirements
- notes
success_assertions:
title_matches: true
markdown_rendered: true
run_failed: false
examples/agent_challenges/report_workflow_challenge/readme.md
-20
View File
@@ -1,20 +0,0 @@
delete and replace file if youve done it.
THE OG. the GOAT. the latest and greatest.
examples\demo_workflow.py
the flow is docs -> foreach summarize -> email by flag.
also look at report_workflow, and other workflows in example.
im expanding it to the biggest and most challenging:
docs collection -> foreach summarize by person / point
|-> render markdown -> send to email
|-> foreach (collect by person -> render markdown -> lookup name -> send to email)
-> save somewhere
this exercises a lot of things.
to make shit easy lets have the docs VERY structured. Each doc is by topic(?), points by person(?), or some better system.
examples/agent_challenges/report_workflow_challenge/results/.gitignore
+2
View File
@@ -0,0 +1,2 @@
*
!.gitignore
examples/agent_challenges/report_workflow_challenge/workspace_template/.gitignore
+2
View File
@@ -0,0 +1,2 @@
*
!.gitignore
examples/agent_challenges/report_workflow_challenge/workspaces/.gitignore
+2
View File
@@ -0,0 +1,2 @@
*
!.gitignore
tests/examples/test_agent_challenge_harness_v2.py
+98
View File
@@ -896,3 +896,101 @@ def test_v2_runner_preserves_report_parse_error_on_malformed_yaml(
"YAMLError",
)
assert result.get("challenge_report") is None
BROWSER_CHALLENGE = (
ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "challenge.yaml"
)
REPORT_CHALLENGE = (
ROOT
/ "examples"
/ "agent_challenges"
/ "report_workflow_challenge"
/ "challenge.yaml"
)
INSTRUCTION_BUNDLE = (
ROOT / "examples" / "agent_challenges" / "instruction_bundles" / "workflow_cli.yaml"
)
@pytest.mark.parametrize(
"manifest_path",
[BROWSER_CHALLENGE, REPORT_CHALLENGE],
ids=["browser_click", "report_workflow"],
)
def test_both_challenges_load_through_same_manifest(manifest_path: Path) -> None:
loaded = load_challenge_manifest(manifest_path)
assert loaded.manifest.version == 1
assert loaded.prompt_path.is_file()
assert loaded.workspace_template.is_dir()
assert loaded.manifest.report.required_fields
@pytest.mark.parametrize(
"manifest_path, expected_id, expected_source_id",
[
(BROWSER_CHALLENGE, "browser_click", "local.browser_click"),
(REPORT_CHALLENGE, "report_workflow", "local.report"),
],
ids=["browser_click", "report_workflow"],
)
def test_both_challenges_prepare_workspaces_under_each_profile(
manifest_path: Path,
expected_id: str,
expected_source_id: str,
tmp_path: Path,
) -> None:
loaded = load_challenge_manifest(manifest_path)
assert loaded.manifest.id == expected_id
assert loaded.manifest.source.id == expected_source_id
for profile in InstructionProfile:
workspace = prepare_v2_trial_workspace(
loaded,
profile=profile,
model="test-model",
index=1,
workspaces_dir=tmp_path / profile.value,
instruction_bundle=INSTRUCTION_BUNDLE,
)
assert workspace.config_path.is_file()
config = json.loads(workspace.config_path.read_text(encoding="utf-8"))
assert config["client"]["target"] == {"kind": "local"}
assert config["server"]["store"]["root"] == loaded.manifest.store_root
if profile in (InstructionProfile.SKILLS, InstructionProfile.ALL):
assert (workspace.root / ".agent/skills/wf-cli/SKILL.md").is_file()
else:
assert not (workspace.root / ".agent").exists()
def test_both_challenges_produce_different_challenge_hashes_but_same_base(
tmp_path: Path,
) -> None:
from examples.agent_challenges.prompts import compose_trial_prompt
browser = load_challenge_manifest(BROWSER_CHALLENGE)
report = load_challenge_manifest(REPORT_CHALLENGE)
hashes: dict[str, str] = {}
for name, challenge in [("browser", browser), ("report", report)]:
for profile in InstructionProfile:
rendered = compose_trial_prompt(
challenge,
profile=profile,
wf_command_prefix="uv run wf --config wf.config.json --local",
server_context="Local mode.",
workspace_path=tmp_path / f"{name}_{profile.value}",
)
hashes[f"{name}_{profile.value}"] = rendered.challenge_sha256
assert hashes["browser_none"] != hashes["report_none"]
for profile in InstructionProfile:
assert hashes[f"browser_{profile.value}"] != hashes[f"report_{profile.value}"]
tests/examples/test_opencode_browser_click_challenge.py
+168 -236
View File
@@ -2,82 +2,79 @@ from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
import yaml
from examples.agent_challenges import reports as generic_reports
from examples.agent_challenges.browser_click_challenge import (
run_opencode_trials,
from examples.agent_challenges.audit import (
main as save_manual_audit_main,
)
from examples.agent_challenges.browser_click_challenge.challenge import (
BROWSER_CLICK_DEF,
LOCAL_WF_COMMAND_PREFIX,
render_prompt,
server_command,
)
from examples.agent_challenges.browser_click_challenge.classification import (
challenge_report_schema_errors,
classify_challenge_report,
classify_output,
extract_challenge_report,
)
from examples.agent_challenges.browser_click_challenge.opencode_io import (
from examples.agent_challenges.manifests import load_challenge_manifest
from examples.agent_challenges.models import InstructionProfile
from examples.agent_challenges.opencode_io import (
build_opencode_command,
parse_opencode_output,
)
from examples.agent_challenges.browser_click_challenge.reports import (
from examples.agent_challenges.reports import (
main as save_trial_report_main,
)
from examples.agent_challenges.reports import (
report_from_result,
save_report,
)
from examples.agent_challenges.browser_click_challenge.run_opencode_trials import (
prepare_trial_workspace,
run_trial,
starting_trial_index,
trial_output_path,
wf_command_prefix_for_config,
)
from examples.agent_challenges.browser_click_challenge.save_manual_audit import (
main as save_manual_audit_main,
)
from examples.agent_challenges.browser_click_challenge.save_trial_report import (
main as save_trial_report_main,
)
from examples.agent_challenges.runner import main as generic_runner_main
from examples.agent_challenges.runner import run_trial
from examples.agent_challenges.workspace import (
ChallengeDef,
TrialConfig,
prepare_trial_workspace,
prepare_v2_trial_workspace,
render_prompt,
server_command,
starting_trial_index,
trial_output_path,
wf_command_prefix_for_config,
write_trial_config,
)
from examples.agent_challenges.workspace import (
prepare_trial_workspace as generic_prepare_trial_workspace,
ROOT = Path(__file__).resolve().parents[2]
LOCAL_WF_COMMAND_PREFIX = (
"uv run wf --config examples/browser_click_workflow/wf.config.json --local"
)
def _valid_challenge_report(**overrides: object) -> dict[str, object]:
report: dict[str, object] = {
"used_product_path": True,
"used_helper_script": False,
"workflow_file": "browser-click.workflow.yaml",
"deployment_id": "browser_click_case_study.default",
"run_id": "run_123",
"before_clicked": False,
"after_clicked": True,
"run_failed": False,
"leftover_processes": False,
"read": {
"skills": True,
"docs": True,
"product_code": False,
"adjacent_attempts": False,
"prior_store": False,
"existing_solution": False,
},
"attempts": {"total": 1, "failed": 0},
"missed_requirements": ["none"],
"notes": "ok",
}
report.update(overrides)
return report
BROWSER_CLICK_DEF = ChallengeDef(
name="browser_click",
source_root=ROOT / "examples" / "browser_click_workflow",
source_id="local.browser_click",
source_module="ops",
source_registry="registry",
store_root=".wf_browser_click_store",
default_workspace_template=ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "workspace_template",
default_workspaces_dir=ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "workspaces",
default_results_dir=ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "results",
default_prompt=ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "challenge-prompt.md",
default_server_port=8772,
server_config_arg="examples/browser_click_workflow/wf.config.json",
)
def test_build_opencode_command_without_attach(tmp_path: Path) -> None:
@@ -123,24 +120,6 @@ def test_build_opencode_command_with_attach(tmp_path: Path) -> None:
assert "http://127.0.0.1:4096" in command
def test_run_opencode_trials_script_supports_direct_execution() -> None:
result = subprocess.run(
[
"uv",
"run",
"python",
"examples/agent_challenges/browser_click_challenge/run_opencode_trials.py",
"--help",
],
text=True,
capture_output=True,
check=False,
)
assert result.returncode == 0
assert "--model" in result.stdout
def test_run_trial_saves_final_report_from_successful_result(
tmp_path: Path,
monkeypatch,
@@ -204,7 +183,12 @@ def test_run_trial_saves_final_report_from_successful_result(
server_context="Use local CLI mode.",
)
result = run_trial(config, index=1, results_dir=tmp_path / "results")
result = run_trial(
config,
index=1,
results_dir=tmp_path / "results",
classify_fn=lambda _text: "success",
)
assert result["classification"] == "success"
assert result["report_path"] == (workspace / "final-report.md").as_posix()
@@ -242,7 +226,12 @@ def test_run_trial_records_report_save_error_for_timeout(
server_context="Use local CLI mode.",
)
result = run_trial(config, index=1, results_dir=tmp_path / "results")
result = run_trial(
config,
index=1,
results_dir=tmp_path / "results",
classify_fn=lambda _text: "unknown",
)
assert result["classification"] == "timeout"
assert result["report_save_error"] == "result file is missing parsed output"
@@ -282,7 +271,12 @@ def test_run_trial_records_parse_error_details(
server_context="Use local CLI mode.",
)
result = run_trial(config, index=1, results_dir=tmp_path / "results")
result = run_trial(
config,
index=1,
results_dir=tmp_path / "results",
classify_fn=lambda _text: "unknown",
)
assert result["classification"] == "parse_error"
assert result["parse_error"]["type"] == "JSONDecodeError"
@@ -339,159 +333,6 @@ def test_parse_opencode_output_prefers_text_event_before_step_finish() -> None:
assert parsed["text"] == "deployment id: demo.default\nbefore.clicked is false"
def test_classify_output_success() -> None:
result = classify_output(
"""
uv run wf-rpc-server --config examples/browser_click_workflow/wf.config.json
uv run wf run start browser_click_case_study.default
deployment id: browser_click_case_study.default
run id: run_123
before.clicked is false
after.clicked is true
"""
)
assert result == "success"
def test_extract_challenge_report_from_yaml_block() -> None:
text = """
The run worked.
```yaml
challenge_report:
used_product_path: true
used_helper_script: false
workflow_file: "browser-click.workflow.yaml"
deployment_id: "browser_click_case_study.default"
run_id: "run_123"
before_clicked: false
after_clicked: true
run_failed: false
leftover_processes: false
read:
skills: true
docs: true
product_code: false
adjacent_attempts: false
prior_store: false
existing_solution: false
attempts:
total: 1
failed: 0
missed_requirements:
- "none"
notes: "ok"
```
"""
report = extract_challenge_report(text)
assert report is not None
assert report["used_product_path"] is True
assert report["before_clicked"] is False
assert report["after_clicked"] is True
def test_classify_challenge_report_success() -> None:
result = classify_challenge_report(_valid_challenge_report())
assert result == "success"
def test_challenge_report_schema_errors_reject_missing_read_block() -> None:
report = _valid_challenge_report()
report.pop("read")
assert challenge_report_schema_errors(report) == ["missing challenge_report.read"]
assert classify_challenge_report(report) == "unknown"
def test_classify_output_prefers_yaml_report() -> None:
result = classify_output(
"""
Some prose that would otherwise be ambiguous.
```yaml
challenge_report:
used_product_path: true
used_helper_script: false
workflow_file: "browser-click.workflow.yaml"
deployment_id: "browser_click_case_study.default"
run_id: "run_123"
before_clicked: false
after_clicked: true
run_failed: false
leftover_processes: false
read:
skills: true
docs: true
product_code: false
adjacent_attempts: false
prior_store: false
existing_solution: false
attempts:
total: 1
failed: 0
missed_requirements:
- "none"
notes: "ok"
```
"""
)
assert result == "success"
def test_classify_challenge_report_detects_helper_script() -> None:
result = classify_challenge_report(
_valid_challenge_report(
used_product_path=False,
used_helper_script=True,
workflow_file="",
)
)
assert result == "workflow_script"
def test_classify_output_workflow_script() -> None:
result = classify_output(
"""
uv run python examples/browser_click_workflow/run_workflow.py
Deployment id: browser_click_case_study.default
Run id: run_123
`before.clicked`: `False`
`after.clicked`: `True`
"""
)
assert result == "workflow_script"
def test_classify_output_workflow_not_used() -> None:
result = classify_output(
"""
I wrote a Playwright script.
before clicked false
after clicked true
"""
)
assert result == "workflow_not_used"
def test_classify_output_run_failed() -> None:
result = classify_output(
"""
wf run start browser_click_case_study.default
error: deployment validation failed
"""
)
assert result == "run_failed"
def test_trial_output_path_is_zero_padded(tmp_path: Path) -> None:
path = trial_output_path(tmp_path, model="opencode/mimo-v2.5-free", index=3)
@@ -509,7 +350,23 @@ def test_prepare_trial_workspace_copies_template_to_model_trial_dir(
source_root = tmp_path / "browser_click_workflow"
source_root.mkdir()
defn = ChallengeDef(
name="browser_click",
source_root=source_root,
source_id="local.browser_click",
source_module="ops",
source_registry="registry",
store_root=".wf_browser_click_store",
default_workspace_template=template,
default_workspaces_dir=workspaces,
default_results_dir=tmp_path / "results",
default_prompt=template / "prompt.md",
default_server_port=8772,
server_config_arg="examples/browser_click_workflow/wf.config.json",
)
prepared = prepare_trial_workspace(
defn,
model="opencode/mimo-v2.5-free",
index=7,
workspaces_dir=workspaces,
@@ -864,12 +721,28 @@ def test_prepare_trial_workspace_uses_next_available_directory(
first.mkdir(parents=True)
stale.write_text("stale", encoding="utf-8")
defn = ChallengeDef(
name="browser_click",
source_root=source_root,
source_id="local.browser_click",
source_module="ops",
source_registry="registry",
store_root=".wf_browser_click_store",
default_workspace_template=template,
default_workspaces_dir=workspaces,
default_results_dir=tmp_path / "results",
default_prompt=template / "prompt.md",
default_server_port=8772,
server_config_arg="examples/browser_click_workflow/wf.config.json",
)
next_index = starting_trial_index(
model="opencode/mimo-v2.5-free",
results_dir=tmp_path / "results",
workspaces_dir=workspaces,
)
prepared = prepare_trial_workspace(
defn,
model="opencode/mimo-v2.5-free",
index=next_index,
workspaces_dir=workspaces,
@@ -915,7 +788,9 @@ def test_main_uses_custom_workspace_template_and_source_root(
monkeypatch.setattr(generic_runner, "run_trial", fake_run_trial)
assert (
run_opencode_trials.main(
generic_runner_main(
BROWSER_CLICK_DEF,
lambda _text: "success",
[
"--model",
"check/model",
@@ -929,7 +804,7 @@ def test_main_uses_custom_workspace_template_and_source_root(
str(workspaces),
"--results-dir",
str(results),
]
],
)
== 0
)
@@ -1044,7 +919,7 @@ def test_generic_workspace_preparation_writes_config_for_arbitrary_challenge_def
server_config_arg="examples/custom/wf.config.json",
)
ws = generic_prepare_trial_workspace(
ws = prepare_trial_workspace(
defn,
model="test-model",
index=1,
@@ -1073,8 +948,8 @@ def test_browser_click_wrapper_produces_expected_paths_and_command_prefix() -> N
assert LOCAL_WF_COMMAND_PREFIX == (
"uv run wf --config examples/browser_click_workflow/wf.config.json --local"
)
assert BROWSER_CLICK_DEF.default_prompt.name == "prompt.md"
assert BROWSER_CLICK_DEF.default_prompt.parent.name == "workspace_template"
assert BROWSER_CLICK_DEF.default_prompt.name == "challenge-prompt.md"
assert BROWSER_CLICK_DEF.default_prompt.parent.name == "browser_click_challenge"
def test_generic_runner_can_be_configured_with_fake_challenge_and_fake_opencode(
@@ -1173,3 +1048,60 @@ def test_generic_write_trial_config_with_custom_source_root(
"module": "test_mod",
"registry": "test_reg",
}
BROWSER_CHALLENGE = (
ROOT
/ "examples"
/ "agent_challenges"
/ "browser_click_challenge"
/ "challenge.yaml"
)
def test_browser_click_manifest_declares_task_success_contract() -> None:
loaded = load_challenge_manifest(BROWSER_CHALLENGE)
assert loaded.manifest.id == "browser_click"
assert loaded.manifest.source.id == "local.browser_click"
assert loaded.manifest.report.success_assertions == {
"before_clicked": False,
"after_clicked": True,
"run_failed": False,
"leftover_processes": False,
}
def test_browser_click_workspace_uses_generic_profile_copy(tmp_path: Path) -> None:
loaded = load_challenge_manifest(BROWSER_CHALLENGE)
bundle = ROOT / "examples/agent_challenges/instruction_bundles/workflow_cli.yaml"
workspace = prepare_v2_trial_workspace(
loaded,
profile=InstructionProfile.SKILLS,
model="opencode/test",
index=1,
workspaces_dir=tmp_path,
instruction_bundle=bundle,
)
assert workspace.config_path.is_file()
assert (workspace.root / ".agent/skills/wf-cli/SKILL.md").is_file()
def test_central_runner_accepts_browser_challenge() -> None:
result = subprocess.run(
[
sys.executable,
"examples/agent_challenges/run_trials.py",
"--help",
],
cwd=ROOT,
text=True,
capture_output=True,
check=False,
)
assert result.returncode == 0, result.stderr
assert "--challenge" in result.stdout
assert "--instruction-profile" in result.stdout
tests/examples/test_report_workflow_challenge.py
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
from pathlib import Path
from examples.agent_challenges.manifests import load_challenge_manifest
ROOT = Path(__file__).resolve().parents[2]
REPORT_CHALLENGE = (
ROOT
/ "examples"
/ "agent_challenges"
/ "report_workflow_challenge"
/ "challenge.yaml"
)
def test_report_workflow_manifest_uses_report_source() -> None:
loaded = load_challenge_manifest(REPORT_CHALLENGE)
assert loaded.manifest.id == "report_workflow"
assert loaded.manifest.source.id == "local.report"
assert loaded.manifest.report.success_assertions == {
"title_matches": True,
"markdown_rendered": True,
"run_failed": False,
}
assert loaded.prompt_path.is_file()
assert loaded.workspace_template.is_dir()
def test_report_challenge_prompt_requires_full_product_lifecycle() -> None:
loaded = load_challenge_manifest(REPORT_CHALLENGE)
prompt = loaded.prompt_path.read_text(encoding="utf-8")
assert "read_notes" in prompt
assert "extract_report" in prompt
assert "render_markdown_report" in prompt
assert "deployment" in prompt.lower()
assert "run_id" in prompt