docs: record staged agent challenge harness
This commit is contained in:
@@ -48,6 +48,12 @@ clear operator feedback before adding more architecture.
|
|||||||
- Completed: `cap call` output is safer for humans through compact/text modes
|
- Completed: `cap call` output is safer for humans through compact/text modes
|
||||||
without changing default JSON semantics. Implementation:
|
without changing default JSON semantics. Implementation:
|
||||||
[`cap call output safety`](historical/superpowers/plans/2026-06-09-cap-call-output-safety.md).
|
[`cap call output safety`](historical/superpowers/plans/2026-06-09-cap-call-output-safety.md).
|
||||||
|
- Completed: raw JSON/YAML workflow plans can be turned into artifacts through
|
||||||
|
JSON-RPC and `wf artifact create-from-plan`, allowing agent/evidence harnesses
|
||||||
|
to use the product-facing CLI path.
|
||||||
|
- Completed: the opencode browser-click challenge harness stages a real
|
||||||
|
`wf-rpc-server`, injects the RPC URL into the prompt, and tears the server
|
||||||
|
down after trials.
|
||||||
- Keep status read-only; do not mutate registry, auth, config, or stores.
|
- Keep status read-only; do not mutate registry, auth, config, or stores.
|
||||||
|
|
||||||
## Priority 2: Durable Run/Resume Hardening
|
## Priority 2: Durable Run/Resume Hardening
|
||||||
|
|||||||
+784
@@ -0,0 +1,784 @@
|
|||||||
|
# RPC Plan Artifact And Staged Agent Harness Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Let remote/CLI users create workflow artifacts from declarative JSON/YAML plans, then make the opencode browser-click challenge stage a real `wf-rpc-server` and give the agent a ready `wf --url` target.
|
||||||
|
|
||||||
|
**Architecture:** Add a protocol-neutral operation already present in local `WorkflowApi` to the JSON-RPC transport and CLI: `workflow.artifacts.create_from_plan` / `wf artifact create-from-plan`. Then update the agent challenge harness so it starts `wf-rpc-server --config examples/browser_click_workflow/wf.config.json` on a free port, waits for `wf status`, injects that URL into the prompt, runs opencode, and tears the server down.
|
||||||
|
|
||||||
|
**Tech Stack:** Python stdlib, Typer, PyYAML, httpx JSON-RPC tests, existing `wf_server`/`wf_transport_rpc_http` test fixtures, opencode CLI harness.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
- Modify `src/wf_transport_rpc_http/models.py`
|
||||||
|
- Add `CreateArtifactFromPlanParams`.
|
||||||
|
- Modify `src/wf_transport_rpc_http/methods/artifacts.py`
|
||||||
|
- Register `workflow.artifacts.create_from_plan`.
|
||||||
|
- Modify `src/wf_transport_rpc_http/client/artifacts.py`
|
||||||
|
- Add `create_artifact_from_plan(...)`.
|
||||||
|
- Modify `src/wf_cli/io.py`
|
||||||
|
- Add `parse_structured_file(path: Path) -> dict[str, Any]` for JSON/YAML object files.
|
||||||
|
- Modify `src/wf_cli/commands/artifacts.py`
|
||||||
|
- Add `wf artifact create-from-plan PLAN_FILE --artifact ...`.
|
||||||
|
- Modify `tests/wf_transport_rpc_http/test_app.py`
|
||||||
|
- Add RPC method test.
|
||||||
|
- Modify `tests/wf_transport_rpc_http/test_client.py`
|
||||||
|
- Add client method test.
|
||||||
|
- Modify `tests/wf_cli/test_artifacts.py`
|
||||||
|
- Add CLI command tests.
|
||||||
|
- Modify `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`
|
||||||
|
- Add staged server lifecycle and prompt rendering.
|
||||||
|
- Modify `examples/agent_challenges/browser_click_challenge/prompt.md`
|
||||||
|
- Add template tokens for `{{rpc_url}}` and `{{wf_command_prefix}}`.
|
||||||
|
- Modify `examples/agent_challenges/browser_click_challenge/README.md`
|
||||||
|
- Document staged server behavior.
|
||||||
|
- Modify `tests/examples/test_opencode_browser_click_challenge.py`
|
||||||
|
- Add server command/prompt rendering tests.
|
||||||
|
- Modify `docs/current_roadmap.md`
|
||||||
|
- Mark both gaps completed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 1: Add JSON-RPC `create_from_plan`
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/wf_transport_rpc_http/models.py`
|
||||||
|
- Modify: `src/wf_transport_rpc_http/methods/artifacts.py`
|
||||||
|
- Modify: `src/wf_transport_rpc_http/client/artifacts.py`
|
||||||
|
- Test: `tests/wf_transport_rpc_http/test_app.py`
|
||||||
|
- Test: `tests/wf_transport_rpc_http/test_client.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add failing RPC app test**
|
||||||
|
|
||||||
|
Add to `tests/wf_transport_rpc_http/test_app.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def test_rpc_create_artifact_from_plan(tmp_path) -> None:
|
||||||
|
server = build_local_static_workflow_server(tmp_path / "store")
|
||||||
|
app = create_rpc_app(server)
|
||||||
|
transport = httpx.ASGITransport(app=app)
|
||||||
|
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||||
|
created = await _rpc(
|
||||||
|
client,
|
||||||
|
"workflow.artifacts.create_from_plan",
|
||||||
|
{
|
||||||
|
"artifact_id": "rpc_plan",
|
||||||
|
"version": 1,
|
||||||
|
"title": "RPC Plan",
|
||||||
|
"plan": _constant_plan().model_dump(mode="json"),
|
||||||
|
"outcomes": ["ok"],
|
||||||
|
"source_bindings": {},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
inspected = await _rpc(
|
||||||
|
client,
|
||||||
|
"workflow.artifacts.inspect",
|
||||||
|
{"artifact_id": "rpc_plan", "version": 1},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert created["result"]["artifact_id"] == "rpc_plan"
|
||||||
|
assert created["result"]["version"] == 1
|
||||||
|
assert inspected["result"]["id"] == "rpc_plan"
|
||||||
|
assert inspected["result"]["plan"]["name"] == "client_constant"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add failing RPC client test**
|
||||||
|
|
||||||
|
Add to `tests/wf_transport_rpc_http/test_client.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def test_rpc_client_creates_artifact_from_plan(tmp_path) -> None:
|
||||||
|
server = build_local_static_workflow_server(tmp_path / "store")
|
||||||
|
app = create_rpc_app(server)
|
||||||
|
transport = httpx.ASGITransport(app=app)
|
||||||
|
async with httpx.AsyncClient(
|
||||||
|
transport=transport,
|
||||||
|
base_url="http://test",
|
||||||
|
) as http_client:
|
||||||
|
client = RpcWorkflowApiClient(
|
||||||
|
url="http://test/rpc",
|
||||||
|
timeout_seconds=5,
|
||||||
|
http_client=http_client,
|
||||||
|
)
|
||||||
|
created = await client.create_artifact_from_plan(
|
||||||
|
artifact_id="client_plan",
|
||||||
|
version=1,
|
||||||
|
title="Client Plan",
|
||||||
|
plan=_constant_plan().model_dump(mode="json"),
|
||||||
|
outcomes=("ok",),
|
||||||
|
source_bindings={},
|
||||||
|
)
|
||||||
|
inspected = await client.inspect_artifact(
|
||||||
|
artifact_id="client_plan",
|
||||||
|
version=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert created["artifact_id"] == "client_plan"
|
||||||
|
assert inspected["id"] == "client_plan"
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Run tests to verify failure**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/wf_transport_rpc_http/test_app.py::test_rpc_create_artifact_from_plan tests/wf_transport_rpc_http/test_client.py::test_rpc_client_creates_artifact_from_plan -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: FAIL because `workflow.artifacts.create_from_plan` and the client method do not exist.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add params model**
|
||||||
|
|
||||||
|
In `src/wf_transport_rpc_http/models.py`, add after `SaveArtifactParams`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class CreateArtifactFromPlanParams(RpcParamsModel):
|
||||||
|
artifact_id: str = Field(min_length=1)
|
||||||
|
version: int = Field(ge=1)
|
||||||
|
title: str = Field(min_length=1)
|
||||||
|
plan: dict[str, Any]
|
||||||
|
outcomes: list[str]
|
||||||
|
kind: Literal["workflow", "wrapper"] = "workflow"
|
||||||
|
description: str | None = None
|
||||||
|
required_capabilities: dict[str, dict[str, Any]] | None = None
|
||||||
|
source_bindings: dict[str, str] | None = None
|
||||||
|
created_from_catalog_version: str | None = None
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Register RPC method**
|
||||||
|
|
||||||
|
In `src/wf_transport_rpc_http/methods/artifacts.py`, import `CreateArtifactFromPlanParams` and add before `workflow.artifacts.save`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@entrypoint.method(
|
||||||
|
name="workflow.artifacts.create_from_plan",
|
||||||
|
errors=[WorkflowRpcError],
|
||||||
|
)
|
||||||
|
async def workflow_artifacts_create_from_plan(
|
||||||
|
params: CreateArtifactFromPlanParams = RpcParams(),
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
return await server.api.create_artifact_from_plan(
|
||||||
|
artifact_id=params.artifact_id,
|
||||||
|
version=params.version,
|
||||||
|
title=params.title,
|
||||||
|
plan=params.plan,
|
||||||
|
outcomes=tuple(params.outcomes),
|
||||||
|
kind=params.kind,
|
||||||
|
description=params.description,
|
||||||
|
required_capabilities=params.required_capabilities,
|
||||||
|
source_bindings=params.source_bindings,
|
||||||
|
created_from_catalog_version=params.created_from_catalog_version,
|
||||||
|
)
|
||||||
|
except (ValueError, KeyError, LookupError, FileNotFoundError) as exc:
|
||||||
|
raise_workflow_rpc_error(exc)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Add RPC client method**
|
||||||
|
|
||||||
|
In `src/wf_transport_rpc_http/client/artifacts.py`, import `Sequence` and add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_artifact_from_plan(
|
||||||
|
self: RpcCaller,
|
||||||
|
*,
|
||||||
|
artifact_id: str,
|
||||||
|
version: int,
|
||||||
|
title: str,
|
||||||
|
plan: dict[str, Any],
|
||||||
|
outcomes: Sequence[str],
|
||||||
|
kind: Literal["workflow", "wrapper"] = "workflow",
|
||||||
|
description: str | None = None,
|
||||||
|
required_capabilities: dict[str, dict[str, Any]] | None = None,
|
||||||
|
source_bindings: dict[str, str] | None = None,
|
||||||
|
created_from_catalog_version: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
return await self._call(
|
||||||
|
"workflow.artifacts.create_from_plan",
|
||||||
|
{
|
||||||
|
"artifact_id": artifact_id,
|
||||||
|
"version": version,
|
||||||
|
"title": title,
|
||||||
|
"plan": plan,
|
||||||
|
"outcomes": list(outcomes),
|
||||||
|
"kind": kind,
|
||||||
|
"description": description,
|
||||||
|
"required_capabilities": required_capabilities,
|
||||||
|
"source_bindings": source_bindings,
|
||||||
|
"created_from_catalog_version": created_from_catalog_version,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run tests**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/wf_transport_rpc_http/test_app.py::test_rpc_create_artifact_from_plan tests/wf_transport_rpc_http/test_client.py::test_rpc_client_creates_artifact_from_plan -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git add src\wf_transport_rpc_http\models.py src\wf_transport_rpc_http\methods\artifacts.py src\wf_transport_rpc_http\client\artifacts.py tests\wf_transport_rpc_http\test_app.py tests\wf_transport_rpc_http\test_client.py
|
||||||
|
git commit -m "feat: expose create artifact from plan over rpc"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 2: Add `wf artifact create-from-plan`
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/wf_cli/io.py`
|
||||||
|
- Modify: `src/wf_cli/commands/artifacts.py`
|
||||||
|
- Test: `tests/wf_cli/test_artifacts.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add failing CLI tests**
|
||||||
|
|
||||||
|
Extend `_ArtifactHandlers` in `tests/wf_cli/test_artifacts.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def create_artifact_from_plan(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
artifact_id: str,
|
||||||
|
version: int,
|
||||||
|
title: str,
|
||||||
|
plan: dict[str, Any],
|
||||||
|
outcomes: tuple[str, ...],
|
||||||
|
kind: str = "workflow",
|
||||||
|
description: str | None = None,
|
||||||
|
required_capabilities: dict[str, dict[str, Any]] | None = None,
|
||||||
|
source_bindings: dict[str, str] | None = None,
|
||||||
|
created_from_catalog_version: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
self.calls.append((artifact_id, version))
|
||||||
|
return {
|
||||||
|
"artifact_id": artifact_id,
|
||||||
|
"version": version,
|
||||||
|
"title": title,
|
||||||
|
"plan_name": plan["name"],
|
||||||
|
"outcomes": list(outcomes),
|
||||||
|
"kind": kind,
|
||||||
|
"source_bindings": source_bindings,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add tests:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_artifact_create_from_plan_reads_yaml(monkeypatch, tmp_path) -> None:
|
||||||
|
handlers = _ArtifactHandlers()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"wf_cli.commands.artifacts.load_cli_context",
|
||||||
|
lambda _ctx: _Context(handlers=handlers),
|
||||||
|
)
|
||||||
|
plan_file = tmp_path / "plan.yaml"
|
||||||
|
plan_file.write_text(
|
||||||
|
"""
|
||||||
|
name: yaml_plan
|
||||||
|
input_schema: {type: object, properties: {}}
|
||||||
|
state_schema: {type: object, properties: {}}
|
||||||
|
output_schema: {type: object, properties: {}}
|
||||||
|
outcomes: [ok]
|
||||||
|
start: __end__
|
||||||
|
nodes: []
|
||||||
|
edges: []
|
||||||
|
""",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
result = CliRunner().invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"artifact",
|
||||||
|
"create-from-plan",
|
||||||
|
str(plan_file),
|
||||||
|
"--artifact",
|
||||||
|
"yaml_artifact",
|
||||||
|
"--version",
|
||||||
|
"1",
|
||||||
|
"--title",
|
||||||
|
"YAML Artifact",
|
||||||
|
"--outcome",
|
||||||
|
"ok",
|
||||||
|
"--binding",
|
||||||
|
"local.ops=local.ops",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.exit_code == 0, result.output
|
||||||
|
payload = json.loads(result.output)
|
||||||
|
assert payload["artifact_id"] == "yaml_artifact"
|
||||||
|
assert payload["plan_name"] == "yaml_plan"
|
||||||
|
assert payload["source_bindings"] == {"local.ops": "local.ops"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_artifact_create_from_plan_rejects_non_object_yaml(monkeypatch, tmp_path) -> None:
|
||||||
|
handlers = _ArtifactHandlers()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"wf_cli.commands.artifacts.load_cli_context",
|
||||||
|
lambda _ctx: _Context(handlers=handlers),
|
||||||
|
)
|
||||||
|
plan_file = tmp_path / "plan.yaml"
|
||||||
|
plan_file.write_text("- not\n- object\n", encoding="utf-8")
|
||||||
|
|
||||||
|
result = CliRunner().invoke(
|
||||||
|
app,
|
||||||
|
[
|
||||||
|
"artifact",
|
||||||
|
"create-from-plan",
|
||||||
|
str(plan_file),
|
||||||
|
"--artifact",
|
||||||
|
"bad",
|
||||||
|
"--version",
|
||||||
|
"1",
|
||||||
|
"--title",
|
||||||
|
"Bad",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.exit_code != 0
|
||||||
|
assert "structured file must contain an object" in result.output.lower()
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify failure**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/wf_cli/test_artifacts.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: FAIL because the command/helper do not exist.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add structured file parser**
|
||||||
|
|
||||||
|
In `src/wf_cli/io.py`, import `yaml` and add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def parse_structured_file(path: Path) -> dict[str, Any]:
|
||||||
|
"""Parse one JSON/YAML object file for declarative workflow inputs."""
|
||||||
|
try:
|
||||||
|
raw = path.read_text(encoding="utf-8")
|
||||||
|
except OSError as exc:
|
||||||
|
raise CliInputError(f"could not read file {path!s}: {exc}") from exc
|
||||||
|
try:
|
||||||
|
payload = yaml.safe_load(raw)
|
||||||
|
except yaml.YAMLError as exc:
|
||||||
|
raise CliInputError(f"invalid YAML/JSON file {path!s}: {exc}") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise CliInputError("structured file must contain an object")
|
||||||
|
return payload
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add CLI command**
|
||||||
|
|
||||||
|
In `src/wf_cli/commands/artifacts.py`, import `Path`, `parse_structured_file`, and add before `delete`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@app.command("create-from-plan")
|
||||||
|
def create_artifact_from_plan(
|
||||||
|
ctx: typer.Context,
|
||||||
|
plan_file: Annotated[Path, typer.Argument(exists=True, dir_okay=False)],
|
||||||
|
artifact_id: Annotated[str, typer.Option("--artifact", help="Artifact id.")],
|
||||||
|
version: Annotated[int, typer.Option("--version", min=1, help="Artifact version.")],
|
||||||
|
title: Annotated[str, typer.Option("--title", help="Artifact title.")],
|
||||||
|
outcome: Annotated[
|
||||||
|
list[str] | None,
|
||||||
|
typer.Option("--outcome", help="Artifact outcome. Repeatable."),
|
||||||
|
] = None,
|
||||||
|
kind: Annotated[
|
||||||
|
Literal["workflow", "wrapper"], typer.Option("--kind", help="Artifact kind.")
|
||||||
|
] = "workflow",
|
||||||
|
description: Annotated[
|
||||||
|
str | None, typer.Option("--description", help="Artifact description.")
|
||||||
|
] = None,
|
||||||
|
binding: Annotated[
|
||||||
|
list[str] | None,
|
||||||
|
typer.Option("--binding", help="Logical=concrete source binding. Repeatable."),
|
||||||
|
] = None,
|
||||||
|
) -> None:
|
||||||
|
"""Create an artifact from a raw JSON/YAML workflow plan file."""
|
||||||
|
try:
|
||||||
|
plan = parse_structured_file(plan_file)
|
||||||
|
source_bindings = parse_bindings(binding or [])
|
||||||
|
except CliInputError as exc:
|
||||||
|
raise typer.BadParameter(str(exc)) from exc
|
||||||
|
context = load_cli_context(ctx)
|
||||||
|
emit_json(
|
||||||
|
run_cli_operation(
|
||||||
|
context,
|
||||||
|
context.handlers.create_artifact_from_plan(
|
||||||
|
artifact_id=artifact_id,
|
||||||
|
version=version,
|
||||||
|
title=title,
|
||||||
|
plan=plan,
|
||||||
|
outcomes=tuple(outcome or ["ok"]),
|
||||||
|
kind=kind,
|
||||||
|
description=description,
|
||||||
|
source_bindings=source_bindings or None,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run tests**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/wf_cli/test_artifacts.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git add src\wf_cli\io.py src\wf_cli\commands\artifacts.py tests\wf_cli\test_artifacts.py
|
||||||
|
git commit -m "feat: add artifact create-from-plan cli"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 3: Stage RPC Server In Opencode Harness
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`
|
||||||
|
- Modify: `examples/agent_challenges/browser_click_challenge/prompt.md`
|
||||||
|
- Modify: `examples/agent_challenges/browser_click_challenge/README.md`
|
||||||
|
- Test: `tests/examples/test_opencode_browser_click_challenge.py`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add failing harness tests**
|
||||||
|
|
||||||
|
Add imports in `tests/examples/test_opencode_browser_click_challenge.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from examples.agent_challenges.browser_click_challenge.run_opencode_trials import (
|
||||||
|
render_prompt,
|
||||||
|
server_command,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Add tests:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_render_prompt_injects_rpc_url_and_command_prefix(tmp_path: Path) -> None:
|
||||||
|
prompt = tmp_path / "prompt.md"
|
||||||
|
prompt.write_text(
|
||||||
|
"Use {{rpc_url}} with {{wf_command_prefix}}",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
rendered = render_prompt(
|
||||||
|
prompt,
|
||||||
|
rpc_url="http://127.0.0.1:8765/rpc",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "http://127.0.0.1:8765/rpc" in rendered
|
||||||
|
assert "uv run wf --url http://127.0.0.1:8765/rpc" in rendered
|
||||||
|
|
||||||
|
|
||||||
|
def test_server_command_uses_example_config_and_requested_port() -> None:
|
||||||
|
command = server_command(port=8765)
|
||||||
|
|
||||||
|
assert command[:3] == ["uv", "run", "wf-rpc-server"]
|
||||||
|
assert "--config" in command
|
||||||
|
assert "examples/browser_click_workflow/wf.config.json" in command
|
||||||
|
assert "--port" in command
|
||||||
|
assert "8765" in command
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify failure**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/examples/test_opencode_browser_click_challenge.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: FAIL because `render_prompt` and `server_command` do not exist.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add prompt template tokens**
|
||||||
|
|
||||||
|
In `examples/agent_challenges/browser_click_challenge/prompt.md`, replace the product-path paragraph with:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
A workflow RPC server is already running at:
|
||||||
|
|
||||||
|
```text
|
||||||
|
{{rpc_url}}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use this command prefix for product-facing operations:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
{{wf_command_prefix}}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use this repository's workflow product path. That means you should use the
|
||||||
|
`wf` CLI against the provided URL, create or reuse a workflow deployment, and
|
||||||
|
run the deployment through the workflow API. Do not solve the challenge with
|
||||||
|
only a standalone Playwright/Python script.
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Implement prompt rendering and server command**
|
||||||
|
|
||||||
|
In `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`, add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
DEFAULT_SERVER_PORT = 8772
|
||||||
|
EXAMPLE_CONFIG = ROOT / "examples" / "browser_click_workflow" / "wf.config.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def rpc_url_for_port(port: int) -> str:
|
||||||
|
return f"http://127.0.0.1:{port}/rpc"
|
||||||
|
|
||||||
|
|
||||||
|
def server_command(*, port: int) -> list[str]:
|
||||||
|
return [
|
||||||
|
"uv",
|
||||||
|
"run",
|
||||||
|
"wf-rpc-server",
|
||||||
|
"--config",
|
||||||
|
str(EXAMPLE_CONFIG.relative_to(ROOT)).replace("\\", "/"),
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--port",
|
||||||
|
str(port),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def render_prompt(prompt_path: Path, *, rpc_url: str) -> str:
|
||||||
|
command_prefix = f"uv run wf --url {rpc_url}"
|
||||||
|
return (
|
||||||
|
prompt_path.read_text(encoding="utf-8")
|
||||||
|
.replace("{{rpc_url}}", rpc_url)
|
||||||
|
.replace("{{wf_command_prefix}}", command_prefix)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Change `build_opencode_command` to call `render_prompt(config.prompt_path, rpc_url=config.rpc_url)`. Add `rpc_url: str` to `TrialConfig`.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add managed server lifecycle**
|
||||||
|
|
||||||
|
Add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class ManagedServer:
|
||||||
|
process: subprocess.Popen[str]
|
||||||
|
rpc_url: str
|
||||||
|
|
||||||
|
|
||||||
|
def start_server(*, port: int, timeout_seconds: int = 30) -> ManagedServer:
|
||||||
|
command = server_command(port=port)
|
||||||
|
process = subprocess.Popen(
|
||||||
|
command,
|
||||||
|
cwd=ROOT,
|
||||||
|
text=True,
|
||||||
|
stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL,
|
||||||
|
)
|
||||||
|
rpc_url = rpc_url_for_port(port)
|
||||||
|
try:
|
||||||
|
wait_for_status(rpc_url=rpc_url, timeout_seconds=timeout_seconds)
|
||||||
|
except Exception:
|
||||||
|
stop_server(process)
|
||||||
|
raise
|
||||||
|
return ManagedServer(process=process, rpc_url=rpc_url)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_status(*, rpc_url: str, timeout_seconds: int) -> None:
|
||||||
|
deadline = time.monotonic() + timeout_seconds
|
||||||
|
command = ["uv", "run", "wf", "--url", rpc_url, "status"]
|
||||||
|
last_stderr = ""
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
completed = subprocess.run(
|
||||||
|
command,
|
||||||
|
cwd=ROOT,
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
if completed.returncode == 0:
|
||||||
|
return
|
||||||
|
last_stderr = completed.stderr
|
||||||
|
time.sleep(0.5)
|
||||||
|
raise RuntimeError(f"wf status did not become ready: {last_stderr}")
|
||||||
|
|
||||||
|
|
||||||
|
def stop_server(process: subprocess.Popen[str]) -> None:
|
||||||
|
if process.poll() is not None:
|
||||||
|
return
|
||||||
|
if sys.platform == "win32":
|
||||||
|
subprocess.run(
|
||||||
|
["taskkill", "/F", "/T", "/PID", str(process.pid)],
|
||||||
|
stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
process.kill()
|
||||||
|
process.wait(timeout=10)
|
||||||
|
```
|
||||||
|
|
||||||
|
Import `sys`. Add CLI flags:
|
||||||
|
|
||||||
|
```python
|
||||||
|
parser.add_argument("--server-url", default=None)
|
||||||
|
parser.add_argument("--start-server", action="store_true", default=True)
|
||||||
|
parser.add_argument("--no-start-server", action="store_false", dest="start_server")
|
||||||
|
parser.add_argument("--server-port", type=int, default=DEFAULT_SERVER_PORT)
|
||||||
|
```
|
||||||
|
|
||||||
|
In `main`, if `args.server_url` is set, use it. Else if `args.start_server`, call `start_server(port=args.server_port)` and stop it in `finally`. Else use `rpc_url_for_port(args.server_port)` and do not start/stop.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Update README**
|
||||||
|
|
||||||
|
Document default behavior:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
By default the harness starts:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run wf-rpc-server --config examples/browser_click_workflow/wf.config.json --host 127.0.0.1 --port 8772
|
||||||
|
```
|
||||||
|
|
||||||
|
It waits until:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run wf --url http://127.0.0.1:8772/rpc status
|
||||||
|
```
|
||||||
|
|
||||||
|
passes, injects that URL into the prompt, runs opencode, then stops the server.
|
||||||
|
Use `--server-url` to target an already-running server or `--no-start-server` to
|
||||||
|
skip lifecycle management.
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run tests**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/examples/test_opencode_browser_click_challenge.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git add examples\agent_challenges\browser_click_challenge\run_opencode_trials.py examples\agent_challenges\browser_click_challenge\prompt.md examples\agent_challenges\browser_click_challenge\README.md tests\examples\test_opencode_browser_click_challenge.py
|
||||||
|
git commit -m "feat: stage rpc server for opencode challenge"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task 4: Docs And Final Verification
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `docs/current_roadmap.md`
|
||||||
|
- Move: this plan to `docs/historical/superpowers/plans/2026-06-15-rpc-plan-artifact-and-staged-agent-harness.md`
|
||||||
|
|
||||||
|
- [ ] **Step 1: Update roadmap**
|
||||||
|
|
||||||
|
Add under Priority 1:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
- Completed: raw JSON/YAML workflow plans can be turned into artifacts through
|
||||||
|
JSON-RPC and `wf artifact create-from-plan`, allowing agent/evidence harnesses
|
||||||
|
to use the product-facing CLI path.
|
||||||
|
- Completed: the opencode browser-click challenge harness stages a real
|
||||||
|
`wf-rpc-server`, injects the RPC URL into the prompt, and tears the server
|
||||||
|
down after trials.
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Archive plan**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
Move-Item -LiteralPath docs\superpowers\plans\2026-06-15-rpc-plan-artifact-and-staged-agent-harness.md -Destination docs\historical\superpowers\plans\2026-06-15-rpc-plan-artifact-and-staged-agent-harness.md
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Final verification**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run pytest tests/wf_transport_rpc_http/test_app.py::test_rpc_create_artifact_from_plan tests/wf_transport_rpc_http/test_client.py::test_rpc_client_creates_artifact_from_plan tests/wf_cli/test_artifacts.py tests/examples/test_opencode_browser_click_challenge.py tests/docs -q
|
||||||
|
uv run ruff check src/wf_transport_rpc_http/models.py src/wf_transport_rpc_http/methods/artifacts.py src/wf_transport_rpc_http/client/artifacts.py src/wf_cli/io.py src/wf_cli/commands/artifacts.py examples/agent_challenges/browser_click_challenge tests/wf_cli/test_artifacts.py tests/examples/test_opencode_browser_click_challenge.py tests/docs
|
||||||
|
uv run basedpyright --level error src/wf_transport_rpc_http/models.py src/wf_transport_rpc_http/methods/artifacts.py src/wf_transport_rpc_http/client/artifacts.py src/wf_cli/io.py src/wf_cli/commands/artifacts.py examples/agent_challenges/browser_click_challenge tests/wf_cli/test_artifacts.py tests/examples/test_opencode_browser_click_challenge.py tests/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
- pytest passes.
|
||||||
|
- ruff passes.
|
||||||
|
- basedpyright reports 0 errors.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Commit**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git add docs\current_roadmap.md docs\historical\superpowers\plans\2026-06-15-rpc-plan-artifact-and-staged-agent-harness.md
|
||||||
|
git commit -m "docs: record staged agent challenge harness"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Manual Trial After Implementation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
uv run python examples/agent_challenges/browser_click_challenge/run_opencode_trials.py `
|
||||||
|
--model opencode/mimo-v2.5-free `
|
||||||
|
--variant high `
|
||||||
|
--trials 1 `
|
||||||
|
--timeout-seconds 900
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
- Harness starts `wf-rpc-server`.
|
||||||
|
- Prompt includes `uv run wf --url http://127.0.0.1:8772/rpc`.
|
||||||
|
- Agent final answer includes YAML `challenge_report`.
|
||||||
|
- Trial JSON classification is one of `success`, `workflow_script`, `run_failed`, `workflow_not_used`, `parse_error`, `timeout`, or `unknown`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review Checklist
|
||||||
|
|
||||||
|
- JSON-RPC exposes the same artifact-from-plan operation local `WorkflowApi` already has.
|
||||||
|
- CLI supports JSON and YAML plan files.
|
||||||
|
- Harness does not require the agent to start the server.
|
||||||
|
- Harness prompt includes the actual RPC URL.
|
||||||
|
- Harness still supports `--server-url` for manually staged servers.
|
||||||
|
- Tests do not invoke opencode.
|
||||||
|
- Tests do not depend on a live network server except through local ASGI transports.
|
||||||
Reference in New Issue
Block a user