docs: record agent challenge skill bundle

This commit is contained in:
lda
2026-06-22 22:55:39 +07:00 Verified
parent 29b343aeea
commit 3c3cd0e241
7 changed files with 148 additions and 14 deletions
+3
View File
@@ -212,6 +212,9 @@ stable.
prompt templates as the agent instruction layer. Strengthen them before
claiming aggregate agent-operability results, and track when trials rely on
product code, prior stores, adjacent attempts, or existing example solutions.
- Completed: workflow/CLI agent instructions now form an explicit copyable
bundle for controlled challenge profiles, use `wf schema` for public shape
discovery, and avoid implementation/test-file guidance.
## Historical References
@@ -0,0 +1,20 @@
version: 1
id: workflow-cli
description: Public workflow lifecycle and CLI instructions for agent trials.
files:
- source: skills/wf-cli/SKILL.md
destination: wf-cli/SKILL.md
- source: skills/wf-workflow/SKILL.md
destination: wf-workflow/SKILL.md
- source: skills/wf-workflow/references/system-model.md
destination: wf-workflow/references/system-model.md
- source: skills/wf-workflow/references/workflow-lifecycle.md
destination: wf-workflow/references/workflow-lifecycle.md
- source: skills/wf-workflow/references/capabilities-and-wrappers.md
destination: wf-workflow/references/capabilities-and-wrappers.md
- source: skills/wf-workflow/references/draft-workspaces.md
destination: wf-workflow/references/draft-workspaces.md
- source: skills/wf-workflow/references/direct-plan-import.md
destination: wf-workflow/references/direct-plan-import.md
- source: skills/wf-workflow/references/troubleshooting.md
destination: wf-workflow/references/troubleshooting.md
+20
View File
@@ -53,6 +53,20 @@ wf run start <deployment_id> --input-file input.json
wf run trace <run_id> --from 0 --limit 25
```
## Public Discovery Order
Use public CLI surfaces before broader documentation or implementation search:
1. `wf status`
2. `wf cap list --format ids`
3. `wf cap inspect <capability>`
4. `wf schema` to list workflow document/component shapes
5. `wf schema draft`, `wf schema raw`, or `wf schema <Component>`
6. `wf explain <diagnostic-code>` after validation failures
Use `wf schema <name> --verbose` only when the complete JSON Schema is required;
the default compact outline is preferred for agent context.
## Rules
- Use explicit `--config <path>` for examples, challenge workspaces, and
@@ -74,3 +88,9 @@ wf run trace <run_id> --from 0 --limit 25
JSON guidance before authoring.
- Add `--verbose` only when a complete JSON Schema document is required; it may
be large.
- Prefer `wf schema` over searching tests or implementation code for draft/raw
plan shape.
- Treat compact schema output as authoring guidance; use validation commands as
the source of truth for a concrete document.
- If public commands and supplied skills are insufficient, report the exact
blocker instead of guessing undocumented fields.
+11 -3
View File
@@ -55,9 +55,17 @@ low-level escape hatch or you already have a complete compiler/generated plan.
## References
Read only the reference needed for the current task. Start with a small preview
or search hit, then open the relevant section; do not dump every reference into
context.
Read only the reference needed for the current task:
- Start with `system-model.md` when lifecycle vocabulary is unclear.
- Use `workflow-lifecycle.md` for operation order.
- Use `capabilities-and-wrappers.md` before selecting a source capability.
- Use `draft-workspaces.md` for iterative editing.
- Use `direct-plan-import.md` only when a complete raw plan is required.
- Use `troubleshooting.md` after a public validation/run failure.
Before authoring JSON, query the live public model with `wf schema`; the
references explain semantics while the command reflects the current shape.
Useful patterns:
@@ -7,17 +7,16 @@ unrunnable, or surprising.
Check in this order:
1. `wf.admin.list_sources`
2. `wf.admin.inspect_source`
3. `wf.workflow.list_capabilities`
4. `wf.workflow.inspect_capability`
1. `wf status`
2. `wf cap inspect <capability>`
3. `wf cap list --format ids`
Remember: MCP control tools are not workflow capabilities. They appear in
MCP `tools/list`, not `wf.workflow.list_capabilities`.
MCP `tools/list`, not `wf cap list`.
## Unrunnable Deployment
Run `validate_deployment` before `run_deployment`.
Run `wf deploy validate <deployment_id>` before `wf run start`.
Common diagnostics:
@@ -27,16 +26,16 @@ Common diagnostics:
- `schema_changed`: saved snapshot no longer matches current source.
- `source_unreachable`: live check could not contact an upstream source.
Use `live_check=true` only when you intentionally want to contact upstream
sources. It may spawn stdio servers or perform network I/O.
Use `wf explain <diagnostic-code>` after validation failures to get
human-readable explanations.
## Run Debugging
If a run fails:
1. Read `status`, `error`, `diagnostics`, and `trace_count`.
2. Use `inspect_run` for stored summary.
3. Use `read_run_trace` with a bounded range.
2. Use `wf run inspect <run_id>` for stored summary.
3. Use `wf run trace <run_id> --from <n> --limit <n>` with a bounded range.
Do not request full traces unless the user explicitly asks and the trace is
known to be small.
@@ -44,5 +43,5 @@ known to be small.
## Harness Problems
Some LLM harnesses do not refresh `tools/list` mid-session. Do not rely on new
saved workflows becoming new tools. Use `run_deployment` and `call_capability`
saved workflows becoming new tools. Use `wf run start` and `wf cap call`
instead.
@@ -0,0 +1,84 @@
from __future__ import annotations
import shutil
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[2]
MANIFEST = (
ROOT / "examples" / "agent_challenges" / "instruction_bundles" / "workflow_cli.yaml"
)
def _bundle() -> dict[str, object]:
loaded = yaml.safe_load(MANIFEST.read_text(encoding="utf-8"))
assert isinstance(loaded, dict)
return loaded
def _entries() -> list[dict[str, str]]:
raw = _bundle()["files"]
assert isinstance(raw, list)
entries: list[dict[str, str]] = []
for value in raw:
assert isinstance(value, dict)
assert isinstance(value.get("source"), str)
assert isinstance(value.get("destination"), str)
entries.append(value)
return entries
def test_workflow_cli_bundle_has_unique_existing_files() -> None:
bundle = _bundle()
entries = _entries()
assert bundle["version"] == 1
assert bundle["id"] == "workflow-cli"
assert len(entries) >= 8
sources = [entry["source"] for entry in entries]
destinations = [entry["destination"] for entry in entries]
assert len(sources) == len(set(sources))
assert len(destinations) == len(set(destinations))
assert all((ROOT / source).is_file() for source in sources)
assert all(not Path(destination).is_absolute() for destination in destinations)
assert all(".." not in Path(destination).parts for destination in destinations)
def test_workflow_cli_bundle_uses_public_surfaces_not_implementation_files() -> None:
contents = "\n".join(
(ROOT / entry["source"]).read_text(encoding="utf-8") for entry in _entries()
)
assert "tests/" not in contents
assert "src/" not in contents
assert "test_" not in contents
assert "wf schema" in contents
assert "wf draft validate" in contents
assert "wf deploy validate" in contents
assert "wf run trace" in contents
assert "empty command group" not in contents
assert "no schema subcommands" not in contents
def test_bundle_destinations_form_two_skills() -> None:
destinations = {entry["destination"] for entry in _entries()}
assert "wf-cli/SKILL.md" in destinations
assert "wf-workflow/SKILL.md" in destinations
assert any(path.startswith("wf-workflow/references/") for path in destinations)
def test_workflow_cli_bundle_copies_to_agent_skill_root(tmp_path: Path) -> None:
destination_root = tmp_path / ".agent" / "skills"
for entry in _entries():
source = ROOT / entry["source"]
destination = destination_root / entry["destination"]
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, destination)
assert (destination_root / "wf-cli" / "SKILL.md").is_file()
assert (destination_root / "wf-workflow" / "SKILL.md").is_file()
assert (
destination_root / "wf-workflow" / "references" / "direct-plan-import.md"
).is_file()