docs: record agent challenge skill bundle
This commit is contained in:
@@ -212,6 +212,9 @@ stable.
|
||||
prompt templates as the agent instruction layer. Strengthen them before
|
||||
claiming aggregate agent-operability results, and track when trials rely on
|
||||
product code, prior stores, adjacent attempts, or existing example solutions.
|
||||
- Completed: workflow/CLI agent instructions now form an explicit copyable
|
||||
bundle for controlled challenge profiles, use `wf schema` for public shape
|
||||
discovery, and avoid implementation/test-file guidance.
|
||||
|
||||
## Historical References
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
version: 1
|
||||
id: workflow-cli
|
||||
description: Public workflow lifecycle and CLI instructions for agent trials.
|
||||
files:
|
||||
- source: skills/wf-cli/SKILL.md
|
||||
destination: wf-cli/SKILL.md
|
||||
- source: skills/wf-workflow/SKILL.md
|
||||
destination: wf-workflow/SKILL.md
|
||||
- source: skills/wf-workflow/references/system-model.md
|
||||
destination: wf-workflow/references/system-model.md
|
||||
- source: skills/wf-workflow/references/workflow-lifecycle.md
|
||||
destination: wf-workflow/references/workflow-lifecycle.md
|
||||
- source: skills/wf-workflow/references/capabilities-and-wrappers.md
|
||||
destination: wf-workflow/references/capabilities-and-wrappers.md
|
||||
- source: skills/wf-workflow/references/draft-workspaces.md
|
||||
destination: wf-workflow/references/draft-workspaces.md
|
||||
- source: skills/wf-workflow/references/direct-plan-import.md
|
||||
destination: wf-workflow/references/direct-plan-import.md
|
||||
- source: skills/wf-workflow/references/troubleshooting.md
|
||||
destination: wf-workflow/references/troubleshooting.md
|
||||
@@ -53,6 +53,20 @@ wf run start <deployment_id> --input-file input.json
|
||||
wf run trace <run_id> --from 0 --limit 25
|
||||
```
|
||||
|
||||
## Public Discovery Order
|
||||
|
||||
Use public CLI surfaces before broader documentation or implementation search:
|
||||
|
||||
1. `wf status`
|
||||
2. `wf cap list --format ids`
|
||||
3. `wf cap inspect <capability>`
|
||||
4. `wf schema` to list workflow document/component shapes
|
||||
5. `wf schema draft`, `wf schema raw`, or `wf schema <Component>`
|
||||
6. `wf explain <diagnostic-code>` after validation failures
|
||||
|
||||
Use `wf schema <name> --verbose` only when the complete JSON Schema is required;
|
||||
the default compact outline is preferred for agent context.
|
||||
|
||||
## Rules
|
||||
|
||||
- Use explicit `--config <path>` for examples, challenge workspaces, and
|
||||
@@ -74,3 +88,9 @@ wf run trace <run_id> --from 0 --limit 25
|
||||
JSON guidance before authoring.
|
||||
- Add `--verbose` only when a complete JSON Schema document is required; it may
|
||||
be large.
|
||||
- Prefer `wf schema` over searching tests or implementation code for draft/raw
|
||||
plan shape.
|
||||
- Treat compact schema output as authoring guidance; use validation commands as
|
||||
the source of truth for a concrete document.
|
||||
- If public commands and supplied skills are insufficient, report the exact
|
||||
blocker instead of guessing undocumented fields.
|
||||
|
||||
@@ -55,9 +55,17 @@ low-level escape hatch or you already have a complete compiler/generated plan.
|
||||
|
||||
## References
|
||||
|
||||
Read only the reference needed for the current task. Start with a small preview
|
||||
or search hit, then open the relevant section; do not dump every reference into
|
||||
context.
|
||||
Read only the reference needed for the current task:
|
||||
|
||||
- Start with `system-model.md` when lifecycle vocabulary is unclear.
|
||||
- Use `workflow-lifecycle.md` for operation order.
|
||||
- Use `capabilities-and-wrappers.md` before selecting a source capability.
|
||||
- Use `draft-workspaces.md` for iterative editing.
|
||||
- Use `direct-plan-import.md` only when a complete raw plan is required.
|
||||
- Use `troubleshooting.md` after a public validation/run failure.
|
||||
|
||||
Before authoring JSON, query the live public model with `wf schema`; the
|
||||
references explain semantics while the command reflects the current shape.
|
||||
|
||||
Useful patterns:
|
||||
|
||||
|
||||
@@ -7,17 +7,16 @@ unrunnable, or surprising.
|
||||
|
||||
Check in this order:
|
||||
|
||||
1. `wf.admin.list_sources`
|
||||
2. `wf.admin.inspect_source`
|
||||
3. `wf.workflow.list_capabilities`
|
||||
4. `wf.workflow.inspect_capability`
|
||||
1. `wf status`
|
||||
2. `wf cap inspect <capability>`
|
||||
3. `wf cap list --format ids`
|
||||
|
||||
Remember: MCP control tools are not workflow capabilities. They appear in
|
||||
MCP `tools/list`, not `wf.workflow.list_capabilities`.
|
||||
MCP `tools/list`, not `wf cap list`.
|
||||
|
||||
## Unrunnable Deployment
|
||||
|
||||
Run `validate_deployment` before `run_deployment`.
|
||||
Run `wf deploy validate <deployment_id>` before `wf run start`.
|
||||
|
||||
Common diagnostics:
|
||||
|
||||
@@ -27,16 +26,16 @@ Common diagnostics:
|
||||
- `schema_changed`: saved snapshot no longer matches current source.
|
||||
- `source_unreachable`: live check could not contact an upstream source.
|
||||
|
||||
Use `live_check=true` only when you intentionally want to contact upstream
|
||||
sources. It may spawn stdio servers or perform network I/O.
|
||||
Use `wf explain <diagnostic-code>` after validation failures to get
|
||||
human-readable explanations.
|
||||
|
||||
## Run Debugging
|
||||
|
||||
If a run fails:
|
||||
|
||||
1. Read `status`, `error`, `diagnostics`, and `trace_count`.
|
||||
2. Use `inspect_run` for stored summary.
|
||||
3. Use `read_run_trace` with a bounded range.
|
||||
2. Use `wf run inspect <run_id>` for stored summary.
|
||||
3. Use `wf run trace <run_id> --from <n> --limit <n>` with a bounded range.
|
||||
|
||||
Do not request full traces unless the user explicitly asks and the trace is
|
||||
known to be small.
|
||||
@@ -44,5 +43,5 @@ known to be small.
|
||||
## Harness Problems
|
||||
|
||||
Some LLM harnesses do not refresh `tools/list` mid-session. Do not rely on new
|
||||
saved workflows becoming new tools. Use `run_deployment` and `call_capability`
|
||||
saved workflows becoming new tools. Use `wf run start` and `wf cap call`
|
||||
instead.
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
MANIFEST = (
|
||||
ROOT / "examples" / "agent_challenges" / "instruction_bundles" / "workflow_cli.yaml"
|
||||
)
|
||||
|
||||
|
||||
def _bundle() -> dict[str, object]:
|
||||
loaded = yaml.safe_load(MANIFEST.read_text(encoding="utf-8"))
|
||||
assert isinstance(loaded, dict)
|
||||
return loaded
|
||||
|
||||
|
||||
def _entries() -> list[dict[str, str]]:
|
||||
raw = _bundle()["files"]
|
||||
assert isinstance(raw, list)
|
||||
entries: list[dict[str, str]] = []
|
||||
for value in raw:
|
||||
assert isinstance(value, dict)
|
||||
assert isinstance(value.get("source"), str)
|
||||
assert isinstance(value.get("destination"), str)
|
||||
entries.append(value)
|
||||
return entries
|
||||
|
||||
|
||||
def test_workflow_cli_bundle_has_unique_existing_files() -> None:
|
||||
bundle = _bundle()
|
||||
entries = _entries()
|
||||
|
||||
assert bundle["version"] == 1
|
||||
assert bundle["id"] == "workflow-cli"
|
||||
assert len(entries) >= 8
|
||||
sources = [entry["source"] for entry in entries]
|
||||
destinations = [entry["destination"] for entry in entries]
|
||||
assert len(sources) == len(set(sources))
|
||||
assert len(destinations) == len(set(destinations))
|
||||
assert all((ROOT / source).is_file() for source in sources)
|
||||
assert all(not Path(destination).is_absolute() for destination in destinations)
|
||||
assert all(".." not in Path(destination).parts for destination in destinations)
|
||||
|
||||
|
||||
def test_workflow_cli_bundle_uses_public_surfaces_not_implementation_files() -> None:
|
||||
contents = "\n".join(
|
||||
(ROOT / entry["source"]).read_text(encoding="utf-8") for entry in _entries()
|
||||
)
|
||||
|
||||
assert "tests/" not in contents
|
||||
assert "src/" not in contents
|
||||
assert "test_" not in contents
|
||||
assert "wf schema" in contents
|
||||
assert "wf draft validate" in contents
|
||||
assert "wf deploy validate" in contents
|
||||
assert "wf run trace" in contents
|
||||
assert "empty command group" not in contents
|
||||
assert "no schema subcommands" not in contents
|
||||
|
||||
|
||||
def test_bundle_destinations_form_two_skills() -> None:
|
||||
destinations = {entry["destination"] for entry in _entries()}
|
||||
|
||||
assert "wf-cli/SKILL.md" in destinations
|
||||
assert "wf-workflow/SKILL.md" in destinations
|
||||
assert any(path.startswith("wf-workflow/references/") for path in destinations)
|
||||
|
||||
|
||||
def test_workflow_cli_bundle_copies_to_agent_skill_root(tmp_path: Path) -> None:
|
||||
destination_root = tmp_path / ".agent" / "skills"
|
||||
for entry in _entries():
|
||||
source = ROOT / entry["source"]
|
||||
destination = destination_root / entry["destination"]
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(source, destination)
|
||||
|
||||
assert (destination_root / "wf-cli" / "SKILL.md").is_file()
|
||||
assert (destination_root / "wf-workflow" / "SKILL.md").is_file()
|
||||
assert (
|
||||
destination_root / "wf-workflow" / "references" / "direct-plan-import.md"
|
||||
).is_file()
|
||||
Reference in New Issue
Block a user