docs: expand report workflow case study

This commit is contained in:
lda
2026-06-16 15:38:28 +07:00 Verified
parent eeb41413db
commit 21880601fc
5 changed files with 310 additions and 88 deletions
+74 -35
View File
@@ -257,7 +257,7 @@ report remain grounded in repository evidence.
The document uses these terms with specific meanings: The document uses these terms with specific meanings:
| Term | Meaning | Example | | Term | Meaning | Example |
| --- | ----- | --- | | --- | ------ | ---- |
| Workflow capability | A workflow-facing callable operation exposed by a source. | `local.report.extract_report` | | Workflow capability | A workflow-facing callable operation exposed by a source. | `local.report.extract_report` |
| `NodeSpec` | The authoring-layer typed contract produced by decorators or source adapters. | a Python `@node` projection | | `NodeSpec` | The authoring-layer typed contract produced by decorators or source adapters. | a Python `@node` projection |
| `NodeDef` | The core-level serializable node contract: input schema, output schema, and declared outcomes. | a workflow plan node definition | | `NodeDef` | The core-level serializable node contract: input schema, output schema, and declared outcomes. | a workflow plan node definition |
@@ -347,7 +347,7 @@ composition.
Representative sources and source families today: Representative sources and source families today:
| Source | Kind | Role | | Source | Kind | Role |
| --- | --- | --- | | --- | --- | ----- |
| `wf.std` | `system` | Built-in workflow nodes and reducers | | `wf.std` | `system` | Built-in workflow nodes and reducers |
| `wf.source` | `system` | Built-in source resource helper | | `wf.source` | `system` | Built-in source resource helper |
| `wf.recipes` | `system` | First-party workflow recipes | | `wf.recipes` | `system` | First-party workflow recipes |
@@ -598,7 +598,7 @@ provider-specific knowledge.
The implementation is organized into focused packages with clear boundaries: The implementation is organized into focused packages with clear boundaries:
| Package | Responsibility | | Package | Responsibility |
| --- | --------- | | --- | -------- |
| `wf_core` | Deterministic workflow kernel: graph execution, state, outcomes, trace, resume | | `wf_core` | Deterministic workflow kernel: graph execution, state, outcomes, trace, resume |
| `wf_authoring` | Authoring primitives: `NodeSpec`, `WorkflowBuilder`, DSL, reducer authoring, recipes | | `wf_authoring` | Authoring primitives: `NodeSpec`, `WorkflowBuilder`, DSL, reducer authoring, recipes |
| `wf_platform` | Neutral source DTOs, source visibility, permission metadata, and policy | | `wf_platform` | Neutral source DTOs, source visibility, permission metadata, and policy |
@@ -839,11 +839,12 @@ expressiveness. Graph features such as interrupts, foreach, subgraphs, joins,
and reducer behavior are covered by targeted tests and code evidence in the and reducer behavior are covered by targeted tests and code evidence in the
evaluation section. evaluation section.
The thesis-critical automated report-workflow run is intentionally narrow: it The thesis-critical automated report-workflow run executes the full deterministic
executes a single deterministic extraction node through the artifact, report pipeline through the artifact, deployment, and run lifecycle:
deployment, and run lifecycle. The same Python source exposes additional `read_notes -> extract_report -> render_markdown_report`. This keeps the case
capabilities for discovery and extensibility evidence, and the supplemental study small enough to audit while still exercising source discovery, multi-node
browser-click example covers a serial three-node workflow. dataflow, state mapping, artifact saving, deployment binding, run output, and
trace inspection.
## Case Study Components ## Case Study Components
@@ -855,7 +856,9 @@ The example bundle lives at
- `input.md` --- fixture Markdown notes with summary, actions, risks, and - `input.md` --- fixture Markdown notes with summary, actions, risks, and
followups sections. followups sections.
- `cap-input.json` --- a capability-call payload generated from the fixture. - `cap-input.json` --- a capability-call payload generated from the fixture.
- `run-input.json` --- a workflow-run payload generated from the fixture. - `run-input.json` --- a workflow-run payload pointing at the fixture.
- `workflow.plan.json` --- the three-node raw workflow plan used for artifact
creation.
- `wf.config.json` --- a local server and client config using the - `wf.config.json` --- a local server and client config using the
`local.report` Python source. `local.report` Python source.
@@ -946,54 +949,70 @@ demonstrate the agent-operable surface:
wf cap call local.report.extract_report --input-file cap-input.json --format compact wf cap call local.report.extract_report --input-file cap-input.json --format compact
``` ```
6. **Draft creation.** A draft workspace is seeded from the capability's 6. **Draft bootstrap.** A draft workspace can be seeded from a capability's
input/output schemas: input/output schemas:
```powershell ```powershell
wf draft create-from-capability report_ws local.report.extract_report wf draft create-from-capability report_ws local.report.extract_report
``` ```
7. **Draft validation.** The draft is checked for schema conformance and source This is intentionally a best-effort bootstrap, not a complete workflow
synthesizer. It creates an editable one-step draft from wrapper hints.
7. **Focused draft edits.** Common edits can be applied without writing JSON
Patch manually:
```powershell
wf draft set-name report_ws --revision 1 --name report_case_study
wf draft set-input report_ws --revision 2 --step call --map input.text=text
wf draft set-output report_ws --revision 3 --step call --map title=state.title --map summary=state.summary
```
These commands edit existing draft fields. Structural edits, such as adding
the `read_notes` and `render_markdown_report` steps around `extract_report`,
still use `wf draft patch` or the raw-plan import path below.
8. **Draft validation.** The draft is checked for schema conformance and source
availability: availability:
```powershell ```powershell
wf draft validate report_ws wf draft validate report_ws
``` ```
8. **Artifact saving.** The draft becomes an immutable artifact with source 9. **Artifact saving.** The tested case study imports the complete three-node
bindings: plan as an immutable artifact with source bindings:
```powershell ```powershell
wf draft save report_ws --artifact report_case_study --version 1 --title "Report Case Study" --binding local.report=local.report wf artifact create-from-plan workflow.plan.json --artifact report_case_study --version 1 --title "Report Case Study" --outcome ok --binding local.report=local.report
``` ```
9. **Deployment saving.** The artifact is bound into a runnable deployment: 10. **Deployment saving.** The artifact is bound into a runnable deployment:
```powershell ```powershell
wf deploy save report_case_study.default --artifact report_case_study --version 1 --binding local.report=local.report wf deploy save report_case_study.default --artifact report_case_study --version 1 --binding local.report=local.report
``` ```
10. **Deployment validation.** The deployment is checked for bound source 11. **Deployment validation.** The deployment is checked for bound source
availability and compatibility: availability and compatibility:
```powershell ```powershell
wf deploy validate report_case_study.default wf deploy validate report_case_study.default
``` ```
11. **Run execution.** A workflow run starts from the deployment: 12. **Run execution.** A workflow run starts from the deployment:
```powershell ```powershell
wf run start report_case_study.default --input-file run-input.json --trace-from 0 --trace-limit 5 wf run start report_case_study.default --input-file run-input.json --trace-from 0 --trace-limit 5
``` ```
12. **Run inspection.** The run status, output, and diagnostics can be 13. **Run inspection.** The run status, output, and diagnostics can be
inspected: inspected:
```powershell ```powershell
wf run inspect <run_id> wf run inspect <run_id>
``` ```
13. **Run trace.** Trace frames recorded during execution can be listed: 14. **Run trace.** Trace frames recorded during execution can be listed:
```powershell ```powershell
wf run trace <run_id> --from 0 --limit 5 wf run trace <run_id> --from 0 --limit 5
@@ -1009,9 +1028,10 @@ The case study produces a structured report with:
- Three action items with owner, task, and due date - Three action items with owner, task, and due date
- Risks mentioning Google Drive MCP quota - Risks mentioning Google Drive MCP quota
- Followups for Markdown rendering and baseline comparison - Followups for Markdown rendering and baseline comparison
- Rendered Markdown beginning with `# Weekly Project Update`
The output matches the typed `ReportOutput` schema, making validation The workflow output includes both the typed `ReportOutput` object and a
deterministic. Markdown rendering produced by the final node, making validation deterministic.
## Automated Test Evidence ## Automated Test Evidence
@@ -1023,19 +1043,16 @@ programmatically:
fixture input. The test asserts the outcome is `ok`, the title matches, and fixture input. The test asserts the outcome is `ok`, the title matches, and
the action items and risks contain expected values. the action items and risks contain expected values.
2. **Artifact/deployment/run path.** A test builds a workflow plan with a 2. **Artifact/deployment/run path.** A test loads `workflow.plan.json`, which
single `extract` node, saves the artifact, creates a deployment with source runs `read_notes -> extract_report -> render_markdown_report`, saves the
bindings, starts a run, and asserts that the run completes with the expected artifact, creates a deployment with source bindings, starts a run, and
typed output. asserts that the run completes with both structured report output and
rendered Markdown output.
The thesis-critical run path therefore demonstrates the full lifecycle using a The thesis-critical run path therefore demonstrates the full lifecycle using a
single deterministic extraction node. The surrounding Python source includes deterministic three-node pipeline. A supplemental browser-click example remains
additional capabilities used for capability discovery and extensibility supporting evidence for human-interaction-style workflows and before/after
evidence; it is not evaluated as a multi-node read -> extract -> render snapshot outputs.
pipeline in this artifact.
A supplemental browser-click example demonstrates a serial three-node workflow
with bounded before/after snapshots; it is supporting evidence, not the main
case study.
(Evidence: `tests/examples/test_report_workflow_example.py`.) (Evidence: `tests/examples/test_report_workflow_example.py`.)
@@ -1437,14 +1454,32 @@ cap call local.report.extract_report `
--input-file examples/report_workflow/cap-input.json --format compact --input-file examples/report_workflow/cap-input.json --format compact
``` ```
### Draft Creation ### Draft Bootstrap And Focused Edits
`create-from-capability` is a best-effort bootstrap. It creates a one-step
draft from the selected capability's wrapper hints. Focused commands then cover
common edits without requiring the agent to write RFC 6902 patches by hand.
```powershell ```powershell
uv run wf --config examples/report_workflow/wf.config.json ` uv run wf --config examples/report_workflow/wf.config.json `
draft create-from-capability report_ws local.report.extract_report ` draft create-from-capability report_ws local.report.extract_report `
--name report_case_study --title "Report Case Study" --name report_case_study --title "Report Case Study"
uv run wf --config examples/report_workflow/wf.config.json `
draft set-name report_ws --revision 1 --name report_case_study
uv run wf --config examples/report_workflow/wf.config.json `
draft set-input report_ws --revision 2 --step call `
--map input.text=text
uv run wf --config examples/report_workflow/wf.config.json `
draft set-output report_ws --revision 3 --step call `
--map title=state.title --map summary=state.summary
``` ```
Structural edits, such as adding `read_notes` before `extract_report` and
`render_markdown_report` after it, use `draft patch` or a complete raw plan.
### Draft Validation ### Draft Validation
```powershell ```powershell
@@ -1454,10 +1489,14 @@ draft validate report_ws
### Artifact Saving ### Artifact Saving
The tested case-study artifact imports the complete three-node plan:
```powershell ```powershell
uv run wf --config examples/report_workflow/wf.config.json ` uv run wf --config examples/report_workflow/wf.config.json `
draft save report_ws --artifact report_case_study --version 1 ` artifact create-from-plan examples/report_workflow/workflow.plan.json `
--title "Report Case Study" --binding local.report=local.report --artifact report_case_study --version 1 `
--title "Report Case Study" --outcome ok `
--binding local.report=local.report
``` ```
### Deployment Saving ### Deployment Saving
+23 -6
View File
@@ -8,7 +8,9 @@ remote OAuth, LLM calls, or provider quota.
- `input.md` — fixture notes. - `input.md` — fixture notes.
- `cap-input.json` — capability-call payload generated from the fixture notes. - `cap-input.json` — capability-call payload generated from the fixture notes.
- `run-input.json` — workflow-run payload generated from the fixture notes. - `run-input.json` — workflow-run payload pointing at the fixture notes.
- `workflow.plan.json` — three-node raw plan:
`read_notes -> extract_report -> render_markdown_report`.
- `ops.py` — Python source exposing `read_notes`, `extract_report`, and - `ops.py` — Python source exposing `read_notes`, `extract_report`, and
`render_markdown_report`. `render_markdown_report`.
- `wf.config.json` — local server/client config using the `local.report` Python - `wf.config.json` — local server/client config using the `local.report` Python
@@ -32,13 +34,11 @@ uv run wf --config examples/report_workflow/wf.config.json cap call local.report
``` ```
The full artifact/deployment/run path is covered by The full artifact/deployment/run path is covered by
`tests/examples/test_report_workflow_example.py`. To exercise the same lifecycle `tests/examples/test_report_workflow_example.py`. To exercise the same
manually through the CLI, use the source capability as the draft seed: three-node lifecycle manually through the CLI, import the raw plan:
```powershell ```powershell
uv run wf --config examples/report_workflow/wf.config.json draft create-from-capability report_ws local.report.extract_report --name report_case_study --title "Report Case Study" uv run wf --config examples/report_workflow/wf.config.json artifact create-from-plan examples/report_workflow/workflow.plan.json --artifact report_case_study --version 1 --title "Report Case Study" --outcome ok --binding local.report=local.report
uv run wf --config examples/report_workflow/wf.config.json draft validate report_ws
uv run wf --config examples/report_workflow/wf.config.json draft save report_ws --artifact report_case_study --version 1 --title "Report Case Study" --binding local.report=local.report
uv run wf --config examples/report_workflow/wf.config.json deploy save report_case_study.default --artifact report_case_study --version 1 --binding local.report=local.report uv run wf --config examples/report_workflow/wf.config.json deploy save report_case_study.default --artifact report_case_study --version 1 --binding local.report=local.report
uv run wf --config examples/report_workflow/wf.config.json deploy validate report_case_study.default uv run wf --config examples/report_workflow/wf.config.json deploy validate report_case_study.default
uv run wf --config examples/report_workflow/wf.config.json run start report_case_study.default --input-file examples/report_workflow/run-input.json --trace-from 0 --trace-limit 5 uv run wf --config examples/report_workflow/wf.config.json run start report_case_study.default --input-file examples/report_workflow/run-input.json --trace-from 0 --trace-limit 5
@@ -47,12 +47,29 @@ uv run wf --config examples/report_workflow/wf.config.json run inspect <run_id>
uv run wf --config examples/report_workflow/wf.config.json run trace <run_id> --from 0 --limit 5 uv run wf --config examples/report_workflow/wf.config.json run trace <run_id> --from 0 --limit 5
``` ```
Draft workspaces are still useful when an agent starts from one capability and
edits toward a complete workflow:
```powershell
uv run wf --config examples/report_workflow/wf.config.json draft create-from-capability report_ws local.report.extract_report --name report_case_study --title "Report Case Study"
uv run wf --config examples/report_workflow/wf.config.json draft set-name report_ws --revision 1 --name report_case_study
uv run wf --config examples/report_workflow/wf.config.json draft set-input report_ws --revision 2 --step call --map input.text=text
uv run wf --config examples/report_workflow/wf.config.json draft set-output report_ws --revision 3 --step call --map title=state.title --map summary=state.summary
uv run wf --config examples/report_workflow/wf.config.json draft validate report_ws
```
`draft create-from-capability` is a best-effort bootstrapper. Focused commands
cover common edits to existing draft fields; structural edits such as adding the
`read_notes` and `render_markdown_report` steps use `draft patch`, or the raw
plan import shown above.
The expected report includes: The expected report includes:
- title: `Weekly Project Update` - title: `Weekly Project Update`
- three action items - three action items
- at least one risk mentioning Google Drive MCP quota - at least one risk mentioning Google Drive MCP quota
- followups for Markdown rendering and baseline comparison - followups for Markdown rendering and baseline comparison
- rendered Markdown beginning with `# Weekly Project Update`
## Thesis Evidence ## Thesis Evidence
+1 -1
View File
@@ -1,3 +1,3 @@
{ {
"text": "# Weekly Project Update\n\nSummary:\nThe workflow platform demo is ready for a deterministic thesis case study. The\nteam wants a repeatable report that does not depend on remote OAuth, LLM output,\nor provider quotas.\n\nActions:\n- Alice | Prepare demo config | Friday\n- Bao | Run five agent attempts | Monday\n- Casey | Capture trace screenshots | Tuesday\n\nRisks:\n- Google Drive MCP quota is too low for regression evidence\n- Unbounded provider output can waste tokens\n\nFollowups:\n- Add optional Markdown renderer\n- Compare direct script baseline against workflow lifecycle\n" "path": "input.md"
} }
+208
View File
@@ -0,0 +1,208 @@
{
"name": "report_case_study",
"input_schema": {
"type": "object",
"properties": {
"path": {
"type": "string"
}
},
"required": [
"path"
]
},
"state_schema": {
"type": "object",
"properties": {
"notes": {
"type": "string",
"reducer": "wf.std.replace"
},
"report": {
"type": "object",
"reducer": "wf.std.replace"
},
"markdown": {
"type": "string",
"reducer": "wf.std.replace"
}
}
},
"output_schema": {
"type": "object",
"properties": {
"report": {
"type": "object"
},
"markdown": {
"type": "string"
}
},
"required": [
"report",
"markdown"
]
},
"outcomes": [
"ok"
],
"start": "read",
"nodes": [
{
"id": "read",
"type": "node",
"node": "local.report.read_notes",
"input": [
{
"path": {
"root": "input",
"parts": [
"path"
]
},
"target": {
"root": "local",
"parts": [
"path"
]
}
}
],
"output": [
{
"source": {
"root": "local",
"parts": [
"text"
]
},
"target": {
"root": "state",
"parts": [
"notes"
]
}
}
]
},
{
"id": "extract",
"type": "node",
"node": "local.report.extract_report",
"input": [
{
"path": {
"root": "state",
"parts": [
"notes"
]
},
"target": {
"root": "local",
"parts": [
"text"
]
}
}
],
"output": [
{
"source": {
"root": "local",
"parts": []
},
"target": {
"root": "state",
"parts": [
"report"
]
}
}
]
},
{
"id": "render",
"type": "node",
"node": "local.report.render_markdown_report",
"input": [
{
"path": {
"root": "state",
"parts": [
"report"
]
},
"target": {
"root": "local",
"parts": [
"report"
]
}
}
],
"output": [
{
"source": {
"root": "local",
"parts": [
"markdown"
]
},
"target": {
"root": "state",
"parts": [
"markdown"
]
}
}
]
}
],
"edges": [
{
"from": "read",
"outcome": "ok",
"to": "extract"
},
{
"from": "extract",
"outcome": "ok",
"to": "render"
},
{
"from": "render",
"outcome": "ok",
"to": "__end__"
}
],
"output": [
{
"path": {
"root": "state",
"parts": [
"report"
]
},
"target": {
"root": "local",
"parts": [
"report"
]
}
},
{
"path": {
"root": "state",
"parts": [
"markdown"
]
},
"target": {
"root": "local",
"parts": [
"markdown"
]
}
}
]
}
+4 -46
View File
@@ -1,5 +1,6 @@
from __future__ import annotations from __future__ import annotations
import json
from pathlib import Path from pathlib import Path
import pytest import pytest
@@ -76,51 +77,7 @@ async def test_report_workflow_artifact_deployment_run_path(tmp_path) -> None:
config.server.store.root = tmp_path / "store" config.server.store.root = tmp_path / "store"
server = build_workflow_server_from_workflow_config(config) server = build_workflow_server_from_workflow_config(config)
plan = { plan = json.loads((EXAMPLE_DIR / "workflow.plan.json").read_text(encoding="utf-8"))
"name": "report_case_study",
"input_schema": {
"type": "object",
"properties": {"text": {"type": "string"}},
"required": ["text"],
},
"state_schema": {
"type": "object",
"properties": {"report": {"type": "object", "reducer": "wf.std.replace"}},
},
"output_schema": {
"type": "object",
"properties": {"report": {"type": "object"}},
"required": ["report"],
},
"outcomes": ["ok"],
"start": "extract",
"nodes": [
{
"id": "extract",
"type": "node",
"node": "local.report.extract_report",
"input": [
{
"path": {"root": "input", "parts": ["text"]},
"target": {"root": "local", "parts": ["text"]},
}
],
"output": [
{
"source": {"root": "local", "parts": []},
"target": {"root": "state", "parts": ["report"]},
}
],
}
],
"edges": [{"from": "extract", "outcome": "ok", "to": "__end__"}],
"output": [
{
"path": {"root": "state", "parts": ["report"]},
"target": {"root": "local", "parts": ["report"]},
}
],
}
await server.api.create_artifact_from_plan( await server.api.create_artifact_from_plan(
artifact_id="report_case_study", artifact_id="report_case_study",
@@ -140,9 +97,10 @@ async def test_report_workflow_artifact_deployment_run_path(tmp_path) -> None:
) )
run = await server.api.run_deployment( run = await server.api.run_deployment(
deployment_id="report_case_study.default", deployment_id="report_case_study.default",
workflow_input={"text": (EXAMPLE_DIR / "input.md").read_text(encoding="utf-8")}, workflow_input={"path": "input.md"},
) )
assert run["status"] == "completed" assert run["status"] == "completed"
assert run["output"]["report"]["title"] == "Weekly Project Update" assert run["output"]["report"]["title"] == "Weekly Project Update"
assert len(run["output"]["report"]["action_items"]) == 3 assert len(run["output"]["report"]["action_items"]) == 3
assert run["output"]["markdown"].startswith("# Weekly Project Update")