docs: add thesis citations and challenge appendix

This commit is contained in:
lda
2026-06-16 00:41:40 +07:00 Verified
parent 32da68e05f
commit e24f28929a
4 changed files with 217 additions and 42 deletions
+20
View File
@@ -41,6 +41,8 @@ Evidence:
- `tests/wf_cli/` - `tests/wf_cli/`
- `tests/wf_transport_rpc_http/` - `tests/wf_transport_rpc_http/`
- `docs/wf_cli.md` - `docs/wf_cli.md`
- `examples/agent_challenges/browser_click_challenge/` — challenge harness for
CLI-operability trials; aggregate model results require manual audit.
## Validation And Diagnostics ## Validation And Diagnostics
@@ -80,6 +82,24 @@ Evidence:
- `examples/agent_challenges/browser_click_challenge/` - `examples/agent_challenges/browser_click_challenge/`
- `tests/examples/test_opencode_browser_click_challenge.py` - `tests/examples/test_opencode_browser_click_challenge.py`
## Agent Challenge Evaluation Protocol
Claim: The project has a repeatable protocol for evaluating whether external
agents can use the product-facing CLI lifecycle, but aggregate model results are
not yet claimed.
Evidence:
- `examples/agent_challenges/browser_click_challenge/workspace_template/prompt.md`
— challenge prompt and self-report schema.
- `examples/agent_challenges/browser_click_challenge/run_opencode_trials.py`
trial runner.
- `examples/agent_challenges/browser_click_challenge/classification.py`
convenience classifier for returned reports.
- `examples/agent_challenges/browser_click_challenge/reports.py` — report
extraction and saving.
- `tests/examples/test_opencode_browser_click_challenge.py`
## Limitations ## Limitations
Claim: This is a prototype platform substrate, not a finished automation product. Claim: This is a prototype platform substrate, not a finished automation product.
+1 -1
View File
@@ -36,7 +36,7 @@ $metatempfile = New-TemporaryFile
try { try {
Set-Content -Path $metatempfile -Value $metadata Set-Content -Path $metatempfile -Value $metadata
& $pandoc_diagram --metadata-file=$metatempfile --embed-resources --standalone @RemainingArgs & $pandoc_diagram --metadata-file=$metatempfile --embed-resources --standalone --citeproc @RemainingArgs
} }
finally { finally {
Remove-Item $metatempfile Remove-Item $metatempfile
+72
View File
@@ -0,0 +1,72 @@
@online{mcp-tools-2025,
title = {Tools},
author = {{Model Context Protocol}},
year = {2025},
url = {https://modelcontextprotocol.io/specification/2025-06-18/server/tools},
urldate = {2026-06-16}
}
@online{mcp-lifecycle-2025,
title = {Lifecycle},
author = {{Model Context Protocol}},
year = {2025},
url = {https://modelcontextprotocol.io/specification/2025-03-26/basic/lifecycle},
urldate = {2026-06-16}
}
@online{langgraph-overview-2026,
title = {LangGraph Overview},
author = {{LangChain}},
year = {2026},
url = {https://docs.langchain.com/oss/python/langgraph/overview},
urldate = {2026-06-16}
}
@online{langgraph-persistence-2026,
title = {Persistence},
author = {{LangChain}},
year = {2026},
url = {https://docs.langchain.com/oss/python/langgraph/persistence},
urldate = {2026-06-16}
}
@online{zapier-operating-constraints,
title = {Zapier Operating Constraints},
author = {{Zapier}},
year = {2026},
url = {https://docs.zapier.com/integrations/build/operating-constraints},
urldate = {2026-06-16}
}
@online{zapier-zap-limits,
title = {Zap Limits},
author = {{Zapier}},
year = {2026},
url = {https://help.zapier.com/hc/en-us/articles/8496181445261-Zap-limits},
urldate = {2026-06-16}
}
@online{swebench-verified,
title = {SWE-bench Verified},
author = {{SWE-bench}},
year = {2025},
url = {https://www.swebench.com/verified.html},
urldate = {2026-06-16}
}
@online{nist-agent-cheating-2025,
title = {Examples of Cheating in CAISI's Agent Evaluations},
author = {{National Institute of Standards and Technology}},
year = {2025},
url = {https://www.nist.gov/caisi/cheating-ai-agent-evaluations/2-examples-cheating-caisis-agent-evaluations},
urldate = {2026-06-16}
}
@online{openai-swebench-audit-2026,
title = {Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities},
author = {{OpenAI}},
year = {2026},
url = {https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/},
urldate = {2026-06-16}
}
+124 -41
View File
@@ -10,6 +10,9 @@ fontsize: 10pt
toc: true toc: true
toc-depth: 2 toc-depth: 2
numbersections: true numbersections: true
bibliography: references.bib
link-citations: true
syntax-highlighting: idiomatic
geometry: geometry:
- top=30mm - top=30mm
- bottom=30mm - bottom=30mm
@@ -150,17 +153,19 @@ The design requirements that follow from this problem statement are:
# Positioning And Related Systems # Positioning And Related Systems
The system occupies a specific position in the automation landscape. It does The system occupies a specific position in the automation landscape. It does
not attempt to replace mature platforms in their strengths, but rather not attempt to replace mature platforms in their strengths, but rather explores
explores a different center of gravity: external AI agents can drive the a different center of gravity: external AI agents can drive the authoring and
authoring and execution lifecycle directly through typed contracts. execution lifecycle directly through typed contracts. The comparison below is
qualitative positioning, not a benchmark across products.
## Direct LLM Tool Orchestration ## Direct LLM Tool Orchestration
Direct tool orchestration through an LLM is the most flexible approach. The Direct tool orchestration through an LLM is the most open-ended approach: the
planner can call any tool in any sequence, and adaptation is immediate. However, planner can choose tools dynamically and adapt immediately. In this report's
this flexibility comes at the cost of durability, validation, and resumability. framing, that flexibility becomes a problem when the tool loop is also expected
The platform argues that durable workspace automation benefits from separating to provide durable lifecycle state, validation, audit structure, and
planning from a typed execution substrate. resumability. The platform argues that reusable workspace automation benefits
from separating planning from a typed execution substrate.
## Generated Scripts ## Generated Scripts
@@ -174,31 +179,46 @@ inspection, and repairable diagnostics.
## Workflow Automation Platforms ## Workflow Automation Platforms
Platforms such as Zapier and RPA tools are stronger today at polished Platforms such as Zapier and RPA tools are stronger today at polished
non-programmer UIs, large integration catalogs, hosted scheduling and non-programmer UIs, large integration catalogs, hosted scheduling and triggers,
triggers, and operational maturity. The prototype does not claim feature parity and operational maturity. Zapier's own documentation describes a hosted,
with these products. Instead, it explores a different trade-off: a platform stateless runtime with explicit execution-time and payload constraints, plus
where external AI agents can operate the full lifecycle directly, where local published Zap limits and rate limits. The prototype does not claim feature
Python and MCP sources share one workflow surface, and where artifacts, parity with these products. Instead, it explores a different trade-off: a
deployments, runs, and traces are first-class inspectable records. platform where external AI agents can operate the full lifecycle directly,
where local Python and MCP sources share one workflow surface, and where
artifacts, deployments, runs, and traces are first-class inspectable records.
## Agent Graph Frameworks ## Agent Graph Frameworks
LangGraph-style durable agent graphs share the idea of typed execution LangGraph-style durable agent graphs share the idea of typed execution
substrates for agent workflows. The emphasis in those systems is typically on substrates for agent workflows. LangGraph's official documentation positions it
agent state machines, memory, and tool-call graphs for autonomous agents. The as an orchestration runtime for long-running, stateful agents, with persistence,
lda.chat platform focuses on reusable workspace workflows backed by explicit human-in-the-loop behavior, and durable execution. The lda.chat platform focuses
source providers, where the workflow is a deployable artifact independent of any on reusable workspace workflows backed by explicit source providers, where the
particular agent instance. workflow is a deployable artifact independent of any particular agent instance.
## Model Context Protocol ## Model Context Protocol
MCP is a useful protocol for exposing tools, resources, and prompts. It is not MCP is a useful protocol for exposing tools, resources, and prompts. Its
itself the workflow artifact, deployment, and run lifecycle. The lda.chat official lifecycle is a client-server connection lifecycle: initialization,
platform treats MCP as one source family behind a provider boundary, not as the operation, and shutdown. It is not itself the workflow artifact, deployment, and
product identity. This distinction is important: MCP demonstrates why run lifecycle. The lda.chat platform treats MCP as one source family behind a
source-provider correctness matters, because a source may require persistent provider boundary, not as the product identity. This distinction is important:
sessions, auth context, catalog refresh, and prompt inventory. The platform MCP demonstrates why source-provider correctness matters, because a source may
places this complexity behind a neutral `CapabilitySource` interface. require persistent sessions, auth context, catalog refresh, and prompt
inventory. The platform places this complexity behind a neutral
`CapabilitySource` interface.
External references used for this positioning include the MCP tools and
lifecycle specifications [@mcp-tools-2025; @mcp-lifecycle-2025], LangGraph's
overview and persistence documentation [@langgraph-overview-2026;
@langgraph-persistence-2026], Zapier's operating constraints and Zap limits
[@zapier-operating-constraints; @zapier-zap-limits], and agent evaluation
discussions such as SWE-bench Verified, NIST CAISI's examples of agent
evaluation cheating, and OpenAI's SWE-bench Verified audit
[@swebench-verified; @nist-agent-cheating-2025; @openai-swebench-audit-2026].
These sources contextualize the comparison; the implementation claims in this
report remain grounded in repository evidence.
# Conceptual Model # Conceptual Model
@@ -952,10 +972,10 @@ The evaluation is organized around criteria derived from the research question:
| Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests | | Representation | Can workflow intent be represented as artifacts, deployments, and runs? | model/API tests |
| Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests | | Validation | Can invalid drafts, deployments, source bindings, and source drift be reported before execution? | validation/diagnostic tests |
| Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests | | Runtime observability | Can runtime failures be persisted as failed run records with inspectable error state? | run API tests |
| Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow case study | | Execution | Can a deterministic workflow execute through the same API/CLI lifecycle used by agents? | report-workflow and browser-click case studies |
| Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests | | Persistence | Are lifecycle records persisted, and can stopped/interrupted runs resume at defined boundaries? | run-store and resume tests |
| Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests | | Source extensibility | Can different source families expose capabilities without changing `wf_core`? | built-in, MCP, and Python source tests |
| Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI and JSON-RPC tests | | Agent-operable surface | Can clients drive the lifecycle through structured CLI/API responses? | CLI/JSON-RPC tests and challenge harness |
This is a prototype system evaluation, not a broad user study or reliability This is a prototype system evaluation, not a broad user study or reliability
benchmark. benchmark.
@@ -964,16 +984,18 @@ benchmark.
| Capability | Direct LLM tool loop | Generated script | Mature automation platform | `lda.chat` prototype | | Capability | Direct LLM tool loop | Generated script | Mature automation platform | `lda.chat` prototype |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
| Versioned workflow artifact | Usually no | Manual | Often yes | Yes | | Versioned workflow artifact | Not inherent | Manual | Often yes | Yes |
| Deployment/source binding | Usually no | Manual config | Platform-specific | Yes | | Deployment/source binding | Not inherent | Manual config | Platform-specific | Yes |
| Typed validation before run | Limited | Custom | Varies | Yes | | Typed validation before run | Tool-schema dependent | Custom | Varies | Yes |
| Durable run record | Conversation/log | Custom | Often yes | Yes, at stopped boundaries | | Durable run record | Not inherent | Custom | Often yes | Yes, at stopped boundaries |
| Source drift diagnostics | No | Custom | Varies | Yes, controlled examples | | Source drift diagnostics | Not inherent | Custom | Varies | Yes, controlled examples |
| Agent-operable repair hints | No | No | Usually human UI | Prototype support | | Agent-operable repair hints | Not inherent | Custom | Usually human UI | Prototype support |
| Scheduling | Depends on agent | External scheduler | Yes | Future work | | Scheduling | Depends on agent | External scheduler | Yes | Future work |
The comparison positions the architecture; it is not a quantitative claim that The comparison positions the architecture; it is not a quantitative claim that
the prototype outperforms mature automation products. the prototype outperforms mature automation products. "Not inherent" means the
feature can be added by surrounding infrastructure, but is not provided by the
bare strategy alone.
## Evidence Package ## Evidence Package
@@ -984,6 +1006,8 @@ The evidence supporting the thesis claims includes:
| Deployment validation catches source drift | `tests/artifacts/test_validation.py` | Missing, disabled, or changed capabilities produce diagnostics | Pass in focused test suite | | Deployment validation catches source drift | `tests/artifacts/test_validation.py` | Missing, disabled, or changed capabilities produce diagnostics | Pass in focused test suite |
| Interrupted runs resume at explicit boundaries | `tests/wf_api/test_run_api.py` and resume-concurrency tests | Stopped run state is persisted and resumed through the run API | Pass in focused test suite | | Interrupted runs resume at explicit boundaries | `tests/wf_api/test_run_api.py` and resume-concurrency tests | Stopped run state is persisted and resumed through the run API | Pass in focused test suite |
| Python source lifecycle works | `tests/examples/test_report_workflow_example.py` | Python capability -> artifact -> deployment -> run completes | Pass in focused test suite | | Python source lifecycle works | `tests/examples/test_report_workflow_example.py` | Python capability -> artifact -> deployment -> run completes | Pass in focused test suite |
| Serial multi-node workflow works | `tests/examples/test_browser_click_workflow_example.py` | `open_click_page` -> `wait_for_click` -> `collect_snapshots` completes with before/after evidence | Pass in focused test suite |
| Agent challenge harness exists | `examples/agent_challenges/browser_click_challenge/` | Agents can be prompted, classified, and manually audited against a browser-click workflow challenge | Harness tests pass; aggregate model results pending |
| CLI and JSON-RPC share the API surface | `tests/wf_transport_rpc_http/` and `tests/wf_cli/` | Transport and CLI delegate to the same workflow operations | Pass in focused test suite | | CLI and JSON-RPC share the API surface | `tests/wf_transport_rpc_http/` and `tests/wf_cli/` | Transport and CLI delegate to the same workflow operations | Pass in focused test suite |
The table summarizes repository evidence; it is not a substitute for rerunning The table summarizes repository evidence; it is not a substitute for rerunning
@@ -1037,11 +1061,16 @@ state.
### Python Source Case Study ### Python Source Case Study
The report workflow example demonstrates the source abstraction is not The report workflow example demonstrates the source abstraction is not
MCP-only. A Python source with three typed capabilities exercises the full MCP-only. A Python source with three typed capabilities is loaded and exposed
lifecycle from config validation through run execution. through the source inventory; the automated lifecycle test runs a deterministic
single-node extraction workflow through artifact, deployment, and run records.
The browser-click example complements this with a serial three-node Python
workflow.
(Evidence: `examples/report_workflow/`, (Evidence: `examples/report_workflow/`,
`tests/examples/test_report_workflow_example.py`.) `examples/browser_click_workflow/`,
`tests/examples/test_report_workflow_example.py`,
`tests/examples/test_browser_click_workflow_example.py`.)
### CLI And Transport Tests ### CLI And Transport Tests
@@ -1103,9 +1132,10 @@ The implementation addresses these evaluation questions:
source-provider boundary is in `wf_platform` and `wf_server`, not in the source-provider boundary is in `wf_platform` and `wf_server`, not in the
core. Python sources were added without modifying the core package. core. Python sources were added without modifying the core package.
5. Are large raw provider payloads bounded in CLI output? --- Yes, the 5. Are large raw provider payloads bounded in CLI output? --- Partially. Source
`SOURCE_PREVIEW_LIMIT` constant and compact output formats bound payload inventory previews are bounded by `SOURCE_PREVIEW_LIMIT`; `wf cap call`
size in CLI responses. offers compact/text rendering with `--max-output-chars`. Raw JSON output
remains intentionally lossless.
6. Can platform sources such as `wf.std` be used without self-bindings? --- Yes, 6. Can platform sources such as `wf.std` be used without self-bindings? --- Yes,
platform sources have `binding_required: False` in their source policy, and platform sources have `binding_required: False` in their source policy, and
@@ -1370,3 +1400,56 @@ See [evidence-index.md](evidence-index.md) for the full claim-to-evidence map.
| Validation/diagnostics | `src/wf_artifacts/validation.py`, `src/wf_api/next_actions.py`, `tests/artifacts/test_validation.py` | | Validation/diagnostics | `src/wf_artifacts/validation.py`, `src/wf_api/next_actions.py`, `tests/artifacts/test_validation.py` |
| Agent-operable surface | `src/wf_cli/`, `tests/wf_cli/` | | Agent-operable surface | `src/wf_cli/`, `tests/wf_cli/` |
| Platform domain | `src/wf_api/service.py`, `tests/wf_api/test_artifact_api.py`, `tests/wf_api/test_run_api.py` | | Platform domain | `src/wf_api/service.py`, `tests/wf_api/test_artifact_api.py`, `tests/wf_api/test_run_api.py` |
## Appendix C: Agent Challenge Harness
The browser-click challenge harness is an evaluation instrument for the
agent-operable CLI surface. It asks an external agent to build and successfully
run a workflow that opens a local page with a visible button, records a
before-click snapshot, performs or waits for a click, records an after-click
snapshot, and returns both snapshots from a deployed workflow run.
The harness deliberately evaluates the product-facing lifecycle rather than
general Python programmability. A valid solution uses `uv run wf ...` commands
for artifact creation, deployment saving, and run execution. Importing
`WorkflowApi`, building `WorkflowServer` directly, calling source functions
directly, or solving the task as a standalone browser script is treated as a
bypass even if the visible output is correct.
The current browser-click prompt allows two product-facing authoring paths:
1. **Draft path.** Create a draft from one capability, apply focused draft edits
or an RFC 6902 patch, validate, save, deploy, and run.
2. **Raw-plan path.** Write a raw workflow plan and load it with
`wf artifact create-from-plan`, then deploy and run.
The challenge report is a YAML self-report with fields for product-path use,
helper-script use, workflow file, deployment id, run id, before/after booleans,
read-behavior flags, attempt counts, missed requirements, and notes. The harness
uses that block for automatic convenience classification, but the official
outcome is manual-reviewed. Manual review checks the command transcript, the
workflow file, the run id, the run output or trace, and whether the agent read
product source code, adjacent attempts, prior stores, or existing solutions.
This distinction is intentional. Agent benchmark literature and practice show
that automated scores and self-reports can be misleading when an agent can
inspect hidden answers, prior artifacts, source code, or evaluator state. The
harness therefore records possible invalidation flags such as helper-script
bypass, adjacent-attempt leakage, prior-store reuse, product-code dependency,
false YAML claims, timeouts, parse failures, and missing run evidence.
At the time of this report, the harness and browser-click workflow are
implemented and unit-tested, and preliminary manual trials have already informed
CLI and prompt improvements. The report does not claim aggregate model success
rates, timeout distributions, command counts, or retry-reduction results yet.
Those require repeated, clean-workspace trials across the selected free opencode
models and manual audit of saved trial reports.
Evidence:
- `examples/browser_click_workflow/`
- `examples/agent_challenges/browser_click_challenge/`
- `tests/examples/test_browser_click_workflow_example.py`
- `tests/examples/test_opencode_browser_click_challenge.py`
# References