lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: show verified diagnostic and result output in thesis walkthrough

Browse Source
This commit is contained in:
lda
2026-09-14 03:29:18 +07:00 Verified
parent 92642fc218
commit 4268c69877
2 changed files with 96 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/thesis/system-design-implementation.md
+86 -57
View File
@@ -8,6 +8,7 @@ papersize: a4
fontsize: 10pt
toc: true
toc-depth: 2
toc-title: "Table of Contents"
lof: true
lot: true
numbersections: true
@@ -176,7 +177,7 @@ assistant backend can execute code in persistent Python shells. Its workflow
authoring integration and developing chat interface are discussed in Future
Work, outside the evaluated workflow case study.
## Report Outline
## Report outline
Chapter 2 derives interaction and execution requirements from the workspace
task. Chapter 3 compares ways to author and operate that task in related
@@ -388,7 +389,7 @@ decisions need a dedicated condition step or an additional declared outcome.
The relevant comparison is where authors express and inspect decisions, not
whether Python or a canvas is inherently the better interface.
## Saved Definitions, Configuration, and Execution Records
## Saved definitions, configuration, and execution records
Lifecycle separation is not unique to the prototype. n8n distinguishes saved
edits from the published version used for production execution, and separates
@@ -652,7 +653,7 @@ executed nodes and their outcomes, helping the operator locate a failed step
or follow a decision through the graph. The implementation chapters explain
how the runtime records this information.
## Working Glossary
## Working glossary
[@tbl:working-glossary] summarizes the core terms through the report example.
@@ -688,7 +689,7 @@ saved definitions versus individual executions, and workflow execution versus
provider-specific calls. These boundaries are visible in the user-facing
objects as well as in the implementation.
## From Authoring to Server Operations
## From authoring to server operations
The Python client is the main programmatic authoring interface. Its `App`
object represents a connection to the workflow service. An author can inspect
@@ -749,7 +750,7 @@ An agent can author or operate a workflow through these interfaces. During a
run, the runtime follows the saved graph. Agent participation within that
execution requires an authored operation that invokes the agent.
## Keeping a Definition Separate from Its Use
## Keeping a definition separate from its use
The client owns the editable graph. The service owns saved artifact,
deployment, and run records. The API coordinates validation and persistence
@@ -772,7 +773,7 @@ unfinished definitions between requests. Python authoring can save directly
from its editable object. Both routes produce a saved definition for deployment
and execution.
## Executing the Graph
## Executing the graph
The execution core is the runtime that runs the saved graph. It holds the
workflow input, working state, and current execution positions. To execute
@@ -832,7 +833,7 @@ concurrent completion order can vary between runs. The runtime's defined
routing and state-update rules should not be confused with reproducibility
of every operation it invokes.
## Iteration and Child Workflow Boundaries
## Iteration and child workflow boundaries
Suppose the report now covers two documents, `A.md` and `B.md`. The
workflow must render a report for each, then assemble the two reports.
@@ -895,7 +896,7 @@ inside and outside the body, or returning past the immediate enclosing loop.
Ordinary cycles within one region remain valid, subject to the run-wide
node-execution limit described in the implementation chapter.
### Calling a Child Workflow for One Document
### Calling a child workflow for one document
Now replace the rendering work with a call to a saved child workflow that
summarizes one document. Consider only the item for `A.md`. The parent may
@@ -934,7 +935,7 @@ The two completion points are separate:
child `END` returns to the calling node, and the calling node's route back
to foreach finishes the item.
### When Items Run Concurrently
### When items run concurrently
In concurrent mode, A and B may be in progress together. A returning from
its child does not permit assembly while B is still running. The foreach
@@ -956,7 +957,7 @@ If a child or item reaches an explicit interrupt, the run preserves which
document was active and where it was waiting. Resume applies the response
to that saved position and continues the same item.
## Connecting External Operations
## Connecting external operations
Providers adapt external functionality to contracts the workflow system can
inspect and invoke. Discovery supplies schemas and declared outcomes;
@@ -985,7 +986,7 @@ This chapter follows those operations into the implementation, including
the additional bookkeeping needed when steps repeat, call child workflows,
or pause for input.
## Following a Python Authoring Request
## Following a Python authoring request
The client package, `wf_client`, exposes the `App` connection object introduced
in the architecture chapter. Its `App.from_http_jsonrpc(...)` constructor
@@ -1016,7 +1017,7 @@ Requests and responses are serialized at the service boundary. Reconstructing
the returned objects in the client gives authors the same inspection methods
after discovery, execution, and later refreshes.
## Executing Nodes and Limiting a Run
## Executing nodes and limiting a run
The `wf_core` package implements the execution cycle shown in
[@fig:node-execution-cycle]. It selects a ready execution position and handles
@@ -1043,7 +1044,7 @@ This bounds the number of admitted node executions. The duration of a single
operation and the cost of its external requests require separate controls.
A handler that blocks indefinitely can still prevent progress after admission.
## Preserving State Across Nested Execution
## Preserving state across nested execution
Consider the two-document iteration in [@fig:foreach-graph]. In serial mode,
document A appends its report to state before document B starts. B can read
@@ -1086,7 +1087,7 @@ contributions are B and C. A later merge applies those writes using the
declared reducer, preserving A once. This bookkeeping supports nested
concurrent iteration even when its intermediate results are merged again.
## Giving Expressions a Consistent Context
## Giving expressions a consistent context
Suppose an outer foreach named `documents` processes documents, and an inner
foreach named `actions` processes each document's action items. A node in the
@@ -1112,7 +1113,7 @@ identifies the active items. Missing owners, parent cycles, or conflicting
aliases are reported as errors before those records are used to construct
context.
## Validation, Persistence, and Diagnostics
## Validation, persistence, and diagnostics
Validation first checks declarations that can be evaluated without running
operations. For example, two output bindings on the same node might write
@@ -1147,7 +1148,7 @@ focused. Definition inspection explains what was saved; deployment
inspection explains environment selection; run inspection and trace explain
what happened during a particular run.
## Server and Provider Responsibilities
## Server and provider responsibilities
The remaining package boundaries put these operations into a service.
`wf_api` coordinates lifecycle operations, `wf_artifacts` supplies storage
@@ -1186,7 +1187,7 @@ The walkthrough below expresses the same procedure through the current
Python client. The example README retains a command-line route for operators
who need that interface.
## Starting with Available Operations
## Starting with available operations
The operator supplies the source identities through the example configuration.
This walkthrough discovers operations within those known sources; source
@@ -1241,7 +1242,7 @@ the graph from those contracts.
Before connecting operations, the author checks their input and output
contracts for compatible fields.
## Describing the Workflow's Data
## Describing the workflow's data
The workflow has one public input, intermediate state, and two public
outputs. They are declared separately so that intermediate notes do not
@@ -1294,7 +1295,7 @@ lets the service validate its data after the author's Python session ends.
The report starts as absent in intermediate state. The public result requires
a report because the successful pipeline produces one before completion.
## Connecting Data and Decisions
## Connecting data and decisions
The author now creates three node uses. Each use selects an operation and
declares its data bindings. In the first mapping, `input_path("text")` selects
@@ -1352,13 +1353,15 @@ one leaves the other unchanged. This requires additional declarations even
for a linear three-step procedure; typed helpers construct the serialized
representation from those declarations.
## Diagnosing and Repairing a Binding
## Diagnosing and repairing a binding
An editable graph can temporarily be invalid. Suppose a final output binding
reads `missing_report`, although the declared state contains `report`:
```python
# Deliberately refer to an undeclared state field.
from textwrap import fill
graph.set_output([
input_from(state_path("missing_report"), "report"),
input_from(state_path("markdown"), "markdown"),
@@ -1367,8 +1370,26 @@ broken = await graph.validate()
assert not broken.ok
assert broken.remote_status == "not_run" # Local rejection skips the server check.
for issue in broken.local.errors:
print(issue.code, issue.path, issue.message)
print(issue.code, issue.path)
print(fill(issue.message, width=72))
```
The example prints the diagnostic code, its location, and the message wrapped
to fit the page:
<!-- walkthrough-output -->
```text
invalid_source_path output[0].path
source path must start with input., state., or context. and reference a
declared root field when applicable
```
The location identifies the first workflow-output mapping. Comparing its
source with the declared state reveals the misspelled reference. Local
validation rejects the graph before a server validation request is made.
The author repairs that reference and validates again:
```python
# Repair the field reference, leaving the nodes and edges unchanged.
graph.set_output([
input_from(state_path("report"), "report"),
@@ -1377,21 +1398,12 @@ graph.set_output([
(await graph.validate()).raise_for_errors()
```
The diagnostic has code `invalid_source_path` and location `output[0].path`.
Its message is:
> source path must start with input., state., or context. and reference a
> declared root field when applicable
The location identifies the first workflow-output mapping. Comparing its
source with the declared state reveals the misspelled reference. Local
validation rejects the graph before a server validation request is made.
The repair changes the workflow's output mapping. The renderer and its
outgoing edge remain unchanged.
This illustrates why data bindings and control routes need separate feedback.
The next section saves only the repaired definition.
## Saving a Version and Choosing Its Environment
## Saving a version and choosing its environment
Before saving, the author can request validation and inspect its diagnostics.
The example stops on errors:
@@ -1446,7 +1458,7 @@ deployment's artifact unchanged. Selecting a different source environment
changes the deployment's bindings. A readiness check
can reject a missing or incompatible binding before a run is attempted.
## Running and Inspecting the Result
## Running and inspecting the result
The client reads the notes and sends their contents. The server therefore
does not need access to the client's file path.
@@ -1465,6 +1477,22 @@ assert result.markdown.startswith("# Weekly Project Update")
run = await run.refresh()
trace = await run.trace(start=0, limit=10)
print("Status:", run.status)
print("Title:", result.report.title)
print("Action items:", len(result.report.action_items))
print(result.markdown.splitlines()[0])
```
The session prints a compact result summary and the first line of the rendered
Markdown:
<!-- walkthrough-output -->
```text
Status: completed
Title: Weekly Project Update
Action items: 3
# Weekly Project Update
```
The run exposes a status, output, and diagnostics independently of the
@@ -1479,12 +1507,13 @@ recorded risks and followups, and a Markdown report headed
example's data reached the intended outputs. The extraction contract requires
the sectioned notes format described at the start of this chapter.
## What This Case Demonstrates
## What this case demonstrates
The executable check in
[`test_thesis_python_walkthrough.py`](../../tests/examples/test_thesis_python_walkthrough.py)
reads and runs this chapter's Python blocks, including the rejected binding
and its repair. It checks the stored report as well as the returned snapshot.
and its repair. It checks the stored report, the returned snapshot, and the
two displayed output excerpts against the session's captured output.
The existing tests in
[`test_report_workflow_example.py`](../../tests/examples/test_report_workflow_example.py)
check the source's input rules, rendering and extraction, capability
@@ -1517,7 +1546,7 @@ the first two through controlled tests and an adapted in-process walkthrough.
The authoring assessment identifies the operations available through the
interface. Usability remains a separate evaluation question.
## Requirements and Evidence
## Requirements and evidence
[@tbl:requirements-evidence] relates the authoring requirements R1–R5 and
execution requirements X1–X5 to the available evidence. E1–E5 refer to the
@@ -1554,7 +1583,7 @@ with a limit of 100 node executions. It completes within that limit and
contains no interruption. The evidence index locates the separate tests for
exhaustion and resume.
## Walkthrough Method and Observations
## Walkthrough method and observations
The retained walkthrough test uses the example server configuration and a
fresh pytest temporary store. It extracts the case study's Python blocks in
@@ -1601,7 +1630,7 @@ lifecycle independently. Document rendering and generated PDF assets are not
part of this command. The test and manuscript must be taken from the same
repository revision because the test reads the manuscript directly.
## Separating Design Comparison from Evaluation
## Separating design comparison from evaluation
The earlier comparison of n8n, Zapier, and LangGraph explains different
authoring and execution choices. The same task has not been measured across
@@ -1620,7 +1649,7 @@ users. Similarly, structured diagnostics and inspection objects may help an
agent avoid trial and error, but reduced retries, token use, and repair time
remain hypotheses rather than measured outcomes.
## Falsifiability Criteria
## Falsifiability criteria
The implementation would fail its stated contracts if, for example:
@@ -1642,7 +1671,7 @@ Its main limitations concern how much authors must understand, which
execution guarantees are provided, and how far the available evidence can
be generalized.
## Authoring and Diagnosis Still Require Technical Knowledge
## Authoring and diagnosis still require technical knowledge
The Python client reduces manual serialization and provides editable graphs
and inspectable objects. It does not remove the need to understand schemas,
@@ -1659,7 +1688,7 @@ users can make these distinctions without assistance.
Inspection also requires judgment. A trace shows recorded execution, not
whether a report is factually correct or a remote side effect was desirable.
## Execution Guarantees Have Defined Boundaries
## Execution guarantees have defined boundaries
Foreach iteration, nested workflow scopes, structured context, and run-wide
node-execution limits are implemented foundations. They do not yet provide general
@@ -1678,7 +1707,7 @@ report may satisfy its schema while containing incorrect information.
Similarly, a fixed graph specifies routing but does not make remote
responses or concurrent completion order reproducible.
## Deployment and Trust Assumptions
## Deployment and trust assumptions
The controlled examples assume trusted operators and trusted Python sources.
Python operations execute in the server process without a sandbox. The
@@ -1710,7 +1739,7 @@ can be used; it is not a distributed execution service.
The provider interface exposes callable operations. It does not reproduce a
provider's interactive widgets or its complete user interface.
## Limits of the Evidence
## Limits of the evidence
The deterministic report fixture demonstrates lifecycle integration, not
broad document understanding or graph expressiveness. Targeted tests cover
@@ -1728,7 +1757,7 @@ require the interaction evaluation described in Future Work.
The remaining questions concern richer execution semantics, the effectiveness
of authoring and diagnosis, and operation beyond the controlled environment.
## Establish General Fork and Gather Semantics
## Establish general fork and gather semantics
General fork/gather requires a rule for identifying which concurrent work
belongs to the same invocation. Existing frames, scopes, iteration activations,
@@ -1748,7 +1777,7 @@ for correlation, merge behavior, and recovery. The authoring contract also
needs to make clear which graphs are rejected before execution and which
decisions remain the author's responsibility.
## Evaluate the Authoring and Recovery Experience
## Evaluate the authoring and recovery experience
A focused usability study should ask participants to discover an operation,
build a small workflow, change its contract, diagnose a broken binding, and
@@ -1770,7 +1799,7 @@ recorded interactions. Human evaluation can test whether the lifecycle
vocabulary and data-binding model are understandable without implementation
knowledge.
## Durable Waiting Within a Run
## Durable waiting within a run
Scheduling starts a new run of a saved deployment. Waiting until a time or
event during an existing run would instead require a durable suspension point
@@ -1779,7 +1808,7 @@ timed execution rather than treat a wait operation as another schedule.
Distributed execution would additionally require an ownership model beyond
the current single-scheduler arrangement.
## An Assistant-Backed Authoring Application
## An assistant-backed authoring application
A companion local-first assistant application has an execution backend for
persistent Python shells, streamed code output and images, execution approvals,
@@ -1796,7 +1825,7 @@ Dedicated views of the client's artifact, deployment, and run objects could
support that interaction. The present case study evaluates the workflow
client directly and includes no agent-driven authoring session.
## Preserving Contracts Across Operational Changes
## Preserving contracts across operational changes
The source catalog already represents resources, and a built-in workflow
operation can read resource text with a size bound. Further authoring work
@@ -1857,9 +1886,9 @@ This appendix maps the evaluation's evidence identifiers to implementation
and tests. Paths identify inspectable evidence; they are not a claim that
all listed suites passed in one newly recorded full-system run.
## Core Workflow Lifecycle
## Core workflow lifecycle
E1: artifacts, deployments, stopped runs, and explicit resume boundaries.
**E1**: artifacts, deployments, stopped runs, and explicit resume boundaries.
- `src/wf_artifacts/models.py`
- `src/wf_artifacts/runs/`
@@ -1867,9 +1896,9 @@ E1: artifacts, deployments, stopped runs, and explicit resume boundaries.
- `tests/wf_api/test_artifact_api.py`
- `tests/wf_api/test_run_api.py`
## Python Authoring and Inspection
## Python authoring and inspection
E2: reconstructed client objects, editable workflows, and the report fixture.
**E2**: reconstructed client objects, editable workflows, and the report fixture.
- `src/wf_client/`
- `tests/wf_client/test_authoring.py`
@@ -1883,9 +1912,9 @@ The example's README retains the command-line route for operators who need
it. That alternative interface is not an additional evaluated case in the
current thesis.
## Validation and Diagnostics
## Validation and diagnostics
E3: structural validation, source compatibility, and repair information.
**E3**: structural validation, source compatibility, and repair information.
- `src/wf_core/validation/`
- `src/wf_artifacts/validation.py`
@@ -1893,9 +1922,9 @@ E3: structural validation, source compatibility, and repair information.
- `tests/core/test_structured_context_validation.py`
- `tests/core/test_foreach_control_regions.py`
## Execution Ownership and Node Limits
## Execution ownership and node limits
E4: nesting, structured context, concurrent iteration, and persisted
**E4**: nesting, structured context, concurrent iteration, and persisted
node-execution limits.
- `src/wf_core/runtime/`
@@ -1906,9 +1935,9 @@ node-execution limits.
- `tests/core/test_run_step_budget_codec.py`
- `tests/core/test_run_step_budget_async.py`
## Source Provider Boundary
## Source provider boundary
E5: source contracts and provider-specific execution behind server composition.
**E5**: source contracts and provider-specific execution behind server composition.
- `src/wf_platform/sources.py`
- `src/wf_server/config.py`
tests/examples/test_thesis_python_walkthrough.py
+10
View File
@@ -73,4 +73,14 @@ async def test_thesis_python_session_preserves_report_and_repair(
diagnostic = capsys.readouterr().out
assert "invalid_source_path" in diagnostic
assert "output[0].path" in diagnostic
# Marked text blocks are verbatim stdout excerpts, checked against the real
# session so displayed diagnostics/results cannot drift from execution.
displayed_outputs = re.findall(
r"^<!-- walkthrough-output -->\n```text\n(.*?)^```",
chapter,
re.MULTILINE | re.DOTALL,
)
assert len(displayed_outputs) == 2, "Retain the diagnostic and result excerpts"
for displayed in displayed_outputs:
assert displayed in diagnostic, "Displayed output differs from the session"
assert namespace["trace"].frames