fix: harden workflow demo and evaluation tooling

This commit is contained in:
lda
2026-07-01 08:20:38 +07:00 Verified
parent 7bdfa3132d
commit 7e5f1c88da
29 changed files with 365 additions and 71 deletions
+4 -2
View File
@@ -10,8 +10,10 @@ implementation plans are kept for context, not as active instructions.
and verification commands.
- [`add/2026-06-workflow-platform-presentation.md`](add/2026-06-workflow-platform-presentation.md):
concise presentation narrative for the current product shape and demo flow.
- [`thesis/thesis-outline.md`](thesis/thesis-outline.md): thesis/report scaffold,
argument structure, evidence checklist, limitations, and future work.
- [`thesis/system-design-implementation.md`](thesis/system-design-implementation.md):
maintained thesis/report document.
- [`thesis/evidence-index.md`](thesis/evidence-index.md): claim-to-code and
claim-to-test evidence map for the thesis.
- [`current_roadmap.md`](current_roadmap.md): active next-work list after the
core type-shape cleanup.
- [`wf_core_architecture.md`](wf_core_architecture.md): kernel package
+2 -1
View File
@@ -134,7 +134,8 @@ clear operator feedback before adding more architecture.
`ux_issues_found: []` so debug-profile reports do not fail by omission.
- Completed: `wf draft bind` now reuses existing workflow input/state schema
fields when binding to step-local inputs, avoiding redundant-schema failures
found by debug challenge runs.
found by debug challenge runs. Implementation:
[`idempotent draft bind inputs`](historical/superpowers/plans/2026-06-29-idempotent-draft-bind-inputs.md).
- Keep status read-only; do not mutate registry, auth, config, or stores.
## Priority 2: Durable Run/Resume Hardening
@@ -2,8 +2,7 @@
Date: 2026-07-01
Status: Approved direction. Needs an executable implementation plan before code
changes.
Status: Implemented. This document is the live interrupt contract.
Related:
@@ -27,23 +26,18 @@ This is required for the Workflow Console and useful for CLI, JSON-RPC, MCP, and
agent clients. It prevents every client from needing workflow-specific code just
to render and answer an approval step.
## Current Gap
## Implemented Contract
The current core interrupt model has useful mechanics but not a complete public
contract.
The core interrupt model now exposes a complete public pause/resume contract.
- `InterruptNode` stores `kind`, request bindings, resume bindings, and resume
outcomes.
outcomes, plus `request_schema` and `resume_schema`.
- Runtime builds an `InterruptRequest` with id, frame id, node id, kind,
payload, route, and resumability.
- `workflow.runs.inspect` serializes the current runtime interrupt request.
- Resume validates the selected outcome against `InterruptNode.outcomes` and
applies resume bindings.
What is missing is a machine-readable schema for the request payload and resume
payload. A client can see data, but it cannot know whether the response should
be `{ "approved": true }`, `{ "selected_issue_ids": [...] }`, or something else
without reading workflow code or challenge-specific docs.
payload, route, resumability, outcomes, request schema, resume schema, and a
`typed` marker.
- `workflow.runs.inspect` serializes that persisted interrupt request.
- Runtime validates request payloads before pausing and validates resume
payloads against the persisted pause-time schema before mutating state.
This is close to the LangGraph-style `interrupt(value)` and `Command(resume=...)`
pattern: flexible and simple, but the response contract is mostly app
@@ -51,7 +45,7 @@ convention. `wf` should keep the flexibility while making the contract explicit.
## Design Summary
Add JSON Schema contracts to interrupt nodes and carry them through persisted run
JSON Schema contracts are carried from interrupt nodes through persisted run
inspection and resume validation.
```mermaid
@@ -73,7 +67,7 @@ sequenceDiagram
## Core Model
Extend `InterruptNode` with two optional schema fields:
`InterruptNode` carries two schema fields:
```python
class InterruptNode(BaseModel):
@@ -347,12 +347,12 @@ The UI selects all issues by default and allows individual deselection.
## Self-Describing Interrupt Dependency
The current interrupt payload exposes `kind` and data but not a complete
machine-readable response contract. The Workflow Console requires explicit
request and resume schemas so it can render and validate arbitrary interrupts
without reading workflow code.
Interrupted run inspection now exposes `kind`, payload, outcomes,
`request_schema`, `resume_schema`, and `typed`. Runtime validates request and
resume payloads against those contracts. The Workflow Console can therefore
render and validate arbitrary interrupts without reading workflow code.
The prerequisite contract is defined in
The implemented contract is defined in
[Self-describing interrupt contracts](2026-07-01-self-describing-interrupt-contracts.md).
## Replay And Failure Handling
+24 -1
View File
@@ -35,6 +35,29 @@ if (-not (Test-Path $pandoc_diagram)) {
Write-Error "pandoc diagram wrapper not found at $pandoc_diagram. Make sure it exists, then rerun this command."
exit 1
}
function Test-RenderNeedsAgentResults([string[]] $arguments) {
for ($index = 0; $index -lt $arguments.Count; $index++) {
$candidate = $null
if ($arguments[$index] -in @("-i", "--input")) {
if ($index + 1 -lt $arguments.Count) {
$candidate = $arguments[$index + 1]
$index++
}
}
elseif ($arguments[$index].EndsWith(".md")) {
$candidate = $arguments[$index]
}
if (
$candidate -and
(Test-Path -LiteralPath $candidate) -and
(Select-String -LiteralPath $candidate -SimpleMatch "include-agent-challenge-results" -Quiet)
) {
return $true
}
}
return $false
}
. $pandoc_diagram
$pandoc_crossref = Get-Command pandoc-crossref -ErrorAction SilentlyContinue
@@ -62,7 +85,7 @@ if (-not (Test-Path $include_markdown_filter)) {
}
$agent_results = Join-Path $PSScriptRoot "agent-challenge-results.md"
if (-not (Test-Path $agent_results)) {
if ((Test-RenderNeedsAgentResults $RemainingArgs) -and -not (Test-Path $agent_results)) {
Write-Error "agent-challenge-results.md is missing. Run generate_agent_challenge_evaluation.py first."
exit 1
}
@@ -54,10 +54,17 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv)
for path in generate(
manifest_path=args.manifest.resolve(), output_dir=args.output_dir.resolve()
):
print(path.relative_to(ROOT))
output_dir = args.output_dir.resolve()
for path in generate(manifest_path=args.manifest.resolve(), output_dir=output_dir):
resolved = path.resolve()
try:
display_path = resolved.relative_to(ROOT)
except ValueError:
try:
display_path = resolved.relative_to(output_dir)
except ValueError:
display_path = resolved
print(display_path)
return 0
+7 -4
View File
@@ -16,10 +16,13 @@ if (-not (Test-Path $file)) {
$file = (Resolve-Path $file).Path
$resourcePath = [System.IO.Path]::GetDirectoryName($file)
$evaluationGenerator = Join-Path $PSScriptRoot "generate_agent_challenge_evaluation.py"
& uv run python $evaluationGenerator
if ($LASTEXITCODE -ne 0) {
throw "agent challenge evaluation generation failed with exit code $LASTEXITCODE"
$needsAgentResults = Select-String -LiteralPath $file -SimpleMatch "include-agent-challenge-results" -Quiet
if ($needsAgentResults) {
$evaluationGenerator = Join-Path $PSScriptRoot "generate_agent_challenge_evaluation.py"
& uv run python $evaluationGenerator
if ($LASTEXITCODE -ne 0) {
throw "agent challenge evaluation generation failed with exit code $LASTEXITCODE"
}
}
# name without extension
+1 -1
View File
@@ -1994,7 +1994,7 @@ product.
Evidence:
- `docs/thesis/thesis-outline.md`
- `docs/historical/thesis/thesis-outline.md`
- `docs/current_roadmap.md`
- Absence of scheduler, visual-editor, and secret-manager production packages
in the current source tree.