fix: harden workflow demo and evaluation tooling
This commit is contained in:
+4
-2
@@ -10,8 +10,10 @@ implementation plans are kept for context, not as active instructions.
|
||||
and verification commands.
|
||||
- [`add/2026-06-workflow-platform-presentation.md`](add/2026-06-workflow-platform-presentation.md):
|
||||
concise presentation narrative for the current product shape and demo flow.
|
||||
- [`thesis/thesis-outline.md`](thesis/thesis-outline.md): thesis/report scaffold,
|
||||
argument structure, evidence checklist, limitations, and future work.
|
||||
- [`thesis/system-design-implementation.md`](thesis/system-design-implementation.md):
|
||||
maintained thesis/report document.
|
||||
- [`thesis/evidence-index.md`](thesis/evidence-index.md): claim-to-code and
|
||||
claim-to-test evidence map for the thesis.
|
||||
- [`current_roadmap.md`](current_roadmap.md): active next-work list after the
|
||||
core type-shape cleanup.
|
||||
- [`wf_core_architecture.md`](wf_core_architecture.md): kernel package
|
||||
|
||||
@@ -134,7 +134,8 @@ clear operator feedback before adding more architecture.
|
||||
`ux_issues_found: []` so debug-profile reports do not fail by omission.
|
||||
- Completed: `wf draft bind` now reuses existing workflow input/state schema
|
||||
fields when binding to step-local inputs, avoiding redundant-schema failures
|
||||
found by debug challenge runs.
|
||||
found by debug challenge runs. Implementation:
|
||||
[`idempotent draft bind inputs`](historical/superpowers/plans/2026-06-29-idempotent-draft-bind-inputs.md).
|
||||
- Keep status read-only; do not mutate registry, auth, config, or stores.
|
||||
|
||||
## Priority 2: Durable Run/Resume Hardening
|
||||
|
||||
@@ -2,8 +2,7 @@
|
||||
|
||||
Date: 2026-07-01
|
||||
|
||||
Status: Approved direction. Needs an executable implementation plan before code
|
||||
changes.
|
||||
Status: Implemented. This document is the live interrupt contract.
|
||||
|
||||
Related:
|
||||
|
||||
@@ -27,23 +26,18 @@ This is required for the Workflow Console and useful for CLI, JSON-RPC, MCP, and
|
||||
agent clients. It prevents every client from needing workflow-specific code just
|
||||
to render and answer an approval step.
|
||||
|
||||
## Current Gap
|
||||
## Implemented Contract
|
||||
|
||||
The current core interrupt model has useful mechanics but not a complete public
|
||||
contract.
|
||||
The core interrupt model now exposes a complete public pause/resume contract.
|
||||
|
||||
- `InterruptNode` stores `kind`, request bindings, resume bindings, and resume
|
||||
outcomes.
|
||||
outcomes, plus `request_schema` and `resume_schema`.
|
||||
- Runtime builds an `InterruptRequest` with id, frame id, node id, kind,
|
||||
payload, route, and resumability.
|
||||
- `workflow.runs.inspect` serializes the current runtime interrupt request.
|
||||
- Resume validates the selected outcome against `InterruptNode.outcomes` and
|
||||
applies resume bindings.
|
||||
|
||||
What is missing is a machine-readable schema for the request payload and resume
|
||||
payload. A client can see data, but it cannot know whether the response should
|
||||
be `{ "approved": true }`, `{ "selected_issue_ids": [...] }`, or something else
|
||||
without reading workflow code or challenge-specific docs.
|
||||
payload, route, resumability, outcomes, request schema, resume schema, and a
|
||||
`typed` marker.
|
||||
- `workflow.runs.inspect` serializes that persisted interrupt request.
|
||||
- Runtime validates request payloads before pausing and validates resume
|
||||
payloads against the persisted pause-time schema before mutating state.
|
||||
|
||||
This is close to the LangGraph-style `interrupt(value)` and `Command(resume=...)`
|
||||
pattern: flexible and simple, but the response contract is mostly app
|
||||
@@ -51,7 +45,7 @@ convention. `wf` should keep the flexibility while making the contract explicit.
|
||||
|
||||
## Design Summary
|
||||
|
||||
Add JSON Schema contracts to interrupt nodes and carry them through persisted run
|
||||
JSON Schema contracts are carried from interrupt nodes through persisted run
|
||||
inspection and resume validation.
|
||||
|
||||
```mermaid
|
||||
@@ -73,7 +67,7 @@ sequenceDiagram
|
||||
|
||||
## Core Model
|
||||
|
||||
Extend `InterruptNode` with two optional schema fields:
|
||||
`InterruptNode` carries two schema fields:
|
||||
|
||||
```python
|
||||
class InterruptNode(BaseModel):
|
||||
|
||||
@@ -347,12 +347,12 @@ The UI selects all issues by default and allows individual deselection.
|
||||
|
||||
## Self-Describing Interrupt Dependency
|
||||
|
||||
The current interrupt payload exposes `kind` and data but not a complete
|
||||
machine-readable response contract. The Workflow Console requires explicit
|
||||
request and resume schemas so it can render and validate arbitrary interrupts
|
||||
without reading workflow code.
|
||||
Interrupted run inspection now exposes `kind`, payload, outcomes,
|
||||
`request_schema`, `resume_schema`, and `typed`. Runtime validates request and
|
||||
resume payloads against those contracts. The Workflow Console can therefore
|
||||
render and validate arbitrary interrupts without reading workflow code.
|
||||
|
||||
The prerequisite contract is defined in
|
||||
The implemented contract is defined in
|
||||
[Self-describing interrupt contracts](2026-07-01-self-describing-interrupt-contracts.md).
|
||||
|
||||
## Replay And Failure Handling
|
||||
|
||||
@@ -35,6 +35,29 @@ if (-not (Test-Path $pandoc_diagram)) {
|
||||
Write-Error "pandoc diagram wrapper not found at $pandoc_diagram. Make sure it exists, then rerun this command."
|
||||
exit 1
|
||||
}
|
||||
|
||||
function Test-RenderNeedsAgentResults([string[]] $arguments) {
|
||||
for ($index = 0; $index -lt $arguments.Count; $index++) {
|
||||
$candidate = $null
|
||||
if ($arguments[$index] -in @("-i", "--input")) {
|
||||
if ($index + 1 -lt $arguments.Count) {
|
||||
$candidate = $arguments[$index + 1]
|
||||
$index++
|
||||
}
|
||||
}
|
||||
elseif ($arguments[$index].EndsWith(".md")) {
|
||||
$candidate = $arguments[$index]
|
||||
}
|
||||
if (
|
||||
$candidate -and
|
||||
(Test-Path -LiteralPath $candidate) -and
|
||||
(Select-String -LiteralPath $candidate -SimpleMatch "include-agent-challenge-results" -Quiet)
|
||||
) {
|
||||
return $true
|
||||
}
|
||||
}
|
||||
return $false
|
||||
}
|
||||
. $pandoc_diagram
|
||||
|
||||
$pandoc_crossref = Get-Command pandoc-crossref -ErrorAction SilentlyContinue
|
||||
@@ -62,7 +85,7 @@ if (-not (Test-Path $include_markdown_filter)) {
|
||||
}
|
||||
|
||||
$agent_results = Join-Path $PSScriptRoot "agent-challenge-results.md"
|
||||
if (-not (Test-Path $agent_results)) {
|
||||
if ((Test-RenderNeedsAgentResults $RemainingArgs) -and -not (Test-Path $agent_results)) {
|
||||
Write-Error "agent-challenge-results.md is missing. Run generate_agent_challenge_evaluation.py first."
|
||||
exit 1
|
||||
}
|
||||
|
||||
@@ -54,10 +54,17 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(argv)
|
||||
for path in generate(
|
||||
manifest_path=args.manifest.resolve(), output_dir=args.output_dir.resolve()
|
||||
):
|
||||
print(path.relative_to(ROOT))
|
||||
output_dir = args.output_dir.resolve()
|
||||
for path in generate(manifest_path=args.manifest.resolve(), output_dir=output_dir):
|
||||
resolved = path.resolve()
|
||||
try:
|
||||
display_path = resolved.relative_to(ROOT)
|
||||
except ValueError:
|
||||
try:
|
||||
display_path = resolved.relative_to(output_dir)
|
||||
except ValueError:
|
||||
display_path = resolved
|
||||
print(display_path)
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -16,10 +16,13 @@ if (-not (Test-Path $file)) {
|
||||
$file = (Resolve-Path $file).Path
|
||||
$resourcePath = [System.IO.Path]::GetDirectoryName($file)
|
||||
|
||||
$evaluationGenerator = Join-Path $PSScriptRoot "generate_agent_challenge_evaluation.py"
|
||||
& uv run python $evaluationGenerator
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "agent challenge evaluation generation failed with exit code $LASTEXITCODE"
|
||||
$needsAgentResults = Select-String -LiteralPath $file -SimpleMatch "include-agent-challenge-results" -Quiet
|
||||
if ($needsAgentResults) {
|
||||
$evaluationGenerator = Join-Path $PSScriptRoot "generate_agent_challenge_evaluation.py"
|
||||
& uv run python $evaluationGenerator
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "agent challenge evaluation generation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
# name without extension
|
||||
|
||||
@@ -1994,7 +1994,7 @@ product.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `docs/thesis/thesis-outline.md`
|
||||
- `docs/historical/thesis/thesis-outline.md`
|
||||
- `docs/current_roadmap.md`
|
||||
- Absence of scheduler, visual-editor, and secret-manager production packages
|
||||
in the current source tree.
|
||||
|
||||
Reference in New Issue
Block a user