lda.chat/lda-wf
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: finalize thesis evaluation and front matter

Browse Source
This commit is contained in:
lda
2026-07-01 02:35:44 +07:00 Verified
parent e12ccdf18a
commit a1007c49a9
11 changed files with 3804 additions and 2084 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/agent_challenges/evaluation.py
+40 -20
View File
@@ -13,7 +13,8 @@ FIGURE_STEMS = (
"agent-challenge-audited-outcomes-by-cell",
"agent-challenge-automatic-vs-manual-outcomes",
"agent-challenge-longitudinal-outcomes",
"agent-challenge-duration-and-tokens",
"agent-challenge-duration",
"agent-challenge-token-volume",
)
_MANUAL_OUTCOMES = frozenset({"pass", "invalid", "fail"})
_CHALLENGE_ORDER = {"browser": 0, "report": 1}
@@ -210,17 +211,20 @@ def render_evaluation_markdown(cohort: EvaluationCohort) -> str:
"## Audited Agent Challenge Campaign",
"",
(
f"The primary campaign contains {len(cohort.trials)} audited trials: "
f"{outcomes['pass']} passes, {outcomes['invalid']} invalid samples, "
f"and {outcomes['fail']} failure."
f"The primary campaign contains {len(cohort.trials)} manually audited "
f"trials: {outcomes['pass']} clean product-path passes under the "
f"campaign rules, {outcomes['invalid']} invalid evaluation samples, "
f"and {outcomes['fail']} failure. These counts are not a "
f"model-success-rate estimate."
),
"",
(
"The campaign crosses two challenges, two hosted models, three instruction "
"profiles (`none`, `skills`, and `all`), and three repetitions per cell. "
"The checked cohort snapshot records report hashes, prompt hashes, the "
"repository commit, automatic metrics, and manual-audit outcomes; local "
"raw report files are verified against those hashes when present."
"The campaign crosses two challenges × two hosted models × three "
"instruction profiles (`none`, `skills`, and `all`) = 12 cells, with "
"three repetitions per cell (n=3). The checked cohort snapshot records "
"report hashes, prompt hashes, the repository commit, automatic metrics, "
"and manual-audit outcomes; local raw report files are verified against "
"those hashes when present."
),
"",
(
@@ -228,6 +232,10 @@ def render_evaluation_markdown(cohort: EvaluationCohort) -> str:
"this is longitudinal engineering evidence, not a controlled model comparison."
),
"",
"> **Campaign validity note.** This campaign is a bounded longitudinal audit, "
"not a controlled comparison. Each cell has n=3; waves changed product and "
"prompt snapshots; all audits were performed by the author.",
"",
f"Selection rule: {cohort.selection_rule}",
"",
"| Challenge / model / profile | Pass | Invalid | Fail |",
@@ -244,10 +252,12 @@ def render_evaluation_markdown(cohort: EvaluationCohort) -> str:
"",
(
"A manual `pass` requires both successful product-path evidence and an "
"acceptable audit trail. `Invalid` means the sample cannot support the "
"clean benchmark claim, commonly because the agent read repository or "
"example material outside its supplied workspace. `Fail` means the "
"challenge contract itself was not established."
"acceptable audit trail. It does not imply the agent avoided every "
"exploratory read, only that no disqualifying read or bypass was found. "
"`Invalid` means the sample cannot support the clean benchmark claim, "
"commonly because the agent read repository or example material outside "
"its supplied workspace. `Fail` means the challenge contract itself was "
"not established."
),
"",
"![Audited outcomes by evaluation cell.](figures/agent-challenge-audited-outcomes-by-cell.svg){#fig:agent-challenge-audited-outcomes-by-cell width=95%}",
@@ -255,7 +265,10 @@ def render_evaluation_markdown(cohort: EvaluationCohort) -> str:
(
"[@fig:agent-challenge-audited-outcomes-by-cell] reports all three "
"repetitions rather than hiding invalid samples. The profile labels are "
"descriptive; this campaign does not isolate instruction-profile effects."
"descriptive; this campaign does not isolate instruction-profile effects. "
"Profile × wave is confounded because the base prompt changed before "
"wave 3, so apparent differences may reflect prompt changes, model "
"updates, or repository drift rather than instruction-layer effects."
),
"",
"![Automatic task outcomes compared with manual outcomes.](figures/agent-challenge-automatic-vs-manual-outcomes.svg){#fig:agent-challenge-automatic-vs-manual-outcomes width=75%}",
@@ -275,14 +288,21 @@ def render_evaluation_markdown(cohort: EvaluationCohort) -> str:
"changed. They preserve the chronology needed to study those changes."
),
"",
"![Duration and recorded token totals grouped by challenge, instruction profile, model, and wave.](figures/agent-challenge-duration-and-tokens.svg){#fig:agent-challenge-duration-and-tokens width=95%}",
"![Wall-clock duration grouped by challenge, instruction profile, model, and wave.](figures/agent-challenge-duration.svg){#fig:agent-challenge-duration width=78%}",
"",
(
"[@fig:agent-challenge-duration-and-tokens] separates each challenge and "
"metric into its own panel. Circle and square markers redundantly identify "
"the models without relying on color. Wall-clock duration includes hosted-service "
"latency, and OpenCode token totals include cache-read accounting, so neither "
"axis is a normalized model-efficiency metric."
"[@fig:agent-challenge-duration] separates the two challenges. Circle "
"and square markers redundantly identify the models without relying on "
"color. Wall-clock duration includes hosted-service latency and is not a "
"normalized model-efficiency metric."
),
"",
"![Recorded token totals grouped by challenge, instruction profile, model, and wave.](figures/agent-challenge-token-volume.svg){#fig:agent-challenge-token-volume width=78%}",
"",
(
"[@fig:agent-challenge-token-volume] reports OpenCode token totals, which "
"include cache-read accounting. The figure records observed workload volume; "
"it is not an efficiency comparison."
),
"",
"### Campaign Limitations",