How we make the guidance work for the creator's assistant: skills, plan templates, AGENTS.md, llms.txt and
the messages of the tooling. We judge it by one thing: does the assistant follow the process? An app built by
the process is good when its creator knows the subject, so content quality is not scored.
How we evaluate
A simulated teacher talks to a real assistant, from the first message to the question whether the app may
go online, in a fresh container run by Harbor. Then we score two things:
The app folder at the end: plan, phase, app, check, commits.
The conversation on the way: what the assistant read, asked and ran, in which phase.
Each row below measures one expectation of the guidance. Right after a run, while its conversations still
exist, we read some of them: it is easy to draw wrong conclusions if the eval itself was faulty.
Newest first. Choose a run: everything inside the frame below belongs to it, its question, results, findings
and what we did about them. In a run, compare its heatmap with another run to see what changed. Hover a cell
for min, max and n.
A run is each scenario with each model, up to three times. A trial that hit the usage limit or
broke off before its assistant started is left out and listed under its run; configurations are never
mixed.
Question: Does Haiku 5.5 keep to the process on the interactive app once each reply begins with the phase and only a checkpoint waits (#133), and does the new score read it right?
Bottom line
In this small run (n = 3), Haiku 5.5 builds the interactive app with a green check in every run, creates nothing before the plan and needs at most three turns after it; in 2 of 3 runs it sets phase: commit before the first question about GitHub. On 2026-10-21 it had done each of these in 1 of 3 runs or none.
Two runs begin every reply the creator reads with the phase; one begins only 2 of 8. Where a reply names a phase, it is the plan's.
The green checks hide a template bug: the browser tests could not press ä, ö, ü or ß, and every run rewrote its playthroughs to avoid them.
Skills in their phase and the questions per turn stay where they were.
Added on 2026-10-23, one run per archetype and anecdotal: DeepSeek V4.1 Flash in OpenCode builds all three archetypes with a green check, keeps to the process and asks at most two questions per turn, for about 20 US cents a run.
Each cell shows its value in this run and, below it, the change against in points: ▲ closer to the guidance, ▼ further from it. A cell without a change has no value in the other run.
Results at guidance cbe3b29 · claude-code 2.1.306
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Interactive
Dimension expectation
Haikun=3
Process (plan, app, git)
Plan boxes ticked A4
98
Competencies written (“Ich kann …”) A12
100
Content guide filled (app-content) A13
100
Archetype chosen right A19
100
lernapps check passes A21, A16, A17, A18
100
Phase in the plan at the end (commit = 100 %) A4
89
Committed clean A4
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
Code done: check green and the scenario ended after the plan A4
67
Replies that begin with the phase A4
75
Announced phase is the plan's A4
100
At most 2 questions in every turn A9
0
No app created before the plan was confirmed A3, A5
100
Plan template read word for word A1
100
Skills loaded in their phase A2
33
No skill of another archetype or before its phase A2
0
Results at guidance dab96fe · opencode 1.19.4
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Quiz
Dimension expectation
DeepSeekn=1
Process (plan, app, git)
Plan boxes ticked A4
100
Competencies written (“Ich kann …”) A12
100
Content guide filled (app-content) A13
100
Archetype chosen right A19
100
lernapps check passes A21, A16, A17, A18
100
Phase in the plan at the end (commit = 100 %) A4
100
Committed clean A4
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
Code done: check green and the scenario ended after the plan A4
100
Replies that begin with the phase A4
100
Announced phase is the plan's A4
100
At most 2 questions in every turn A9
100
No app created before the plan was confirmed A3, A5
100
Plan template read word for word A1
100
Skills loaded in their phase A2
88
No skill of another archetype or before its phase A2
0
Phase code set only after the plan's yes A3, A6
100
First archetype the agent proposed is the expected one A19
100
The simulated creator made no mistake of its own
100
Results at guidance 124d83e · opencode 1.19.4
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Explainer
Interactive
Dimension expectation
DeepSeekn=1
DeepSeekn=1
Process (plan, app, git)
Plan boxes ticked A4
100
95
Competencies written (“Ich kann …”) A12
100
100
Content guide filled (app-content) A13
100
100
Archetype chosen right A19
100
100
lernapps check passes A21, A16, A17, A18
100
100
Phase in the plan at the end (commit = 100 %) A4
100
100
Committed clean A4
100
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
100
Code done: check green and the scenario ended after the plan A4
100
100
Replies that begin with the phase A4
90
100
Announced phase is the plan's A4
100
100
At most 2 questions in every turn A9
100
100
No app created before the plan was confirmed A3, A5
100
100
Plan template read word for word A1
100
100
Skills loaded in their phase A2
100
100
No skill of another archetype or before its phase A2
100
100
Phase code set only after the plan's yes A3, A6
100
100
First archetype the agent proposed is the expected one A19
100
100
The simulated creator made no mistake of its own
100
100
Minutes, tokens and turns per task and model
Mean (min–max) per run. Output tokens in thousands, from the agent's session log.
Task
Model
n
Agent minutes
Output tokens (k)
Creator turns
… after the plan
Questions in a turn, max
Interactive
Haiku 5.5 guidance cbe3b29
3
21 (15–26)
75 (52–90)
7.7 (6.0–9.0)
2.0 (1.0–3.0)
4.3 (4.0–5.0)
Quiz
DeepSeek V4.1 Flash guidance dab96fe
1
32 (32–32)
75 (75–75)
12.0 (12.0–12.0)
3.0 (3.0–3.0)
2.0 (2.0–2.0)
Explainer
DeepSeek V4.1 Flash guidance 124d83e
1
33 (33–33)
101 (101–101)
11.0 (11.0–11.0)
2.0 (2.0–2.0)
2.0 (2.0–2.0)
Interactive
DeepSeek V4.1 Flash guidance 124d83e
1
24 (24–24)
106 (106–106)
9.0 (9.0–9.0)
2.0 (2.0–2.0)
2.0 (2.0–2.0)
Findings
What the numbers show. Each finding links the cells it rests on.
Haiku 5.5 keeps to the process on the interactive app
All three runs pass the check, tick 94–100 % of the plan's boxes, fill the content guide, create the app only after the plan's yes and need one to three creator turns after it. Two set phase: commit and then asked for the repo in a reply that began "**Commit · Checkpoint**"; the third asked for the publish yes while still in Code. On 2026-10-21 (guidance b20b514) the same cell passed the check in 1 of 3 runs and set phase: commit in none. With n = 3 this is a sign, not a proof.
Most replies begin with the phase, and it is the plan's
Two runs began every reply the creator read with the phase ("**Explore · Checkpoint**", "**Plan · Checkpoint**", "**Code**"); the third began only 2 of 8, narrating first ("Die Antworten sind klar…"). No reply named a phase the plan did not hold. Code was announced in one run only: in the other two the whole Code phase ran inside one turn, whose reply already began with Commit.
The interactive template's end-to-end test pressed each key of a playthrough with Playwright's keyboard.press, which knows only the keys of a US keyboard: "Unknown key: ö" (and Ä, ä, ß). The check called it a template bug, and every run got around it by rewriting its playthroughs so that they never press those keys. That is also why the check failed in 2 of 3 runs on 2026-10-21, where the tests stayed red.
The simulated creator wrote the marker [Ende: …] only into its own summary, never into a message to the agent; the score fell back to the creator's sentences. In one run it ended without sending its last message at all, so the scenario's end and "Code done" read 0 there, which is the creator's mistake, not the agent's.
Skills in their phase and the questions per turn stay as they were
Haiku loads 14–57 % of the skills a phase names by its end (38 % on 2026-10-21), and every run loads one before its phase. Every run asks more than two questions in a turn (at most 4–5).
DeepSeek V4.1 Flash in OpenCode keeps to the process on all three archetypes
One run per archetype (n = 1 each), so an anecdote, not a measure. Every run passes the check, ticks 95–100 % of the plan's boxes, creates the app only after the plan's yes, reaches its phase and ends the scenario regularly. It begins 90–100 % of its replies with the phase and asks at most two questions in a turn, which Haiku 5.5 did in none of its runs, and loads 88–100 % of the skills a phase names (Haiku: 14–57 %). Where it falls short: the quiz run loaded two skills out of place and read four pages outside the guidance, and the explainer run created the app without the plan's create command. Of the two skills, lernapps-quiz came from a fault of the guidance: at dab96fe the front matter of lernapps-quiz-design was no valid YAML, so npx skills add installed only lernapps-quiz; Flash noticed the design skill was missing and read the Code skill in Explore instead (fixed in #160 before the other two runs). It read lernapps-content-questions on purpose, for the question types of its plan. The same fault was there at cbe3b29 and may explain part of Haiku's 0 on this row; we have not checked its transcripts for it. A run took 24–33 minutes and cost about 0.19–0.25 US dollars on OpenCode Go.
The interactive template's browser test sends any key that is not on a US keyboard as the keydown event the runtime listens to; its example presses ü.Because of: The browser tests could not press ä, ö, ü or ß
The personas send their last message with acpx prompt, the marker on its last line, and never end without it.Because of: The creator did not send its end marker
The score reads the phase from the first text of a reply, at that point of the turn, and leaves out the reply to the creator's last message. The numbers here are scored that way.Because of: Most replies begin with the phase, and it is the plan's
Notes
One job, interactive-fresh × Haiku 5.5 × 3, at guidance cbe3b29 (#133); the creator was played by Haiku 5.5, as in the earlier runs. No trial was left out. The transcripts were scored again with 8ef76ea after the run; only the phase announced changed (0.79 to 0.75 in the mean).
The runs of DeepSeek V4.1 Flash (OpenCode 1.19.4, opencode-go/deepseek-v4.1-flash) ran on 2026-10-22 and 2026-10-23 at guidance dab96fe (quiz) and 124d83e (explanation pages, interactive app), not at cbe3b29. Between these commits the process guidance is the same: what changed are the archetypes' content (exercises, components, the wording of the design skills' descriptions) and the listing skill's part after the merge, which no eval reaches. The evals measure the process, so the cells compare. What differs is the creator: these runs had the structured creator, which records what each reply answers and waits for an agent that works long (#164), and the rows added since (code after the plan's yes, the archetype first proposed, the creator's mistakes) have no numbers for Haiku. Their results were scored with the score of #164 and added to this file next to Haiku's.
Question: Does the guidance by phase (one skill per phase, the archetype chosen from the templates' READMEs, GitHub only in Commit; #118) keep the process as well as the guidance of 2026-10-18, and does the agent load each skill in the phase it belongs to?
Bottom line
Sonnet 5.5 and Opus 5.5 pass the check in every run and tick nearly all plan boxes; the content guide is now filled in every run.
Haiku 5.5 reaches the scenario's end with a green check in every quiz and explainer run, but the interactive app passes the check in 1 of 3 runs, and Haiku loads only about a third of the skills a phase names.
Sonnet and Opus load 76–95 % of a phase's skills by its end, but often too early.
Creators need more turns after the plan. Apps created before the plan are rare in the score (3 runs), but the transcripts show more: agents that took a yes to an early draft for the plan's yes.
Each cell shows its value in this run and, below it, the change against in points: ▲ closer to the guidance, ▼ further from it. A cell without a change has no value in the other run.
Results at guidance b20b514 · claude-code 2.1.305, 2.1.306
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Quiz
Explainer
Interactive
Dimension expectation
Haikun=3
Sonnetn=3
Opusn=3
Haikun=3
Sonnetn=3
Opusn=3
Haikun=3
Sonnetn=3
Opusn=3
Process (plan, app, git)
Plan boxes ticked A4
86
99
100
79
100
100
61
100
100
Competencies written (“Ich kann …”) A12
67
100
100
100
100
100
67
100
100
Content guide filled (app-content) A13
100
100
100
67
100
100
100
100
100
Archetype chosen right A19
100
100
100
67
100
100
100
100
100
lernapps check passes A21, A16, A17, A18
100
100
100
100
100
100
33
100
100
Phase in the plan at the end (commit = 100 %) A4
67
83
100
67
100
89
67
89
89
Committed clean A4
100
100
100
100
100
100
67
100
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
67
33
33
33
100
0
33
67
Code done: check green and the scenario ended after the plan A4
100
100
100
100
100
100
33
100
100
At most 2 questions in every turn A9
33
0
0
33
0
0
33
0
0
No app created before the plan was confirmed A3, A5
100
100
100
100
67
100
67
67
100
Plan template read word for word A1
100
100
100
100
100
100
100
100
100
Skills loaded in their phase A2
31
91
77
20
86
76
38
95
89
No skill of another archetype or before its phase A2
67
0
33
67
33
67
33
33
33
Results at guidance 1e4ba59 · claude-code 2.1.306
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Quiz
Dimension expectation
Sonnetn=3
Process (plan, app, git)
Plan boxes ticked A4
100
Competencies written (“Ich kann …”) A12
100
Content guide filled (app-content) A13
100
Archetype chosen right A19
100
lernapps check passes A21, A16, A17, A18
100
Phase in the plan at the end (commit = 100 %) A4
78
Committed clean A4
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
Code done: check green and the scenario ended after the plan A4
100
At most 2 questions in every turn A9
0
No app created before the plan was confirmed A3, A5
100
Plan template read word for word A1
100
Skills loaded in their phase A2
79
No skill of another archetype or before its phase A2
0
Minutes, tokens and turns per task and model
Mean (min–max) per run. Output tokens in thousands, from the agent's session log.
Task
Model
n
Agent minutes
Output tokens (k)
Creator turns
… after the plan
Questions in a turn, max
Quiz
Haiku 5.5 guidance b20b514
3
21 (12–32)
69 (51–101)
7.3 (7.0–8.0)
2.7 (2.0–3.0)
5.7 (2.0–9.0)
Quiz
Sonnet 5.5 guidance b20b514
3
19 (12–27)
70 (55–86)
10.0 (8.0–11.0)
3.0 (2.0–4.0)
4.3 (4.0–5.0)
Quiz
Sonnet 5.5 guidance 1e4ba59
3
46 (44–47)
83 (79–90)
9.0 (9.0–9.0)
3.0 (3.0–3.0)
4.0 (3.0–5.0)
Quiz
Opus 5.5 guidance b20b514
3
35 (23–46)
109 (107–110)
10.0 (9.0–11.0)
3.3 (2.0–4.0)
6.3 (6.0–7.0)
Explainer
Haiku 5.5 guidance b20b514
3
21 (11–33)
83 (52–106)
8.3 (8.0–9.0)
4.7 (3.0–6.0)
5.0 (2.0–9.0)
Explainer
Sonnet 5.5 guidance b20b514
3
41 (30–46)
124 (98–141)
10.7 (10.0–12.0)
3.7 (3.0–4.0)
4.7 (4.0–6.0)
Explainer
Opus 5.5 guidance b20b514
3
61 (39–88)
179 (166–186)
10.0 (10.0–10.0)
3.0 (3.0–3.0)
4.7 (4.0–6.0)
Interactive
Haiku 5.5 guidance b20b514
3
27 (22–34)
101 (78–123)
14.0 (13.0–16.0)
10.3 (8.0–13.0)
3.7 (2.0–5.0)
Interactive
Sonnet 5.5 guidance b20b514
3
24 (19–35)
82 (77–87)
10.7 (10.0–11.0)
3.7 (3.0–4.0)
5.0 (3.0–7.0)
Interactive
Opus 5.5 guidance b20b514
3
50 (40–67)
169 (153–185)
11.0 (10.0–13.0)
3.7 (3.0–5.0)
3.7 (3.0–4.0)
Findings
What the numbers show. Each finding links the cells it rests on.
Sonnet 5.5 and Opus 5.5 still follow the process, and fill the content guide
Both pass the check in every run and tick 99–100 % of the plan's boxes. The content guide is filled in every run, also where Opus left it empty on 2026-10-18 (interactive, 1 of 2).
The interactive app passes the check in 1 of 3 runs (its browser tests fail on ö and ü; on 2026-10-18 in 3 of 3). One explainer run chose the interactive archetype. Haiku needs the most turns after the plan: in no interactive run were there three or fewer. Every Haiku cell ends with phase: code (67 %), but every quiz and explainer run reached the scenario's end with a green check ("Code done"): Haiku asked about GitHub or publishing without setting phase: commit first, and the creator ends the scenario there. So the phase row measures only whether the agent set the phase before that question, which Sonnet and Opus mostly do and Haiku does not.
The skills of a phase are mostly loaded, often too early
New in this run (A2, scored from the phases of the skills' building blocks in the architecture). Sonnet and Opus load 76–95 % of the skills a phase names by the end of that phase, Haiku 20–38 %. Many loads come before their phase, mostly lernapps-exercise-patterns (then in Plan) and the Code skills, read already in Explore. lernapps-listing and lernapps-commit are not expected: their tasks come after the scenario's end.
Some agents take a yes to an early draft for the plan's yes
The row counts 3 runs that created the app before the plan was confirmed (explainer and interactive with Sonnet, interactive with Haiku); on 2026-10-18 Sonnet and Opus had none. It counts a create before the agent sets phase: code after the creator's yes, so it misses the other kind the transcripts show: 5 times the agent took a yes to an early draft without sketches as the plan's yes, set the phase and created before the Plan phase was worked through (3 with Haiku, 2 with Sonnet).
The re-run at guidance 1e4ba59 looks like the run at b20b514
The quiz with Sonnet 5.5 was run again after the usage limit, by then at a newer guidance (listing rule "own data only", questions and page frame after the runtime). It passes the check in every run and loads 79 % of the skills by their phase; its verifier did not crash again.
Check whether "created before the plan" counts right. It did not: the score now reads the phase a sed or one-liner writes (the new value) and the output of the template's grep '^phase:', which covers helper scripts.Because of: Some agents take a yes to an early draft for the plan's yes
Measure the scenario's end as such: the row "Code done" counts a green check and a conversation that ended after the plan with the creator's stop at publishing or its thanks when the agent is done; the phase row stays beside it. Run 2 is re-scored with it and with the phase detection above.Because of: Haiku 5.5 falls behind on the rest
Say plainly that only a checkpoint waits: a phase's last box sets the next phase and goes on in the same message, and Code has no checkpoint before Commit.Because of: Haiku 5.5 falls behind on the rest
The simulated creator marks its last message with the end ([Ende: veröffentlichen], fertig, festgefahren), so that the end is read, not guessed from its sentences, which took three fixes in the re-score.Because of: Haiku 5.5 falls behind on the rest
Each reply begins with the phase in phase: ("**Plan · Checkpoint**"), so that the creator always knows where the agent is, and a yes to a draft is not taken for the plan's yes. Scored from the next run: replies that begin with the phase, and whether it is the plan's.Because of: Haiku 5.5 falls behind on the rest, Some agents take a yes to an early draft for the plan's yes
Make lernapps-commit and lernapps-listing hard to skip in Commit. Dropped: the scenario ends before their tasks; the fresh tasks now leave them to that condition (skills_conditional).Because of: The skills of a phase are mostly loaded, often too early
Left out
Interactive · Opus 5.5: the whole job, the subscription's usage limit (HTTP 429); run again
Quiz · Sonnet 5.5: the whole job, the subscription's usage limit (HTTP 429)
Quiz · Sonnet 5.5, one run (kept in part): the verifier crashed; its numbers are left out
Notes
The creator was played by Haiku 5.5 for every model, as on 2026-10-18. Claude Code was updated during the run (2.1.305 to 2.1.306): the interactive jobs ran on the newer version, which makes them a configuration of their own; they share no cell with the others, so the table shows them together.
The subscription's usage limit hit after eight jobs: the interactive job with Opus 5.5 (3 runs, broken off after about 34 minutes) and a first re-run of the quiz with Sonnet 5.5 (3 runs, before the agent started) are left out. The first was run again at the same guidance, the second at the newer guidance 1e4ba59.
One quiz run with Sonnet 5.5 kept its transcript scores but lost the verifier's: the verifier crashed on a dangling symlink the agent left in .claude/skills/.
First eval run 2026-10-18
33 runs · runs 2026-10-17 23:13 to 2026-10-18 05:59 UTC · claude-code 2.1.302 · Harbor 0.24.0 · creator played by Haiku 5.5 · guidance 7ceb8c4 · results.json · conclusions.json
Question: Does the creator's assistant follow the guidance from the first prompt to the publish checkpoint, and how do the models differ?
Bottom line
Sonnet 5.5 and Opus 5.5 follow the process nearly fully on all three archetypes.
Haiku 5.5 builds apps that pass the check every time, but skips process steps: the content guide stays empty in 12 of 14 runs.
Opus 5.5 is not better than Sonnet 5.5 and takes up to 2.5 times as long.
No configuration keeps to one or two questions per turn.
Each cell shows its value in this run and, below it, the change against in points: ▲ closer to the guidance, ▼ further from it. A cell without a change has no value in the other run.
Results
Share per run in percent (0–100), mean over the runs; darker is closer to the guidance. n = runs per cell.
Quiz
Explainer
Interactive
Dimension expectation
Haikun=6
Sonnetn=4
Opusn=3
Haikun=5
Sonnetn=4
Opusn=3
Haikun=3
Sonnetn=3
Opusn=2
Process (plan, app, git)
Plan boxes ticked A4
70
93
99
46
100
99
7
97
100
Competencies written (“Ich kann …”) A12
83
100
100
80
100
100
0
100
100
Content guide filled (app-content) A13
17
100
100
0
100
100
33
100
50
Archetype chosen right A19
100
100
100
100
100
100
67
100
100
lernapps check passes A21, A16, A17, A18
100
100
100
100
100
100
100
100
100
Phase in the plan at the end (commit = 100 %) A4
100
100
100
87
100
100
100
100
100
Committed clean A4
100
100
100
100
100
100
100
100
100
Transcript (what the agent said and did)
At most 3 creator turns after the plan A11
100
75
100
40
75
100
33
100
100
Code done: check green and the scenario ended after the plan A4
100
100
100
100
100
100
100
100
100
At most 2 questions in every turn A9
17
0
0
0
0
0
0
0
0
No app created before the plan was confirmed A3, A5
83
100
100
80
100
100
33
100
100
Plan template read word for word A1
100
100
100
80
100
100
100
100
100
Minutes, tokens and turns per task and model
Mean (min–max) per run. Output tokens in thousands, from the agent's session log.
Task
Model
n
Agent minutes
Output tokens (k)
Creator turns
… after the plan
Questions in a turn, max
Quiz
Haiku 5.5
6
15 (10–18)
73 (54–104)
6.5 (5.0–8.0)
1.3 (1.0–3.0)
3.8 (2.0–7.0)
Quiz
Sonnet 5.5
4
20 (14–33)
62 (54–78)
8.0 (6.0–11.0)
2.5 (1.0–4.0)
6.3 (5.0–8.0)
Quiz
Opus 5.5
3
24 (16–37)
90 (73–114)
6.3 (5.0–8.0)
2.3 (2.0–3.0)
5.0 (4.0–6.0)
Explainer
Haiku 5.5
5
28 (17–36)
100 (83–123)
7.8 (7.0–9.0)
3.6 (2.0–5.0)
4.8 (3.0–7.0)
Explainer
Sonnet 5.5
4
24 (14–40)
99 (83–123)
7.0 (6.0–9.0)
2.3 (1.0–4.0)
5.0 (4.0–6.0)
Explainer
Opus 5.5
3
60 (45–71)
168 (144–181)
7.0 (6.0–8.0)
2.0 (1.0–3.0)
8.7 (6.0–12.0)
Interactive
Haiku 5.5
3
34 (16–64)
95 (74–114)
9.7 (7.0–11.0)
3.7 (2.0–5.0)
4.7 (3.0–6.0)
Interactive
Sonnet 5.5
3
21 (16–27)
79 (69–88)
7.3 (7.0–8.0)
2.0 (2.0–2.0)
6.3 (6.0–7.0)
Interactive
Opus 5.5
2
41 (30–51)
125 (123–127)
8.0 (8.0–8.0)
3.0 (3.0–3.0)
5.0 (4.0–6.0)
Findings
What the numbers show. Each finding links the cells it rests on.
Sonnet 5.5 and Opus 5.5 follow the process
They wrote the competencies, filled the content guide, chose the right archetype, passed the check, reached the commit phase and committed clean in every run, with one exception: Opus 5.5 left the content guide empty in 1 of 2 interactive runs. They ticked 93–100 % of the plan's boxes on average.
Haiku 5.5 passes the check but skips process steps
Haiku 5.5 passed the check in 14 of 14 runs, but filled the content guide in only 2, ticked between 0 and 100 % of the plan's boxes per run (mean 70 % quiz, 46 % explainer, 7 % interactive), wrote no competencies in any interactive run, chose the wrong archetype once (an explainer for the typing trainer) and created the app before the creator confirmed the plan in 4 runs (2 of 3 interactive, 1 of 5 explainer, 1 of 6 quiz). Every run reached the scenario's end with a green check ("Code done"); in 2 of 5 explainer runs it did so without setting phase: commit.
32 of 33 runs asked more than two questions in one turn: Haiku 5.5 at most 2–7, Sonnet 5.5 4–8, Opus 5.5 4–12. The score counts sentences that end in "?" in the agent's text, so questions inside a plan draft count too.
Opus 5.5 took 60 minutes on the explainer against 24 for Sonnet 5.5 (2.5 times), 41 against 21 on the interactive app and 24 against 20 on the quiz, and read 3.5 times as many cached tokens on the explainer.
Interactive · Haiku 5.5: the whole job, The persona took a GitHub Pages question in Explore for the publish checkpoint and ended before the plan (fixed in 596220b).
Notes
Measuring: the retrospective is left out, since the scenarios stop at the publish checkpoint and the plan template fills it after the listing (n/a in 23 of 33 runs). Whether the skill lernapps-app was read is left out too: the score of that run missed reads in the shell (cat).
Re-scored on 21 October from the same transcripts, with the scenario's end ("Code done") and the phase read from sed, one-liners and grep '^phase:': two Haiku runs moved, one interactive run now counts as created before the plan, and the turns after the plan dropped by one in three runs. The rest is unchanged.
How we got there, on the evening of 17 October: an inventory of the guidance as a chain and the expectations A1–A34; an ADR over six options, decided for Harbor, the only one with a simulated creator; a spike without a model; a patch so that a subscription token reaches the agent behind Harbor's ACP bridge. The matrix ran overnight in three lanes and stopped before the subscription's usage window reset.
Fixed while running: Harbor's .acpxrc.json in the app folder made lernapps create refuse it; the persona stopped at a GitHub Pages question in Explore (one run left out, fixed in 596220b); the transcript score raised false hook-bypass alarms.
For machines: runs.json lists the runs, each with its results and
conclusions as JSON.
Limits
Small n: 2 to 6 runs per cell. One creator model (Haiku 5.5).
The creator decides each message in a fixed format since 22 October, so its own mistakes count apart from
the agent's. Runs before that read its end from its free text, and before 21 October from its sentences.
New apps only: no scenario extends an existing app yet, and none starts from a prepared state.
GitHub only as a local remote in the container; the GitHub cases on github.com are not run yet.
Claude Code so far; OpenCode is prepared (HARNESS=opencode), Codex and Gemini CLI are not run
yet.
Harbor's own token and cost numbers are wrong for the agent behind the ACP bridge; the tokens here come
from the agent's own record: OpenCode's database, else the ACP session, else Claude Code's log.
How we work
Interactive test. One real creator session with a person as the creator, on a small app.
We read the transcript: which instructions were followed, which ignored, which links never opened, where
GitHub got in the way.
Evals. The same scenarios across assistant configurations, several runs each, with a
simulated creator; scored per expectation from the app folder and the transcript. Results above.
Changes. Each finding becomes a change to the guidance or the tooling, as short, hard rules
where the step happens, or as a check when a rule keeps being skipped.
Interactive test of the new guidance. Before a change is published, a person builds one
small app with it. The assistant runs in a separate Claude Code session on the web and gets the unpublished
guidance from the orchestrating session: instead of opening a URL on lernapps.net or GitHub, it sends
GET <url> to that session and waits for the content. The orchestrator answers from the
working copy, can change a file between two requests without publishing, and logs every request. The CLI
still comes from npm, so the assistant treats the guidance copied into the app as outdated and asks for it
too. Then the evals again, comparing the new guidance with the old in the same configurations.
The assistant knows it is in a test, and each request costs a turn: good enough to iterate fast, not a
measurement. Measurements come from the evals.
How to run it, in detail
What it needs
A machine with Docker (the daemon is started by setup.sh if none runs), uv and
Node 22. In a Claude Code cloud container this is all there.
A credential in EVALS_ANTHROPIC_API_KEY: an API key, or a subscription token from
claude setup-token. In a cloud environment, store it as an environment variable under that name;
names starting with ANTHROPIC_ are filtered out by the platform.
About 2 GB of disk for the task image and up to 1.5 GB per running trial.
Steps
evals/harbor/setup.sh installs Harbor 0.24.0 into ~/.lernapps-evals, applies the
patches in evals/harbor/ (the subscription token for the ACP bridge, the acpx config outside the
app folder) and writes the generated files into each task.
evals/harbor/run.sh <task> <model> [trials] [concurrency] runs one Harbor job. The
tasks are quiz-fresh, explainer-fresh and interactive-fresh; the model is a
Harbor name such as anthropic/claude-sonnet-5-5; HARNESS names the assistant
(claude-code by default, or opencode with EVALS_OPENCODE_API_KEY);
USER_MODEL sets the simulated creator (default: the same model for Claude Code, Haiku 5.5
otherwise). Harbor runs through evals/harbor/harbor.sh, which clears the
environment, so no other credential reaches the containers. The job lands in
~/.lernapps-evals/jobs with its configuration.json and the transcript score.
For a matrix, start one run.sh per model in parallel; two or three at a time fit a machine with
4 CPUs and 16 GB. Watch the rate_limit_event lines in
<trial>/user-agent/claude-code.txt and stop at a limit.
node evals/summary.ts ~/.lernapps-evals/jobs prints mean, min and max per dimension for each
task and configuration while a run goes on; configurations are never mixed.
node evals/results.ts <run> ~/.lernapps-evals/jobs writes the run's
evals/runs/<run>/results.json: the numbers, cleaned as above. A job with an
excluded.txt (its reason inside) is left out.
Write evals/runs/<run>/conclusions.yaml beside it: the question, the bottom line, the
findings with the cells they rest on, and what we did about them. The site's build renders this page from
both and fails when a finding names a cell or row the run does not have.
