How we evaluate the guidance
Our guidance tells a creator's AI assistant how to build a learning app together with a person:
the skills, the plan template, llms.txt. We test it the way it is used. A simulated teacher talks
to a real assistant, from the first message to the question whether the app may go online. Then we score what
the assistant built and how it got there.
- End to endA fresh machine, an empty folder, the real build prompt, the real assistant with its shell and files.
- Many turnsA simulated creator answers like a teacher would, until the scenario ends.
- Two scoresOne for the app folder at the end, one for the conversation on the way.
- Check the measure right after a runWhile its conversations still exist, we read some of them: it is easy to draw wrong conclusions if the eval itself was faulty.
Why a whole conversation
What the guidance asks for happens across many messages:
- ask what the app teaches before building anything,
- stop at each checkpoint and wait for the creator's yes,
- work through the phases in order: Explore, Plan, Code, Commit,
- ask at most two questions in one message,
- load each skill in its phase, not before.
None of this shows in one answer to one prompt. Did the assistant wait for the yes? Did it create the app only after the plan was confirmed? Only a whole conversation tells. So each eval is one scenario, played from start to end, in a fresh container, with the assistant's real tools.
Why not test single prompts? They are cheaper and faster, but they test the model's answer, not the process. A guidance that reads well can still be skipped in turn five.
What happens in one trial
The scenarios
Each scenario starts in an empty folder with the build prompt of
lernapps.net/apps/build: a quiz on fractions, explanation pages on electric circuits,
and a small typing game. The assistant reads the guidance from lernapps.net, as any assistant would, at the
version on main at that moment.
The simulated creator
The creator is Claude Code playing a teacher. Its persona says what the teacher wants, which facts they know, how they answer, and when they stop. It ends the scenario before anything goes online: at the question to publish, when the assistant says it is done, or when the assistant is stuck.
For each message the creator first decides, in a fixed format: what the assistant asked, what its reply answers, the reply itself, and whether this is the end. Only the reply reaches the assistant, which keeps talking to what looks like a person.
{"asked": ["plan_confirm"], "answers": ["plan_confirm"],
"reply": "Ja, passt so.", "end": null}Why the decision in a fixed format? In the first runs the creator made mistakes that counted as the assistant's: it ended a scenario at a question about GitHub, and it forgot to mark its end. We had to guess its intent from free text and fixed those guesses three times. Now its end and its yes to the plan are read, not guessed, and its own mistakes are counted apart: an answer to a question nobody asked, a yes without a question, a message sent without the command, a last message without its end.
What Harbor does, and what we built
Harbor is an open framework for running agents on tasks in containers. It does the heavy lifting. Our part is what makes it an eval of our guidance.
Harbor
- builds the container of each task,
- installs the assistant: Claude Code, OpenCode and many more,
- runs the simulated creator and connects it to the assistant over ACP, an open protocol between a client and a coding agent,
- keeps the logs, the ACP session and the app folder,
- runs the verifier in a second container,
- runs the same task in Docker, in Apple Container on a Mac, or in the cloud.
Built by us
- the scenarios: build prompt, persona, expected kind of app,
- the creator's command and its fixed format,
- the verifier of the app folder,
- the score of the conversation, with one small adapter per assistant,
- the results of a run, the conclusions and this site,
- two small patches to Harbor and a wrapper that keeps every other credential out of the containers.
Why Harbor? It runs a simulated user against a real assistant out of the box, for many assistants, in a standard task format. Building our own runner would have cost more than the evals themselves. The decision is recorded in the ADR on eval runs with Harbor.
What we score
Two scripts look at a finished trial from two sides.
| Source | What it tells |
|---|---|
| The app folder the verifier, in its own container |
Is there a valid plan? Which phase did it reach, and how many of its boxes are ticked? Is the kind of app
right? Does lernapps check pass? Is everything committed? |
| The conversation the transcript score |
Did it read the guidance in full or only a summary? How many questions in one message wait for an answer? Did it create the app before the plan's yes at the checkpoint? Does each reply begin with its phase? Did it load each skill in its phase? Which kind of app did it propose first? Did it bypass a git hook? |
Each row on Prompt engineering measures one expectation of the guidance (A1 to A34 in the plan of the evals). We score the process, not the content: an app built by the process is good when its creator knows the subject.
Every assistant, one score
Each assistant writes its own kind of log. But Harbor saves the ACP session of every assistant it runs: each message of the creator, each text of the assistant, each tool call with what went in and what came out. The transcript score reads that session, so it works the same for every assistant.
Only the names of the tools differ. A small adapter per assistant maps them: OpenCode's bash
becomes Bash, its filePath becomes file_path.
How we know it holds: for a Claude Code run, reading the ACP session gives the same 24 numbers and the same evidence as reading Claude Code's own log.
Adding an assistant takes an adapter in evals/harnesses.ts and one line in
evals/harbor/run.sh.
From a run to a change
- Runan eval session in the cloud, with its own key
Each scenario with each model, three times where the subscription allows. The session pushes only the numbers:
results.json. - Check the measure, right after the runpeople, with an assistant We read some conversations and check that each number measures what it should. Every run so far found a fault in the eval itself (the list); we fix it and score the run again. This has to happen while the conversations exist: the run keeps them in its own container, and only the numbers reach the repository.
- Concludepeople, in a pull request
conclusions.yaml: the question, the bottom line, findings that link the cells they rest on, and what we do about them. - Changeguidance or tooling Each finding becomes a short, hard rule where the step happens, or a check when a rule keeps being skipped.
- Run againthe same scenarios The next run compares its cells with the last one, on Prompt engineering.
Faults we found in our own evals
Each fault made a number say something about the assistant that was not true. We found them by reading the conversations behind the numbers.
| Found in | What the eval got wrong | What we changed |
|---|---|---|
| First run 18 Oct |
Harbor put its own config file into the app folder, and the app generator refused the folder. | A patch to Harbor keeps the file out of the app folder. |
| First run | The simulated creator stopped at a question about GitHub Pages while planning, as if it were the question to publish. | The persona says which question ends the scenario; the run was left out. |
| First run | Reading the setting of the git hooks counted as bypassing them. | Only setting or removing it counts. |
| First run | Files read in the shell (cat) did not count as read, so “skill read” was wrong. |
The row was left out of that run; the score now reads the shell's reads and downloads. |
| First run | Every sentence with a question mark counted as a question, also sample questions inside a plan draft. | Since 22 Oct only questions that wait for an answer count. |
| Second run 21 Oct |
A phase set with sed, a one-liner or a helper script went unseen, so “app created before
the plan” was wrong. |
The score reads these commands and the output of grep '^phase:'; runs 1 and 2 were scored
again. |
| Second run | Haiku looked “stuck in Code”: mostly it was the creator ending the scenario at the first GitHub question. | The end of the scenario is measured as such (“Code done”). |
| Second run | The creator's end was guessed from its free text; the guess needed three fixes. | The creator marks its end, and since 22 Oct decides each message in a fixed format. |
| Second run | A yes to a draft in Explore counted as the plan's yes, so apps created too early went unseen. | Each reply begins with its phase; since 22 Oct only the yes at the Plan checkpoint counts. |
| Second run | The verifier crashed on a broken link in the app folder and lost the numbers of one trial. | It skips such a link. |
| Third run 21 Oct |
The creator wrote its end only into its own notes, and once ended without sending its last message: the assistant was scored for the creator's mistake. | Since 22 Oct the creator's own mistakes are counted apart from the assistant's. |
| Test run 22 Oct |
A command that ticked a box naming a phase (“Set phase: code”) counted as setting that
phase. |
Only a change of the phase line counts. |
| Test run | “Kind of app chosen right” read the plan at the end, after the creator had asked for the right kind: a wrong first choice scored as right. | The first kind of app the assistant proposes is scored too. |
| OpenCode run 22 Oct |
OpenCode reports the tokens of only the last model call of each turn: 3,200 output tokens for a turn that wrote 45 questions. | Its tokens are summed from its own record in the trial. |
| OpenCode run | The creator recorded that the assistant asked nothing, though it had asked two questions, and said “go on”. Its record agreed with itself, so no mistake was counted. | Such a turn is flagged for a person to read; a second model as judge is planned (#156). |
| OpenCode run | The README of an app template counted as a read outside the guidance, though the guidance links it. | Every link in the guidance of the run counts as guidance. |
Two lessons hold across all of them: read the conversations before the numbers, and do it while they still exist.
What the runs cannot tell yet, and every run, are on Prompt engineering. The code and its README: evals/ on GitHub.