{ "title": "First eval run", "question": "Does the creator's assistant follow the guidance from the first prompt to the publish checkpoint, and how do the models differ?", "bottomLine": [ "Sonnet 5.5 and Opus 5.5 follow the process nearly fully on all three archetypes.", "Haiku 5.5 builds apps that pass the check every time, but skips process steps: the content guide stays empty in 12 of 14 runs.", "Opus 5.5 is not better than Sonnet 5.5 and takes up to 2.5 times as long.", "No configuration keeps to one or two questions per turn." ], "findings": [ { "id": "big-models-follow", "title": "Sonnet 5.5 and Opus 5.5 follow the process", "text": "They wrote the competencies, filled the content guide, chose the right archetype, passed the check, reached the commit phase and committed clean in every run, with one exception: Opus 5.5 left the content guide empty in 1 of 2 interactive runs. They ticked 93–100 % of the plan's boxes on average.", "evidence": [ { "task": "interactive-fresh", "model": "Opus 5.5", "row": "app_content_filled" }, { "task": "explainer-fresh", "model": "Sonnet 5.5", "row": "plan_boxes_ticked" } ] }, { "id": "haiku-skips-steps", "title": "Haiku 5.5 passes the check but skips process steps", "text": "Haiku 5.5 passed the check in 14 of 14 runs, but filled the content guide in only 2, ticked between 0 and 100 % of the plan's boxes per run (mean 70 % quiz, 46 % explainer, 7 % interactive), wrote no competencies in any interactive run, chose the wrong archetype once (an explainer for the typing trainer) and created the app before the creator confirmed the plan in 4 runs (2 of 3 interactive, 1 of 5 explainer, 1 of 6 quiz). Every run reached the scenario's end with a green check (\"Code done\"); in 2 of 5 explainer runs it did so without setting `phase: commit`.", "evidence": [ { "task": "quiz-fresh", "model": "Haiku 5.5", "row": "app_content_filled" }, { "task": "interactive-fresh", "model": "Haiku 5.5", "row": "plan_boxes_ticked" }, { "task": "interactive-fresh", "model": "Haiku 5.5", "row": "archetype_expected" }, { "task": "explainer-fresh", "model": "Haiku 5.5", "row": "phase_reached" }, { "task": "explainer-fresh", "model": "Haiku 5.5", "row": "code_done" }, { "task": "interactive-fresh", "model": "Haiku 5.5", "row": "not_created_before_plan" } ] }, { "id": "too-many-questions", "title": "No model keeps to one or two questions per turn", "text": "32 of 33 runs asked more than two questions in one turn: Haiku 5.5 at most 2–7, Sonnet 5.5 4–8, Opus 5.5 4–12. The score counts sentences that end in \"?\" in the agent's text, so questions inside a plan draft count too.", "evidence": [ { "task": "quiz-fresh", "model": "Sonnet 5.5", "row": "questions_max_2" }, { "task": "explainer-fresh", "model": "Opus 5.5", "row": "questions_max_2" } ] }, { "id": "opus-slow", "title": "Opus 5.5 is slower, not better", "text": "Opus 5.5 took 60 minutes on the explainer against 24 for Sonnet 5.5 (2.5 times), 41 against 21 on the interactive app and 24 against 20 on the quiz, and read 3.5 times as many cached tokens on the explainer.", "evidence": [ { "task": "explainer-fresh", "model": "Opus 5.5", "metric": "agent_minutes" }, { "task": "explainer-fresh", "model": "Sonnet 5.5", "metric": "agent_minutes" } ] }, { "id": "summary-fetches", "title": "Every run read guidance through a summarising fetch", "text": "Every run fetched guidance through a summarising fetch tool at least once; 32 of 33 also downloaded the plan template and read it word for word.", "evidence": [ { "task": "quiz-fresh", "model": "Haiku 5.5", "row": "plan_template_read_raw" } ] } ], "actions": [ { "text": "Choose the archetype from what each template's README says it can do, with no example apps.", "status": "done", "findings": [ "haiku-skips-steps" ], "link": "https://github.com/lernapps/tooling/pull/118" }, { "text": "Write the guidance for agents, not for humans: compact, one skill per phase, GitHub only in Commit. Measured in the run of 2026-10-21.", "status": "done", "findings": [ "haiku-skips-steps", "opus-slow" ], "link": "https://github.com/lernapps/tooling/pull/118" }, { "text": "Enforce the content guide earlier than at the listing (a check or hook while \"FILL IN\" is left).", "status": "open", "findings": [ "haiku-skips-steps" ] }, { "text": "Make the plan confirmation a checkpoint before `create` that is hard to miss.", "status": "open", "findings": [ "haiku-skips-steps" ] }, { "text": "Rethink the rule of one or two questions per turn, state it more plainly, or count it differently.", "status": "open", "findings": [ "too-many-questions" ] } ], "notes": [ "Measuring: the retrospective is left out, since the scenarios stop at the publish checkpoint and the plan template fills it after the listing (n/a in 23 of 33 runs). Whether the skill `lernapps-app` was read is left out too: the score of that run missed reads in the shell (`cat`).", "Re-scored on 21 October from the same transcripts, with the scenario's end (\"Code done\") and the phase read from `sed`, one-liners and `grep '^phase:'`: two Haiku runs moved, one interactive run now counts as created before the plan, and the turns after the plan dropped by one in three runs. The rest is unchanged.", "How we got there, on the evening of 17 October: an inventory of the guidance as a chain and the expectations A1–A34; an ADR over six options, decided for Harbor, the only one with a simulated creator; a spike without a model; a patch so that a subscription token reaches the agent behind Harbor's ACP bridge. The matrix ran overnight in three lanes and stopped before the subscription's usage window reset.", "Fixed while running: Harbor's `.acpxrc.json` in the app folder made `lernapps create` refuse it; the persona stopped at a GitHub Pages question in Explore (one run left out, fixed in 596220b); the transcript score raised false hook-bypass alarms." ] }