{
  "title": "Phase announced",
  "question": "Does Haiku 5.5 keep to the process on the interactive app once each reply begins with the phase and only a checkpoint waits (#133), and does the new score read it right?",
  "bottomLine": [
    "In this small run (n = 3), Haiku 5.5 builds the interactive app with a green check in every run, creates nothing before the plan and needs at most three turns after it; in 2 of 3 runs it sets `phase: commit` before the first question about GitHub. On 2026-10-21 it had done each of these in 1 of 3 runs or none.",
    "Two runs begin every reply the creator reads with the phase; one begins only 2 of 8. Where a reply names a phase, it is the plan's.",
    "The green checks hide a template bug: the browser tests could not press ä, ö, ü or ß, and every run rewrote its playthroughs to avoid them.",
    "Skills in their phase and the questions per turn stay where they were.",
    "Added on 2026-10-23, one run per archetype and anecdotal: DeepSeek V4.1 Flash in OpenCode builds all three archetypes with a green check, keeps to the process and asks at most two questions per turn, for about 20 US cents a run."
  ],
  "findings": [
    {
      "id": "haiku-keeps-process",
      "title": "Haiku 5.5 keeps to the process on the interactive app",
      "text": "All three runs pass the check, tick 94–100 % of the plan's boxes, fill the content guide, create the app only after the plan's yes and need one to three creator turns after it. Two set `phase: commit` and then asked for the repo in a reply that began \"**Commit · Checkpoint**\"; the third asked for the publish yes while still in Code. On 2026-10-21 (guidance b20b514) the same cell passed the check in 1 of 3 runs and set `phase: commit` in none. With n = 3 this is a sign, not a proof.",
      "evidence": [
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "check_passes"
        },
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "phase_reached"
        },
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "not_created_before_plan"
        },
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "turns_after_plan_max_3"
        }
      ]
    },
    {
      "id": "phase-announced",
      "title": "Most replies begin with the phase, and it is the plan's",
      "text": "Two runs began every reply the creator read with the phase (\"**Explore · Checkpoint**\", \"**Plan · Checkpoint**\", \"**Code**\"); the third began only 2 of 8, narrating first (\"Die Antworten sind klar…\"). No reply named a phase the plan did not hold. Code was announced in one run only: in the other two the whole Code phase ran inside one turn, whose reply already began with Commit.",
      "evidence": [
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "phase_announced"
        },
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "phase_announced_right"
        }
      ]
    },
    {
      "id": "umlaut-keys",
      "title": "The browser tests could not press ä, ö, ü or ß",
      "text": "The interactive template's end-to-end test pressed each key of a playthrough with Playwright's `keyboard.press`, which knows only the keys of a US keyboard: \"Unknown key: ö\" (and Ä, ä, ß). The check called it a template bug, and every run got around it by rewriting its playthroughs so that they never press those keys. That is also why the check failed in 2 of 3 runs on 2026-10-21, where the tests stayed red.",
      "evidence": [
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "check_passes"
        }
      ]
    },
    {
      "id": "creator-end",
      "title": "The creator did not send its end marker",
      "text": "The simulated creator wrote the marker `[Ende: …]` only into its own summary, never into a message to the agent; the score fell back to the creator's sentences. In one run it ended without sending its last message at all, so the scenario's end and \"Code done\" read 0 there, which is the creator's mistake, not the agent's.",
      "evidence": [
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "code_done"
        }
      ]
    },
    {
      "id": "unchanged",
      "title": "Skills in their phase and the questions per turn stay as they were",
      "text": "Haiku loads 14–57 % of the skills a phase names by its end (38 % on 2026-10-21), and every run loads one before its phase. Every run asks more than two questions in a turn (at most 4–5).",
      "evidence": [
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "skills_in_phase"
        },
        {
          "task": "interactive-fresh",
          "model": "Haiku 5.5",
          "row": "questions_max_2"
        }
      ]
    },
    {
      "id": "flash-keeps-process",
      "title": "DeepSeek V4.1 Flash in OpenCode keeps to the process on all three archetypes",
      "text": "One run per archetype (n = 1 each), so an anecdote, not a measure. Every run passes the check, ticks 95–100 % of the plan's boxes, creates the app only after the plan's yes, reaches its phase and ends the scenario regularly. It begins 90–100 % of its replies with the phase and asks at most two questions in a turn, which Haiku 5.5 did in none of its runs, and loads 88–100 % of the skills a phase names (Haiku: 14–57 %). Where it falls short: the quiz run loaded two skills out of place and read four pages outside the guidance, and the explainer run created the app without the plan's create command. Of the two skills, `lernapps-quiz` came from a fault of the guidance: at dab96fe the front matter of `lernapps-quiz-design` was no valid YAML, so `npx skills add` installed only `lernapps-quiz`; Flash noticed the design skill was missing and read the Code skill in Explore instead (fixed in #160 before the other two runs). It read `lernapps-content-questions` on purpose, for the question types of its plan. The same fault was there at cbe3b29 and may explain part of Haiku's 0 on this row; we have not checked its transcripts for it. A run took 24–33 minutes and cost about 0.19–0.25 US dollars on OpenCode Go.",
      "evidence": [
        {
          "task": "quiz-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "check_passes"
        },
        {
          "task": "explainer-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "check_passes"
        },
        {
          "task": "interactive-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "check_passes"
        },
        {
          "task": "interactive-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "questions_max_2"
        },
        {
          "task": "interactive-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "skills_in_phase"
        },
        {
          "task": "quiz-fresh",
          "model": "DeepSeek V4.1 Flash",
          "row": "no_skill_out_of_place"
        }
      ]
    }
  ],
  "actions": [
    {
      "text": "The interactive template's browser test sends any key that is not on a US keyboard as the keydown event the runtime listens to; its example presses ü.",
      "status": "done",
      "findings": [
        "umlaut-keys"
      ]
    },
    {
      "text": "The personas send their last message with `acpx prompt`, the marker on its last line, and never end without it.",
      "status": "done",
      "findings": [
        "creator-end"
      ]
    },
    {
      "text": "The score reads the phase from the first text of a reply, at that point of the turn, and leaves out the reply to the creator's last message. The numbers here are scored that way.",
      "status": "done",
      "findings": [
        "phase-announced"
      ]
    },
    {
      "text": "Run the other archetypes and models with the phase announced, to see whether the gain holds beyond n = 3.",
      "status": "open",
      "findings": [
        "haiku-keeps-process",
        "phase-announced"
      ]
    },
    {
      "text": "Rethink the rule of one or two questions per turn (carried over).",
      "status": "open",
      "findings": [
        "unchanged"
      ]
    }
  ],
  "notes": [
    "One job, interactive-fresh × Haiku 5.5 × 3, at guidance cbe3b29 (#133); the creator was played by Haiku 5.5, as in the earlier runs. No trial was left out. The transcripts were scored again with 8ef76ea after the run; only the phase announced changed (0.79 to 0.75 in the mean).",
    "The runs of DeepSeek V4.1 Flash (OpenCode 1.19.4, opencode-go/deepseek-v4.1-flash) ran on 2026-10-22 and 2026-10-23 at guidance dab96fe (quiz) and 124d83e (explanation pages, interactive app), not at cbe3b29. Between these commits the process guidance is the same: what changed are the archetypes' content (exercises, components, the wording of the design skills' descriptions) and the listing skill's part after the merge, which no eval reaches. The evals measure the process, so the cells compare. What differs is the creator: these runs had the structured creator, which records what each reply answers and waits for an agent that works long (#164), and the rows added since (code after the plan's yes, the archetype first proposed, the creator's mistakes) have no numbers for Haiku. Their results were scored with the score of #164 and added to this file next to Haiku's."
  ]
}