How it's tested

The release gate asks a broader question than whether each command works: can an agent that did not build the tool complete real work through it? That makes testing part of the product contract. The interface must carry a project across discovery, authoring, play, diagnosis, repair, and verification without a human filling in missing steps.

Every one of the 330 commands is exercised against a running editor when it is added or changed, and the result is read back rather than trusted: node get after node set, scene tree after a structural change, a screenshot when the question is visual. The whole sweep re-runs whenever command registration or the shared helpers change, because tool bugs cluster there.

That check ran clean for six releases. Then one night of real use found twelve bugs it had never seen. So there are now two checks.

Every command, driven live#

The per-command sweep catches contract bugs: a parameter that doesn’t coerce, a result that misreports, an undo step that doesn’t undo. It is fast, it localizes a failure to one command, and it runs constantly during development.

What it cannot do is put the editor into the situations real work creates. It tests runtime screenshot with a game that has been running for a while, not one that started 40 milliseconds ago. It tests runtime get against a live game, not one parked at a debugger breakpoint. Each command runs once, with time to breathe, in a clean scene. Real sessions have none of that slack.

A full game build before each release#

Before a release is tagged, an agent that did not write the tool builds a complete game slice through it: discovery, scene assembly, scripting, lighting, playtesting, cleanup. Anything surprising in that session is treated as a bug in the tool, not something to work around, and gets fixed or filed before the tag.

The rule comes from a specific night. Six releases had shipped with every command individually verified. Building a small lighthouse-keeper game end to end then surfaced twelve real defects:

• runtime screenshot taken right after scene play read a frame nothing had drawn into yet, and returned an all-black image with no error.
• A game paused at a debugger breakpoint made every runtime command time out, and the timeout message blamed a missing autoload that the same call had already verified was present.
• The editor and the game raced over the IPC response file. The failure was intermittent and an immediate retry always cured it, which is exactly why nobody had chased it.
• scene create wrote a scene without opening it, so the next node add built into whatever scene happened to be open. And nothing could close a tab: scene delete refused an open scene with “close the scene tab first,” advice only a person with a mouse could follow.
• --filter on project settings, line ranges on script read, and a --path alias on scene instance were accepted and silently ignored. Testing that those commands answered proved nothing about the flags.

All twelve are fixed in 0.8.2. None of them was a broken command. They were broken sequences, broken states, bad timing, and missing steps, and testing one command at a time cannot produce any of those. Building a game produces all of them.

What a build session looks like#

The session follows the rules the agent skill teaches: ask the live engine instead of guessing, verify state by reading it back, and stop to root-cause anything odd, including a retry that “fixes” a failure. A wrong number is a finding. An error with useless advice is a finding. A flag that does nothing is a finding.

Each session also leaves a playable sample behind in the test project, so the next sweep has one more real scene to run against.

The build gate now also runs in a scripted, repeatable form: fixed scenarios handed to a blind worker agent and graded by deterministic checks against the live editor. Agent evals describes the harness.