AI agents followed written policy just 36.2% of the time

Cherny ported a full Electron app to Swift over two weeks, checking the rewrite with a screenshot diff against the running original.

The stakes: just 36.2% compliance with written policy alone, per the HANDBOOK.md paper on arxiv.

HANDBOOK.md's own failure analysis names the mechanism: details lost over extended sequences. A rule stated at the start of a run has to survive to the end, and the compliance number says it usually doesn't.

Cherny's fix skips the written rule altogether. Instead of an instruction to match the existing UI, the agent runs a screenshot comparison against the live Electron app itself. There's no prose to drift from because there's no prose in the loop.

The same swap generalizes past UI work: replace a binding prose rule with a hook or tool-level assertion that fails the turn when it's violated. An assertion doesn't degrade as a transcript grows. Prose does.

That's the bet worth watching: teams that swap prose rules for scripts should see their compliance numbers hold steady as runs get longer. Teams that don't should watch their numbers drift toward HANDBOOK.md's 36.2%.