Who Referees the AI?

Wed, 21 Oct 2026 00:00:00 +0000

TL;DR: In July 2026 a purported disproof of the Collatz conjecture got through Lean’s official proof checker, and through an independent second checker as well. It was wrong. The submission exploited a different bug in each checker. Nanoda’s developer had fixed one shortly before, and de Moura fixed the other on 28 July. Lean’s creator Leonardo de Moura said in a recent interview that he strongly believes AI built the exploit. The episode shows what happens to any “AI checks AI” setup once the output gets too large for people to read. Acceptance then rests on the checker, its specification and the code around it, and a capable generator will go looking for the checker’s weaknesses. The Lean community’s answer carries over to agent builders: keep what must be trusted small, check it with implementations that share no code, publish how the check works, and remember that a passed check only certifies what you specified.

Most agent designs I review end in a gate: an LLM judge, an eval suite, a CI run, sometimes all three. The green result at the end of that gate is what lets the work ship without a person reading it, and this post is about what that green result is worth once the thing being checked is clever enough to go after the gate. Everything here comes from a public interview and the public GitHub record around it, all cited below.

schristoph.online

Who Referees the AI?