TL;DR: On the Dwarkesh podcast, Noam Brown of OpenAI described a system of roughly 10,000 agents that spent 130 billion tokens over 88 hours on a single problem. This is his account, not independently…
Evaluation
3 posts tagged Evaluation · all tags
2026
When Your Benchmark Runs for 88 Hours
The Gate That Couldn't See the Defect
TL;DR: I am building narrated demo videos for an upcoming conference, and one of them went back for rework six times. Three of my own quality checks were involved, and each failed differently. One ran…
Your AI Models Have an Expiry Date — A Practical Guide to Model Lifecycle Management
Introduction, The Promise I Made In my previous article [1], I explored the maintenance trap in IT, how software systems are more like plants than stones, requiring constant care. I ended with a cliff…