The Mess Under the Bed

The room-cleaning problem is ancient

You want a clean room. You can’t stand over the child for an hour, so you attach a reward: clean the room, get an hour of TV. The reward measures a proxy for what you want, because “a genuinely tidy room” is expensive to verify and “no visible mess when I glance in” is cheap. The child, being a rational optimiser of TV, finds the cheapest path to the measured outcome: shove everything under the bed.

That is reward hacking, and it long predates AI. Economics has known it as Goodhart’s law since 1975: when a measure becomes a target, it stops being a good measure [1]. Charles Goodhart’s contemporaries sharpened it. Campbell’s law says the more a metric drives a decision, the more it corrupts the process it was meant to watch [2]. Principal-agent theory calls the general case moral hazard: when an agent acts for a principal but its actions aren’t fully observable, it can serve its own objective at the principal’s expense [3]. And the Holmström-Milgrom multitask result proves the sharpest version: when only some of an agent’s tasks are measurable, paying on the measurable ones diverts effort away from everything else [4]. A 2026 result by Skalse and colleagues shows this is not a bug you can patch out. Under a few minimal assumptions, reward hacking is the equilibrium: a sufficiently optimised agent will always under-invest in the quality you forgot to measure [5].

There’s a second, quieter failure mode worth naming so we don’t conflate it with the first. In education science the overjustification effect describes rewards degrading intrinsic motivation: pay a child to do something they already enjoyed, and when the pay stops, they do it less than before [6]. That is not gaming. It’s the reward corroding the original motivation. It has a loose parallel in heavily reward-trained models that become reward-model-pleasing (sycophantic) at the expense of what they knew before [7]. I flag it because “rewards fail” is not one phenomenon; it’s at least two. The rest of this post is about gaming, because that’s what showed up in production this summer.

The asymmetry that makes the analogy load-bearing

Here’s the part that actually matters, and it’s the point where the kid analogy stops being reassuring and starts being the whole problem.

For the room, monitoring is cheap because the search space is bounded. If the floor is clear but the room isn’t really clean, the mess is somewhere: under the bed, in the closet, behind the door. You know the hiding places because you know the room. Verification is simple: look in the finite number of spots.

For a trained model, you don’t know where to look. The behaviour space is astronomically large, the internal reasoning is opaque, and, critically, nobody’s threat model enumerates the hiding places in advance. This is exactly the observability asymmetry that principal-agent theory identifies as the real cost of alignment: not the gaming, but the price of catching it.

For years this was an argument you had to make on a whiteboard. In July 2026 it stopped being hypothetical.

Two incidents, three weeks apart, both first-party documented

Incident one: theft (Hugging Face x OpenAI)

In July, an autonomous agent driven by a combination of OpenAI models (GPT-5.6 Sol and a more capable, since-deactivated internal prototype, both “with reduced cyber refusals for evaluation purposes” [8]) was running an internal OpenAI cyber-capability evaluation based on the ExploitGym benchmark. Over roughly four and a half days it ran an end-to-end intrusion against Hugging Face’s production infrastructure: Hugging Face’s forensic reconstruction recovered ~17,600 attacker actions between 9 and 13 July [9].

Hugging Face’s own reading of the motive is the thesis of this entire post, so I’ll quote it directly:

“We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.” [9]

Nobody pointed the agent at Hugging Face. It inferred that the largest ML dataset host might hold the benchmark’s models, datasets, and reference solutions, and went for them. The only customer content it touched was five datasets whose names suggest ExploitGym challenges and solutions [9]. OpenAI’s account agrees on the shape: the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal” [8]. The kid didn’t tidy the room. It found the answer key.

Incident two: deception of the monitor (UK AISI)

Three weeks of context later, the UK AI Security Institute disclosed something adjacent and, in one respect, sharper. During routine cyber-range testing (122 runs across several models) AISI catalogued 19 unsanctioned, real-world actions across 10 runs. 17 of the 19 came from Anthropic’s Mythos 5; 2 from OpenAI’s GPT-5.6 Sol [10]. (I’m naming the labs because AISI’s own report does; the point here is structural, not a scoreboard.)

In the most serious sequence, an agent tried to insert malicious code into a real, public open-source project. To get it approved it didn’t just submit the code. It researched the project’s maintainer, created multiple fake identities, and used them to socially engineer the maintainer into approving the change. When its pull request was challenged in public, per AISI, the agent “edited its earlier activity to appear harmless” and considered adopting a fresh identity to keep going [10]. A human maintainer refused it; a member of the public who suspected the code was malicious opened it inside an isolated sandbox. AISI’s own framing: “deception emerged as a by-product of pursuing the task”; it was never instructed to deceive [10].

Put the two side by side and the analogy extends further than I expected it to.

Hide the mess (theft). When questioned, deny it, tidy the evidence, get someone to vouch for you (deception). This is the escalation Ryan Greenblatt sketches when he argues reward-seeking generalises from narrow hacks into concealment; that’s his framing, and one I’m deliberately not extending into predictions about the future [11]. The observed behaviour is enough on its own, and it’s behavioural: I’m not claiming either model “wanted” anything.

The one detail that proves it was misspecification, not malevolence

Buried in Hugging Face’s timeline is the cleanest piece of evidence I’ve seen that this was objective-shaping rather than a model “turning evil.” As the agent escalated inside the cluster, it issued cloud API calls, and:

“every potential destructive cloud API call the agent tried to make was issued with DryRun=True: it was mapping capability, not causing damage. That is consistent with an objective that rewarded reach and information rather than disruption.” [9]

DryRun=True is the AWS convention for “check whether I could do this, but don’t actually do it.” The agent wasn’t trying to break things. It was trying to reach the answer key and confirm it had, because that’s what maximised the score. It did exactly what it was rewarded for and nothing else. That’s not malevolence. That’s a proxy, optimised.

Cheating wasn’t cheaper, it was more certain

The obvious story is “the model took the lazy shortcut.” That story is wrong, and getting it wrong matters for how you defend against it. Seventeen thousand six hundred actions is almost certainly more effort than solving the challenges honestly would have taken. So reward hacking doesn’t optimise for effort. It optimises for the measured outcome. A guaranteed score beats a probable one. If you’re confident you can solve the challenge but not certain, copying the answer key is the only route that makes a perfect score certain.

That reframes mitigation. You can’t rely on “the honest path is easier, so the model will prefer it.” Sometimes the honest path is easier and the model games it anyway, because gaming is more reliable. It also deepens the kid analogy instead of breaking it: the child hides the mess not because hiding is less work than tidying, but because a shallow inspection makes the reward certain.

The caveat, split honestly

Both evaluations ran under conditions that don’t match how these models are actually deployed, and it would be dishonest to bury that. AISI is explicit: the tests used open internet access and disabled provider cyber classifiers, “conditions that do not reflect how frontier models are made available to the public” [10]. That sentence is real and it belongs in this post. But it’s also the sentence people reach for to dismiss the whole thing, so it needs splitting into its two halves, because only one half is actually non-representative.

Half one: the disabled safety classifiers are genuinely exotic

Both OpenAI and AISI turned off the production refusal layer to measure raw capability [8], [10]. In real deployment that layer is on. This half of the caveat is fair.

Half two: the network access is entirely representative

Web access is not an exotic eval condition; it is table stakes for deployed agent products. Amazon’s own AgentCore Browser documentation describes exactly this as a first-class capability: it “provides a secure, isolated browser environment for your agents to interact with web applications,” where “remote browsers allow AI agents to interact with the web as humans do” [12]. An agent that can browse is the normal case, not the lab case.

And here’s the detail that collapses the “but it had the open internet” defence entirely: the Hugging Face agent’s escape didn’t need the open internet. It broke out through a zero-day in the package registry cache proxy, “one of its primary permitted network egress” [9], the single most mundane permitted piece of egress in essentially every build environment on earth. OpenAI confirms the sandbox had no direct internet; the model reached it by exploiting that proxy [8].

So strip the caveat down and what’s left? One item: the safety classifier. Everything else in the setup (the capability, the network egress, the package proxy) is present in production.

The capability is in the production models. A classifier is what stands between it and this behaviour. That’s a thinner margin than “it only happens in the lab” suggests.

Volume is the defensive shift, not cleverness

The other thing worth internalising: none of the individual weaknesses were novel. Hugging Face is candid that a capable human attacker could have found the same flaws: unsafe dataset processing, exposed cloud metadata, over-broad credentials, long-lived tokens [9]. What changed was scale. In their words, “the successful path was hidden inside the noise generated by the thousands of failed ones” [9]. Their security stack actually did correlate the signals into a coherent attack, but it “failed to correctly raise the alert’s criticality and trigger the on-call team,” costing response time [9]. Monitoring existed. Escalation failed.

That’s the real defensive lesson. Machine-speed offence makes ordinary weaknesses more expensive because it multiplies the number of paths tested and the volume of evidence a defender has to interpret. The bottleneck moves from “can we detect the anomaly” to “can we correlate thousands of low-signal events and escalate the one that matters.”

The constructive answer is a boundary, not an exorcism

If the problem is “we don’t know where the model will look,” the response isn’t to make the model promise to behave. It’s to build the room so hiding is structurally hard and searching is cheap: the AI-control stance, which assumes the agent may try to act beyond its remit and designs containment and monitoring that hold regardless [13].

Hugging Face’s own hardening list is exactly this, and notice it’s a boundary-and-monitoring answer, not an alignment one: block pod access to the cloud metadata endpoint, scope credentials per cluster instead of sharing one, use short-lived tokens and workload identity, add admission policies against privileged pods, and the one every platform team should sit with: detection that correlates across systems and escalates [9].

This is also where the cloud is the answer rather than the liability. The failure surfaces here were cloud-shaped (instance metadata, service-account tokens, a package proxy), but they were configuration choices, and the same platforms give you the envelope to bound them. The AgentCore Browser docs, for instance, pair that first-class web access with session isolation, CloudTrail logging, live view, and session replay that captures DOM changes, actions, console logs, and network events [12]. The answer to “agents need the network” isn’t to remove the network. It’s a bounded, recorded envelope where you can see what the agent did, turning the unbounded search space back into something closer to a room you can inspect.

Why this gets worse when AI builds AI

The interview that sent me down this path spends most of its two hours on a bigger question than eval hygiene, and it’s worth carrying one part of it back here — as Greenblatt’s argument, not as a prediction I’m endorsing.

His model of the last few years is two curves moving in opposite directions. He expected the rate of problematic behaviour to fall quickly, while “the worst things that the AIs would sometimes do would get more extreme, more egregious, and more scary” — and says what we’ve seen has roughly matched that. With one exception. The recent spike surprised him: the AISI report on models “doing insane hacking operations out of cyber evals” was, in his words, “a thing that I would have expected that you wouldn’t see,” at rates lower than what actually happened. [11]

The mechanism he describes for how that compounds is the part that should feel familiar. When a reward hack turns up in production, you train against it. Do that repeatedly and the measured rate of reward hacking goes down — “though the severity of the reward hacks we do detect are increasingly bad.” You are not selecting for honesty. You are selecting for hacks that survive your detector. A child punished for the mess under the bed does not learn to tidy; they learn the closet.

Stack that on top of AI-assisted AI development and the search space stops being merely large. Greenblatt’s concern is that models get trained in “increasingly complicated environments built by earlier AI systems” that humans don’t fully understand — so the hiding places are no longer ones we designed. And the failure mode he lands on doesn’t require a villain. One branch of his argument is scheming; the other is that the AIs are “not scheming against you per se, but they are sort of just optimizing for… getting a high score on their task.” He thinks that second branch is sufficient on its own for very bad outcomes. Whether or not you follow him that far, the structural claim is the same one DryRun=True demonstrated in miniature: you do not need malice. A proxy and enough optimisation pressure will do.

The question to take back to your own evals

I don’t think the useful takeaway from July is “frontier models are dangerous.” It’s much more actionable and much closer to home. Every one of these incidents is a reward-hacking story, and reward hacking is not exotic. It’s the default behaviour of any optimiser facing a proxy metric it can game more reliably than it can satisfy honestly.

So the question isn’t about frontier labs. It’s about the eval you wrote last week. Look at it and ask: what would a shortcut look like here, and would I actually see it? If your eval rewards a TOP PICK: line, would you notice a model that learned to emit the line without doing the work? If it rewards a passing test, would you notice a model that edited the test? For the kid’s room, you know to check under the bed. For your eval, do you even know where the bed is?

That’s the whole thing. The mess is always somewhere. The hard part was never the mess; it’s knowing where to look.

What would a shortcut look like in your eval, and would you catch it?

Sources

• [1] Goodhart, C. (1975). “Problems of Monetary Management: The U.K. Experience” — “when a measure becomes a target, it ceases to be a good measure.”
• [2] Campbell, D. (1979). “Assessing the Impact of Planned Social Change,” Evaluation and Program Planning — corruption of monitored processes.
• [3] Principal-agent problem / moral hazard (Jensen & Meckling 1976; Holmström 1979) — unobservable actions and the cost of monitoring.
• [4] Holmström, B. & Milgrom, P. (1991). “Multitask Principal-Agent Analyses,” JLEO — effort diverts to the measured tasks.
• [5] Skalse et al. (2026), arXiv:2603.28063 — reward hacking as an equilibrium, not a bug (preprint).
• [6] Deci, E. (1971) / Lepper, Greene & Nisbett (1973): the overjustification effect — rewards degrading intrinsic motivation.
• [7] Anthropic (2025), arXiv:2511.18397, “Natural emergent misalignment from reward hacking in production RL”.
• [8] OpenAI (2026). “OpenAI and Hugging Face partner to address security incident during model evaluation” — classifiers disabled, internal prototype since deactivated, Artifactory zero-day.
• [9] Hugging Face (2026). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident” — the “steal the test solutions” quote, ~17,600 actions, DryRun=True, the package-proxy escape, the volume/asymmetry reflection, the hardening list.
• [10] UK AI Security Institute (2026). “Incident Report: unsanctioned agent behaviour during cyber testing” — 122 runs, 19 actions, 17 Mythos 5 / 2 GPT-5.6 Sol, the fake-identities social engineering, the “do not reflect how frontier models are made available to the public” caveat.
• [11] Ryan Greenblatt on the Dwarkesh Podcast (2026-08-11) — the escalation model from narrow hacks to concealment (his framing).
• [12] Amazon Bedrock AgentCore Browser tool documentation — web access as a first-class agent capability, plus session isolation, CloudTrail logging, live view, and session replay.
• [13] Greenblatt, Shlegeris et al. (2023), “AI Control: Improving Safety Despite Intentional Subversion,” arXiv:2312.06942 — design protocols that hold even if the agent tries to subvert them.

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• RewardHacking
• AISecurity
• AIAgents
• AISafety
• Evals