Compression Is Intelligence

written by Stefan Christoph

- 8 minutes read
TL;DR: Information theory says prediction and compression are the same thing measured two ways. That gives the cleanest mental model I know for what a large language model does: it is a compressor of its training text, and predicting the next token well is the same as compressing that text well. The framing is not mine. It traces to Marcus Hutter’s compression prize, and a 2024 paper showed that how well a model compresses tracks its capability almost linearly. The practical payoff sits one level up: if the model is a fixed compressor you rent by the token, then context is the real lever, and curating it well is feeding the compressor better priors.

Last September I wrote a short post asking whether intelligence is just compression. A Machine Learning Street Talk episode had planted the idea and I could not shake it, so I posted the question and left it open [1]. Recently 3Blue1Brown released a video that gives the cleanest mechanical intuition I have seen for why the question is even reasonable [2]. It also changed how I think about the part of my own work that actually moves the needle: the context I hand a model.

This is the second piece in a short “Whiteboard to Cloud” series, where I take an idea that lives on an academic whiteboard and follow it to where you would actually build on it. The first part looked at vector search.

The whiteboard: shorter codes for likely things

Grant Sanderson opens with a deceptively small puzzle [2]. Imagine a robot on a faraway moon that takes four instructions: up, down, left, right. The instructions are not equally likely. Half are “up”, a quarter are “down”, an eighth are “left”, an eighth are “right”. You have to send them as a stream of bits, and bandwidth is expensive. What is the most efficient encoding?

The obvious answer gives every instruction two bits: 00, 01, 10, 11. Clean, but wasteful, because it spends as many bits on the rare “left” as on the common “up”. The clever answer uses a short code for the frequent instruction and longer codes for the rare ones. Send a single bit for “up”, a couple more for “down”, and so on. Because “up” dominates the stream, the average cost per instruction drops below two bits.

Push that idea to its limit and you rediscover a quantity Claude Shannon named in the 1940s: entropy. Entropy is the floor. It is the smallest average number of bits you can use to encode a source, and it depends entirely on how predictable that source is. A stream where one symbol is almost certain carries little information and compresses to almost nothing. A stream where every symbol is a coin-flip surprise carries maximum information and resists compression. The more predictable the data, the fewer bits you need to describe it.

Flow from patterns in the data, to a model that learns them, to a shorter description (better compression), which is the same as better prediction

Learning the regularities in data is what lets you describe it in fewer bits, and that is the same skill as predicting it.

Prediction and compression are the same coin

Here is the turn that makes the whole thing click. One of the conclusions of information theory is that prediction and compression are mathematically equivalent [2]. They are two sides of the same coin. To compress data well you need a good model of what comes next, and a good model of what comes next is exactly what lets you assign short codes to likely things.

That equivalence runs straight into how language models are trained. The pre-training objective is usually described as next-token prediction, measured with something called cross-entropy loss. The term cross-entropy comes directly from Shannon’s information theory. So you can reframe the objective: a language model is not really being trained to predict the next token for its own sake. It is being trained to be the most efficient possible compressor of its training text. Predicting the next word well and compressing the text well are the same achievement, scored differently.

The video is careful here, and I want to keep its caution. “Compression is intelligence” as a flat statement is hard to judge, because intelligence is a squishy, ill-defined word. The safer and more defensible claim is that the mathematics of compression is strangely relevant to how these systems work. I am not arguing that a compressor is a mind. I am saying it is the most useful intuition I have for what the model is doing under the hood.

Where the thesis comes from

The slogan is neither mine nor new. Marcus Hutter has run a compression prize since 2006, with a simple premise: take a fixed snapshot of Wikipedia and compress it as small as you can, because better compression of human knowledge is a reasonable proxy for understanding it [3]. The competition treats “make this smaller without losing anything” as a stand-in for “model the world better”.

For a long time that was a philosophical stance more than a measured fact. A 2024 paper closed some of that gap. “Compression Represents Intelligence Linearly” tested actual language models and found that how well a model compresses text tracks its score on capability benchmarks in a nearly linear relationship [4]. The better compressor is, on average, the more capable model. That does not prove the slogan, but it gives it empirical legs in the specific domain we care about.

The cloud: if the model is a fixed compressor, context is the lever

This is where the whiteboard meets the thing you deploy. Once a model is trained, its compressor is fixed. The regularities of its training data are baked into the weights, and you rent that frozen compressor by the token. The part you control at run time is what you feed it: the context.

Read through the compression lens, good context engineering is feeding the compressor better priors. You are supplying the specific facts, examples, and constraints that let the model predict your situation instead of the average of the internet. That is the same conclusion I reached writing about context engineering, from a different direction [5]: better context is more precise context, not more of it. The compression view explains why. Irrelevant context is noise the compressor still has to account for, and it competes with the signal you actually wanted.

It also explains an economic pattern I keep running into. The cheapest model with the right context usually beats the most expensive model guessing without it, because you have done the compressor’s hardest work for it by narrowing what it has to predict.

Two earlier posts fit the same frame. “LLMs Don’t Do Math” [6] makes more sense once you see the model as a compressor of how math is written rather than a calculator that runs the arithmetic. “What Reasoning Actually Means” [7] reads as spending extra compute at inference to find a better-compressed path to an answer, rather than emitting the first one.

The honest edge: no compression algorithm for experience

Where does this stop? Andy Jassy has a line I keep coming back to: there is no compression algorithm for experience. Compression captures the regularities in what has already been written down. It is very good at the seen and the sayable. It says much less about the part of intelligence that comes from acting in the world, being wrong, and updating. That is the boundary I would draw, and it is the same tension I left open last September [1]. The compression lens is the best intuition I have for what a language model does. It is not a theory of mind, and the people who explain it best are the first to say so.

If You’re Running This on AWS

If context is the lever, then re-sending the same large context on every call is paying to compress the same priors over and over. Amazon Bedrock prompt caching is the practical fix: you mark a stable prefix of the context, and repeated calls reuse the cached prefix instead of reprocessing it, which cuts latency and cost on the part of the input that does not change [8]. It is the operational version of the idea in this post. Curate the priors once, then feed them cheaply on every turn. The model stays a fixed compressor; you just stop paying full price to hand it the same context twice.

What is your mental model for what a language model actually does under the hood? I am curious whether the compression framing lands for you, or whether you reach for something else.

Sources

Stefan Christoph

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

❤️ Created with the support of AI (Kiro)