Compression Is Intelligence

Mon, 06 Jul 2026 00:00:00 +0000

TL;DR: Information theory says prediction and compression are the same thing measured two ways. That gives the cleanest mental model I know for what a large language model does: it is a compressor of its training text, and predicting the next token well is the same as compressing that text well. The framing is not mine. It traces to Marcus Hutter’s compression prize, and a 2024 paper showed that how well a model compresses tracks its capability almost linearly. The practical payoff sits one level up: if the model is a fixed compressor you rent by the token, then context is the real lever, and curating it well is feeding the compressor better priors.

Last September I wrote a short post asking whether intelligence is just compression. A Machine Learning Street Talk episode had planted the idea and I could not shake it, so I posted the question and left it open [1]. Recently 3Blue1Brown released a video that gives the cleanest mechanical intuition I have seen for why the question is even reasonable [2]. It also changed how I think about the part of my own work that actually moves the needle: the context I hand a model.

schristoph.online

Compression Is Intelligence