The Quadratic Wall: The Algorithmic Race to Beat O(n²) Attention

Wed, 22 Jul 2026 00:00:00 +0000

TL;DR: Standard self-attention costs on the order of n² in the sequence length n, in both arithmetic and the attention-score memory it materialises. Almost every long-context architecture headline of the last few years is an attempt to beat that wall, and the honest way to read them is by which cost axis each one moves. FlashAttention keeps exact O(n²) arithmetic but cuts materialised auxiliary memory from O(n²) to O(n) and sharply reduces the memory it moves. Sparse attention (when each token attends to a sublinear number of positions), linear attention, and state-space models like Mamba (which its authors report reaching roughly linear time in the sequence length, for a fixed state width) lower the arithmetic exponent itself by constraining or replacing exact all-pairs attention. Those axes — arithmetic, memory, data movement, latency — do not move together, so no single technique simply “wins,” and the field has not crowned one. Neither is a free win, and the field has not crowned a winner. The practical takeaway is older than any of these papers: the cheapest quadratic is the one you never trigger, so context discipline and retrieval can beat brute-forcing a longer prompt — when the retrieval earns its keep, which is an end-to-end question to measure.

Back at university, one exercise showed up in every algorithms course in a slightly different costume: here is the obvious quadratic solution, now can you do better? Sometimes you could drop it to n log n, sometimes only to a smaller constant, and sometimes the honest answer was that the quadratic was inherent and the interesting work was elsewhere. That instinct, look at an O(n²) and ask what it would take to beat it, turns out to be the cleanest way I know to read the modern architecture race.

schristoph.online

The Quadratic Wall: The Algorithmic Race to Beat O(n²) Attention