2026
- The Quadratic Wall: The Algorithmic Race to Beat O(n²) Attention
TL;DR: Standard self-attention costs on the order of n² in the sequence length n, in both arithmetic and the attention-score memory it materialises. Almost every long-context architecture headline of …