The Beach Test: Why Big Neural Networks Stop Getting Stuck

Tue, 11 Aug 2026 00:00:00 +0000

TL;DR: Dry sand flows; densely packed, confined sand jams into a rigid pile - and the jamming transition that describes that rigidity is, in the work of physicist Matthieu Wyart, analogous to a transition a neural network’s loss surface crosses as it becomes overparameterized. Below a size threshold the loss surface is rough and full of holes to get stuck in; above it, in the model classes Wyart’s group studied, it develops flat near-zero-loss regions where gradient descent can usually find a way down. So one theoretical reason to make models big is not to memorize more but to make optimization more tractable - and, in overparameterized diffusion models on structured data, learning the structure can even precede memorizing individual examples. (A real wet sandcastle, to be precise, is held together mostly by capillary bridges between grains - a related but distinct effect from jamming.)

It’s beach season

It is holiday season, which in my house means a beach, a bucket, and the eternal engineering debate: do we build a castle or dig a hole. My kids do both, usually at the same time, usually undermining each other’s foundations. And somewhere down the beach there is always a group of our lovely Dutch neighbours who - the running family joke goes - have come to reclaim a bit of coastline by flooding it one bucket at a time. It isn’t true, of course. It’s just the kind of fond nonsense you tell on a beach. Everyone digs, everyone builds, everyone’s castle eventually loses to the tide.

schristoph.online

The Beach Test: Why Big Neural Networks Stop Getting Stuck

It’s beach season