The Beach Test: Why Big Neural Networks Stop Getting Stuck

It’s beach season

It is holiday season, which in my house means a beach, a bucket, and the eternal engineering debate: do we build a castle or dig a hole. My kids do both, usually at the same time, usually undermining each other’s foundations. And somewhere down the beach there is always a group of our lovely Dutch neighbours who - the running family joke goes - have come to reclaim a bit of coastline by flooding it one bucket at a time. It isn’t true, of course. It’s just the kind of fond nonsense you tell on a beach. Everyone digs, everyone builds, everyone’s castle eventually loses to the tide.

But the beach is hiding a genuinely deep piece of physics, and it is closely analogous to the physics behind one of the most-asked questions I get from engineering leaders: why do our models have to be so enormous - and doesn’t “bigger” just mean “memorizes more”?

The answer came into focus for me listening to Matthieu Wyart, a theoretical physicist at EPFL and Johns Hopkins, on Machine Learning Street Talk [1]. Wyart’s home discipline is glassy and granular matter - window glass, suspensions, and yes, sand. He spent years studying exactly what makes a pile of grains flow or lock up. Then, about nine years ago, he pointed that same lens at deep learning. The connection he found is the subject of this post.

The physics of a good sandcastle

Start with the sand itself. Loose, dry sand flows: tilt it and it slides, because each grain has room to rearrange and there is no rigid structure holding it in place. Pack the grains densely enough - or confine them - and something changes qualitatively: the grains lock into a force-bearing contact network, the pile becomes rigid, and it can hold a steep wall. Physicists call the onset of that rigidity a jamming transition: a disordered heap of particles rigidifies once it is packed densely enough for the grains to constrain each other [7]. (A real sandcastle gets extra help from water - capillary bridges between grains add cohesion, which is what lets castles be sculpted at all [8]. That is a related but distinct effect; the piece of physics we need here is the packing-driven rigidity, not the water.)

Here is how Wyart describes the flowing side of it:

“If you tilt a layer of sand at some point it’s going to flow. It means that the energy landscape was rough and you are in a metastable state, but you tilted this energy landscape, you had a phase transition, and then the entire system flows.” [1]

That idea - a rough energy surface, full of dips and pockets - is the bridge to machine learning. A rough one is full of dips and pockets: a ball rolling around in it can come to rest in any number of little holes, none of them the true bottom. Physicists call those metastable states. Sand near jamming is a system with a rough surface and a lot of holes.

An analogous transition, in a neural network

Gradient-based training is usually pictured as a ball rolling downhill. You define a loss function - a number that is small when the model fits your data well - and gradient descent adjusts the parameters to make it smaller (with a stochastic wrinkle: it’s SGD, so the ball jitters). So the natural question a physicist asks is: what does that hill actually look like?

Wyart’s group answered it, and the answer is the punchline of this whole post:

“This landscape has exactly the same phase transition as sand. When you are underparameterized, when you don’t have enough parameters, you have a rough landscape with many metastable states, and if you train your machine many times it will end up in different positions where it’s actually stuck. But if you have enough parameters, then suddenly the system can flow - your landscape has many flat valleys which have essentially zero energy.” [1]

In the paper that formalizes this, the language is precise: “a phase transition, analogous to the jamming transition of granular media, delimits the over- and under-parametrized regimes where fitting can or cannot be achieved” [2]. Below the threshold - too few parameters - the network is jammed. The loss surface is rough, training gets stuck in poor minima, and running the same training twice can land you in different bad spots. Cross the threshold - enough parameters - and, in the regimes studied, it unjams: near-zero-loss solutions appear, the poor minima seen below the threshold stop obstructing training, and gradient descent can usually find a way down.

A quick note on the word “analogous,” because it matters and the physics deserves precision. Wyart is not claiming a neural network is a bucket of sand, and he is careful about it. What he is claiming is that the two problems are structurally analogous - both are constraint-satisfaction problems with continuous degrees of freedom - and that this shared structure is why the same kind of transition appears. (In the strict statistical-physics sense, a “universality class” is defined by matching critical behaviour - exponents and scaling functions. The published work [2] deliberately says the network transition is “analogous to” jamming; it does not claim the two are provably the same universality class.) Both systems are doing the same abstract job:

“In both cases what you have are essentially degrees of freedom. In one case those are the particles of sand, in the other case the parameters of your large model, and in both cases the systems are trying to satisfy constraints… if you have a problem of satisfiability of constraints and you have continuous degrees of freedom that can change continuously, then you have a universality class.” [1]

For sand the constraint is “grains must avoid each other.” For a network the constraint is “parameters must fit the data.” More data means more constraints, much as denser packing means more contacts between grains - a correspondence between the two systems, not an identity. It is an analogy, but a load-bearing one: the network transition has been characterized quantitatively, and in the models studied in [2] the test-error cusp near the interpolation threshold coincides with this jamming-like transition - one concrete theoretical realization of the broader phenomenon machine learning calls “double descent” [9], not a claim that every double-descent peak is a granular jamming transition.

Why “bigger” is what makes training work

This reframes what over-parameterization is for. The instinct - mine too, once - is to read a model with billions of parameters as wasteful: surely most of that capacity is slack. The physics says the opposite. The extra parameters are what turn a jammed, holey surface into a flowing one. In the over-parameterized regime, the paper shows, “poor minima of the loss are not encountered during training, because the number of constraints that hinders the dynamics is insufficient to allow for the emergence of stable minima” [2]. There simply aren’t enough constraints to build a wall you get stuck behind.

That reframes part of the answer to “why so big?” In the model classes studied, extra parameters aren’t only about storing more - they change the optimization geometry, suppressing the poor minima that trap a smaller model and opening up near-zero-loss solutions. Size buys trainability, not just capacity. That isn’t the whole story - more parameters also add representation capacity and, yes, more room to memorize - and it is not a general convergence guarantee for arbitrary deep-network training. But within these results, a too-small model behaves less like a leaner big one and more like a jammed one, sitting in the rough regime where training is more of a gamble.

I’ll add the precise caveat, because the same work spells it out: right at the transition, generalization actually gets worse - there is a cusp where classical overfitting lives - and it improves again as you push well past the threshold and the network widens [2]. “Bigger is better for training” is a statement about reaching the over-parameterized regime, not a promise that every point on the way is monotonically better. For trainability, in these regimes, the place you want to be is comfortably over-parameterized, not balanced on the edge - what size you can afford to train and serve is a separate engineering question.

Bigger doesn’t mean memorizing

Which brings us to the fear behind the question. If a model has vastly more parameters than training examples, why doesn’t it just store the training set and call it a day? Isn’t a huge model just a very expensive lookup table?

The surprising answer, at least in the setting where it has been studied most carefully, is that generalization tends to arrive before memorization does. In work on highly over-parameterized diffusion models, Wyart’s group found that “generalization in natural data domains is progressively achieved during training before the onset of memorization,” and that “memorization time is proportional to the dataset size” [3]. Generalization and memorization are best understood as a race between two clocks. Early in training the model discovers the structure of the data - the rules that generate it. Only much later, and much more slowly for larger datasets, does it start to nail down individual examples. Stop at the right time and you get the structure without the rote copying.

That is a result about over-parameterized diffusion models learning specific kinds of structured data, not a universal law of every LLM. But within that setting it punctures the “it just memorizes” intuition: discovering the data’s structure is the fast thing the model does first, and pinning down individual examples is the slow thing it only gets around to later.

The hidden hierarchy of data

So what is this “structure” the model finds early? Wyart’s working hypothesis is that real data - language, images - is approximately compositional: a hierarchy of parts within parts. A scene is made of objects; objects of features; features of edges and textures. A sentence is made of phrases; phrases of words; words of characters. Whether real data satisfies this exactly is an empirical question - but it is a productive modelling assumption.

Depth is the tool that matches this shape - if the data has it. In the Random Hierarchy Model, a synthetic task built with exactly this parts-within-parts structure, Wyart’s group showed that deep networks learn by developing internal representations invariant to exchanging equivalent groups - peeling off one layer of the hierarchy at a time - and that within this model those invariant representations are how they overcome the curse of dimensionality a learner ignoring the structure would face [4]. Crucially, they could estimate how much data it takes: learning becomes possible where correlations between low-level features and the class first become statistically detectable [4]. Whether real language and images meet the model’s assumptions well enough for this to transfer is the empirical open question.

This is also a quiet answer to an old objection from linguistics. Noam Chomsky’s “poverty of the stimulus” argument holds that children can’t possibly learn language from the limited examples they hear, so the structure must be largely innate. Wyart’s counter is that a deep enough architecture has “a huge implicit bias” toward exactly these coarse-grained variables, and that the abstractions “emerge from statistics alone” [1]. You don’t have to build the hierarchy in by hand; depth plus data can recover it when the data is genuinely hierarchical. This is a suggestive counter-argument, not a resolution - I’m reporting a physicist’s conjecture demonstrated on a synthetic model, not closing a debate that has run for sixty years - but it’s a striking one.

Predict your own latents, not the next token

The most forward-looking part of the conversation is about what you ask the model to predict. Today’s dominant recipe - predict the next token - is powerful but hungry. Wyart’s blunt framing: humans “learn languages with 100,000 times less exposition” than machines do [1]. He doesn’t spell out the exact comparison, so treat the number as rhetorical - but the gap is large enough that it says something is architecturally different about how we learn.

His candidate for the difference is where the prediction happens. Instead of predicting raw tokens, predict your own internal, latent representation of the data - the approach behind architectures like JEPA [5]. Wyart puts it plainly:

“Those algorithms that are introspective, that learn from their own latent, are much more powerful in terms of sample complexity.” [1]

And this is now more than intuition. In a recent paper, his group proves a separation on a tractable model of hierarchical language. Holding the other model parameters and the target error fixed, to recover the hidden tree of depth L that generates the data, token-level learning needs a number of samples that grows exponentially in L, while latent prediction needs only polylogarithmic dependence on L [6]. Exponential versus polylogarithmic in the depth is an asymptotic separation between the two methods in that model - not, by itself, a claim about arbitrary architectures, where constants and finite-data regimes also decide what is feasible.

The tempting leap is “so that’s how brains work.” I’d resist planting the flag there. The sample-efficiency gap is real and the latent-prediction mechanism is a serious, testable hypothesis for part of it - but “networks think like brains” is a much bigger claim than “predicting in latent space is more data-efficient on hierarchical data,” and only the second one has a proof attached.

If you’re building on AWS

For those of us who help teams choose and deploy models, this theory cashes out in two practical stances.

When someone asks “why do we need such a large model?” you have a principled part of the answer that isn’t hand-waving about capacity. Over-parameterization pushes a model into a regime where the loss surface is easier to optimize - the size buys trainability, not only memory. That reframes a cost conversation: a large model isn’t just a bigger lookup table, it’s partly paying to be on the trainable side of a transition. The jamming picture is one theoretical lens on why frontier models are large - not a demonstrated cause of their sizing; representation capacity and empirical scaling behaviour matter too. Note what it explains: why the model was big at training time. It is not an argument for picking a bigger model on Amazon Bedrock - what you serve is a separate decision, made on measured task quality, latency, throughput, and price.

When you evaluate where architectures are heading, latent-prediction and world-model approaches are worth taking seriously rather than dismissing as research exotica. The sample-efficiency argument for predicting latents over tokens has a proof behind it now, at least on structured data [6]. If a future generation of models needs far less data to reach the same capability, that changes the economics of everything downstream - fine-tuning, domain adaptation, the amount of proprietary data a customer needs before a model is useful to them. It’s a direction to watch, not a product to buy today.

Make mistakes

I’ll leave the last word to Wyart, because it landed with me as much as the physics did. Asked about how science actually moves, he said:

“If you take risk, I think you should make mistakes. If you never make mistakes, maybe it’s a sign that you’re staying a bit on the beaten path in science. Some of us want to explore the jungle - and in the jungle, you can be wrong.” [1]

A physicist spent a career on sand and glass, then took the risk of pointing that lens at deep learning, and found a transition hiding in the loss surface. The bet paid off. Most beaches, most castles, and most research ideas end up reclaimed by the tide anyway. That’s not a reason to stop building. It’s the whole point of getting to build at all.

So: dig the hole, build the castle, let the neighbours “flood” the beach. And the next time someone frets that a model is too big, tell them about the sand.

Sources

• [1] Machine Learning Street Talk, “Why Deep Networks Don’t Need to Memorize Everything - Matthieu Wyart” (2026-08-10) - the source interview.
• [2] Spigler, Geiger, d’Ascoli, Sagun, Biroli, Wyart, “A jamming transition from under- to over-parametrization affects generalization in deep learning” (2019) - the jamming / loss-surface result.
• [3] “Bigger Isn’t Always Memorizing: Early Stopping Overparameterized Diffusion Models” (arXiv 2505.16959) - generalization before memorization; memorization time scales with dataset size.
• [4] Cagnetta, Petrini, Tomasini, Favero, Wyart, “How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model” (arXiv 2307.02129) - depth recovers hierarchy, beating the curse of dimensionality.
• [5] Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (JEPA)” (arXiv 2301.08243) - a latent-prediction architecture.
• [6] Korchinski, Favero, et al., “Learn from your own latents and not from tokens” (arXiv 2605.27734) - in the paper’s hierarchical-language model, latent prediction has polylogarithmic sample dependence on hierarchy depth L, versus exponential dependence for token-level learning.
• [7] Liu & Nagel, “Jamming is not just cool any more”, Nature 396, 21-22 (1998) - the canonical framing of jamming as a common transition across disordered systems.
• [8] Scheel et al., “Morphological clues to wet granular pile stability”, Nature Materials 7, 189-193 (2008) - liquid surface tension stiffens wet sand, which “enables sand castles to be sculpted.”
• [9] Belkin, Hsu, Ma, Mandal, “Reconciling modern machine learning practice and the bias-variance trade-off” (2019) - the paper that named the “double descent” curve.

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• ComplexityTheory
• DeepLearning
• Generalization
• FoundationModels