When I say "toy models", what do I mean?

Author: Ziming Liu (刘子鸣)

Motivation

When I talk with colleagues and students, they are often confused by what I call “toy models.” The confusion usually comes from two questions:
(1) What exactly is a toy model, and how does one come up with it?
(2) Toy models sound useless—what are they good for?

The second question is essentially a special case of the first: only after you become familiar with a tool do you understand what it is useful for.

The goal of this blog is therefore twofold: to clearly explain what a toy model is (for AI researchers), and to provide a methodology—how to construct toy models.

What does a toy model mean in physics?

The term toy model comes from physics—at least, that is where I first encountered and grew fond of it during my physics training. It precisely captures a physicist’s research aesthetic: striking a delicate balance between simplicity and relevance.

For example, magnetic materials exhibit a critical temperature (the Curie temperature): above it they are paramagnetic, below it ferromagnetic. To understand this phenomenon, Ernest Ising proposed the Ising model. The simplest Ising model, which only considers interactions between neighboring atoms, is already sufficient to produce this phase transition and even predict Curie temperatures for some materials.

As another example, although animal shapes vary enormously, scientists have found that heat dissipation and energy consumption depend primarily on an animal’s size, not its detailed shape. For studying heat dissipation, a “spherical cow” (or “spherical chicken”) is therefore an excellent toy model.

So what is the philosophy behind toy models in physics?

Einstein: Everything should be made as simple as possible, but not simpler.

Max Tegmark (my PhD advisor):
If we don’t understand something, we should resort to a simpler thing that we still don’t understand… until we get to something we can start to understand.

You might argue that constructing toy models sounds like it requires genius. For instance, how did Ising come up with the Ising model? My argument is that building toy models requires at most 10% talent and 90% hard work—provided you have the right methodology. With the right methodology, anyone with 90% effort can outperform a “genius” relying on 10% talent alone. This is precisely the purpose of this article: to teach a methodology for constructing toy models.

Before doing so, however, I must emphasize that toy models from physics cannot be transplanted wholesale into AI. We need to redefine what a toy model means in AI.

Toy model in AI? Model = Network + Data

In AI, models and data must be clearly distinguished.

In physics, a toy model is just a model whose purpose is to explain data. Here, “data” refers to physical phenomena or laws, which are fixed in our universe and cannot be arbitrarily controlled.

In AI, however, a “toy model” can refer either to the model (neural network) or to the data—and these two factors can be controlled independently. In particular, we have the superpower of designing and manipulating data, something that physics does not allow. To avoid ambiguity, when I specifically mean a neural network, I will use the word Network rather than Model. In short:

Model = Network + Data

This gives rise to four regimes: the network can be simple or complex, and the data can be simple or complex. The four regimes correspond to existing research paradigms:

Zone 1 — simple data, simple network.
This is where much of ML theory lives. To prove theorems, both the data and the model must satisfy strong simplifying assumptions.
Zone 2 — simple data, complex network.
This is the regime of Zeyuan Zhu’s Physics of LLMs: carefully designed, controlled tasks paired with real-world LLMs.
Zone 3 — complex data, simple network.
This is the vibe of traditional science. Given complex observed data, scientists propose simple models that can generate it.
Zone 4 — complex data, complex network.
This is mainstream modern AI.

Definition of a Toy Model
Anything outside Zone 4—i.e., Zones 1, 2, and 3—belongs to the realm of toy models. Which toy model one should study depends entirely on one’s goal. Here are a few concrete examples:

Zone 1: I want to study the emergence of sparse attention and gain as much quantitative, possibly analytic, understanding as possible. Both the network and the data should be as simple as possible.
See: “The emergence of sparse attention: impact of data distribution and benefits of repetition”.
Zone 2: I want to decompose LLM capabilities by studying their behavior on simpler, more controllable tasks.
See Zeyuan Zhu’s Physics of LLM.
Zone 3: I want to test whether data possesses a certain structure. For example, if an equivariant network fits the data well, the data likely has that symmetry; if not, it probably doesn’t. This closely resembles hypothesis testing in traditional science.
See: Physics-augmented learning, AI Feynman.

An important mindset: interpolation

A common criticism of toy models is:

What works in a toy model may not work in a real model—and even if it does, it may work for entirely different reasons.

At its core, this criticism argues that toy models and real models feel completely disconnected, making transfer nontrivial.

The solution: interpolation.
We should find a path connecting toy models and real models—akin to the notion of homotopy in mathematics.

Why interpolate?

Knowledge is created at the boundary between the known and the unknown. If we fully understand a toy model and the real model behaves differently, there must be a phase change (sharp or smooth) along the path connecting them. Understanding the phase change is key to understanding the difference between the toy setup and the real setup. Starting from the toy model (known) and gradually moving toward the real model (unknown) mirrors how humans learn. Fully known is trivial; fully unknown is confusing. The sweet spot—half-known, half-unknown—is where information gain is maximized. Research is, at its core, an information-gathering game.

How to interpolate?

Directionality: move from simple to complex, or from complex to simple.
Locality: change one feature at a time; avoid overly large jumps.

Here are a few (abstracted) examples from projects I’ve supervised:

A complex model shows strange behavior on real data (Zone 4). We hypothesize similar behavior might appear on simpler data. A student constructs a simple dataset (Zone 2), but the behavior differs. After failing to “complexify” the simple data to match reality, we realize the real data itself has a tunable parameter that can simplify it.
Lesson: not only can you make simple data more complex—you can also make complex data simpler.
A diffusion model exhibits an intriguing phenomenon on CIFAR-10 (Zone 4). A student builds a toy setup (Zone 1): simple 2D data and a simple MLP. The phenomenon appears, but it’s unclear whether the mechanisms match. The toy metric doesn’t transfer. I then emphasize interpolation: network and data complexity can be adjusted independently. For example, keep a complex network (U-Net or DiT) but simplify the data to one prototype image per CIFAR class—only ten images total. This yields complex networks with simple data (Zone 2).
Lesson: maintain an interpolation mindset. Simplicity vs. complexity is not binary—it is continuous and multidimensional.

The map of my “physics of AI”

Finally, how does the Zone 1–4 framework guide my Physics of AI program?

The ultimate goal is, of course, Zone 4. But I argue that we should start from foundations in Zone 1—while distinguishing this effort from ML theory. ML theory is theory-driven, prioritizing rigor and carefully “feeling the elephant’s trunk.” Physics of AI is experiment- and phenomenon-driven, prioritizing intuition and breadth—touching many sides of the elephant and assembling a coherent picture.

I view Physics of AI as progressing through three stages:

Stage 1: Study training dynamics on simple data and simple models. This may sound trivial, but it isn’t—there is a rich landscape of dynamics and many failure modes, some of which appear connected to failures in LLMs. The goal is to catalog phenomena (e.g., grokking). In physics terms: what are the behaviors of the fundamental particles? In AI, the “particles” (attention, MLPs, etc.) are already given; we only need to characterize them.
Stage 2: Study training dynamics on simple data but complex models. This is actually almost trivial once Stage 1 is done: we combine well-understood modules and add corrections to capture their interactions. At this stage, we begin to build systematic recipes for architectures and optimizers. Physics analogy: what are the interactions between particles?
Stage 3: The hardest step—moving from Stage 2 to Stage 3—requires understanding the data itself and therefore draws on domain expertise across fields. Physics analogy: what is the Standard Model of AI?

Citation

If you find this article useful, please cite it as:

BibTeX:

@article{liu2026methodology-toy,
  title={When I say "toy models", what do I mean?},
  author={Liu, Ziming},
  year={2026},
  month={February},
  url={https://KindXiaoming.github.io/blog/2026/toy/}
}