AI finally enters its Tycho stage

Author: Ziming Liu (刘子鸣)

---

Introduction

Astronomy went through three stages: Tycho — Kepler — Newton. If we count the era before Tycho, there were four.

Today, the “Physics of AI” is still in the pre-Tycho era — we have observed a handful of phenomena, such as grokking, scaling laws, and double descent, but they are only a few stars in a vast phenomenological sky.

Today, we are finally entering the Tycho era. We are releasing ComfyResearch, which allows us to discover new phenomena in training dynamics at scale. For the first time, we have the ability to see the entire sky.

A field truly begins when it has puzzles at scale.

---

AI Is Still in the Pre-Tycho Era

The development of astronomy went through three important stages: Tycho — Kepler — Newton. Tycho accumulated a vast body of astronomical phenomena through systematic and precise observations. Kepler distilled empirical laws of planetary motion from these observations. Newton then uncovered the unified mechanism behind these laws: universal gravitation.

So where is AI today?

An optimist might say that AI has already entered the Kepler era, because the Scaling Law is an empirical law. I don’t think so. We are still in the pre-Tycho era.

A star is not an experiment; a star is a phenomenon. The Scaling Law is just one star in the phenomenological sky. To this day, there are very few phenomena — stars — that we can actually name: Grokking, Scaling Law, Double Descent, Attention Sink, and a handful of others. In this sense, both our ability and our willingness to observe fall far short of Tycho’s. Most AI practitioners are excessively obsessed with a single star — the Scaling Law. AI theorists may see a few more, but most stars remain hidden in the darkness of the universe.

Why don’t we see more stars? I think there are four reasons:

• Willingness to observe — Most people look only at the performance curves they care about, such as loss and accuracy.
• Ability to observe — How do we define internal observables, such as gradient norm, attention entropy, and so on?
• Willingness to record — If a phenomenon does not tell a good story, we tend not to record it. This is closely related to today’s publication culture.
• Ability to record — The entire experimental setup must be recorded before others can reproduce the result.

So how can we see more stars? We need to solve these four problems, one by one.

---

Today, ComfyResearch Takes Us into the Tycho Era

Scalable Discovery of Phenomena, Enabled by ComfyResearch

We built ComfyResearch to address the four problems above:

• Willingness to observe: Make observation easy and fun.
• Ability to observe: We predefine many observables, while allowing users to define their own.
• Willingness to record: Surface a rich variety of phenomena that can be shared with the community.
• Ability to record: We define a “DSL for AI research.” An experiment is a graph, making experiments easy to record and reproduce.

In ComfyResearch, an experiment is a graph (Blog, Github, or Try online). Take Grokking as an example. The nodes in the graph correspond to the dataset (modular addition), model (MLP), optimizer (Adam), loss function (Cross-entropy Loss), and observables (loss and accuracy). All tunable parameters are clearly exposed, while lower-level implementation details are hidden, allowing users to focus on understanding how these parameters affect the phenomenon. For Grokking, try adjusting the weight decay of the Adam optimizer and see how the onset time of grokking depends on weight decay.

ComfyResearch can do more than reproduce known phenomena. It can also synthesize many new experiments, revealing new phenomena along the way. ComfyResearch has a Lottery feature that allows users to randomly generate an experimental graph, train it, and observe what happens. We have already observed many new phenomena. In one example, on a simple world-modeling task — predicting uniform linear motion — the model eventually generalizes, but seems to get confused twice along the way, producing an M-shaped test loss curve.

And there is a little Easter egg: when the Lottery string contains kaiming, the automatically generated experiment will include a ResNet.

---

A Field Truly Begins When It Has Puzzles at Scale

Thousands of years of progress in physics have been driven by one puzzle after another. Many of the greatest breakthroughs in physics did not begin with “we need a better theory.” They began with a concrete anomaly — sometimes a seemingly tiny puzzle. For example:

• Why does Mercury’s perihelion precess by an extra 43 arcseconds per century? Newtonian mechanics was extraordinarily successful, yet it could not explain this tiny discrepancy. The anomaly eventually became an important test of general relativity.
• Why doesn’t blackbody radiation lead to an “ultraviolet catastrophe”? Classical theory predicted that the energy radiated at high frequencies should diverge, but experiments showed otherwise. This very specific contradiction forced Planck to introduce energy quanta, becoming one of the starting points of quantum mechanics.
• Why does the photoelectric effect have a threshold frequency? If light were purely a classical wave, sufficiently intense light should always be able to eject electrons. Experiments showed that below a certain frequency, no amount of intensity would work. This helped motivate the concept of light quanta.
• Why do different elements produce discrete spectral lines? The Balmer formula initially appeared to be little more than an empirical pattern: “Why do these numbers happen to obey this formula?” Only later did we understand that the answer lies in quantized energy levels.
• Why do stars in the outskirts of galaxies rotate so fast? Based on visible matter alone, stars farther from the galactic center should orbit more slowly — but they do not. This became one of the central observational puzzles behind dark matter.

What Physics of AI should learn from these examples is not primarily the answers that were eventually found, but the form of the questions themselves. A mature science first needs a collection of puzzles that are clear, robust, and reproducible. Only then can it attract the brightest minds to solve them.

With ComfyResearch, we can generate puzzles at scale. Our small team spent just one day exploring with ComfyResearch and came up with several new puzzles. If you are curious, visit our Blog, Github, or Try online. We will continue releasing more puzzles and invite the community to solve them together.

Compared with AI, physics is an extraordinarily mature science. Perhaps that maturity has made us forget something fundamental: physics is not a book of answers. It began as a collection of questions driven by curiosity.

We must first enter the Tycho era before we can reach the Kepler era, and eventually, the Newton era.