Neurons evolve in many ways: branching generalization and unrelated coexistence (superposition) exist side by side.
神经元的演化机制是多样的，分支泛化与无关特征的共存重叠共同存在。
Tap the dots to explore the scene点击光点，看看每棵树代表什么We tracked 240 neurons of Pythia-160M across 40 training checkpoints to see how their features appear and change.我们在 40 个训练检查点上追踪了 Pythia-160M 的 240 个神经元，观察它们的特征如何出现和变化。
By tracking neurons in Pythia-160M throughout training, we found that superposition and recursive generalization are not mutually exclusive. Instead, they happen together most of the time.
我们在整个训练过程中追踪 Pythia-160M 的神经元，发现重叠和递归泛化并不互斥，反而在大多数时候同时发生。
We also observed two previously unreported learning patterns: replacement, where a neuron drops what it has learned and switches to something new, and sporadic features, where weak features appear only briefly.
我们还观察到两种此前未被报道的学习模式：替换，即神经元放弃已经学会的东西，转向新的东西；以及零散特征，即微弱的特征只短暂出现。
One tentative explanation for neurons’ evolution is that neurons start out noisy and gradually settle on a few features—a process we call “denoising.”
对神经元演化的一个初步解释是：神经元起初充满噪声，随后逐渐稳定在少数几个特征上，我们称这个过程为“去噪”。
Nick Cammarata’s recent essay, Neural Networks Are Already Clean, sparked a debate about why neurons respond to seemingly unrelated things. He argues that much of the evidence usually attributed to superposition was misread, partly because of weak tools and older model designs. Instead, he proposes recursive generalization as the main explanation.
Nick Cammarata 最近的文章《Neural Networks Are Already Clean》引发了一场讨论：神经元为什么会响应看似无关的东西？他认为，通常被归因于重叠的许多证据其实被误读了，部分原因在于工具较弱、模型设计较旧。他转而提出，递归泛化才是主要解释。
This post builds on that discussion with further experiments and thoughts on language models. Our investigation centers on these two ideas:
本文在这一讨论的基础上，补充了在语言模型上的进一步实验与思考。我们的研究围绕以下两个概念展开：
A neuron starts by responding to a simple pattern, then gradually extends to related things, sometimes branching in several directions. Each step follows a connection, but the endpoints may look unrelated if we cannot see how the neuron got there.
神经元起初只响应一个简单的模式，随后逐渐延伸到相关的事物，有时会朝多个方向分叉。每一步都沿着某种联系，但如果看不到神经元是怎么走到那里的，终点之间可能显得毫不相关。
In Toy Models of Superposition, networks represent more features than they have dimensions by placing features along overlapping, non-orthogonal directions. This definition does not require the features to be unrelated. Nick uses the term more narrowly: unrelated features share neurons because they can coexist with little interference, rather than because they share a meaningful connection.
在 Toy Models of Superposition 中，网络把特征放在相互重叠、非正交的方向上，从而表示比维度更多的特征。这一定义并不要求特征彼此无关。Nick 对这个词的用法更窄：无关的特征共享神经元，是因为它们可以几乎互不干扰地共存，而不是因为它们之间有什么有意义的联系。
We follow Nick’s usage here as we want to investigate the distinction he proposes—whether a neuron’s different responses develop through connected steps or bring together unrelated features.
我们在这里沿用他的用法，因为我们想检验他提出的区分：神经元的不同响应，是通过一步步相连的过程发展出来的，还是把无关的特征放到了一起。
Nick directly follows neurons through training in vision models, but his language-model analysis mainly works backward from already-trained models. Direct evidence of how these patterns develop during language-model training is still limited. Do we see recursive generalization, superposition, or both when we follow neurons as they learn? We explored this with Pythia-160M, whose public training data and intermediate checkpoints let us track how individual neurons change throughout training.
Nick 在视觉模型中直接跟踪了神经元在训练中的变化，但他对语言模型的分析主要是从已经训练好的模型往回推。关于这些模式在语言模型训练中如何发展，直接证据仍然有限。当我们跟随神经元的学习过程时，看到的是递归泛化、重叠，还是两者都有？我们用 Pythia-160M 探究了这个问题：它公开的训练数据和中间检查点，让我们能追踪单个神经元在整个训练过程中的变化。
Our overall process looks like this: at each checkpoint, we collected the text passages that most strongly activated each neuron. An AI agent then described what the neuron responded to. A second agent read these descriptions in training order to identify how features appeared and evolved.
整体流程是：在每个检查点，我们收集最能激活每个神经元的文本片段；由一个 AI agent 描述该神经元在响应什么；再由第二个 agent 按训练顺序阅读这些描述，识别特征如何出现和演化。
Our results suggest that superposition and recursive generalization are not mutually exclusive. Of the 240 neurons we tracked, only 14 showed recursive generalization alone, and only 11 showed superposition alone. Another 180 showed a mixture of learning patterns.
我们的结果表明，重叠和递归泛化并不互斥。在追踪的 240 个神经元中，只有 14 个仅表现出递归泛化，只有 11 个仅表现出重叠；另有 180 个表现出多种学习模式的混合。
Classified by the AI agent. Click a row to list its neurons.由 AI agent 分类。点击一行可列出其中的神经元。
The figure below shows one example of a mixed neuron. By the end of training, it responds to both medical and programming terms. We traced these responses back to two seemingly unrelated early patterns: the verb “send” and capital letters. Each pattern then developed through a series of related steps.
下图是一个混合型神经元的例子。训练结束时，它同时响应医学术语和编程术语。我们把这些响应追溯到两个看似无关的早期模式：动词 “send” 和大写字母。此后，每个模式都经过一系列相关的步骤发展而来。
Still, our observations support Nick’s view that most features present at the end of training develop through generalization. This was the case for 213 of the 240 neurons. Superposition appeared more often early in training and became less common at later stages.
不过，我们的观察支持 Nick 的观点：训练结束时存在的大多数特征是通过泛化发展而来的，240 个神经元中有 213 个是这样。重叠在训练早期更常见，到后期变得较少。
The mechanism behind most of each neuron’s strongest responses after step 16,000.第 16,000 步之后，每个神经元大部分最强响应背后的机制。
Average over the 240 neurons of how much of the 10 strongest responses comes from features of each mechanism.在 240 个神经元上平均，最强 10 段响应中来自各机制特征的占比。
Beyond the two most commonly discussed patterns—generalization and superposition—we also observed two other ways neurons evolved:
除了最常被讨论的两种模式，即泛化和重叠，我们还观察到神经元演化的另外两种方式：
A neuron abruptly stops responding strongly to a previously learned feature and begins responding to a new, unrelated one.
神经元突然不再强烈响应之前学会的特征，转而响应一个新的、无关的特征。
Weak, unrelated features appear briefly but never become a lasting part of what the neuron responds to.
微弱、无关的特征短暂出现，但从未成为神经元持久响应的一部分。
The example shows both patterns. Colors indicate how each feature developed: branch, unrelated coexistence, replacement, sporadic. Let’s focus on yellow and pink.
这个例子同时展示了这两种模式。颜色表示每个特征是如何发展出来的：分支、无关共存、替换、零散。我们重点看黄色和粉色。
Around training step 6,000, a new feature involving approximate quantities, such as “about 10,” replaced the neuron’s earlier response to “MOT” strings.
大约在第 6,000 步，一个与约数有关的新特征（如“about 10”）替换了神经元早先对“MOT”字符串的响应。
This replacement feature, shown in yellow, soon generalized to expressions involving “around” (the red arrow).
这个以黄色表示的替换特征很快泛化到含“around”的表达（红色箭头）。
That branch eventually developed into the neuron’s dominant final response: “about” introducing the phrase that follows. The “MOT” feature did not reappear in the later checkpoints we examined.
这条分支最终发展成神经元训练结束时的主要响应：用“about”引出后面的短语。在我们检查的后续检查点中，“MOT”特征再也没有出现。
Pink marks sporadic features, such as the “Caf” and “Café/Cafe” word forms, observed only at step 512. These appeared unrelated to the neuron’s other features and tended to involve surface patterns, such as spelling fragments. They looked like early features that never developed further. In our sample, they rarely persisted for more than one checkpoint.
粉色表示零散特征，比如只在第 512 步出现的“Caf”和“Café/Cafe”词形。它们看起来与神经元的其他特征无关，往往涉及拼写片段之类的表层模式，像是从未进一步发展的早期特征。在我们的样本中，它们很少持续超过一个检查点。
In this sense, superposition and recursive generalization are both ways a neuron can become polysemantic, and neither provides a complete explanation on its own. Together with replacement and sporadic features, they often appear within the same neuron’s training history and may interact in complex ways. There may also be patterns we have not yet identified.
从这个意义上说，重叠和递归泛化都是神经元变得多语义的途径，但单独哪一个都不是完整的解释。它们与替换和零散特征一起，常常出现在同一个神经元的训练历史中，并可能以复杂的方式相互作用。也可能还有我们尚未发现的模式。
So a broader question is: what general principles govern how neurons learn?
因此，一个更大的问题是：神经元的学习遵循什么普遍规律？
Here is a tentative idea we call “denoising.” Early in training, a neuron responds without a clear pattern. As training progresses, these scattered responses gradually give way to strong responses to a few specific features.
这里提出一个初步的想法，我们称之为“去噪”。训练早期，神经元的响应没有清晰的模式；随着训练推进，这些零散的响应逐渐让位于对少数几个特定特征的强烈响应。
Drag the slider or press play. Each bar is a feature; its height is how strongly the neuron responds to it.拖动滑块或点击播放。每根柱子是一个特征，高度是神经元对它的响应强度。
Why call it denoising? At initialization, a neuron has not yet learned clear preferences. In several examples in the interactive panel, its early responses appear weak and scattered. Most of the complex changes occur during early and middle training. Later, a clearer and more stable pattern emerges. This resembles a gradual clearing of noise. This picture is also consistent with previous work showing that, under certain conditions, neuron weights move from random initialization toward a few directions determined by the data.
为什么叫去噪？初始化时，神经元还没有学到明确的偏好。在交互面板的若干例子中，它早期的响应显得微弱而分散。大多数复杂的变化发生在训练前期和中期，之后才出现更清晰、更稳定的模式。这就像噪声被逐渐清除。这一图景也与已有工作一致：在一定条件下，神经元权重会从随机初始化移向由数据决定的少数几个方向。
One appealing part of this idea is that it can accommodate both recursive generalization and superposition. Think of images emerging from noise in an image generator. Initially, they all look similarly noisy. One eventually becomes a tree: its branches are connected, like features developing through recursive generalization. Another becomes a collection of separate sticks, like unrelated features coexisting through superposition. Even with the same training data, different neurons may settle on different patterns. In practice, a neuron may end up with several “trees” and several “sticks” together. The image at the top of this page shows such a mix.
这个想法吸引人的一点是，它能同时容纳递归泛化和重叠。想象图像生成模型从噪声中生成图像：起初它们都同样嘈杂。一张最终变成一棵树，枝干彼此相连，就像通过递归泛化发展出来的特征；另一张变成一堆分开的木棍，就像通过重叠共存的无关特征。即使训练数据相同，不同神经元也可能稳定到不同的模式。实际上，一个神经元最后可能同时拥有几棵“树”和几根“木棍”。本页顶部的图画就是这样的混合。
The limitation is that it can’t explain replacement. An abrupt switch from one feature to another is hard to understand as a gradual clearing of noise. We therefore distinguish this denoising picture from evolution involving replacement, where a neuron’s direction changes along the way. We do not yet have a clear sense of what drives those changes.
它的局限在于无法解释替换。从一个特征突然切换到另一个，很难理解为噪声的逐渐清除。因此，我们把这种去噪图景与包含替换的演化区分开来：后者中，神经元的方向在途中发生了改变。我们目前还不清楚是什么驱动了这些改变。
We found that of the 240 neurons, 68 fit the denoising picture, 118 underwent feature replacement, and 54 showed no clear pattern. Here is one example from each group. Click a group to switch the panel below.
我们发现，240 个神经元中，68 个符合去噪图景，118 个经历了特征替换，54 个没有清晰的模式。以下每组各举一例，点击一组即可切换下方的面板。
The denoising picture fits only about a quarter of the neurons. About half undergo replacement, which we can observe but do not yet understand. For roughly a fifth, we cannot even identify a clear learning pattern. To us, this suggests that the debate over how neurons learn—and why they become polysemantic—is far from settled.
去噪图景只适用于大约四分之一的神经元。约一半经历了替换，我们能观察到它，但还不理解它。约五分之一的神经元，我们甚至无法识别出清晰的学习模式。在我们看来，这说明关于神经元如何学习、以及为什么会变得多语义的争论远未结束。
These are some early results from a paper we are working on. They leave us with several open questions:
这些是我们正在撰写的论文中的一些早期结果。它们留给我们几个开放问题：
Do generalization, replacement, and the coexistence of unrelated features share a common cause, or do they arise from different mechanisms?
泛化、替换和无关特征的共存，是有共同的成因，还是源于不同的机制？
Our intuition is that messy or changing features may suggest that individual neurons need more training, even when the model as a whole appears well trained. Would further training help them settle into stable functions?
我们的直觉是，混乱或不断变化的特征可能意味着个别神经元还需要更多训练，即便模型整体看起来已经训练得很好。继续训练能否帮助它们稳定到固定的功能？
They might lose out through local competition during generalization, then be taken over by other neurons or disappear altogether. This could happen even before the network reaches its capacity limit. But tracing what happens to them remains difficult.
它们可能在泛化过程中的局部竞争中落败，随后被其他神经元接管，或者彻底消失。这甚至可能在网络达到容量上限之前就发生。但追踪它们的去向仍然很困难。
Does it change the model’s internal representations or how easily we can steer its behavior?
它是否会改变模型的内部表征，或改变我们操控模型行为的难易程度？
@misc{gao2026neuronevolution,
title = {Recursive Generalization and Superposition Are Both Right},
author = {Gao, Lang and Zhang, Jinghui and Chen, Xiuying},
year = {2026},
url = {https://heartyhaven.github.io/blog/neuron-evolution/}
}