In an Oct. 15 essay, Goodfire researcher Nick Cammarata revisits an InceptionV1 neuron once thought to respond to both cat faces and car fronts. He argues that it may respond to their shared V-shaped feature. In an Oct. 17 LessWrong response, writer leogao says coherent-looking examples still need tests on broader inputs and model outputs. That leaves a practical question: can an explanation predict what a neuron does beyond examples chosen to illustrate it?
Mechanistic interpretability aims to explain how parts of a neural network contribute to its calculations. A neuron is one component whose activation changes with input. When it appears to respond to unrelated patterns, researchers may call it polysemantic. One explanation, superposition, is that a network with limited capacity can represent multiple features in overlapping patterns. Sparse autoencoders and similar methods try to separate those features for study.
Goodfire reinterprets a neuron and reports a preliminary language-model audit
Cammarata argues that some famous examples of apparent superposition may have simpler explanations. Neuron 4e:55 in InceptionV1 was described as responding to cat faces and car fronts. In Cammarata’s new interpretation, those images share a pointed, V-like shape; he says the neuron’s strongest negative responses instead tend toward round shapes. He also points to related examples such as fox faces and flames. This is a reinterpretation of an established vision-model example, not a result showing that the same explanation applies to modern language models.
Goodfire’s essay also reports a preliminary audit of Qwen3-32B. A “pole” is one direction of a neuron’s response: its positive side fires strongly in one direction, while its negative side runs strongly the other way. Cammarata says roughly 90% of the audited neuron poles had one main job, about 7% had two related jobs, and 2% to 4% were not yet understood. He cautions that the estimates depend on how weak activations are counted and what qualifies as one job. The essay calls itself a working hypothesis; a technical paper with more detail on the language-model experiments is forthcoming.
What would make an explanation convincing?
Leogao’s response agrees that some neurons labeled as arbitrary mixtures may have a coherent pattern. But looking at a neuron’s strongest examples can miss counterexamples and weaker activations that still matter to the model. A unit that fires on some images of V-shaped objects, for instance, might respond only to a narrower subset than the short explanation suggests.
The response proposes a more demanding check: write a rule that predicts the neuron’s activation, apply it to a broader set of inputs, and temporarily replace the real activation with the rule’s prediction. Researchers could then compare the model’s output probabilities with and without that replacement. A large shift would suggest the explanation missed something important. Leogao recommends measuring that downstream change with KL divergence, a measure of how much two output distributions differ, while acknowledging that no single metric is perfect.
Even a successful test of one neuron would leave open how circuits combine signals from many neurons. Cammarata calls his broadest claim about interpretability becoming much more tractable highly speculative. The forthcoming technical paper and further tests will show how broadly the essay’s reinterpretation applies; this exchange does not establish that modern language models are understood as complete systems or that they are safe.
Sources
- Nick Cammarata’s “Neural Networks Are Already Clean” (working hypothesis, posted Oct. 15, 2026).
- Goodfire research listing (publication date).
- Leogao’s “Neural networks are not already clean” (Oct. 17, 2026).
- Image credit: “Neural Network.svg” by QuantuMechaniX8, Wikimedia Commons, CC0 1.0. Generic explanatory diagram.
