More Features Than Neurons

One Neuron, Three Meanings

Researchers looking inside an image-recognition network called InceptionV1 found a neuron that fired for cat faces, for the fronts of cars and for cat legs. It was not responding to something the three share. It simply had three unrelated jobs.

Such neurons are called polysemantic, and they are common. They make a network hard to read: you can’t point to a neuron and say what it means. In 2022, researchers at Anthropic offered an explanation, using a model small enough to watch at every step.

A two-neuron model holding five features · each rod is a feature, each etched line a neuron

Five Things, Two Numbers

The model is tiny. Five numbers go in, each one a “feature” that is either absent or present with some strength. The model has to squeeze them through just two neurons, then rebuild all five on the other side.

Each feature gets a direction in the plane the two neurons span, drawn here as a rod. Train the model on features that are always present and it does the safe thing: it gives the two most important features a direction each, at right angles, and lets the other three go.

Rare Things Can Share

Now make every feature rare, present only one time in twenty. The world is like that: in any one picture or sentence, almost everything is absent. Train again, from scratch.

This time all five survive. They fan out into a pentagon, about 72 degrees apart, squeezed into a plane with room for two. The paper calls this superposition. Nobody told the model to make a pentagon; training found it, one small step at a time.

The Price Is Interference

Five directions can’t all be at right angles in a plane, so every feature leaks a little into its neighbours. Switch one feature on: the bright point in the middle is the pair of numbers the two neurons now hold, and the five glass tubes show what the model reads back. A threshold before the last step (a ReLU with a negative bias) turns the negative leaks to zero and shrinks the small positive ones.

That works because two rare things seldom happen together. When they do, the model can be confidently wrong. Switch on two features that point almost opposite ways and their signals cancel: the model reports neither of them, and reports a third that was never there.

Shapes in Three Dimensions

Give the model three neurons and eight rare features, and the rods settle into a square antiprism: two squares, one twisted 45 degrees against the other. Six features pair off in opposite directions, like the three axes of an octahedron.

The researchers found such tidy shapes again and again, and warned that they may be a quirk of toy models. What carries over is the idea behind them: if features are rare enough, a network can hold more of them than it has neurons.

Pulling Features Apart

If features are stacked on top of one another, they can be pulled apart again. A sparse autoencoder looks only at the numbers in the neurons and learns a set of directions, so that each moment is built from as few of them as possible. The white wires here are one, learning as you watch. Never told what the features are, they swing into line with all six.

Anthropic did this to a small real model in 2023 (one layer of 512 neurons, read as 4,096 features in the run they studied most closely), and in 2024 to its large model Claude 3 Sonnet, with up to 34 million features. One fired for the Golden Gate Bridge, in English, in other languages and in pictures. Clamped at ten times its maximum, it made the model start calling itself the bridge.