If a network stores more concepts than it has neurons, using nearly orthogonal directions (superposition), looking neuron by neuron does not work: each one responds to several things. Another basis is needed, larger than the original, in which each activation uses few elements. This is the old problem of dictionary learning; Olshausen & Field (1996) had already shown that a sparse code learned from natural images yields filters resembling those of simple cells in the visual cortex.
A sparse autoencoder is an autoencoder turned inside out: its hidden layer is much wider than its input. The encoder projects the activation (say, a vector from the residual stream) onto thousands or millions of units through a ReLU; the decoder reconstructs it as a linear sum of "feature directions." The loss function combines reconstruction error with an L1 penalty that forces almost all units to be zero (Templeton et al., 2024). Each unit that survives is a candidate interpretable feature.
Templeton et al. (2024) trained dictionaries of ≈1, 4 and 34 million features on Claude 3 Sonnet: on average, fewer than 300 active per token, with a reconstruction explaining at least 65% of the variance. They found multilingual and multimodal features, from the Golden Gate Bridge to sycophancy, and clamping them changes behavior (see steering). The limits are equally clear: the unexplained variance, the share of "dead" features (up to 65% in the largest dictionary), and the dependence on the chosen size, since larger dictionaries split a feature into finer variants.
The authors themselves warn against reading too much into the mere existence of a feature: knowing about lies, being capable of lying and actually lying are different things (Templeton et al., 2024). A feature is a direction that activates on certain contents, not a belief or a disposition. What else would it take for a direction to count as a concept the model has?