The hypothesis that networks represent high-level concepts as directions in their activation space, so that they can be read, added and manipulated with linear algebra.
According to this hypothesis, a concept such as "is Python code" or "is in French"
corresponds to a direction in a layer's activation space: how much of it is present in
a token is read by projecting the activation onto that direction (a dot product)
(Millière & Buckner, 2024b, sec. 2.5.3). If true, interpretability becomes tractable: linear
probes suffice for reading, adding a vector suffices for intervening
(steering), and sparse autoencoders can search for a dictionary
of directions. Templeton et al. (2024) take it, together with superposition, as an
explicit starting point.
The famous precedent is the analogies of word2vec:
vector("King") − vector("Man") + vector("Woman") yields a vector whose nearest neighbor is
"Queen" (Mikolov, Chen, et al., 2013, sec. 1.1). The most cited evidence comes from Othello-GPT.
Li et al. (2023) found the board with nonlinear probes; Nanda et al. (2023) showed
that it was linearly encoded if one asked about "mine," "theirs" and "empty" instead of
"black" and "white," and that a simple vector addition changed the model's moves
(Millière & Buckner, 2024b, sec. 2.5.3). Marks & Tegmark (2024) found a linear structure separating
true from false simple factual statements in large models.
There are caveats. The hypothesis admits several technical formulations, and the fact that
something can be decoded linearly does not stop the network from also using nonlinear
information (2024b, sec. 2.5.3). Engels et al. (2025) found irreducibly
multidimensional features: circles for the days of the week and the months, which models
use to do modular arithmetic with them.
The idea recalls the conceptual spaces of Gärdenfors (2000), where concepts are regions of a
geometric space. But a direction is not yet a concept: the "truth direction" might track
what is true or only what is usually said to be true. What fixes the content of a
direction: its correlation with the world, its effect on the model's behavior, or our
labeling?
Sources
- Engels, J., Michaud, E. J., Liao, I., Gurnee, W., & Tegmark, M. (2025). Not all language model features are one-dimensionally linear. In 13th International Conference on Learning Representations (ICLR 2025) (pp. 84591–84622). https://proceedings.iclr.cc/paper_files/paper/2025/hash/d3221cdb27e49d9c1cd35ad254feccfe-Abstract-Conference.html §
- Gärdenfors, P. (2000). Conceptual spaces: The geometry of thought. MIT Press. https://doi.org/10.7551/mitpress/2076.001.0001 §
- Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Emergent world representations: Exploring a sequence model trained on a synthetic task. In 11th International Conference on Learning Representations (ICLR 2023). https://openreview.net/forum?id=DeG07_TcZvT §
- Marks, S., & Tegmark, M. (2024). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling (COLM 2024). https://openreview.net/forum?id=aajyHYjjsk §
- Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). arXiv. https://doi.org/10.48550/arXiv.1301.3781 §
- Millière, R., & Buckner, C. (2024b). A philosophical introduction to language models—Part II: The way forward (arXiv:2405.03207). arXiv. https://doi.org/10.48550/arXiv.2405.03207 §
- Nanda, N., Lee, A., & Wattenberg, M. (2023). Emergent linear representations in world models of self-supervised sequence models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, & H. Mohebbi (Eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP (pp. 16–30). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.blackboxnlp-1.2 §
- Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin, A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., . . . Henighan, T. (2024). Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html §
