An intervention technique that swaps a component's activation in one run for the one it had in another, to measure whether that component is causally responsible for a behavior.
The recipe has three steps (Millière & Buckner, 2024b, sec. 2.5). Run the model on "The capital of
France is…" and cache its activations; run it on "The capital of Germany is…", which gives
"Berlin"; and run Germany again, but replace the activation of a single component (an
attention head, a feedforward layer, a position in the residual stream) with the
cached one. If the probability of "Paris" recovers, measured for instance by the difference
in logits between the two answers, that component carries causally relevant
information. Repeated over every component, the method draws a map of circuits.
The same idea goes by several names: causal mediation analysis (Vig et al., 2020),
causal tracing (Meng et al., 2022), interchange intervention (Geiger et al., 2021). Meng's team used it to locate factual associations in middle feedforward
layers and then edit them; Wang et al. (2023) explained how GPT-2 small identifies
the indirect object in sentences like "When Mary and John went to the store, John gave a
drink to…" with 26 attention heads grouped into seven classes.
Unlike probing, which only shows that information can be read off, patching shows that
it is used. Its philosophical frame is Woodward's interventionism: X causes Y if
intervening on X changes Y with everything else held fixed (Millière & Buckner, 2024b, sec. 2.4); and
its formalization is causal abstraction, which measures how well a high-level algorithm
reproduces the effects of interventions on the network (2024b, sec. 2.6).
The caveats are technical and conceptual. Patched activations can push the network into
states it would never visit; some layers compensate for the ablation of others (the "Hydra
effect" of McGrath et al. (2023)); and Hase et al. (2023) showed that where a
fact is localized does not predict where it is best edited. If causation is distributed and
redundant, what does it mean to say that a component "represents" something?
Sources
- Geiger, A., Lu, H., Icard, T., & Potts, C. (2021). Causal abstractions of neural networks. In Advances in Neural Information Processing Systems (Vol. 34, pp. 9574–9586). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2021/hash/4f5c422f4d49a5a807eda27434231040-Abstract.html §
- Hase, P., Bansal, M., Kim, B., & Ghandeharioun, A. (2023). Does localization inform editing? Surprising differences in causality-based localization vs. knowledge editing in language models. In Advances in Neural Information Processing Systems (Vol. 36, pp. 17643–17668). Curran Associates. https://doi.org/10.52202/075280-0774 §
- McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., & Legg, S. (2023). The Hydra effect: Emergent self-repair in language model computations (arXiv:2307.15771). arXiv. https://doi.org/10.48550/arXiv.2307.15771 §
- Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (Vol. 35, pp. 17359–17372). Curran Associates. https://doi.org/10.52202/068431-1262 §
- Millière, R., & Buckner, C. (2024b). A philosophical introduction to language models—Part II: The way forward (arXiv:2405.03207). arXiv. https://doi.org/10.48550/arXiv.2405.03207 §
- Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., & Shieber, S. (2020). Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems (Vol. 33, pp. 12388–12401). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html §
- Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. (2023). Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. In 11th International Conference on Learning Representations (ICLR 2023). https://openreview.net/forum?id=NpsVSN6o4ul §
