Standard attention compares each token's query with the keys of all earlier tokens and normalizes with a softmax: the cost grows with the square of the length, and during generation every key and value is stored in the KV cache. Linear attention replaces the similarity exp(q·k) with a product of feature maps φ(q)·φ(k), that is, with a kernel. Since matrix multiplication is associative, Σⱼ vⱼ·(φ(kⱼ)ᵀφ(q)) can be regrouped as (Σⱼ vⱼ·φ(kⱼ)ᵀ)·φ(q), and the bracketed sum no longer depends on the query (Katharopoulos et al., 2020, sec. 3.2). With a causal mask, this is a recurrence:
Sₜ = Sₜ₋₁ + vₜ·φ(kₜ)ᵀ, oₜ = Sₜ·φ(qₜ)
Hence the paper's title, "Transformers are RNNs": any causal attention layer can be written as a network that modifies an internal state and emits an output (2020, sec. 3.4). Memory no longer grows with the context, and the authors report generation up to 4,000 times faster on very long sequences (2020).
A superposed memory. Schlag et al. (2021) showed that the formula is equivalent to the "fast weight programmers" of the early 1990s: a slow network, trained by gradient descent, writes key–value associations at each step into a matrix that serves as the weights of another network. From this they derived its ceiling: d dimensions hold at most d orthogonal keys, and beyond d associations what comes back is a blend (2021, sec. 4.1). The remedies are to forget (a decay αₜ multiplying Sₜ₋₁: this is how Mamba-2 can be read, as "linear attention with decay" (S. Yang et al., 2025, sec. 2.1)) or to correct before writing, with the delta rule; Gated DeltaNet does both.
The price. No fixed state can copy arbitrarily long texts: Jelassi et al. (2024) prove this for the whole class of fixed-memory models, which includes linear attention and state-space models. This is why hybrids interleave a few full attention layers among many linear-attention or Mamba layers. The underlying contrast would then be not recurrence versus attention, but verbatim archive versus compressed summary.