The idea comes from control theory: a system has an internal state h that evolves with the inputs x and from which the outputs y are read. In discrete form:
hₜ = Ā·hₜ₋₁ + B̄·xₜ, yₜ = C·hₜ
It is a recurrence, but a linear one. In pre-Mamba versions, moreover, the matrices were the same at every step; this allows the whole sequence to be computed at once, as a convolution, and trained in parallel, while generation proceeds step by step with a fixed-size state. The best of both worlds: the parallel training of transformers and the constant memory of recurrent networks.
Mamba. Gu & Dao (2023) pinpointed these models' weakness with language: they could not decide what to keep based on each token's content. Their solution was to make the parameters depend on the input, so as to "selectively propagate or forget" information, an idea akin to the gating of the LSTM. This rules out the convolution, so they designed a hardware-aware parallel algorithm. According to the authors, Mamba scales linearly with sequence length, has about five times a transformer's inference throughput, and its 3-billion-parameter version matches transformers twice its size (Gu & Dao, 2023). Alammar & Grootendorst (2024) present it, alongside RWKV, as an alternative that seeks transformer-level performance with longer contexts or faster inference (2024, p. 24).
The trade-off. A transformer keeps everything it has read (in its KV cache, which grows with every token) and can look back at any exact word. A state-space model compresses the past into a fixed hidden state: cheaper, but what it did not keep is gone. Hence the hybrids that interleave attention layers.
A philosophical angle. It is the old tension between memory as archive and memory as summary. We do not recall verbatim what we read ten pages ago, but we retain what is relevant; which design is closer to that, and whether it matters, is left for discussion.