Mamba is a sequence architecture that does without attention: it processes text with a state-space model, a form of recurrence that carries the history compressed into a fixed-size state. Gu and Dao introduced it in December 2023 as the first linear-time alternative to match transformers at language modeling: their 2.8-billion-parameter model beat transformers of the same size and matched ones twice as large (Gu & Dao, 2023). Alammar & Grootendorst (2024) list it among the new architectures that aim for transformer-level performance with larger context windows or faster inference (2024, p. 24).
How it works
A state-space model, in its discrete form, is a linear recurrence: hₜ = Ā·hₜ₋₁ + B̄·xₜ, yₜ = C·hₜ. The vector h is the hidden state; the idea comes from control theory, from the Kalman filter (Gu & Dao, 2023, sec. 2). Its predecessors (S4 and kin) had parameters that were fixed over time, which allowed computing them as one big convolution and training them in parallel, but prevented them from deciding what to remember based on content: they solved the task of copying a sequence at a fixed distance, but not “selective copying,” where only certain tokens scattered among noise must be retained (Gu & Dao, 2023, sec. 3.1).
Mamba's novelty is selection: Δ, B and C become functions of the input, so that at every token the model decides how much of the state to keep and how much of the token to take in. The authors show that the gates of classic recurrent networks are a special case of this mechanism (Gu & Dao, 2023, sec. 3.5). The price is losing the convolution; they pay it with a parallel scan that runs in the GPU's fast memory without materializing the expanded state (Gu & Dao, 2023, sec. 3.3). The resulting block (linear projection, short convolution, selective SSM and multiplicative gate) is stacked homogeneously, with no separate attention or feedforward layers (Gu & Dao, 2023, sec. 3.4).
What changed
Compared with the LSTM, the recurrence is linear in the state, and that is what makes it trainable in parallel. Compared with the Transformer, the authors frame the problem as compression: attention is effective and inefficient because it does not compress anything (it stores the whole context in the KV cache, which grows with every token), whereas a recurrent network is efficient because it compresses everything into a finite state, but is only as good as that compression (Gu & Dao, 2023, sec. 3.1). Mamba bets on compressing with judgment. The result: compute and memory linear in length during training, constant cost per token during generation, about five times the generation throughput of a similarly sized transformer, and improvements with contexts of up to a million elements on synthetic tasks and DNA (Gu & Dao, 2023, sec. 1).
Limits
The original paper stops at 2.8 billion parameters and 300 billion tokens; its authors warn that it remains to be seen whether the advantage holds at larger scales (Gu & Dao, 2023, sec. 5). Besides, a fixed state cannot keep everything. Jelassi et al. (2024) proved that a two-layer transformer can copy strings of exponential length while these models are limited by their state, and found that in practice they do much worse at copying and retrieving information from context. Industry's answer was hybrids: Jamba (Lieber et al., 2024) interleaves one attention layer for every seven Mamba layers and shrinks the KV cache eightfold relative to a plain transformer. Since then hybrids have grown from an experiment into a family of their own: SSM hybrid (Jamba, Qwen3.5).