No. 052

SSM hybrid (Jamba, Qwen3.5)

SSM–attention hybrid · hybrid linear attention · Jamba · Nemotron-H · Qwen3-Next · Qwen3.5

Chimera

It has the body of a snake and, every so often, an eye that looks back. Seven out of eight vertebrae digest the text without going over it again; the eighth rereads all of it when something must be quoted word for word. It runs almost like Mamba, remembers almost like a transformer, and argues with both about which one is its real parent.

Height: Jamba: 32 (4 blocks × 8: 28 Mamba, 4 attention)
Weight: Jamba: ≈ 104 GB (bf16); fits on one 80 GB GPU with int8 weights
Parameters: Jamba: 52 B total · 12 B active per token (Qwen3.5: 397 B · 17 B)
Context: Jamba: 256K tokens, with a 4 GB KV cache (Mixtral: 32 GB)
Training data: Jamba: web, books and code up to March 2024; size undisclosed
Year: 2024
Generation: 6 · Assistants, reasoning, and agents
Creators: Lieber et al. (AI21 Labs, Jamba, 2024); De et al. (Google DeepMind, Griffin, 2024); NVIDIA (Nemotron-H, 2025); Qwen Team (Alibaba, Qwen3-Next, 2025, and Qwen3.5, 2026)
Kind: Architecture

Base stats

Compare with

Parameters: 70
Data: 85
Compute: 80
Context: 85
Generality: 75
Transparency: 35
Total: 430

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

The KV cache of the attention layers still grows with the context: more slowly, but it grows.
How many attention layers, and where, is settled by ablations on small models, not by theory.
It runs best with special compute kernels for the recurrent layers, which cards such as Jamba's and Qwen3-Next's recommend installing separately.
Almost all the family's figures come from the labs' own reports.

Habitat

It was born at AI21 Labs (Jamba, 2024) and in the labs of NVIDIA and Google DeepMind. Today it lives in open-weight models such as Qwen3.5 and Nemotron 3, especially where whole documents are read or reasoning runs for thousands of tokens.

Moves

Abilities

The ratio: One attention layer in eight (Jamba) or in four (Qwen3.5): a few suffice to retrieve from context, and the KV cache shrinks by the same factor.
Two memories: Most layers carry a compressed, fixed-size state; a few keep every token as it was and can read it again.
No positions: Jamba uses no explicit positional encoding: the recurrence in the Mamba layers already knows what came first.

Anatomy

Left: the Jamba block, repeated four times. Of its eight layers, seven are Mamba and one (the fifth) is attention; the feedforward networks alternate between a plain MLP and a mixture of 16 experts that uses two per token. Each layer has its residual connection and normalization (omitted). Right: while generating, the Mamba layers carry a state that does not grow; only the four attention layers store keys and values for every token in the context.

Evolution chain

most layers stay recurrent; a few attention layers come back for recall043 Mamba
keeps a handful of attention layers (one in eight in Jamba)025 Transformer

052 SSM hybrid (Jamba, Qwen3.5)

A hybrid is a language model that interleaves two kinds of layer. Most run in linear time and carry the history compressed into a fixed-size state: a state-space model (SSM) such as Mamba, or a "linear attention." A few are full attention layers, like the Transformer's. Recent models usually add a mixture of experts (MoE), as in Mixture of experts (Mixtral). The specimen that made the family known is AI21 Labs' Jamba (March 2024): 52 billion parameters, 12 billion active per token, and only 4 of its 32 layers are attention (Lieber et al., 2024). Its authors presented it as the first production-grade attention–SSM hybrid (Lieber et al., 2024, sec. 1); Google DeepMind's Griffin had appeared a month earlier (De et al., 2024).

Why mix

Each parent fails where the other succeeds. Attention stores keys and values for every previous token in the KV cache, which grows with the context: at 256,000 tokens, Mixtral needs a 32 GB cache; Jamba, 4 GB (Lieber et al., 2024, sec. 2). A fixed state, by contrast, cannot keep everything: Jelassi et al. (2024) proved that a two-layer transformer can copy strings of length exponential in its size, while any fixed-state model, whether an SSM or a linear attention, is bounded by the size of its state. In the Jamba ablations, on 1.3-billion-parameter models, pure Mamba often did not respect the format of the examples in the prompt (it answered "Very Good" where it had to choose between "Positive" and "Negative"); the hybrid did, and in its attention layers the authors found heads that behave like induction heads, evidence they themselves call anecdotal (Lieber et al., 2024, sec. 6.2). Dao and Gu hypothesize a division of labor: the SSM layers act as a general sequence-to-sequence map and attention as a retrieval mechanism that spares the model from compressing the whole context into its state (Dao & Gu, 2024, sec. 9.2.3).

The family

Ratios as given in each model's report (De et al., 2024; Waleffe et al., 2024; NVIDIA, 2025a; MiniMax, 2025; Qwen Team, 2025; Qwen Team, 2026; Kimi Team, 2025; NVIDIA, 2025b):

Model	Year	Fixed-state layer	State : attention	MoE
Jamba	2024	Mamba	7 : 1	yes
Griffin	2024	gated linear recurrence	2 : 1 (local attention)	no
Mamba-2-Hybrid 8B	2024	Mamba-2	24 : 4	no
MiniMax-Text-01	2025	linear attention	7 : 1	yes
Nemotron-H 56B	2025	Mamba-2	54 : 10	no
Qwen3-Next, Qwen3.5	2025–26	Gated DeltaNet	3 : 1	yes
Kimi Linear	2025	refined Gated DeltaNet	3 : 1	yes
Nemotron 3	2025	Mamba-2	a few attention layers	yes

Gated DeltaNet combines two ways of writing to memory: a gate that erases quickly, and the delta rule, which corrects only the entry associated with a key (S. Yang et al., 2025). The Qwen3.5 flagship has 397 billion parameters, 17 billion active, and open weights (Qwen Team, 2026).

What is debated

How many. Ablations, mostly on small models, put the optimum between 8 and 25% attention layers, though not everyone counts the same way: about 10% for Dao and Gu (2024, sec. 9.2.3); about 8% for Waleffe et al. (2024), who include the MLP layers in the total (2024, sec. 4.1); three linear layers per attention layer for Kimi Linear (Kimi Team, 2025, sec. 5.2). Jamba found no substantial difference between 1 : 3 and 1 : 7 (Lieber et al., 2024, sec. 6.1).

Where. Spread evenly, with no better pattern in sight, and with a recurrent layer first, which encodes position and makes positional encoding unnecessary (Waleffe et al., 2024, sec. 4.1). Not everyone drops it: the attention layers of Qwen3-Next and Qwen3.5 apply rotary positions (RoPE) to a quarter of each head's dimensions (Qwen Team, 2025; Qwen Team, 2026).

Whether they are two things. Mamba-2 grew out of showing that SSMs and certain variants of attention are two decompositions of the same class of matrices (Dao & Gu, 2024), and the Gated DeltaNet paper presents Mamba-2 as "linear attention with decay" (S. Yang et al., 2025, sec. 2.1). The real opposition would then be not recurrence versus attention but compressing versus not compressing. Kimi Linear claims to be the first hybrid to beat full attention in a fair comparison (Kimi Team, 2025): a claim made by the lab itself.

Professor Dendrett's note

A hybrid has two memories: a compressed one, rewritten token by token and forgetful, and a literal one, which keeps every token and can reread it. The temptation is to read it through the psychology of memory. Buckner (2024) presents the complementary learning systems model of McClelland et al. (1995): two interacting networks, a fast-learning one that records a few recent episodes exactly (the hippocampus) and a slow one that integrates them into abstract, durable knowledge (the cortex) (2024, pp. 160–161, 166–167). The analogy limps at a decisive point: for McClelland, the two memories differ in how fast their weights learn, and the fast one passes what it learned on to the slow one; in the hybrid, the parameters are frozen during generation and nothing passes from one memory to the other. Unless another reading is taken seriously: the Gated DeltaNet paper treats the compressed state as a matrix of fast weights that the delta rule adjusts token by token through gradient descent (S. Yang et al., 2025, sec. 3.1). If that is learning, the fast-learning memory would be the compressed one, and the literal one would learn nothing: McClelland's map, inverted.

The question of architecture remains. Buckner recommends modeling the faculties with semi-independent modules, each with its own wiring motifs or learning algorithms (2024, p. 32), and the hybrid seems to bear him out: two distinct operations sharing the work. But Mamba-2's duality suggests they are two regimes of one mathematical family, and Buckner himself grants that, on reflection, one faculty might reduce to a mode of the others (2024, p. 32, n. 31). Does the hybrid have two memories, or one memory in two doses? And is a ratio of one attention layer in four or eight a discovery about memory, or about the price of GPUs?

Sources

Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality (arXiv:2405.21060). arXiv. https://doi.org/10.48550/arXiv.2405.21060 §
De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., De Freitas, N., & Gulcehre, C. (2024). Griffin: Mixing gated linear recurrences with local attention for efficient language models (arXiv:2402.19427). arXiv. https://doi.org/10.48550/arXiv.2402.19427 §
Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2024). Repeat after me: Transformers are better than state space models at copying. In Proceedings of the 41st International Conference on Machine Learning (Vol. 235, pp. 21502–21521). PMLR. https://proceedings.mlr.press/v235/jelassi24a.html §
Kimi Team. (2025). Kimi Linear: An expressive, efficient attention architecture (arXiv:2510.26692). arXiv. https://doi.org/10.48550/arXiv.2510.26692 §
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Rozen, N., . . . Shoham, Y. (2024). Jamba: A hybrid Transformer-Mamba language model (arXiv:2403.19887). arXiv. https://doi.org/10.48550/arXiv.2403.19887 §
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3), 419–457. https://doi.org/10.1037/0033-295X.102.3.419 §
MiniMax. (2025). MiniMax-01: Scaling foundation models with lightning attention (arXiv:2501.08313). arXiv. https://doi.org/10.48550/arXiv.2501.08313 §
NVIDIA. (2025a). Nemotron-H: A family of accurate and efficient hybrid Mamba-Transformer models (arXiv:2504.03624). arXiv. https://doi.org/10.48550/arXiv.2504.03624 §
NVIDIA. (2025b). NVIDIA Nemotron 3: Efficient and open intelligence (arXiv:2512.20856). arXiv. https://doi.org/10.48550/arXiv.2512.20856 §
Qwen Team. (2025). Qwen3-Next-80B-A3B-Instruct [Model card]. Hugging Face. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct §
Qwen Team. (2026). Qwen3.5-397B-A17B [Model card]. Hugging Face. https://huggingface.co/Qwen/Qwen3.5-397B-A17B §
Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., Kulshreshtha, G., Singh, V., Casper, J., Kautz, J., Shoeybi, M., & Catanzaro, B. (2024). An empirical study of Mamba-based language models (arXiv:2406.07887). arXiv. https://doi.org/10.48550/arXiv.2406.07887 §
Yang, S., Kautz, J., & Hatamizadeh, A. (2025). Gated delta networks: Improving Mamba2 with delta rule. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025) (pp. 29687–29707). https://proceedings.iclr.cc/paper_files/paper/2025/hash/4904fad153f6434a7bcf04465d4be2cc-Abstract-Conference.html §