No. 023

Bahdanau attention

RNNsearch · attentional encoder–decoder

Soft Alignment

Before writing each word, it looks back over the original sentence and lights up only what it needs. Its glances leave a diagonal trail that linguists frame like a painting. Every transformer carries a little of this Pokémon's blood.

Height: 1 BiGRU layer (2 × 1,000) + 1 GRU layer (1,000)
Weight: ≈ 320 MB (fp32)
Parameters: ≈ 80 M (estimate from Appendix A)
Context: Sentences up to 50 words (RNNsearch-50)
Training data: WMT'14 EN–FR: 348 M words; 30,000-word vocabulary
Year: 2014
Generation: 3 · Deep learning
Creators: Bahdanau, Cho & Bengio (Jacobs University Bremen / Université de Montréal)
Kind: Architecture

Base stats

Compare with

Parameters: 44
Data: 49
Compute: 41
Context: 18
Generality: 15
Transparency: 22
Total: 189

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

It scores every source word for every output word: cost grows with the product of the lengths.
It is still recurrent: slow to train and hard to parallelize.
A 30,000-word vocabulary per language: everything else is [UNK].
Its attention maps look like explanations, but it is unclear that they are.

Habitat

It grew up in Montreal, on a single GPU per model, from 348 million words of parallel English–French text (European Parliament proceedings, UN documents, news). Today it no longer lives as a separate species: its backward glance survives as cross-attention inside every encoder–decoder transformer.

Moves

Abilities

Made-to-measure context: For each word it writes, it computes a different context vector: an average of the source annotations, weighted by their relevance.
Bidirectional annotations: Each source word is represented by a vector that summarizes what comes before it and what comes after it.
Aligning untaught: Nobody tells it which word corresponds to which: the alignment emerges from training the translation end to end.

Anatomy

RNNsearch (Bahdanau et al., 2015): the bidirectional encoder leaves one annotation per word. At every step, the decoder scores all the annotations against its previous state, normalizes the scores with a softmax, and builds from them a context of its own for the word it is about to emit.

Evolution chain

attention: the decoder looks back at every encoder state022 Seq2seq

023 Bahdanau attention

drops recurrence entirely: attention is all it needs025 Transformer

Bahdanau attention (named after its first author; the model itself was called RNNsearch) is a recurrent encoder–decoder with one addition: before writing each word, the decoder looks back over the whole input sentence and decides which parts matter. Bahdanau, Cho and Bengio presented it in September 2014, under a title that says it all: neural machine translation "by jointly learning to align and translate" (Bahdanau et al., 2015). It is the first appearance of the Attention type in this Pokédex and the direct ancestor of the Transformer.

How it works

An encoder that annotates. One recurrent network reads the sentence left to right and another right to left; concatenating their states yields, for each word xⱼ, an annotation hⱼ that summarizes the whole sentence with a focus on that word's neighborhood (Bahdanau et al., 2015, sec. 3.2). There is no longer a single vector: there is one per word.
An attention step before every word. To produce word i, a small feedforward (MLP) block network, the alignment model, scores each annotation against the decoder's previous state: eᵢⱼ = a(sᵢ₋₁, hⱼ). A softmax turns these scores into weights αᵢⱼ that sum to 1, and the context vector is the weighted average cᵢ = Σⱼ αᵢⱼ hⱼ (Bahdanau et al., 2015, sec. 3.1).
A decoder that writes. The new state sᵢ depends on the previous one, on the last word emitted and on cᵢ; from it comes the distribution over the next word.

The authors read αᵢⱼ as the probability that target word i is aligned to source word j, and add: "Intuitively, this implements a mechanism of attention in the decoder" (2015, sec. 3.1). Unlike in statistical translation, where alignment was a latent variable, here it is a soft, differentiable quantity trained together with everything else by backpropagation. Buckner (2024) notes a subtlety: the context is still a fixed-size vector, but it is now recomputed for every word (2024, p. 270).

The numbers are modest by today's standards: gated recurrent units (the GRUs of Cho et al. (2014)) with 1,000 dimensions, 620-dimensional embeddings and 30,000-word vocabularies per language (Bahdanau et al., 2015, appendix A), trained on the WMT'14 English–French corpus filtered down to 348 million words, for about five days per model (Bahdanau et al., 2015, secs. 4.1 and 4.2).

What changed

The paper's diagnosis is that the fixed vector of Seq2seq is a bottleneck; its remedy is to spread the information across many annotations and retrieve it selectively (Bahdanau et al., 2015, sec. 3.1). With the same data and training procedure, the basic encoder–decoder's BLEU collapses as sentences get longer, while RNNsearch-50 shows no deterioration even on sentences of 50 words or more: 26.8 versus 17.8 BLEU. On sentences without unknown words, its longer-trained version (36.2) matched Moses, the statistical baseline (35.6), which also used a 418-million-word monolingual corpus (Bahdanau et al., 2015, sec. 5.1).

The unexpected gift was the alignments. Drawn as matrices, the weights αᵢⱼ trace a diagonal, because English and French tend to preserve word order, but with deviations a linguist will recognize: to translate European Economic Area, the model first jumps to Area to write zone and then steps back one word at a time for économique européenne; to choose between le, la, les or l' for the man, it also looks at man (Bahdanau et al., 2015, sec. 5.2.1).

The lineage is direct. Vaswani et al. (2017) call this mechanism "additive attention," and their encoder–decoder attention layers mimic that of seq2seq models (2017, secs. 3.2.1 and 3.2.3): it is today's cross-attention. The transformer was born from a question: if the decoder can already see every encoder state, why keep the recurrence? (Buckner, 2024, pp. 271–272; Alammar & Grootendorst, 2024, pp. 14–15).

Limits

Quadratic cost. Every source word must be scored for every target word. With sentences of 15 to 40 words this is not serious, but the authors warn that it may limit other applications (Bahdanau et al., 2015, sec. 6.1). The transformer inherits this cost.
Still recurrent. Encoder and decoder advance step by step, so training cannot be parallelized along the sentence (Alammar & Grootendorst, 2024, p. 14).
Rare words. Anything outside the 30,000 words is [UNK]; the authors flag this as the main piece of unfinished business (Bahdanau et al., 2015, sec. 7).
Tempting maps. The alignment matrices invite being read as explanations of the translation (see the professor's note).

Professor Rortiga's note

"Everyone knows what attention is," wrote William James, and Buckner (2024) opens his chapter on attention with that line (2024, p. 239). Bahdanau and colleagues were more cautious: they said their mechanism "intuitively" implements attention (Bahdanau et al., 2015, sec. 3.1). Buckner reads it as a form of soft attention directed at internal representations rather than at locations in the visual field, loosely inspired by the way a reader looks for cues in the context to disambiguate bank (2024, pp. 270–271); of the later self-attention he warns that its similarity to biological attention is questionable (2024, p. 273). Is "attention" here a metaphor, a functional analogy, or a mere homonym?

The second question is epistemological. The alignment maps are the paper's best-remembered image, and it is tempting to read them as explanations: the model wrote zone because it looked at Area. Jain & Wallace (2019) argued that attention is not explanation: in classification, question-answering and inference tasks, the weights are often uncorrelated with gradient-based measures of importance, and very different distributions yield the same prediction. Wiegreffe & Pinter (2019) replied that this depends on what one means by explanation, and proposed more demanding tests. One detail: Jain and Wallace deliberately left out sequence-to-sequence tasks such as translation.

And in translation, alignment is a relation between expressions that linguists studied long before neural networks. Is showing where a system "looked" the same as showing why it translated as it did? What would an alignment lack to count as a reason?

Sources

Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR 2015). https://doi.org/10.48550/arXiv.1409.0473 §
Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation. In A. Moschitti, B. Pang, & W. Daelemans (Eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1724–1734). Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1179 §
Jain, S., & Wallace, B. C. (2019). Attention is not explanation. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 3543–3556). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1357 §
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html §
Wiegreffe, S., & Pinter, Y. (2019). Attention is not not explanation. In K. Inui, J. Jiang, V. Ng, & X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 11–20). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1002 §