Many language tasks turn one sequence into another of a different length: translating, summarizing, transcribing. The encoder–decoder architecture splits them in two: an encoder converts the input into representations and a decoder generates the output step by step from them (Vaswani et al., 2017, sec. 3; Witten et al., 2017, p. 460).
The recurrent version and its bottleneck. In Seq2seq (2014), two recurrent networks passed the baton: the first read the whole sentence and condensed it into a single fixed-size vector; the second wrote the translation from it. Bahdanau et al. (2015) conjectured that this vector was a bottleneck, and showed it: the performance of these models plummeted as sentences grew longer (2015, secs. 1 and 5.1). Their remedy, attention, lets the decoder consult all the encoder states, not only the last one (Bahdanau attention).
The transformer version. The Transformer kept the scheme (six encoder blocks, six decoder blocks, joined by cross-attention) but without recurrence (Vaswani et al., 2017, sec. 3.1). T5 took it to the limit, recasting every task as "text-to-text". Then the family branched out: BERT kept only the encoder; the GPT line, only the decoder (Alammar & Grootendorst, 2024, pp. 18–21). Outside language, the VAE has the same shape, with a latent space in the middle.
A philosophical angle. A Toronto group called whole-sentence representations "thought vectors", and Geoffrey Hinton went as far as saying: "I think you can capture a thought by a vector" (M. Mitchell, 2019, ch. 11). That a sentence's content fits into one fixed vector is a strong thesis about meaning. Bahdanau's result works as a small empirical refutation: for long sentences, it pays to keep one representation per word and consult it as needed, rather than compress everything into a single point.