Seq2seq (sequence to sequence) is an architecture for turning one sequence into another of a different length: a recurrent network, the encoder, reads the input sentence word by word and summarizes it in a single vector; a second network, the decoder, unfolds that vector into the output sentence. It was proposed almost simultaneously in 2014 by Sutskever, Vinyals and Le at Google (Sutskever et al., 2014) and by Cho and Bengio's group in Montreal, who called it the RNN Encoder–Decoder (Cho et al., 2014). Its proving ground was machine translation.
How it works
The encoder is an LSTM that consumes one token per step and keeps rewriting its
hidden state; at the end, that state is a composite reflecting which words occurred and in
what order (Buckner, 2024, pp. 269–270). The decoder is another LSTM that works as a
conditioned language model: it starts from the encoder's vector, emits a word, receives it
as input at the next step, and goes on until it produces the end symbol <EOS>
(Sutskever et al., 2014, sec. 2; Alammar & Grootendorst, 2024, pp. 11–12). In probabilistic notation:
p(y₁…y_T′ | x₁…x_T) = ∏ₜ p(yₜ | v, y₁…yₜ₋₁), where v is the vector left by the encoder. The
translation is chosen with beam search; a beam of size 2 already delivered
most of the benefit (Sutskever et al., 2014, sec. 3.2).
Google's model had four layers of 1,000 cells on each side, 1,000-dimensional embeddings, and vocabularies of 160,000 words (English) and 80,000 (French). Of its 384 million parameters, only 64 million were recurrent connections: most of the rest were the embedding tables and the output softmax layer. The whole input sentence was squeezed into 8,000 numbers (Sutskever et al., 2014, sec. 3.4). It was trained on 12 million sentence pairs from the WMT'14 English–French corpus, for about ten days, on a machine with eight GPUs (Sutskever et al., 2014, secs. 3.1 and 3.5).
One trick proved decisive: reading the source backwards. Reversing the order of the input words lowered perplexity from 5.8 to 4.7 and raised BLEU from 25.9 to 30.6. The authors conjecture that this puts the first source words close to the first words of the translation, shortening the dependencies that backpropagation has to traverse, though they admit they lack a complete explanation (Sutskever et al., 2014, sec. 3.3).
What changed
A single LSTM can map sequences to sequences easily when it is known in advance which output corresponds to which input; translation offers no such luxury, since lengths differ and word order changes (Sutskever et al., 2014, sec. 2). Seq2seq solves this by separating reading from writing. Until then, neural networks entered translation mostly as auxiliary parts of statistical phrase-table systems (Bahdanau et al., 2015, sec. 6.2); seq2seq translated end to end. An ensemble of five LSTMs scored 34.8 BLEU against 33.3 for the statistical baseline, and when used to rescore that system's 1,000 best hypotheses it reached 36.5, half a point from the best published result (Sutskever et al., 2014, sec. 3.6). In 2016, Google Translate, which had run on statistical translation since 2006, switched to a neural encoder–decoder system (M. Mitchell, 2019, ch. 12; Mollick, 2026).
The recipe, moreover, knew nothing about languages: the authors predicted that it would work for any sequence-to-sequence task with enough data (Sutskever et al., 2014, sec. 5).
Limits
- The bottleneck. Whatever its length, the sentence has to fit into a fixed vector. The Montreal group reported that performance dropped quickly on long sentences (Bahdanau et al., 2015, sec. 1); Sutskever and colleagues, with deeper networks and the reversed source, saw only a minor degradation (2014, sec. 3.7). The dispute was settled in practice by the attention of Bahdanau attention, which lets the decoder consult the whole input (Alammar & Grootendorst, 2024, p. 13).
- Closed vocabulary. Any word outside the chosen 80,000 becomes
UNK. - Structural slowness. Recurrence forces the sentence to be processed step by step, which precludes parallelization within each example (Vaswani et al., 2017, sec. 1). The Transformer was born to remove that constraint.