The LSTM (long short-term memory) is a recurrent architecture that adds to each unit a protected memory cell and a set of multiplicative gates that decide what gets written into it, what is kept and what is read out. It was Sepp Hochreiter and Jürgen Schmidhuber's answer to the great defect of networks like Elman's: they could not learn relations between events many steps apart.
The problem
When a recurrent network is trained with backpropagation through time, the error signal traveling backward is multiplied at each step by factors that depend on the weights and on the derivative of the activation function. If those factors are below 1, the error decays exponentially; if above, it blows up (Hochreiter & Schmidhuber, 1997, secs. 1, 3). This is the vanishing gradient, which Hochreiter had analyzed in his 1991 thesis.
How it works
The 1997 solution is almost an engineering fix: a linear unit connected to itself with weight 1, the constant error carousel, through which error flows without decaying or exploding. To keep this memory from filling up with noise, an input gate shields it from irrelevant inputs and an output gate shields the rest of the network from contents that are not yet relevant (Hochreiter & Schmidhuber, 1997, secs. 4, 7). On synthetic tasks, the LSTM bridged time lags of more than 1000 steps, where earlier methods failed (1997, sec. 1).
The version used today also has a forget gate, which Gers et al. (2000) introduced so that the cell could reset itself when processing continuous streams. At each step, three sigmoid gates (with values between 0 and 1) and a candidate computed with tanh combine as follows:
- cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ: forget part of the old, write part of the new;
- hₜ = oₜ ⊙ tanh(cₜ): read part of the cell out as the hidden state.
Chollet suggests picturing the cell as a conveyor belt running parallel to the sequence: information gets on at one point, travels intact, and gets off when needed (Chollet, 2021, sec. 10.3). Because the belt is updated by addition rather than by multiplication with a weight matrix, the gradient travels along it without vanishing (it can still explode).
Golden age
According to Chollet, the algorithm has barely changed since 1997 (Chollet, 2021, sec. 1.3): what was missing was data and GPUs. It already led cursive handwriting recognition when, in 2013, a deep LSTM achieved what its authors called the best recorded score on the TIMIT phoneme benchmark (Graves et al., 2013). In 2014, Sutskever et al. (2014) used two stacks of four LSTM layers, one reading the source sentence and one writing the translation, with 384 million parameters and 12 million English–French sentence pairs: Seq2seq was born (2014, secs. 3.1, 3.4). A curious trick: reversing the order of the source words improved results considerably (2014, sec. 3.3). The attention of Bahdanau et al. (2015) relieved the bottleneck of squeezing a whole sentence into a single vector (Alammar & Grootendorst, 2024, pp. 13–14; Buckner, 2024, p. 270), and in 2016 Google Translate moved to a network with eight encoder and eight decoder LSTM layers (Wu et al., 2016). The GRU of Cho et al. (2014) is a lighter form, with two gates.
The stats on this entry describe an LSTM from that era (Sutskever and colleagues' model), not the 1997 ones, which had between a hundred and ten thousand weights (Hochreiter & Schmidhuber, 1997, sec. 5.7).
Limits
Recurrence is sequential and wastes the parallelism of the hardware (Alammar & Grootendorst, 2024, p. 14). That is why the Transformer, which dispenses with it, displaced the LSTM from 2017 on (Vaswani et al., 2017; Buckner, 2024, p. 272). Effective memory is shorter than theoretical memory: Khandelwal et al. (2018) found that an LSTM language model used about 200 tokens of context, was highly sensitive to word order within the most recent sentence, and ignored order beyond about 50 tokens. Gradients still explode and must be clipped (Sutskever et al., 2014, sec. 3.4). And, as its authors admitted, counting steps precisely (did it happen 99 or 100 steps ago?) is beyond its reach (Hochreiter & Schmidhuber, 1997, sec. 6).