A recurrent network has connections that form cycles, and therefore an internal state (Witten et al., 2017, p. 456). At each step t it combines the current input xₜ with its previous hidden state:
hₜ = act(W·xₜ + U·hₜ₋₁ + b)
with the same matrices W and U at every step (Witten et al., 2017, p. 457). "Unrolled" in time, it is a very deep network with weights shared across layers. Whereas feedforward networks compute ever more abstract features layer by layer, in recurrent ones processing is more like the steps of an algorithm (Witten et al., 2017, p. 457).
Strengths. It handles sequences of any length with fixed memory, and order comes built in: it needs no positional encoding. It is a natural inductive bias for time series and language: Chollet (2021) shows a temperature-forecasting problem where a dense network cannot beat a trivial baseline and a recurrent one can (2021, secs. 5.3.2 and 10.2).
Species. Simple recurrent network (Elman) (Elman, 1990) showed that a recurrent network trained to predict the next word discovered categories like noun and verb on its own (Elman, 1990). LSTM added gating; Seq2seq and Bahdanau attention used it to translate; ELMo, to produce contextual embeddings.
Why the transformer dropped it. Its sequential nature "precludes parallelization" within each example, and between two distant words the signal must pass through as many steps as there are words between them (Vaswani et al., 2017, secs. 1 and 4). Add to that the vanishing gradient.
The comeback. State-space models such as Mamba recover recurrence with a trick: an update that can be trained in parallel. And, in another sense, every generative model is recurrent in its outer loop: each token produced is fed back in as input (Alammar & Grootendorst, 2024, pp. 75–76).