A recurrent network applies the same matrices at every step of a sequence. If it is unrolled, one copy per step, the result is a very deep feedforward (MLP) block network whose layers share weights (Witten et al., 2017, p. 457). Ordinary backpropagation is then applied to this unrolled network: the error at each step travels back through all previous steps, and since the weights are the same, their gradients are added up. Rumelhart et al. (1986) already described this equivalence, with two provisos: store the whole history of states, and keep corresponding weights equal by averaging their derivatives (1986, p. 535) (see weight sharing). Werbos (1990) wrote its classic exposition.
Costs. Memory grows with sequence length, because every hidden state must be kept. Hence the truncated version: the error goes back only k steps (Williams & Peng, 1990). Elman's network was the extreme case: the copy into the context units was fixed, and the error did not travel back in time. The forward alternative, Williams and Zipser's (1989) real-time recurrent learning, avoids storing the history but costs far more computation.
Instability. Going backward, the error is multiplied again and again by the same matrix; over many steps it tends to zero or to infinity (Witten et al., 2017, p. 457). This is the vanishing gradient (or exploding one). Against explosion, the gradient norm is clipped (2017, p. 458); against vanishing, it took the LSTM (Hochreiter & Schmidhuber, 1997).
The Transformer sidesteps all this: with the causal mask it trains every position in parallel, and any pair of tokens is one step apart (Vaswani et al., 2017, sec. 4). Seen through credit assignment, BPTT solves the temporal version of the problem: which past input is responsible for the present error. As a biological model it is even less plausible than backpropagation, itself contested (Buckner, 2024, p. 132): it requires remembering the whole history and running through it in reverse.