Backpropagation computes how much each weight contributed to the error by multiplying derivatives layer by layer, from the output backward. If those factors tend to be smaller than 1, the product shrinks exponentially: 0.5 multiplied thirty times is less than one billionth. The early layers receive an almost null signal and barely learn. If the factors are larger than 1, the gradient "explodes".
In recurrent networks the problem is worse, because the same matrix is used at every time step: over many steps the gradient tends to zero or to infinity, just like any number other than 1 raised to a large power (Witten et al., 2017, p. 457). That is why a network like Simple recurrent network (Elman) struggled with long-range dependencies (a subject at the start of a sentence, its verb many words later). Clipping the gradient is enough against explosion; against vanishing, the architecture had to change: the LSTM was designed specifically for this, with memory cells protected by gates (Witten et al., 2017, pp. 457–458; Hochreiter & Schmidhuber, 1997).
In deep networks generally, Chollet (2021) notes that until the late 2000s the training signal faded as the number of layers grew, which kept networks shallow. The fixes were better activation functions (such as ReLU), better initializations and optimizers, and later batch normalization and residual connections (2021, sec. 1.3.3).
In transformers it reappears in miniature: attention dot products grow with dimension and push the softmax into regions of near-zero gradient; hence the division by √dₖ (Vaswani et al., 2017, sec. 3.2.1). And self-attention shortens the paths between distant positions, which was part of the original problem (Vaswani et al., 2017, sec. 4).
A historical angle. Neural-network "winters" are usually told as sociological episodes; Witten et al. (2017) recall that in the early 2000s having "neural networks" in a paper's title correlated with its rejection at NIPS (2017, p. 461). But there were technical reasons too, and this was one of the central ones.