A residual connection replaces "the layer's output is F(x)" with
output = x + F(x)
The layer no longer has to rebuild the whole representation: it only learns what should be added to it, the "residual". If it has nothing useful to contribute, it can push F toward zero and let x pass through untouched. The idea became famous with the ResNets of He et al. (2016), vision networks of up to 152 layers that won the 2015 ImageNet competition (3.6% top-5 error, per the table in (Witten et al., 2017, p. 439)).
What problem it solves. In a deep network, the error signal that backpropagation carries back to the first layers is multiplied by many factors and tends to fade (see vanishing gradient). The sum offers a direct route: the derivative of x + F(x) with respect to x is 1 + F′(x), and that "1" ensures the gradient gets through. Chollet (2021) lists residual connections among the 2014–2016 innovations that improved gradient propagation and made it possible to train models of almost arbitrary depth from scratch (2021, sec. 1.3.3; see also sec. 9.3.2).
In transformers. Every sublayer of the Transformer is wrapped in a residual
connection: LayerNorm(x + Sublayer(x)) (Vaswani et al., 2017, sec. 3.1). This is what allows
GPT-3's 96 blocks to be stacked and, at the same time, what gives rise to the picture
of the residual stream: one vector per token that runs through the whole network, with
each layer adding its contribution (Elhage et al., 2021). In BERT's diagram they appear as
"Add & normalize".
An analogy. It is the difference between rewriting a draft from scratch every round and correcting it with notes in the margin: the second is more stable, and the original text is never entirely lost.