Dropout, proposed by Hinton and his students in Toronto (Srivastava et al., 2014), is one of the most widely used techniques against overfitting. At every training step, each unit in a layer is set to zero with some probability, the dropout rate, usually between 0.2 and 0.5 (Chollet, 2021, sec. 5.4.4):
layer output: [0.2 · 0.5 · 1.3 · 0.8 · 1.1] → with dropout: [0 · 0.5 · 1.3 · 0 · 1.1]
At inference time nothing is switched off; instead, activations are rescaled to compensate for the fact that they now all work together (in practice, the rescaling is usually done during training). This is why libraries explicitly distinguish "training" mode from "use" mode (Chollet, 2021, sec. 7.4.1).
Why does breaking the network on purpose help? Hinton said he was inspired by his bank: the tellers kept changing, and he concluded it was so that no group of employees could cooperate to defraud it. Switching off a different random subset of neurons on each example prevents such "conspiracies," that is, happenstance patterns the network would memorize if nothing disturbed them (Chollet, 2021, sec. 5.4.4). Another reading: a network with n units and dropout behaves like an ensemble of 2ⁿ smaller networks that share parameters, averaged at the end (Witten et al., 2017, p. 436).
The base model of the original Transformer applied dropout at a rate of 0.1 to the output of every sub-layer (Vaswani et al., 2017, sec. 5.4), and InstructGPT's supervised fine-tuning used 0.2 (Ouyang et al., 2022, sec. 3.5). The idea connects with an old connectionist thesis: in a distributed representation no node is indispensable, and the system degrades gracefully instead of breaking. Dropout turns that robustness into a discipline imposed during learning: it forces every useful feature to be represented in more than one way.