In 2013, a year after AlexNet's triumph, Szegedy and colleagues showed that pixel changes invisible to a person were enough for that network to take a school bus for an ostrich, with very high confidence (M. Mitchell, 2019, ch. 6). Goodfellow et al. (2015) gave the canonical example: a photo of a panda, classified as a panda with 57.7% confidence, becomes a "gibbon" at 99.3% after calculated noise is added. The recipe is simple: nudge each pixel slightly in the direction that most increases the loss, x′ = x + ε · sign(∇ₓ loss).
Their explanation was surprising: the problem is not the networks' nonlinearity but their excessive linearity. In a high-dimensional space, thousands of tiny changes add up in a dot product until they cross the decision boundary. The examples transfer across architectures, work in the physical world (stickers that turn a stop sign into a speed-limit sign) and also exist as "rubbish images," noise the network classifies confidently (Buckner, 2024, pp. 66–67). Automated jailbreaks are their textual version (Zou et al., 2023); the classic defense is adversarial training.
The interpretation is disputed. For some, they prove that networks do not see as we do: in a phrase Buckner quotes, they build "a Potemkin village" that works on natural data and collapses outside it (2024, p. 67). Ilyas et al. (2019) proposed the opposite: they are not bugs but features, real and predictive patterns that we do not perceive. Buckner (2024) adds that people can guess, above chance, the "wrong" label the network picks, and asks for fair play: before penalizing a network for its response to an artificial stimulus, one must justify which response is correct (2024, p. 89). If the network detects something real that we cannot see, who is making the mistake?