The phrase has two related senses.
1. GANs. A generative adversarial network pits two networks against each other. The generator G turns noise z, drawn from a latent space, into fake data; the discriminator D tries to tell them from real ones. Each is trained against the other in a zero-sum (minimax) game (Goodfellow et al., 2014):
min_G max_D E[log D(x)] + E[log(1 − D(G(z)))]
The authors compare it to counterfeiters and police: competition improves both until the counterfeits are indistinguishable (Goodfellow et al., 2014, sec. 1); Buckner (2024) uses the image of a forger facing a bank (2024, pp. 205–208). What is remarkable is that the discriminator works as a learned loss function: nobody defines what a "realistic face" is; the opponent does. The typical risk is mode collapse: the generator finds a few outputs that fool the judge and stops varying (Goodfellow and colleagues jokingly called it "the Helvetica scenario", (2014, sec. 6)). See GAN; today image generation relies mostly on diffusion.
2. Robustness against adversarial examples. A year after AlexNet's win, Szegedy and colleagues showed that pixel changes imperceptible to a human could make that network mistake a school bus for an ostrich, with high confidence (M. Mitchell, 2019, ch. 6). Training on such examples to make the model robust is also called adversarial training (Goodfellow et al., 2015). Buckner (2024) notes that a GAN's discriminator, trained against forgeries, ends up less vulnerable to such attacks (2024, p. 206). The jailbreaks of language models are the textual version of the problem (Zou et al., 2023).
A general angle. Witten et al. (2017) warn that spam filtering is not ordinary classification: the data are not chosen at random but crafted to evade the filter, and the adversary can learn too (2017, pp. 524–525). When there is an agent with goals on the other side, the assumption that the world does not change because we are modeling it no longer holds.