A GPU (graphics processing unit) is a processor specialized in performing the same operation on a huge amount of data at once, such as computing the color of every pixel in a 3D scene. An ordinary processor (CPU) does a few complicated things in sequence; a GPU does many simple things in parallel. Neural networks consist mostly of matrix multiplications, exactly that kind of work.
The history is a case of "built for something else" (Mollick, 2026). Nvidia was founded in 1993 to make graphics chips for gamers and sold the GeForce 256 (1999) as "the world's first GPU." Before 2007, anyone who wanted to use these chips for scientific computing had to disguise their equations as graphics operations; that year, CUDA opened them to any computation. In 2012, AlexNet was trained on two 3 GB GTX 580 cards, built for gaming, over five to six days, in Alex Krizhevsky's bedroom at his parents' house (2026). As Chollet (2021) puts it, the gaming market subsidized supercomputing for the next generation of AI (2021, sec. 1.3.1).
The Transformer architecture was designed partly to exploit that parallelism: the original was trained on 8 GPUs for 3.5 days (Vaswani et al., 2017, sec. 5.2), and LLaMA-65B on 2,048 A100 GPUs for about 21 days (Touvron, Lavril, et al., 2023). Today the card's memory (VRAM) decides which models one can run, and those without access to powerful GPUs are, in the jargon, "GPU-poor" (Alammar & Grootendorst, 2024, pp. 28–29). The p(doom) song calls for a "Hundred thousand GPU" and bets on Nvidia's stock going to the moon (NVDA to the moon) (osmarks, n.d.).
Sara Hooker (2021) coined the term "hardware lottery" for when a research idea wins because it suits the available hardware rather than because it is superior. How much of what we now call artificial intelligence is the kind of intelligence that GPUs make cheap?