AlexNet is an eight-layer convolutional network that sorts photographs into a thousand categories. Its victory in the 2012 ImageNet challenge is usually taken as the start of the deep learning boom (Buckner, 2024, p. 55; Chollet, 2021, sec. 1.2.5). It invented almost nothing: it brought together, for the first time and at the right scale, ideas that had been waiting for two decades.
How it works
The network takes a 224 × 224 image with three color channels and processes it through five convolutional and three dense layers: about 60 million parameters and 650,000 neurons (Krizhevsky et al., 2012). Each convolutional layer slides a bank of small filters across the image (96 filters of 11 × 11 pixels in the first one) that respond to edges, orientations or blobs of color; a rectification lets through only what was detected, and pooling summarizes each neighborhood by its maximum value. Buckner (2024) calls this sequence an "abstraction sandwich," stacked again and again (2024, p. 127). Two dense layers of 4,096 units combine the features, and a softmax spreads the probability over the thousand classes.
The authors single out several ingredients (Krizhevsky et al., 2012, secs. 3–4):
- ReLU: the activation function max(0, x), with which the network trained several times faster than with tanh.
- Two GPUs: the network did not fit in the 3 GB of a GTX 580, so it was split across two cards that communicate only at certain layers.
- Data augmentation: random crops and reflections multiply the training images.
- Dropout: in the first two dense layers each neuron is switched off with probability 0.5 at every step; without it, the network overfit substantially. It amounts to training, all at once, an exponential ensemble of subnetworks that share weights (Witten et al., 2017, p. 436).
Training took five to six days (Krizhevsky et al., 2012, sec. 5).
What changed
The pieces were old: convolution and backpropagation were well understood by 1990 (Chollet, 2021, sec. 1.3), and Convolutional network (LeNet) was already reading handwritten digits. What was new was scale. In 2010 and 2011, the ImageNet winners, built on support vector machines and hand-designed features, got 72% and 74% right with five guesses (M. Mitchell, 2019, ch. 5). AlexNet brought the top-5 error down to 15.3%, against 26.2% for the runner-up (Krizhevsky et al., 2012); non-convolutional methods seemed stuck at around 25% (Witten et al., 2017, p. 439). It was not the first GPU-trained network to win competitions (Dan Cireşan had been doing so since 2011), but it was the one that won the field over (Chollet, 2021, sec. 1.2.5).
The deeper change is that features were no longer designed by hand: the network learns them together with the classifier (Chollet, 2021, sec. 1.2.6). Its first-layer filters turned out to be selective for orientation and frequency (Krizhevsky et al., 2012, sec. 6.1), like the "simple" cells of cat visual cortex described by Hubel & Wiesel (1962), whose discovery inspired the whole convolutional lineage (Buckner, 2024, pp. 122–124).
Everything was built for something else
GPUs were designed for video games: the gaming market, as Chollet puts it, subsidized supercomputing for AI (2021, sec. 1.3.1). ImageNet's labels were obtained through Amazon Mechanical Turk, a system built to weed out duplicate product pages; some 49,000 people in 167 countries worked on them between 2007 and 2010 (Mollick, 2026). The name alludes to a chess automaton with a human hidden inside. Soon afterwards, Google bought Hinton, Krizhevsky and Sutskever's tiny company in an auction that stopped at 44 million dollars (Mollick, 2026).
Limits
A year later, Szegedy and colleagues showed that imperceptible pixel changes made AlexNet see an ostrich where there was a school bus, and that the fragility affected other networks too (M. Mitchell, 2019, ch. 6). As a model of vision, such a network covers at most the first hundred milliseconds of perception, with no recurrence or top-down control (Buckner, 2024, p. 124). And its design aged quickly: local normalization was dropped, and its successors won with more depth, from VGG (19 layers, 7.3%) to ResNet (152 layers, 3.6%) (Witten et al., 2017, p. 439; He et al., 2016). Even the "5% human error" figure these networks were compared with comes from a single person, Andrej Karpathy, who trained himself on 500 images (M. Mitchell, 2019, ch. 5).