A convolutional network (CNN) is a feedforward neural network whose first layers do not connect each unit to the whole input: they slide one small filter across the entire image and then reduce the resolution. LeNet-5, described by LeCun, Bottou, Bengio and Haffner in 1998, is the canonical specimen: a seven-layer network that reads handwritten digits (LeCun et al., 1998).
How it works
According to its authors, the architecture combines three ideas: local receptive fields, shared weights, and spatial subsampling (LeCun et al., 1998, sec. II.A).
- Convolution. Each unit looks at a 5×5-pixel window, and all the units in a "feature map" share the same 25 weights: they detect the same feature (an edge, a corner, a stroke ending) at every position. It is like passing a stencil over the image and noting where it fits (Buckner, 2024, p. 125).
- Subsampling. Each unit in the next layer summarizes a 2×2 patch of the previous map. Resolution halves and the network tolerates small shifts: what matters is that there is an edge around here, not at which exact pixel (Witten et al., 2017, p. 438).
- Classifier. After two rounds of convolution and subsampling, fully connected layers (an ordinary MLP) decide the digit.
A 32×32 image goes through C1 (6 maps of 28×28), S2 (6 of 14×14), C3 (16 of 10×10), S4 (16 of 5×5), C5 (120 units), F6 (84) and an output of 10 units. There are 340,908 connections but only about 60,000 trainable parameters, because the weights are shared (LeCun et al., 1998, secs. II.A–B). Everything, filters included, is tuned by backpropagation: the network builds its own feature extractor (LeCun et al., 1998, sec. II.A; Witten et al., 2017, p. 441). Earlier solutions for these digits relied on hand-crafted features, such as the number of loops in the stroke (Chollet, 2021, sec. 5.4.2).
What changed
An ordinary MLP treats the image as a list of pixels with no geometry: it would have to learn separately to recognize a stroke at each position. The CNN builds in two assumptions from the start: that relevant features are local, and that a detector useful in one place is useful everywhere. This is an inductive bias (Chollet speaks of architecture priors) that saves parameters and data (Chollet, 2021, sec. 5.3.2; Buckner, 2024, p. 130).
The genealogy is neuroscientific. Hubel & Wiesel (1962) distinguished, in cat visual cortex, "simple" cells, sensitive to an edge at one position, and "complex" cells, which respond to the same feature with some spatial invariance. Fukushima carried that division into his Neocognitron (Fukushima, 1980), which alternated convolution and subsampling but had no global supervised learning procedure like backpropagation (LeCun et al., 1998, sec. II.A; Buckner, 2024, pp. 122–124). In 1989 LeCun brought the two together at Bell Labs, and that network read ZIP codes for the US Postal Service (LeCun et al., 1989; Chollet, 2021, sec. 1.2.2). LeNet-5 is its mature form.
On MNIST (60,000 training digits, 10,000 test digits), LeNet-5 made 0.95% errors, and 0.8% when also trained on deformed versions of the images (LeCun et al., 1998, sec. III.B). Compared with other methods, such as k-nearest neighbors or SVMs (Witten et al., 2017, p. 421):
| Classifier (MNIST) | Test error |
|---|---|
| Linear | 12.0% |
| k-nearest neighbors | 5.0% |
| MLP with 300 hidden units | 4.7% |
| SVM with Gaussian kernel | 1.4% |
| LeNet-5 | 0.95% |
Training it took two to three days of CPU time (LeCun et al., 1998, sec. III.D), and from 1996 a system built on it read millions of checks a day in American banks (1998, sec. X).
Limits
LeNet-5 needs characters already centered and normalized, works on small grayscale images, and learns a single task. CNNs had to wait for ImageNet and GPUs to reach real photographs with AlexNet (Witten et al., 2017, p. 462). Even then, they model at most the first hundred milliseconds or so of vision, without recurrence or top-down control (Buckner, 2024, p. 124), and they need far more examples than a person does (2024, pp. 131–132).