The Vision Transformer (ViT) is a transformer encoder applied to images with as few changes as possible. The trick is to treat the image like a text: it is cut into 16 × 16-pixel patches and each patch plays the role of a word (Dosovitskiy et al., 2021, sec. 1). As the paper's title announces, an image is worth 16 × 16 words.
How it works
A 224 × 224-pixel image yields 196 patches. Each is flattened into a vector and passed through a learned linear projection; as in BERT, a special [class] token is prepended and learned position encodings are added. The sequence enters a standard stack of multi-head attention and MLP blocks, with normalization and residual connections, and the final state of the [class] token feeds the classifier (Dosovitskiy et al., 2021, sec. 3.1). Alammar & Grootendorst (2024) point out the detail that forces a projection instead of a vocabulary: unlike words, a patch almost never recurs exactly in another image, so it cannot be given a dictionary ID (2024, pp. 261–263).
The Base, Large and Huge variants copy BERT's configurations and add a bigger one: from 86 to 632 million parameters (Dosovitskiy et al., 2021, sec. 4.1).
What changed
Convolutional networks come with three built-in assumptions about images: locality, two-dimensional neighborhood structure and translation equivariance, repeated at every layer. ViT has almost none of them. The only moments at which the architecture "knows" its input is two-dimensional are the patching and the adjustment of positions when the resolution changes; everything else must be learned (Dosovitskiy et al., 2021, sec. 3.1).
The outcome was a lesson about inductive bias. Trained on ImageNet alone (1.3 million images), ViT falls a few points below comparable residual networks. With ImageNet-21k or JFT-300M the picture flips: the best model reaches 88.55% on ImageNet with about four times less pretraining compute than BiT-L, a large residual network trained on the same data (2021, secs. 1, 4.2). On subsets of JFT, ViT overfits more than a ResNet with 9 million images and overtakes it from 90 million on: the convolutional assumption helps with little data, but with a lot of it, learning the patterns directly is enough, even better (2021, sec. 4.3).
Looking inside is revealing. The learned positions reconstruct the grid of rows and columns, and some attention heads in the first layers become local, as if the network were reinventing convolution; others attend to the whole image from the start (2021, sec. 4.5).
Offspring
Since ViT, cutting into patches has become a standard way of turning images into tokens. It is the typical image encoder of CLIP (Millière & Buckner, 2024b, sec. 3.1.1), Gato uses it to mix images with text and actions (Reed et al., 2022, sec. 2.1), and multimodal models such as BLIP-2 connect a frozen ViT to a language model (Alammar & Grootendorst, 2024, pp. 274–275).
Limits
Data hunger is the price of flexibility. Moreover, the cost of attention grows with the square of the number of patches, so smaller patches make the model much more expensive (Dosovitskiy et al., 2021, sec. 4.1), and the best results depend on a private dataset.