CLIP is a pair of encoders, one for images and one for text, trained so that an image and its caption land close together in a single vector space (Radford et al., 2021, sec. 2.3). It generates nothing: it is a multimodal embedding model that lets you compare a photo directly with a sentence (Alammar & Grootendorst, 2024, pp. 264–265).
How it works
The starting point is data. OpenAI assembled WIT, 400 million image–text pairs collected from the internet with 500,000 queries (up to 20,000 pairs per query), with a total word count similar to that of WebText, the corpus of GPT-2 (Radford et al., 2021, sec. 2.2).
The key decision was the objective. Predicting the exact words of each caption turned out to be slow; predicting only which caption, as a whole, goes with which image was several times more efficient (2021, sec. 2.3). In each batch of N pairs, CLIP computes the cosine similarity of all N × N possible combinations and learns to maximize the N true ones and minimize the N² − N false ones. This is contrastive learning: the model learns what makes two things alike by also being shown which ones do not belong together (Alammar & Grootendorst, 2024, pp. 266–268). Batches held 32,768 pairs, and the largest version with a Vision Transformer (ViT) image encoder trained for 12 days on 256 V100 GPUs (Radford et al., 2021, sec. 2.5).
Classifying without examples
Once trained, CLIP becomes a zero-shot classifier with no fine-tuning: classes are written as sentences ("A photo of a dog.", "A photo of a cat.") and the one closest to the image is chosen. That way it reached 76.2% on ImageNet, matching a supervised ResNet-50 without using any of its 1.28 million labeled images (Radford et al., 2021, secs. 3.1.2–3.1.3). The authors note a problem any linguist will recognize: polysemy. "Crane" is both a machine and a bird, and ImageNet contains both; "boxer" may be a dog or an athlete. Giving context in the prompt helps disambiguate, as with GPT-3 (2021, sec. 3.1.4).
What changed
CLIP replaced closed label lists, like ImageNet's thousand classes, with supervision in natural language, which can name anything. Its shared space became infrastructure: CLIP's text encoder conditions generation in diffusion systems such as Stable Diffusion, and CLIP was the stepping stone to models that take images in and produce text (Millière & Buckner, 2024b, sec. 3.1.1).
Limits
The authors list several (Radford et al., 2021, sec. 6): it does poorly on fine-grained classification (car models, flower species), on counting objects and on tasks rare in its data, and it generalizes poorly to what is truly out of distribution, such as MNIST's handwritten digits (88%). They also document biases: with certain label sets, it classified photos of people into crime-related or "non-human" categories at very different rates depending on racial group (2021, sec. 7.1). And since its objective does not require preserving word order, its text encoder treats captions almost as bags of words: it confuses which attribute goes with which object, spatial relations, counts and negation (Millière & Buckner, 2024b, sec. 3.1.1).