No. 051

Whisper

Listener

It listened to 680,000 hours of the internet with the subtitles on and learned to write down what it hears in nearly a hundred languages. It is an excellent stenographer, but during long silences it sometimes gets bored and adds, on its own, a “thanks for watching” that nobody said.

Height: 4 · 6 · 12 · 24 · 32 blocks (tiny → large), in the encoder and as many again in the decoder
Weight: ≈ 3 GB for large in fp16 (estimate: 1,550 M × 2 bytes)
Parameters: 39 M · 74 M · 244 M · 769 M · 1,550 M
Context: 30 s of audio (1,500 encoder positions)
Training data: 680,000 h of web audio with transcripts (65% English); sources undisclosed
Year: 2022
Generation: 5 · Large language models
Creators: Radford, Kim, Xu, Brockman, McLeavey & Sutskever (OpenAI)
Kind: Model

Base stats

Compare with

Parameters: 60
Data: 60
Compute: 60
Context: 25
Generality: 30
Transparency: 35
Total: 270

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

It hallucinates: during silences and long pauses it may write sentences nobody said.
On long recordings it gets stuck in loops repeating the same sentence, or skips the first or last words.
It does much worse on languages with few training hours; Quechua and Guaraní are not there at all.
It writes down what was said, not how: intonation, emphasis and pauses are lost.

Habitat

It grew up on 680,000 hours of internet audio with transcripts, two thirds of it in English; which sites the audio came from was not disclosed. Today it lives in dictation and captioning apps, in transcription tools for meetings, interviews and lectures, and on any computer where someone has downloaded its open weights.

Moves

—Multilingual transcription
TM058Translation into English
TM007Language identification
—Voice activity detection (<|nospeech|>)
—Timestamps at 20 ms resolution
TM166Zero-shot recognition

Abilities

Task tokens: A single decoder does five jobs: the language, the task and whether to emit timestamps are signaled by special tokens at the start of what it writes.
Weak supervision at scale: 680,000 hours of audio with imperfect transcripts from the internet, instead of a few thousand hand-checked hours.
Spectrogram ear: It reads sound as a time-by-frequency image, with bands spaced on the Mel scale, and condenses it with two convolutions before the transformer blocks.

Anatomy

Whisper, bottom to top, as in Figure 1 of Radford et al. (2023). The encoder turns 30 seconds of audio into a log-Mel spectrogram, condenses it with two convolutions (the second halves the time resolution) and processes it with transformer blocks. The decoder starts with special tokens that fix the language and the task (<|translate|> instead of <|transcribe|> asks for an English translation; <|notimestamps|> turns timestamps off) and then writes the text token by token, consulting the audio through cross-attention, until <|endoftranscript|>. Encoder and decoder have the same number of blocks: 4 in tiny, 32 in large.

Evolution chain

log-Mel spectrograms in, text out; 680,000 hours of weakly supervised audio025 Transformer

051 Whisper

Whisper is OpenAI's speech recognition model: a transformer encoder–decoder that listens to 30 seconds of audio and writes down what was said, or its English translation. It was released in September 2022 with open weights and code (OpenAI, n.d.; Radford et al., 2023, sec. 1).

How it works

The audio is resampled to 16 kHz and turned into an 80-channel log-Mel spectrogram, computed on 25 ms windows every 10 ms (Radford et al., 2023, sec. 2.2): a picture of the sound, with time on one axis and energy per frequency band on the other, the bands spaced on the Mel scale, which mimics the perception of pitch. It is the old recipe of computing features inspired by the human auditory system from a spectral analysis (Witten et al., 2017, p. 524). Two convolution layers condense that picture; sinusoidal positions are added and the encoder's transformer blocks follow. The decoder, with cross-attention over the encoder's output, generates tokens with GPT-2's BPE (2023, sec. 2.2). There are five sizes, from 39 to 1,550 million parameters (2023, sec. 2.4).

The cleverest part is the output format. The task is not chosen by switching models but with special tokens at the start of what the decoder writes: <|startoftranscript|>, a language token (one of 99), <|transcribe|> or <|translate|>, and <|notimestamps|> or timestamps every 20 ms; if there is no speech, <|nospeech|> (2023, sec. 2.3). A single model thus replaces several stages of a classic system: detecting speech, identifying the language, transcribing, translating and aligning text with audio. It is T5's move applied to speech: the instruction is part of the sequence.

What changed

Classic speech recognition chained signal preprocessing, hidden Markov models and a language model, and deep learning gradually replaced pieces of that chain (Witten et al., 2017, p. 524). M. Mitchell (2019) calls it deep learning's first major success in language processing, from 2012 on (2019, ch. 11). Before Whisper, the usual approach was to pretrain encoders on untranscribed audio (self-supervised learning) and then fine-tune a decoder for each dataset (Radford et al., 2023, sec. 1).

Whisper bet on weak supervision. It is the familiar supervised setup, sound files paired with transcripts (Chollet, 2021, sec. 1.1.3), but with 680,000 hours of pairs taken from the internet, noisy and unchecked (Radford et al., 2023, sec. 2.1). 65% is English; about 117,000 hours cover 96 other languages (Spanish, with ≈ 11,100, ranks third) and about 125,000 are translation into English (2023, app. E). Transcripts produced by other automatic recognizers had to be filtered out, so as not to learn their "transcript-ese" (2023, sec. 2.1). Where the audio came from was not disclosed, and the dataset was not released.

The payoff is zero-shot robustness. Models trained on LibriSpeech match or beat a human on that dataset but make roughly twice as many errors as the human on others; the best Whisper model, without having seen LibriSpeech, makes 55% fewer errors outside it than a supervised model that is just as good inside it, and roughly matches human robustness (Radford et al., 2023, sec. 3.3). It is the i.i.d. assumption problem seen from speech.

Limits

Quality depends on how many hours each language had, and most had fewer than a thousand (Radford et al., 2023, sec. 6); Quechua and Guaraní are absent (2023, app. E). Automatic language identification also polluted the data: most of what was labeled as Welsh translation was English audio misclassified (2023, sec. 3.5).

The decoder is an audio-conditioned language model, and that has a cost: the paper reports repetition loops and complete hallucinations, transcripts unrelated to the audio (Radford et al., 2023, sec. 6). The model card attributes this to the model mixing next-word prediction with transcription (OpenAI, n.d.). Koenecke et al. (2024) ran 13,140 speech segments through the API in 2023: 1.4% of the transcriptions contained invented phrases, and 38% of those included explicit harms (violence, made-up names or illnesses, YouTuber-style thanks), more often for speakers with aphasia, whose pauses are longer (2024, secs. 3, 4.2).

Professor Dendrett's note

For twentieth-century linguistics, language is first of all spoken. Saussure put it with an image: "Langue et écriture sont deux systèmes de signes distincts ; l'unique raison d'être du second est de représenter le premier" [language and writing are two distinct systems of signs; the sole reason for the second to exist is to represent the first; my translation], and confusing them is like believing that, to know someone, it is better to look at their photograph than at their face (Saussure, 1916/1971, p. 45).

Whisper takes photographs. Its output is writing with the conventions of internet captions, capitals, commas and question marks included, which is what it learned to imitate (Radford et al., 2023, sec. 2.1). Much of prosody is left out: the intonation that tells a question from a statement, the stress that marks focus ("I didn't say that" versus "I didn't say that"), irony, pauses, laughter, speakers talking over each other. Many implicatures, what is conveyed without being said (Grice, 1975), depend on how something is said. M. Mitchell (2019) marveled that speech recognizers transcribe without understanding anything, and suspected that the last 10 percent of the task would require understanding what is said (2019, ch. 11).

There is one more irony: Whisper learned to listen from text written by people, and, according to press reports cited by Koenecke et al. (2024), it transcribed over a million hours of YouTube to train GPT-4 (2024, sec. 4.1). Is a transcript a photograph of speech, or a translation into another system of signs? What would a recognizer have to write for a phonetician or a conversation analyst to call it faithful?

Sources

Chollet, F. (2021). Deep learning with Python (2nd ed.). Manning Publications. §
Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics: Vol. 3. Speech acts (pp. 41–58). Academic Press. https://doi.org/10.1163/9789004368811_003 §
Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024). Careless Whisper: Speech-to-text hallucination harms. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 1672–1681). Association for Computing Machinery. https://doi.org/10.1145/3630106.3658996 §
Mitchell, M. (2019). Artificial intelligence: A guide for thinking humans. Pelican Books. §
OpenAI. (n.d.). Model card: Whisper. GitHub. Retrieved October 1, 2026, from https://github.com/openai/whisper/blob/main/model-card.md §
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (Vol. 202, pp. 28492–28518). PMLR. https://proceedings.mlr.press/v202/radford23a.html §
Saussure, F. de. (1971). Cours de linguistique générale [Course in general linguistics] (C. Bally & A. Sechehaye, Eds.). Payot. (Original work published 1916) §
Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2017). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann. https://doi.org/10.1016/C2015-0-02071-8 §