ElemedexA pocket encyclopedia of AI architectures
ENES
Gemini
No. 047

Gemini

Gemini 1.0 (Ultra · Pro · Nano) → 1.5 → 2.0 → 2.5 → 3 → 3.x

Omnivore

MultimodalDecoder

It was raised watching, listening and reading all at once. It learned to translate some Kalamang, a language with fewer than 200 speakers, by reading its grammar and dictionary in a single sitting. In 2025 it took 813 hours to become Pokémon League Champion; the second time, half that.

Height
? (undisclosed)
Weight
? (undisclosed; Nano, 4-bit quantized, runs on a phone)
Parameters
Nano: 1.8 B and 3.25 B · Pro and Ultra: undisclosed (mixture of experts since 1.5)
Context
32K (1.0) · 1–2 M (1.5 onward) · 10 M in research tests
Training data
? (web documents, books and code, plus images, audio and video; size undisclosed)
Year
2023
Generation
6 · Assistants, reasoning, and agents
Creators
Google DeepMind (Gemini Team, Google)
Kind
Model

Base stats

Parameters
90
Data
92
Compute
93
Context
98
Generality
90
Transparency
6
Total
469

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

  • Almost everything is secret: sizes, detailed architecture and data of Pro and Ultra are not published.
  • Its headline number, 90% on MMLU, used 32 chains of thought per question; with five examples, the way GPT-4 was measured, it fell short.
  • The launch video did not show a real-time interaction: it was assembled from still frames and text prompts.
  • In 2024 the app's image generator overcorrected for diversity, and Google paused images of people.

Habitat

It was trained on Google TPUs on web documents, books, code, images, audio and video. It replaced Bard, first under the hood (December 2023) and then in name too (February 2024). It lives in the Gemini app, in Search, on Android phones, where Nano runs on the device, and in the developer API.

Moves

Abilities

Multimodal from birth
It is pretrained from the start on text, images, audio and video interleaved in a single sequence, instead of stitching separate models together.
Million-token context
Gemini 1.5 Pro retrieves a “needle” with 99.7% recall at a million tokens and 99.2% at ten million, though it does much worse when there are a hundred needles.
Sparse experts
Since 1.5 it is a mixture of experts: a router sends each token to a subset of the parameters, so compute per token does not grow with total size.

Anatomy

Decoder block×?TextImageAudioVideoSentencePiecetokensVisualencoderUSMfeatures(16kHz)Frames→imagesInterleaved sequenceCausal attention (e.g. MQA)FFN (MoE since 1.5)TextImage tokensGemini 1.0 (what is published)

Four modalities, one sequence. According to the Gemini 1.0 report, text, images (with a visual encoding inspired by Flamingo, CoCa and PaLI), audio (features from the USM speech model, at 16 kHz) and video (as a succession of frames) enter interleaved into a single transformer decoder with efficient attention and a 32K context, which can answer with text or with images made of discrete tokens. Layers, sizes and details of the visual encoder were not published; since 1.5, the feedforward layers are a mixture of experts.

Evolution chain

  • everything as tokens, now at frontier scale and with a million-token context038 Gato
047 Gemini

Gemini is Google DeepMind's family of models, which Google introduced on December 6, 2023 in three sizes: Ultra, Pro and Nano (Pichai & Hassabis, 2023). That same day Bard, Google's chatbot, began using a fine-tuned version of Gemini Pro; in February 2024 Bard was renamed Gemini (Hsiao, 2024). Then came 1.5 (February 2024), 2.0 and 2.5 (December 2024 and March 2025) (Gemini Team, 2024; Gemini Team, 2025), 3 (November 2025) and, in 2026, 3.1 Pro and a series of Flash models numbered 3.5 to 3.8 (Google, n.d.).

How it works

The promise was to be “natively multimodal”: instead of training separate components for each modality and stitching them together afterwards, Gemini was pretrained on several from the start (Pichai & Hassabis, 2023). According to its technical report, it is a transformer decoder with efficient attention (multi-query attention, for example) and a 32K context, which takes text interleaved with images, charts, screenshots, PDFs, audio and video, and answers with text or with images generated as discrete tokens. Video goes in as a succession of frames; audio, as features from a speech model at 16 kHz (Gemini Team, 2023, sec. 2). It is Gato's recipe, everything turned into tokens in a single sequence (Reed et al., 2022), brought to frontier scale. Compared with CLIP, which aligns images and texts in a shared space, or GPT-4, which accepts images with an undisclosed encoding, Gemini at least describes the general scheme of its multimodality.

The numbers, however, are missing. Only Nano's are known (1.8 and 3.25 billion parameters, obtained by distillation and quantized to 4 bits) (Gemini Team, 2023, sec. 2); the data are web documents, books and code, plus images, audio and video, with no figures (2023, sec. 4). Epoch AI tentatively estimates about 5 × 10²⁵ FLOP for Ultra (Epoch AI, 2026); the base stats of this entry reflect those outside estimates, not published data. Since 1.5, Pro has been a sparse mixture of experts (Gemini Team, 2024, sec. 3.1), and 2.5 continues that path (Gemini Team, 2025, sec. 2.1).

Millions of tokens

Gemini 1.5 Pro pushed the context window to several million tokens: it retrieves a hidden sentence (a “needle”) with 99.7% recall at a million tokens and 99.2% at ten million (Gemini Team, 2024, sec. 5.2). The test is simple, as the authors admit: with a hundred needles at once, recall drops to about 70% at 128K and just over 60% at a million (-@gemini2024, sec. 5.2.1.5). The most striking example is linguistic: given a field grammar of about 500 pages, a 2,000-entry dictionary and some 400 parallel sentences in its context, the model translated from English into Kalamang, a Papuan language with fewer than 200 speakers, at a quality close to that of a person who studied the same materials (Gemini Team, 2024, sec. 5.2.2.1; Tanzer et al., 2024). Without those materials, the output was chance.

With 2.5 came the models that “think”: trained with reinforcement learning to spend more test-time compute before answering, with an adjustable thinking budget (Gemini Team (2025), sec. 2.5) (see Reasoning model (o1, R1)). An independent developer built it a harness to play Pokémon Blue, and Gemini 2.5 Pro finished the game after 813 hours, with the harness adjusted along the way (2025, sec. 4.1).

Stumbles

The launch came with fine print. The 90% on the MMLU benchmark, presented as the first result to beat human experts, used a technique with 32 chains of thought per question; with five examples, the format in which GPT-4 was measured, Ultra scored 83.7% against 86.4% (Gemini Team, 2023, sec. 5.1.1). The demo video, in which Gemini seemed to comment live on what it saw, had been made with still frames and text prompts (Coldewey, 2023). And in February 2024 the app's image generator (built on another model, Imagen 2) produced inaccurate historical images by overcorrecting for diversity; Google paused images of people (Raghavan, 2024).

Sources