ElemedexA pocket encyclopedia of AI architectures
ENES
DeepSeek-V3
No. 048

DeepSeek-V3

DeepSeek-V3 → V3-0324 → V3.1 → V3.2 → V4

Thrifty Expert

DecoderGenerative

It was trained in under two months on 2,048 export-trimmed GPUs. Its rental bill, $5.6 million, became the most quoted (and misquoted) number of 2025: it left out everything that came before. A month later, when its offspring R1 arrived, Nvidia lost nearly $600 billion in a single day.

Height
61 layers
Weight
≈ 685 GB (FP8, as released) · ≈ 1.4 TB (BF16)
Parameters
671 B total · 37 B active per token (+ 14 B MTP module)
Context
128K tokens (4K during pretraining)
Training data
14.8 trillion tokens, with more math and code than V2 and more languages beyond English and Chinese; composition undisclosed
Year
2024
Generation
6 · Assistants, reasoning, and agents
Creators
DeepSeek-AI (Hangzhou)
Kind
Model

Base stats

Parameters
86
Data
91
Compute
86
Context
80
Generality
78
Transparency
40
Total
461

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

  • The famous cost is partial: the $5.6 million covers only the final run, at rental prices, without the prior research and experiments.
  • Open weights, closed data: nobody outside knows what is in its 14.8 trillion tokens.
  • A US government evaluation found that DeepSeek's models echo Chinese Communist Party narratives four times as often as US reference models.
  • Serving it well requires a large deployment unit, as its own report admits.

Habitat

It was trained on a cluster of 2,048 Nvidia H800 GPUs, the version of the H100 with its chip-to-chip transfer rate cut so that it could be sold in China. Its weights are freely downloadable; it lives in DeepSeek's app and API, and on any server with enough memory to hold its 671 billion parameters.

Moves

Abilities

Multi-head latent attention
Keys and values are reconstructed from a compressed 512-dimensional vector per token and layer; the cache stores that rather than the keys and values of all 128 heads.
Shared and routed experts
Each expert layer has one shared expert, which processes every token, and 256 small routed ones, of which eight are chosen per token.
Auxiliary-loss-free balancing
Instead of a loss that penalizes imbalance, each expert has a bias that only affects selection: it goes down if the expert is overloaded and up if it is idle.

Anatomy

Transformer block×61in detailTokens (128K BPE)Embeddings (7,168 dim.)RMSNormLatent attention (MLA)RMSNormRouter: top-8 of 256SharedE1E7…E131…E256ΣNext tokenMTP: the oneafterhₜ (7,168 dim.)Compressed c_KV (512)Cache: c_KV + RoPE keyK (128 heads)V (128 heads)softmax(QKᵀ/√d)·VDeepSeek-V3MLA (detail)

Each of the 61 blocks combines latent attention (MLA) with an expert layer (the first three layers instead have a dense feedforward network). The router scores the 256 routed experts and picks eight (in color, together with the shared one); the output adds up the shared expert, which sees every token, and the eight chosen ones, weighted by their normalized scores. On the right, MLA: instead of storing keys and values for each of the 128 heads, the KV cache stores a compressed 512-dimensional vector plus a 64-dimensional positional (RoPE) key, and keys and values are reconstructed from it. A separate module (MTP) also learns to predict the token after the next one.

Evolution chain

048 DeepSeek-V3

DeepSeek-V3 is an open weights language model that the Chinese company DeepSeek released on December 26, 2024: a mixture of experts with 671 billion parameters, of which each token uses 37 billion, pretrained on 14.8 trillion tokens (DeepSeek, 2024; DeepSeek-AI, 2024). Its technical report is unusually detailed, and the model was the base of R1, the January 2025 reasoning model (Guo et al., 2025).

How it works

It is a transformer decoder with two pieces inherited from DeepSeek-V2 (DeepSeek-AI, 2024, sec. 2.1). The first is multi-head latent attention (MLA). In ordinary attention, generating text requires storing the keys and values of every head for every previous token: the KV cache. MLA compresses both into a single 512-dimensional latent vector, from which the keys and values of the 128 heads are reconstructed (2024, secs. 2.1.1 and 4.2). With the report's hyperparameters, the cache holds 576 numbers per token and layer, against 32,768 for standard attention of the same size (our own calculation).

The second is the DeepSeekMoE expert architecture: many small experts plus a few “shared” ones that process every token, so that the rest can specialize (Dai et al., 2024). Where Mixtral has 8 experts, V3 has in each layer 1 shared and 256 routed experts, of which it picks 8. To keep the router from always favoring the same ones, instead of the usual balancing loss it uses a per-expert bias adjusted after each step (DeepSeek-AI, 2024, secs. 2.1.2 and 4.2). In addition, a multi-token prediction (MTP) module learns to anticipate the token after the next; in use, that second prediction is accepted 85% to 90% of the time and speeds up generation 1.8 times (2024, secs. 2.2 and 5.4.3). The large matrix multiplications were done in FP8, 8-bit numbers, which the report presents as the first validation at this scale (2024, sec. 3.3). Post-training combined supervised tuning, reinforcement learning and reasoning distilled from R1 (2024, sec. 5).

What it cost

The report gives the total: 2.788 million H800 GPU hours, of which 2.664 million went to pretraining, completed in under two months on 2,048 cards. At an assumed rental price of $2 an hour, that is $5.576 million, a figure that, the report itself notes, excludes prior research and experiments (DeepSeek-AI, 2024, sec. 1). Epoch AI estimates about 3.3 × 10²⁴ FLOP (Epoch AI, 2026), nearly an order of magnitude below the estimates for GPT-4. The H800 is the version of the H100 that Nvidia trimmed, mainly in chip-to-chip transfer speed, so that it could be sold in China (Nellis & Lee, 2023); much of the report's ingenuity goes into working around that bottleneck, for instance by limiting each token to four nodes (DeepSeek-AI (2024), sec. 3.2).

The “DeepSeek moment”

R1 came out on January 20, 2025 (DeepSeek, 2025); days later DeepSeek's app overtook ChatGPT as the most-downloaded free app on the US App Store, and on January 27 Nvidia fell 17% and lost nearly $600 billion in value, the largest one-day drop for a US company (Subin, 2025). Dario Amodei, who runs Anthropic and advocates export controls, replied that V3 was an expected point on the cost-reduction curve (Amodei, 2025). A US government evaluation found that DeepSeek's models echo more official Chinese narratives than US reference models (NIST, 2025). The family went on with V3.1 and V3.2 (2025), which introduced its own sparse attention, and V4 (April 2026), whose Pro version has 1.6 trillion total parameters, 49 billion active and a million-token context (DeepSeek, 2026).

Sources