LLaMA is a family of four transformer decoders, from 7 to 65 billion parameters, that Meta AI introduced in February 2023. Its architecture is conventional; its importance lies in two decisions: training small models for much longer, and using only publicly available data so that the weights could be released to the research community (Touvron, Lavril, et al., 2023).
How it works
It is an autoregressive language model like GPT-3, with three tweaks borrowed from other models: it normalizes the input of each sublayer (pre-normalization, with RMSNorm), uses the SwiGLU activation in the feedforward network, and replaces absolute positional encoding with rotary embeddings (RoPE), applied inside attention (Touvron, Lavril, et al., 2023, sec. 2.2). Alammar & Grootendorst (2024) present exactly these changes as the features of the typical 2024 transformer block, that of Llama 3 (2024, pp. 101–103). The tokenizer is BPE and splits numbers into single digits.
The data add up to 1.4 trillion tokens: two thirds filtered CommonCrawl, plus C4, GitHub, Wikipedia in twenty languages, books, arXiv and StackExchange. The 7B and 13B models saw 1 trillion tokens; the 33B and 65B models, 1.4 trillion. The largest was trained for about 21 days on 2,048 A100 GPUs (Touvron, Lavril, et al., 2023, secs. 2.1 and 2.4).
What changed
Chinchilla had shown how to split a fixed compute budget between parameters and data (Hoffmann et al., 2022). Touvron and colleagues objected that this calculation ignores the cost of using the model: for a given level of performance, a smaller model trained longer is preferable, even if it costs more to train. Where Chinchilla recommended a 10-billion model on 200 billion tokens, they observed that a 7-billion model kept improving after a trillion (Touvron, Lavril, et al., 2023, sec. 1). The upshot: LLaMA-13B beat GPT-3 (175 billion) on most benchmarks while being ten times smaller, and LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B.
The leak
Meta released it on February 24, 2023, under a noncommercial license, on request and case by case. On March 3 someone posted a torrent of the weights on 4chan. What followed was an ecosystem: on March 10 Georgi Gerganov released llama.cpp, whose stated goal was to run the model quantized to 4 bits on a MacBook; on March 13 Stanford presented Alpaca, a LLaMA-7B fine-tuned on 52,000 instructions generated by an OpenAI model, for under 600 dollars. Quantizing means storing each weight with fewer bits, like saying “it's 14:16” without the seconds (Alammar & Grootendorst, 2024, pp. 200–201): the 7-billion model shrinks from about 13 GB to about 4. In July came Llama 2, trained on 2 trillion tokens (Alammar & Grootendorst, 2024, p. 26), with a doubled window and released, in its own paper's words, “for research and commercial use.”
Limits
LLaMA is a base model: it neither follows instructions nor converses. Its window is 2,048 tokens, and the paper itself documents biases and toxicity inherited from the web. Besides, “open” is a contested term: the weights are distributed, but rarely the training code or the data, and licenses impose restrictions (Alammar & Grootendorst, 2024, p. 30). One of its corpora, Books3, came from a pirated-books site and was taken down in July 2023 over copyright claims.