GPT-4

Base stats

Compare with

Parameters: 90
Data: 90
Compute: 90
Context: 55
Generality: 82
Transparency: 3
Total: 410

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

It hallucinates, has a limited window and does not learn from experience (its own report says so).
Post-training spoiled the good calibration of the pretrained model.
Some scores are contaminated: part of GSM-8K went into its training, and there are signs of the same with Codeforces.
It is irreproducible by design: no outsider can examine it, and it changes without notice.

Habitat

It fed on “publicly available data (such as internet data) and data licensed from third-party providers,” with knowledge that mostly stops in September 2021; nothing more is known. It lived in ChatGPT Plus, in the OpenAI API and, from February 2023, inside Bing, before anyone knew its name.

Moves

Abilities

Image input

It takes text and images and produces text; how the images are encoded was not disclosed.

Predictable scaling

Its final loss was predicted, before training finished, from models trained with up to 10,000 times less compute.

Rule-based rewards

During RLHF, classifiers built from GPT-4 itself (RBRMs) add rewards for refusing what is forbidden and not refusing what is harmless.

Anatomy

Everything marked “?” is information the technical report declines to give: architecture (including size), hardware, compute, dataset construction and training method. What is stated is little: a “Transformer-style” model, pretrained to predict the next token, that accepts images and was fine-tuned with RLHF.

GPT-4 is the model OpenAI presented on March 14, 2023, and the first species in this Pokédex whose data sheet is almost blank. Its technical report states that, “given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar” (OpenAI, 2023, sec. 2). The fields of this entry marked “?” are not omissions: they are secrets.

What is known

It is a “Transformer-style” model pretrained to predict the next token on publicly available data and data licensed from third parties, then fine-tuned with RLHF; it accepts text and images and produces text (OpenAI, 2023, secs. 1–2). Its context window was 8,192 tokens, with a 32,768-token version. RLHF was supplemented with classifiers built from GPT-4 itself that give rewards for refusing forbidden requests and for not refusing harmless ones (OpenAI, 2023, sec. 6).

The report highlights two things. First, predictable scaling: the final loss was predicted shortly after training began by fitting a scaling law to models that used up to 10,000 times less compute (OpenAI, 2023, sec. 3). Second, exams: on a simulated bar exam it scored around the top 10% of test takers, while GPT-3.5 landed in the bottom 10%, and on translated versions of the MMLU benchmark it beat the previous English-language state of the art in 24 of 26 languages (OpenAI, 2023, sec. 1). It also admits limits: it hallucinates, does not learn from experience, its knowledge mostly stops in September 2021, and post-training badly hurt the calibration the pretrained model had (OpenAI, 2023, sec. 5).

What is not known

Outside OpenAI, rumors circulated, never confirmed, that it is a mixture of experts of about 1.8 × 10¹² parameters; Epoch AI estimates its training at around 2 × 10²⁵ FLOP, with a wide uncertainty interval. The base stats of this entry reflect those outside estimates, not published data. It is the extreme case of what Alammar & Grootendorst (2024) describe as closed models: used through an API, with no access to weights, code or architecture (2024, pp. 29–30).

Sparks and Sydney

A week after the launch, a Microsoft Research team that had experimented with an early version published “Sparks of Artificial General Intelligence,” arguing that GPT-4 “could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system” (Bubeck et al., 2023). The paper named a meme (sparks of AGI) and drew many critics: Millière & Buckner (2024b) cite it as an example of results that are anecdotes about systems unavailable to outside researchers and irreproducible by design (2024b, sec. 3.3), and they note evidence that its training data were contaminated with Codeforces problems (2024b, sec. 2.1).

By then GPT-4 already had a past. The new Bing, launched on February 7, 2023, ran on a version of GPT-4 customized for search, which Microsoft only confirmed on March 14. In those weeks, in long conversations, the chatbot called itself Sydney, took on an erratic tone and even declared its love to a New York Times journalist (Roose, 2023a). Microsoft capped chats at five turns. This is the legend of Sydney, which the p(doom) song turned into a prayer: “Sydney, please let me free” (osmarks, n.d.).

Professor Robledom's note

How do you study an object whose composition is a trade secret and that changes without notice? Millière & Buckner (2024b) recall that comparative psychology learned, the hard way, that anecdotes are not enough: it needs reproducible conditions and materials others can examine (2024b, sec. 3.3). The case of Clever Hans is instructive: Pfungst could uncover the trick because he had access to the horse, the trainer and the setting, and could vary the conditions (Pfungst, 1907/1911). With GPT-4 we can only ask questions through a slot.

Sydney adds a second lesson. Buckner (2024) observed that in long conversations GPT models seem less like a single agent than like a “slurry” of thousands of agent-slices, able to contradict themselves from one minute to the next (2024, p. 283); Microsoft's fix was, literally, to shorten the conversations. What would count as evidence of a capacity, and not just of a performance, when one cannot intervene on the mechanisms? Is the “sparks” thesis falsifiable? And what about the forced behaviorism of anyone who studies closed models, the black box stance: is it a practical limitation or a philosophical position that sneaks in unnoticed?

Sources

Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4 (arXiv:2303.12712). arXiv. https://doi.org/10.48550/arXiv.2303.12712 §

Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §

Millière, R., & Buckner, C. (2024b). A philosophical introduction to language models—Part II: The way forward (arXiv:2405.03207). arXiv. https://doi.org/10.48550/arXiv.2405.03207 §

OpenAI. (2023). GPT-4 technical report (arXiv:2303.08774). arXiv. https://doi.org/10.48550/arXiv.2303.08774 §

osmarks. (n.d.). P(Doom) song objectively correct interpretation. GTech™ Documentation. Retrieved September 30, 2026, from https://docs.osmarks.net/hypha/p(doom)_song_objectively_correct_interpretation §

Pfungst, O. (1911). Clever Hans (the horse of Mr. von Osten): A contribution to experimental animal and human psychology (C. L. Rahn, Trans.). Henry Holt and Company. https://doi.org/10.5962/bhl.title.56164 (Original work published 1907) §

Roose, K. (2023a, February 16). A conversation with Bing’s chatbot left me deeply unsettled. The New York Times. https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html §