A language model is said to hallucinate when it confidently produces incorrect information (Alammar & Grootendorst, 2024, p. 177): a biography with wrong dates, an academic paper that does not exist. The technical literature defines it as content that is nonsensical or unfaithful to the provided source, and distinguishes intrinsic hallucination (contradicting the source) from extrinsic hallucination (unverifiable from it) (Ji et al., 2023, secs. 2–2.1).
It is not an occasional glitch but a consequence of the training objective: the model learns to produce plausible continuations, with no reference to whether they are true (Mahowald et al., 2024). When launching ChatGPT, OpenAI admitted that the system sometimes gives plausible-sounding but incorrect answers, partly because in reinforcement-learning training "there's currently no source of truth" (OpenAI, 2022). Retrieval augmentation reduces the problem by giving the model documents to lean on, but does not eliminate it (Alammar & Grootendorst, 2024, pp. 249–250).
The word is contested. It comes from computer vision, where it had a positive sense: "hallucinating" details to increase an image's resolution (Ji et al., 2023, sec. 1). Applied to language models, it faces three objections. It suggests perception, and the model perceives nothing; hence Smith et al. (2023) propose confabulation, the clinical term for false accounts the patient does not recognize as such. It uses a psychiatric symptom as a metaphor, which may stigmatize; Østergaard & Nielbo (2023) prefer names of fallacies, such as non sequitur. And it suggests an exception, when true and false answers come out of exactly the same process. Hicks et al. (2024) go further: in Frankfurt's sense, what these systems produce is bullshit, speech indifferent to truth.
Shanahan (2024) supplies the ground for the discussion: a bare-bones model has no notion of truth or falsehood, because it lacks the means to exercise those concepts the way we do (2024, p. 72). If so, does it make sense to speak of error, or only of a text that fails to match the world? See anthropomorphism.