Berglund et al. (2024) describe a failure of generalization: if an autoregressive generation model is trained on sentences of the form "A is B," it does not go on to answer with A when asked about B (2024, p. 1). Fine-tuned on fictitious facts, GPT-3 175B reaches up to 96.7% accuracy in the training order and close to 0% in the reverse one, no better than picking a random name (2024, sec. 2.1). With real facts, GPT-4 identifies a celebrity's parent 79% of the time (Tom Cruise's mother is Mary Lee Pfeiffer), but given the parent it names the child only 33% of the time (2024, sec. 2.2).
It is not an inability to reason: if "A is B" is in the prompt, models reverse the relation almost without error (2024, sec. 1, app. B.6). The problem is what training leaves in the weights: on the authors' sketch of an explanation, the gradient update is "myopic," adjusting the prediction of B given A, not that of A given B (2024, sec. 4). Kitouni et al. (2024) generalize it as a "factorization curse": next-token prediction learns the distribution in one order only, and neither scale nor reversed training text cures it; an order-agnostic objective does mitigate it, masked modeling with a uniform masking rate, which is a discrete diffusion model (2024, abstract, secs. 2.1–2.2). That is how LLaDA is trained, and it recalls the preceding line of a Chinese poem almost as well as the following one (45.6% against 51.8%), whereas GPT-4o drops from 82.7% to 34.3% (Nie et al., 2025, sec. 3.3).
Identity and symmetry. The authors suggest seeing a failure of deduction here: b = a follows from a = b by symmetry. They warn that this is a simplification, since the model learns to predict what people would write, not what is true (2024, sec. 1). Frege (1892/1998) started from a different asymmetry: a = a holds a priori, whereas a = b can be a discovery, such as that the morning star is the evening star (1892/1998, pp. 84–85). The curse adds a third: for the model, a = b and b = a are not equally available. Does it store identities, or directed associations? We too recite the alphabet backwards less well, though the authors know of no human effect this stark (2024, sec. 3).