The embeddings of current models live in spaces of 384, 768 or 3,072 dimensions (Alammar & Grootendorst, 2024, pp. 60, 62, 79). Nobody can visualize them, and analogies with the plane mislead: as Witten et al. warn, our intuitions tend to lead us astray in high-dimensional spaces (2017, p. 87).
Tiny example. In a cube of side 1, what fraction of the volume lies within 0.05 of some face? It is 1 − 0.9ᵈ, where d is the dimension. For d = 2, 19%; for d = 10, 65%; for d = 100, 99.997%. In high dimensions almost all the volume is “in the shell”, and almost no point is in the “middle”.
Hence the so-called curse of dimensionality, a phrase usually credited to the mathematician Richard Bellman: the volume of the space grows exponentially with the number of axes. If each axis is divided into 10 intervals, there are 10ᵈ cells: a thousand for d = 3, an astronomical number for d = 768. No dataset can “fill” that space, and distance-based methods such as k-nearest neighbors suffer especially when there are many irrelevant attributes (Witten et al., 2017, p. 290).
Why does deep learning work, then? One influential answer is the manifold hypothesis: natural data do not occupy the whole space but a low-dimensional region within it, and learning consists largely in discovering that region and “uncrumpling” it (Chollet, 2021, sec. 5.1). Chollet presents it as a strong claim about the structure of information, probably true; and he warns that interpolating within that region yields local generalization, not the extreme generalization humans are capable of. See also latent space.