Bengio, Ducharme, Vincent and Jauvin's neural probabilistic language model is the first neural language model that worked at a serious scale: a network that, given the last few words, estimates the probability of the next one and, in order to do so, simultaneously learns a vector for every word in the vocabulary. A first version was presented at NIPS 2000; the journal version is from 2003.
The problem
An n-gram model estimates probabilities by counting short sequences. The obstacle is the curse of dimensionality: with a 100,000-word vocabulary, modeling the joint distribution of ten consecutive words would take on the order of 10⁵⁰ free parameters, and almost any test sentence differs from every sentence seen in training (Bengio et al., 2003, p. 1137). N-grams generalize by gluing together short overlapping fragments; nothing tells them that "cat" and "dog" are alike.
How it works
The proposal has two pieces (Bengio et al., 2003, pp. 1139–1143):
- A table C of |V| × m numbers: each row is the vector (today we would say the embedding) of one word, with m = 30, 60 or 100 dimensions for vocabularies of about 17,000 words.
- A network that takes the vectors of the previous n − 1 words, concatenates them, passes them through a hidden layer with a hyperbolic tangent, and ends in a softmax over the whole vocabulary: y = b + Wx + U·tanh(d + Hx), where the matrix W, which connects the words directly to the output, is optional.
Table and network are trained together, by gradient descent on the log-likelihood of the corpus. Since the probability is a smooth function of the vectors, and words that play similar roles end up with nearby vectors, having seen "The cat is walking in the bedroom" also raises the probability of "A dog was running in a room" and of a combinatorial number of neighboring sentences (2003, pp. 1139–1140). The authors trace the idea of representing symbols with distributed vectors to early connectionism, citing Hinton (1986) and Elman (1990) (2003, p. 1140).
Results
It was tested on the Brown corpus (≈ 1.18 million words) and on Associated Press news from 1995–1996 (≈ 14 million training words) (Bengio et al., 2003, p. 1147). Compared with the best n-gram, test perplexity dropped by about 24% on Brown and 8% on AP News, and averaging the network with a trigram helped further: the two made their errors in different places (2003, p. 1148). The cost was large for the time: five passes over AP News took about three weeks on 40 CPUs (2003, p. 1148), and 99.7% of the computation went into the output layer (2003, p. 1145). By the paper's own formula, the AP News network has about 12 million parameters (our calculation); the authors mention almost 100 MB for their largest network (2003, p. 1144).
Offspring and limits
Table C survives: every current language model stores an embedding matrix, randomly initialized and adjusted during training (Alammar & Grootendorst, 2024, pp. 57–58). Mikolov, Chen, et al. (2013) took this network as the reference "NNLM" and noted that most of its cost lay in the nonlinear hidden layer; by removing it they obtained word2vec (2013, secs. 2.1, 3).
The limits are flagged in the paper itself: the window is fixed (n − 1 words), the softmax over the whole vocabulary is very expensive, and each word occupies a single point in the space, so polysemy is poorly served (Bengio et al., 2003, p. 1152). Bank is the same vector by the river and at the counter, to use the example of Alammar & Grootendorst (2024, p. 11). Recurrent networks and attention would tackle the fixed window; contextual embeddings, polysemy.