Jeffrey Elman's simple recurrent network is a neural network that reads a sequence one element at a time and, at each step, also receives a copy of its own previous state. With that minimal trick (a loop) time stops being a spatial dimension of the input and is represented implicitly, by the effect it has on processing (Elman, 1990, pp. 180–182).
How it works
The architecture starts from a multilayer perceptron with one hidden layer and adds a bank of context units, as many as there are hidden units. At time t, the hidden layer receives the current input and the contents of the context units, which are exactly the hidden state from time t − 1. The copy connections are fixed at 1 and are not trained; the remaining weights are adjusted with ordinary backpropagation (Elman, 1990, pp. 182–183). The idea came from a network by Michael I. Jordan (1986), the cognitive scientist rather than the basketball player, which fed back the output instead of the hidden state.
Chollet describes any recurrent network as a loop that reuses, at each iteration, what it computed at the previous one (Chollet, 2021, sec. 10.3); Elman's is the most austere version of that recurrence. Since the same weights are applied at every step, the network handles sequences of any length with a fixed number of parameters.
What it found
The paper became a classic because of its simulations, all built on the same task: predict the next element.
- Words without spaces. Given a stream of letters with no separators, prediction error rose at the beginning of each word and fell as the word became predictable: error peaks marked lexical boundaries, although in a graded way and with ambiguities (Elman, 1990, pp. 191–193).
- Lexical categories. A generator produced 10,000 two- and three-word sentences over 29 distinct words, coded as orthogonal vectors that said nothing about their class. A network with 150 hidden units learned to predict the next word; Elman then averaged the hidden states for each word and submitted them to hierarchical clustering. The tree separated verbs from nouns, transitives from intransitives, animates from inanimates, humans from animals, and edibles from breakables (Elman, 1990, pp. 195–200).
- A new word. When man was replaced throughout the corpus by an invented word, zog, the network, without any further learning, gave it a representation located where man had been (Elman, 1990, p. 201).
Elman stressed that these categories were soft and implicit: regions of a space, not named symbols. And since each hidden state represents a word in context, the network offers a way of telling types from tokens (Elman, 1990, pp. 200–203). He himself added a caveat: averaging states can produce spurious clusters, so they should be checked against individual tokens (1990, p. 199, n. 6).
Limits
Its memory is short. In the original network the error does not travel backward in time (the copy is fixed), and when such networks were trained with backpropagation through time, the vanishing gradient showed up: the error signal shrinks or blows up exponentially with distance, and long dependencies become almost impossible to learn, as analyzed by Hochreiter (1991) and by Bengio et al. (1994) (Hochreiter & Schmidhuber, 1997, sec. 3). That is why this entry's context stat is low. Moreover, reading one step at a time rules out parallel training (Alammar & Grootendorst, 2024, p. 14).
Its offspring is huge: the LSTM added gates; recurrent encoder–decoder networks translated text before the transformer (Alammar & Grootendorst, 2024, pp. 11–12); and recent architectures such as RWKV or Mamba are betting on recurrence again (2024, p. 24).