In a recurrent network, the hidden state hₜ is what the network "remembers"
at step t. It is computed from the current input and the previous state,
hₜ = f(xₜ, hₜ₋₁) (Bahdanau et al., 2015, sec. 2.1), and reused at the next step
(Witten et al., 2017, p. 457). It is "hidden" in the same sense as a hidden layer: it is neither
input nor output, but an intermediate representation that nobody specifies and that is
adjusted during training.
A fixed-size summary. The state always has the same dimension, whether the network has read three words or three hundred. This makes it cheap and, at the same time, limited. In sequence-to-sequence task translators, the encoder's last hidden state was the only summary of the sentence the decoder received; with long sentences, information was lost (Bahdanau et al., 2015, sec. 1; M. Mitchell, 2019, ch. 12). Attention arose to hand the decoder all the hidden states, not just the last one (Alammar & Grootendorst, 2024, pp. 13–14).
Variants. In an LSTM, alongside the hidden state there is a "memory cell" cₜ that gates protect from being overwritten (Witten et al., 2017, pp. 458–459). In state-space models such as Mamba, the state is again the protagonist: fixed in size, as opposed to a transformer's KV cache, which grows with every token. In transformer jargon, by contrast, "hidden states" are the activations of each layer for each token, even though they no longer pass from one time step to the next.
What was found inside. Elman (1990) trained a simple recurrent network to predict the next word and, clustering its hidden states, found that they were organized by category: nouns and verbs, animate and inanimate (Simple recurrent network (Elman)). Nobody had taught it grammar. For linguistics, it was one of the first demonstrations that abstract categories can emerge from pure distribution; whether that amounts to having those categories is another debate.