ElemedexA pocket encyclopedia of AI architectures
ENES
Hopfield network
No. 049

Hopfield network

Content-addressable (associative) memory

Attractor

RecurrentNeural

It keeps no memory in any drawer: it keeps memories in the shape of the landscape. Show it half a face and it rolls downhill until the face is whole. Teach it too many faces and it starts mixing them up, remembering some it never saw.

Height
1 recurrent layer: every unit connected to every other
Weight
1 bit of state per unit + the weight matrix (kilobytes)
Parameters
N(N−1)/2 symmetric weights: 4,950 with N = 100
Context
no sequence: one N-bit state
Training data
the patterns to remember: ≈ 0.15 N (≈ 15 with N = 100)
Year
1982
Generation
2 · Classical machine learning
Creators
John J. Hopfield (Caltech / Bell Laboratories)
Kind
Architecture

Base stats

Parameters
18
Data
5
Compute
8
Context
0
Generality
10
Transparency
68
Total
109

Approximate, ordinal scales meant for comparing species with each other; the hard numbers are in the data sheet.

Weaknesses

  • Low capacity: with N units it stores about 0.14 N patterns, and when overloaded it forgets everything at once.
  • It has spurious minima: valleys nobody stored, often blends of real memories.
  • Memories that are too similar get confused and end up merged into one.
  • It retrieves what it has, but it does not learn a function that generalizes to new cases.

Habitat

It was born in 1982 in the Proceedings of the National Academy of Sciences, written by a physicist working between Caltech and Bell Labs. Today it lives in textbooks of computational neuroscience and statistical physics and, transformed, inside every attention layer. In 2024 it visited Stockholm.

Moves

  • —Complete a pattern from a fragment
  • —Error correction
  • —Energy descent
  • TM026Hebbian learning
  • —Recognizing the unfamiliar
  • —Combinatorial optimization (traveling salesman)

Abilities

Content-addressable memory
There is no need to know where a memory is stored: a large enough part of it, or a version with errors, is enough to retrieve the whole.
Energy function
With symmetric weights, every update lowers the energy E = −½ Σ wᵢⱼ sᵢ sⱼ or leaves it unchanged; the network cannot circle forever and comes to rest in a minimum.
Hebbian weights
The weights are set in one go by adding up the correlations between units across the patterns to be stored: no gradient, no backpropagation.

Anatomy

One unit at random↻the same weightsPatterns ξ¹ … ξᵖHebb: wᵢⱼ = Σ ξᵢ ξⱼSymmetric weights, wᵢᵢ = 0Cue: “& Wannier, (1941)”s₁s₂s₃s₄s₅hᵢ = Σⱼ wᵢⱼ sⱼsᵢ ← sign(hᵢ)E = −½ Σ wᵢⱼ sᵢ sⱼ dropsAttractor: the full referenceStoring (once)Recalling

A five-unit Hopfield network. On the left, storage: the weights are computed once from the patterns (Hebb's rule) and are symmetric. On the right, recall: the network starts from the partial cue (the example is Hopfield (1982)'s own); at each step a randomly chosen unit adds up what all the others send it and aligns with that sum. Every change lowers the energy E or leaves it unchanged, so the network ends in a valley: the stored pattern closest to the cue. Hopfield used 0/1 states; here they are written ±1, today's usual convention.

Evolution chain

049 Hopfield network

A Hopfield network is a recurrent neural network in which every unit is connected to every other unit and whose dynamics drag it, step by step, toward stable states. Those states are its memories. John Hopfield introduced it in 1982 as a physical model of "content-addressable" memory: a system that retrieves a whole memory from a large enough part of it (Hopfield, 1982).

How it works

Each unit is binary, like the McCulloch–Pitts neuron (Hopfield used 0 and 1; today it is usually written ±1). At random moments a unit adds up its incoming signals, hᵢ = Σⱼ wᵢⱼ sⱼ, and switches on if the sum exceeds the threshold. Updates are asynchronous, with no global clock (Hopfield, 1982, p. 2555).

To store a set of patterns, the weights are set in one go: wᵢⱼ adds up, over all patterns, the product of the values of units i and j, with wᵢᵢ = 0 (1982, p. 2555). This is Hebb's idea that neurons that fire together strengthen their connection (Hebb, 1949; Buckner, 2024, p. 183). There is no gradient descent and no backpropagation: learning is adding up correlations.

The decisive point is that, if the weights are symmetric (wᵢⱼ = wⱼᵢ), the quantity E = −½ Σᵢⱼ wᵢⱼ sᵢ sⱼ never increases with an update. The network must come to rest in a local minimum of this "energy," and Hopfield noted that the system is isomorphic to an Ising model, the physics of magnets, with the weights playing the role of the coupling between spins (1982, p. 2556). Stored patterns are valleys in the energy landscape (attractors): if the network starts near one, it falls into it. The paper's example is a bibliographic reference, "H. A. Kramers & G. H. Wannier Phys. Rev. 60, 252 (1941)," which should be retrievable from "& Wannier, (1941)" or even from "Vannier, (1941)" (1982, p. 2554).

Capacity is modest: about 0.15 N patterns for N units before recall becomes badly flawed (1982, p. 2556); later analysis with the physics of spin glasses put it at ≈ 0.138 N (Ramsauer et al., 2021, sec. 1). If one keeps loading the network, it does not forget gradually: all memories become irretrievable (Hopfield, 1982, p. 2557).

What changed

The unit is almost the 1943 one; what is new is the wiring. Hopfield set his model apart from the Perceptron, with its forward-only connections and synchronized neurons: "all our interesting results arise as consequences of the strong back-coupling" (1982, p. 2555). Computation stops being a sequence of instructions and becomes the relaxation of a physical system, a collective or emergent property. With David Tank, Hopfield used an analog version of the network to find good solutions to the traveling-salesman problem (Hopfield & Tank, 1985).

Its stochastic relative is the Boltzmann machine of Ackley, Hinton and Sejnowski: units change state with a probability that depends on a temperature, hidden units are added, and the goal is no longer to store individual patterns but to learn the distribution of the data, as a generative model (Ackley et al., 1985; Witten et al., 2017, pp. 449–450). In 2024 Hopfield and Hinton received the Nobel Prize in Physics "for foundational discoveries and inventions that enable machine learning with artificial neural networks" (The Royal Swedish Academy of Sciences, 2024b).

Modern form: memory and attention

Ramsauer et al. (2021) proposed a Hopfield network with continuous states and an exponential energy, which stores a number of patterns exponential in the dimension and usually retrieves them in a single step. Its update rule, ξ′ = X softmax(β Xᵀ ξ), is the attention of the Transformer with β = 1/√dₖ: the partial cue acts as the query and the stored patterns as keys and values (2021, sec. 2). The equivalence is formal and should not be overstated: in a transformer the "memories" are the other tokens in the context, and each layer takes a single retrieval step (see queries, keys and values).

Limits

Besides capacity, the network has spurious minima that nobody stored (Ramsauer et al., 2021, sec. 1), and memories that are too similar get confused and tend to merge (Hopfield, 1982, p. 2557). As a practical memory it has been superseded; as a model of how a network can remember without addresses, it is still current.

Sources

  • Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), 147–169. https://doi.org/10.1207/s15516709cog0901_7 §
  • Bartlett, F. C. (1932). Remembering: A study in experimental and social psychology. Cambridge University Press. §
  • Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
  • Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. Wiley. §
  • Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. https://doi.org/10.1073/pnas.79.8.2554 §
  • Hopfield, J. J., & Tank, D. W. (1985). “Neural” computation of decisions in optimization problems. Biological Cybernetics, 52(3), 141–152. https://doi.org/10.1007/BF00339943 §
  • Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2021). Hopfield networks is all you need. In 9th International Conference on Learning Representations (ICLR 2021). https://openreview.net/forum?id=tL89RnzIiCd §
  • The Royal Swedish Academy of Sciences. (2024b, October 8). The Nobel Prize in Physics 2024 [Press release]. NobelPrize.org. https://www.nobelprize.org/prizes/physics/2024/press-release/ §
  • Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2017). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann. https://doi.org/10.1016/C2015-0-02071-8 §