Attention is a way of consulting context: to represent one element, the network computes how relevant each of the others is and averages their contents with those weights, which change with every sentence.
Origin
It was born in machine translation. The encoder–decoder networks of 2014 squeezed the
whole source sentence into a single vector and did worse and worse as sentences grew longer.
Bahdanau et al. (2015) proposed that, when writing each word of the translation, the decoder should
consult all the encoder states: it scores every source word, normalizes the scores with a
softmax, and builds a tailor-made context, cᵢ = Σⱼ αᵢⱼ·hⱼ. The authors say this
"implements a mechanism of attention" (2015, sec. 3.1). One of their examples: to
translate the man as l'homme, the model looks at the and man together, because the
French article depends on the noun that follows (2015, sec. 5.2.1)
(Bahdanau attention).
How it works today
In the Transformer the idea is generalized with queries, keys and values and applied within a single sequence (self-attention). In "The dog chased the squirrel because it…", the representation of it must absorb information from squirrel (or from dog) before the model can predict what comes next. Attention does this in two steps: it scores the relevance of each token and combines their contents according to those scores (Alammar & Grootendorst, 2024, pp. 88–90). You can experiment with it in the attention lab.
Is it like human attention?
Partly. Nearly all psychological theories treat attention as selectivity: a filter at a capacity bottleneck (Broadbent), a glue that binds an object's features (Treisman), selection in the service of action, or the outcome of a competition among processes (Buckner, 2024, pp. 243–247). Machine attention shares the idea of selectively reweighting information, and the softmax forces candidates to share a "budget" that sums to 1. But there is no limited capacity (nothing goes unprocessed), no voluntary control, no link to consciousness. In his chapter on attention, Buckner (2024) calls its similarity to biological attention "questionable" and takes up Grace Lindsay's suggestion that its closest psychological analogue is priming (2024, pp. 273–275). The name is a fertile metaphor, not a thesis about the mind.