ElemedexA pocket encyclopedia of AI architectures

Glossary (TM) › Minimal mathematics

TM039

Dot product

Also: scalar product, inner product

In Spanish: Producto punto (producto escalar)

An operation that takes two vectors of the same length, multiplies them component by component and adds up the results, returning a single number.

Given two vectors of the same length, the dot product multiplies the components in matching positions and adds everything up: a·b = a₁b₁ + a₂b₂ + … + aₙbₙ. The result is a number, not a vector, and the operation is only defined when both vectors have the same number of components (Chollet, 2021, sec. 2.3.3).

Tiny example. (1, 2, 3)·(4, 0, −1) = 1·4 + 2·0 + 3·(−1) = 4 + 0 − 3 = 1.

Two intuitions. The first is almost logical: if the vectors are zeros and ones marking features, the dot product counts the shared features. With the features (animal, domestic, four-legged, flies), dog = (1, 1, 1, 0), cat = (1, 1, 1, 0) and pigeon = (1, 0, 0, 1); then dog·cat = 3 and dog·pigeon = 1. The second is geometric: the dot product is large when the vectors point in similar directions, zero when they are perpendicular, and negative when they point in opposite directions. Divided by both lengths, it yields cosine similarity (Alammar & Grootendorst, 2024, p. 125).

It is probably the most frequently executed operation in contemporary AI. An artificial neuron computes a dot product between its inputs and its weights and adds a bias; a dense layer computes many at once, W·x + b (Chollet, 2021, sec. 2.3.5). In transformer attention, the relevance of each previous token is obtained by multiplying the current position's query vector by the keys, and those scores are then normalized with softmax (Alammar & Grootendorst, 2024, p. 94).

A caveat for logicians: the dot product is a graded measure of overlap, not a consequence relation. It says how much two feature profiles resemble each other; it says nothing about whether one entails the other, and it is not asymmetric like entailment: a·b = b·a always.

Sources

  • Alammar, J., & Grootendorst, M. (2024). Hands-on large language models: Language understanding and generation. O’Reilly Media. §
  • Chollet, F. (2021). Deep learning with Python (2nd ed.). Manning Publications. §