A matrix is a table of numbers with rows and columns: a 2 × 3 matrix has two rows and three columns. In deep learning parlance, a tensor is simply an array with any number of axes: a single number is a rank-0 tensor, a vector is rank 1, a matrix rank 2, a “cube” of numbers rank 3. Every tensor has a rank (how many axes), a shape (how many elements lie along each axis) and a data type (Chollet, 2021, sec. 2.2). Readers coming from physics or geometry will notice that this usage is thinner than theirs: here “tensor” requires no transformation law, only a grid.
Tiny example. Multiplying a matrix by a vector means taking one dot product per row: [[1, 2], [0, 1]] · (3, 1) = (1·3 + 2·1, 0·3 + 1·1) = (5, 1). Geometrically, applying a matrix transforms space linearly: it rotates, stretches or shears it. The matrix [[0, −1], [1, 0]] turns the vector (1, 0) ninety degrees, to (0, 1) (Chollet, 2021, sec. 2.3.5).
This is nearly all a neural network does: multiply by matrices of parameters
and apply nonlinearities. Two concrete examples. Phi-3's embedding table is a
matrix with 32,064 rows, one per token in the vocabulary, and 3,072 columns; “looking up”
a token's embedding means reading its row (Alammar & Grootendorst, 2024, p. 79). And when a small BERT-like
model processes “Hello world”, it returns a tensor of shape [1, 4, 384]: one sentence, four
tokens (counting the special [CLS] and [SEP]), 384 numbers per token
(Alammar & Grootendorst, 2024, p. 60).
Because matrix multiplication consists of a great many mutually independent products, it parallelizes very well; this is why GPUs, designed for graphics, became the engine of deep learning.