"Feedforward" designates any network in which information flows in one direction, without loops: the 1986 multilayer perceptron is the classic case. Inside a transformer block, the term names the second sublayer: two linear transformations with an activation function in between, applied to each position separately and identically (Vaswani et al., 2017, sec. 3.3):
FFN(x) = max(0, x·W₁ + b₁)·W₂ + b₂
The vector is expanded (from 512 to 2048 dimensions in the original) and compressed again. In GPT-3 the inner layer is always four times as wide as the model (Brown et al., 2020, sec. 2.1); recent models swap the ReLU for SwiGLU (Alammar & Grootendorst, 2024, p. 101; Touvron, Lavril, et al., 2023, sec. 2.2).
Division of labor. Attention mixes information between tokens; the feedforward network does not look at neighbors, it transforms what each token already has. With the usual proportions, its two matrices make up about two thirds of each block's parameters (our own arithmetic: 8d² against attention's 4d²). Alammar & Grootendorst (2024) describe it as housing much of the processing capacity and, probably, of the memorization: that "The Shawshank" continues with "Redemption" would depend on information stored there (2024, pp. 86–87). Geva et al. (2021) proposed reading these layers as key–value memories: the first matrix detects textual patterns and the second pushes toward certain output words.
Variants. In a mixture of experts, this block is replaced by many parallel blocks and a router that picks a few per token (Mixture of experts (Mixtral)).
A caveat. That "facts" live in these layers is a useful hypothesis, not a map: the same machinery that memorizes also interpolates between examples and generalizes (Alammar & Grootendorst, 2024, p. 87), and information tends to be distributed across many layers and superposed in the same neurons.