An autoencoder has two halves. The encoder maps the input x to a lower-dimensional code z; the decoder tries to rebuild x from z. The loss measures how far the reconstruction is from the original, for instance ‖x − d(e(x))‖². The model is thus trained to predict its own input, but forced to pass through the narrow representation of its hidden units (Witten et al., 2017, p. 445). No labels are needed: it is self-supervised learning in its purest form.
What it learns. Without nonlinearities, an autoencoder finds the same subspace as principal component analysis (PCA). With several nonlinear layers it can compress better than PCA with the same number of dimensions and arrange the data into more natural groupings, as Hinton and Salakhutdinov (2006) showed (Witten et al., 2017, pp. 446–447). Around that time it was used to pretrain deep networks layer by layer, because optimizing them directly proved difficult (2017, p. 448).
Variants. The denoising autoencoder receives a corrupted input (for instance, with some values set to zero) and must restore the original (Vincent et al., 2008). It is a direct ancestor of masked language modeling: BERT defines itself by contrast with it, since it predicts only the masked words rather than the whole input (Devlin et al., 2019, p. 4174). The VAE encodes distributions instead of points, and latent diffusion models work in the latent space of an autoencoder (Rombach et al., 2022). Chollet (2021) warns that the classical autoencoder, by contrast, does not yield particularly useful or well-structured latent spaces (2021, sec. 12.4).
A philosophical angle. Buckner (2024) uses the generative half of these networks to answer a challenge Fodor posed to Hume: how does an empiricist mind produce particular ideas from general representations? The decoder can reconstruct an exemplar it has seen, or average the codes of a category and render a "typical" member (2024, pp. 210–211). Whether reconstructing is the same as imagining is, of course, the open question.