No finite set of examples determines a unique generalization. Tom Mitchell (1980) called bias any basis for choosing one generalization over another other than strict consistency with the observed examples, and showed that without bias a system cannot make the inductive leap to new cases. Witten et al. (2017) distinguish three sources: the language in which concepts can be expressed, the order in which the hypothesis space is searched, and the way overfitting is avoided, for instance by preferring simplicity (2017, p. 33). They conclude that bias is the only thing that makes learning feasible and that there is no universally best method (2017, p. 35).
In neural networks, bias lives mostly in the architecture. Chollet (2021) puts it bluntly: to learn from data you have to make assumptions about it, and the architecture encodes those assumptions, the prior knowledge the model starts with (2021, sec. 3.6.2). A convolutional network assumes that a pattern means the same thing anywhere in the image; attention treats its input as an unordered set (Buckner, 2024, p. 23), which is why a positional encoding has to be added. Vision Transformer (ViT), with fewer built-in visual assumptions than a convolutional network, needs much more data (Dosovitskiy et al., 2021).
The philosopher will recognize Goodman's (1954/1983) new riddle of induction: the same observed emeralds confirm "all emeralds are green" and "all emeralds are grue," and only a prior preference for certain predicates decides. They will also recognize the debate between empiricists and nativists. Gary Marcus read the translation invariance of convolutional networks as a priori knowledge at odds with its authors' empiricism; Buckner (2024) replies that a general bias, applicable to any data with the right structure, is a far cry from the innate ideas of Descartes or Leibniz, and that the real question is how much specific knowledge can be extracted from experience with general biases (2024, pp. 20–23) (see empiricism and nativism).