Chollet (2021) defines machine learning as the search for useful representations within a predefined space of possibilities, guided by a feedback signal: algorithms do not invent transformations, they search through a set fixed in advance (2021, sec. 1.1.3).
Learning is search. Witten et al. (2017) show this with a toy problem, deciding whether to play given the weather: there are 288 possible rules, and rule sets of up to 14 rules number about 2.7 × 10³⁴. Each positive example rules out the descriptions that fail to cover it; each negative one, those that do (2017, p. 32). What survives is Mitchell's (1982) version space. In practice nothing is enumerated: the search is heuristic, as when gradient descent moves through the weights.
What fixes it. In a network, the architecture: choosing a topology restricts the hypotheses to a certain series of operations, and training is the search for values of the weights (Chollet, 2021, sec. 3.6.2). A layer without an activation function can only represent linear transformations, and stacking linear layers does not enlarge the space (2021, sec. 4.1.3). With a nonlinearity, a single hidden layer with enough units can already approximate any continuous function (Hornik et al., 1989).
The dilemma. A poor space may not contain the target concept; a universal one, which allows arbitrary disjunctions, contains the "hypothesis" that merely lists the examples, and that is not generalizing (Witten et al., 2017, p. 34). Hence the need for an inductive bias, which restricts the space or sets preferences within it.
For linguists. Gold (1967) proved that no class of languages containing all the finite languages and at least one infinite one can be identified in the limit from positive examples alone. Many read this as an argument for a space of grammars restricted in advance (see poverty of the stimulus); what the theorem implies for actual acquisition is still debated.