Parameters are adjusted by gradient descent; hyperparameters are chosen by a person, or by a search procedure, before training starts. Witten et al. (2017) give a simple example: the k in k-nearest neighbors, how many neighbors are consulted to classify (2017, p. 171). Chollet (2021) draws the line for networks: the number and size of the layers are hyperparameters; the weights are parameters (2021, sec. 5.2.1). Other typical ones are the learning rate, the batch size, the number of epochs, or the dropout rate. The table for GPT-3 lists them next to the architecture: 96 layers, batches of 3.2 million tokens, a learning rate of 0.6 × 10⁻⁴ (Brown et al., 2020, sec. 2.1).
They are chosen by trial: variants are trained and compared on the validation set, never on the test set, because peeking at the test data to make choices biases the final estimate (Witten et al., 2017, p. 171) (see training and test sets). M. Mitchell (2019) stresses what this says about the supposed autonomy of learning: it is inaccurate to say a network learns "on its own," because hyperparameters are everything humans must set for learning even to begin, and setting them well is a craft knowledge that Eric Horvitz likened to alchemy (2019, ch. 6).
Philosophically, hyperparameters are where the "innate" part of a system that presents itself as empiricist hides. When Gary Marcus objected that AlphaZero, the successor of AlphaGo, had the size of its convolutional layers tuned to Go before training, Buckner (2024) granted that this is more suspect than a general bias, while noting that it is unclear whether the size of a layer amounts to the kind of innate knowledge rationalists have championed (2024, p. 20) (see inductive bias and empiricism and nativism).