A linear classifier only draws straight boundaries (hyperplanes). But if the data are first transformed by a nonlinear function Φ, a straight boundary in the transformed space can be curved in the original (Witten et al., 2017, p. 252). The problem is cost: with 10 attributes and all products of 5 factors, there are more than 2000 coefficients to learn (Witten et al., 2017, pp. 252–253).
The trick. Algorithms like the Support vector machine use the data only through dot products between pairs of examples. So it suffices to have a function K(x, y) that directly gives the dot product in the transformed space:
K(x, y) = Φ(x)·Φ(y)
without ever computing Φ (Witten et al., 2017, pp. 255–256). Example with two attributes: the
polynomial kernel (x·y)² equals Φ(x)·Φ(y) with Φ(x) = (x₁², √2·x₁x₂, x₂²). Computing
(x·y)² is cheap; the space of Φ can be huge, even infinite for the Gaussian (RBF) kernel.
With the sigmoid kernel, an SVM amounts to a perceptron with one hidden layer
(Witten et al., 2017, p. 256).
Why it mattered. Combined with the maximum-margin hyperplane of support vector machines (Cortes & Vapnik, 1995), the trick gave nonlinear boundaries with theoretical guarantees and little overfitting (Witten et al., 2017, p. 255). SVMs were for years among the most popular methods, until the rise of deep learning (Chollet, 2021, sec. 1.2.3).
The contrast with networks. Chollet (2021) stresses that the kernel is usually crafted by hand: in an SVM only the hyperplane is learned, not the representation (2021, sec. 1.2.3). A kernel is, at bottom, a similarity measure chosen by the designer, and it can be defined over strings, trees or sets (Witten et al., 2017, p. 273); string kernels were used to classify texts (Lodhi et al., 2002). Deep networks, instead, learn their own representation. From the standpoint of the empiricism and nativism debate, the kernel is an innate bias about what resembles what. Curiously, transformer attention can be read as a learned kernel: a query–key similarity used to smooth over the values (Tsai et al., 2019).