Three functions mark the history:
- Step: 1 if the sum exceeds 0, otherwise 0. It is the function of the Perceptron and of McCulloch and Pitts: it turns numbers into truth values.
- Sigmoid: 1/(1 + e⁻ˣ), a smooth version of the step, with values between 0 and 1. It took over because the step is not differentiable and gradient descent needs derivatives (Witten et al., 2017, p. 264).
- ReLU: max(0, x). It lets positive values through and zeroes out negative ones. Today it is the default: its derivative is 0 or 1, which helps against the vanishing gradient problem (Witten et al., 2017, pp. 424–425).
Why is it needed? Because a composition of linear functions is linear. If each layer merely multiplied by a matrix and added a vector, stacking ten layers would be the same as having one: a "deep" network without activations would be a linear model in disguise (Chollet, 2021, sec. 2.3.5). Nonlinearity is what enlarges the hypothesis space and gives depth its point (Chollet, 2021, sec. 4.1.3). The classic example is XOR: no straight line separates its positive cases from its negative ones, so no linear classifier can compute it; with a nonlinear hidden layer, it can be done (Witten et al., 2017, p. 262).
At the network's output, activations are chosen by task: sigmoid for a binary probability, softmax for a distribution over many classes, none for regression.
A note for logicians: the step function makes each unit a small truth function, and in that sense the first networks were hard-wired logic. By smoothing the step in order to differentiate, networks traded bivalence for degrees: what they gained in learnability they lost in immediate legibility. You can compare both in the perceptron lab.