ElemedexA pocket encyclopedia of AI architectures

Glossary (TM) › Machine learning fundamentals

TM005

Backpropagation

Also: backprop, backward propagation of errors

In Spanish: Retropropagación

The algorithm that computes, by applying the chain rule backwards, how much each weight in a network contributes to the final error.

Gradient descent needs the derivative of the loss with respect to each weight. In a single-layer network that is easy; with hidden layers a problem arises: we know what the output should have produced, but not what the intermediate units should have produced (Witten et al., 2017, p. 264). Backpropagation solves this with the chain rule: since the network is a composition of simple operations with known derivatives, the error is computed at the output and "propagated" layer by layer back towards the input (Chollet, 2021, sec. 2.4.4). M. Mitchell (2019) describes it as distributing the blame for the error among all the weights of the network (2019, ch. 2). Today software libraries do it automatically over computation graphs (automatic differentiation).

Minsky and Papert doubted that an effective method for training multilayer networks could be found, and their skepticism helped dry up funding in the 1970s (M. Mitchell, 2019, ch. 2). The algorithm was discovered several times, perhaps first by Werbos in 1974 (Buckner, 2024, p. 120), and it was popularized by Rumelhart et al. (1986), who showed that hidden layers learned useful representations. Without it there would be no Multilayer perceptron (backprop) and no deep learning. In very deep or recurrent networks, the product of many small derivatives can wipe out the signal: this is the vanishing gradient problem.

Two clarifications. Backpropagation and gradient descent are not synonyms: one computes the gradient, the other uses it to update, although introductory texts often run them together (Chollet (2021) has the optimizer implement backpropagation (2021, sec. 1.1.5)). Nor is it a model of how the brain learns: neurons do not seem able to send error signals backwards across a synapse, and the algorithm's biological plausibility is still debated (Buckner, 2024, p. 132). Its philosophical interest lies elsewhere: it is a technical solution to the old credit-assignment problem, that of apportioning the merit or blame for an outcome among many intermediate causes.

Sources

  • Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
  • Chollet, F. (2021). Deep learning with Python (2nd ed.). Manning Publications. §
  • Mitchell, M. (2019). Artificial intelligence: A guide for thinking humans. Pelican Books. §
  • Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0 §
  • Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2017). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann. https://doi.org/10.1016/C2015-0-02071-8 §