ElemedexA pocket encyclopedia of AI architectures
ENES

Glossary (TM) › Architectural parts

TM077

Convolution

Also: convolutional filter, convolutional layer

In Spanish: Convolución

An operation that slides a small filter of learned weights across the whole input and records where a local pattern appears, reusing the same weights everywhere.

A convolution sweeps the input (an image, an audio signal) with a small filter: at each position, it multiplies the values in a neighborhood by the filter's weights and adds them up. In one dimension:

y[n] = Σₖ w[k] · x[n + k], with k from −K to K

(Witten et al., 2017, p. 440). The result is a "feature map" showing where the pattern (a vertical edge, a curve) was detected. Because the same filter is applied everywhere, weights are shared: a convolutional network has far fewer parameters than a fully connected one, and each neuron responds only to its "receptive field" (Witten et al., 2017, pp. 438–440; Chollet, 2021, sec. 8.1.1). (Technically, what is computed is a cross-correlation, but the name stuck.)

Genealogy. Hubel and Wiesel found "simple" cells in the cat's visual cortex, tuned to an edge at one position, and "complex" cells, tuned to the same edge across nearby positions (Hubel & Wiesel, 1962). Fukushima carried that division over to his Neocognitron: nodes that convolve and nodes that aggregate (Fukushima, 1980; Buckner, 2024, pp. 122–124). Then came Convolutional network (LeNet), which read checks, and AlexNet, which in 2012 launched the deep learning boom. AlphaGo's policy and value networks (Silver et al., 2016) and the U-Net of Diffusion model models (Ho et al., 2020, sec. 4) are also convolutional. Vision Transformer (ViT), by contrast, does almost entirely without them.

A built-in bias. Convolution, pooling and local connectivity implement two assumptions: that dependencies among features tend to be local, and that a feature means the same wherever it appears (translation invariance) (Buckner, 2024, p. 130). Gary Marcus flagged this as smuggled-in innate knowledge; Buckner (2024) replies that it is a domain-general inductive bias, applicable to any data with local structure, and hence compatible with a moderate empiricism (2024, pp. 20–21) (see empiricism and nativism).

Abstraction. For Buckner (2024), convolutional filters perform "abstraction-as-composition" (building complex features from simple parts) and pooling "abstraction-as-subtraction"; stacked, they form ever more abstract "sandwiches" (2024, p. 127). See abstraction.

Sources

  • Buckner, C. J. (2024). From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. https://doi.org/10.1093/oso/9780197653302.001.0001 §
  • Chollet, F. (2021). Deep learning with Python (2nd ed.). Manning Publications. §
  • Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4), 193–202. https://doi.org/10.1007/BF00344251 §
  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (Vol. 33, pp. 6840–6851). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html §
  • Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1), 106–154. https://doi.org/10.1113/jphysiol.1962.sp006837 §
  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. https://doi.org/10.1109/5.726791 §
  • Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489. https://doi.org/10.1038/nature16961 §
  • Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2017). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann. https://doi.org/10.1016/C2015-0-02071-8 §