Sharing weights means forcing several connections in a network to always hold the same value. During training, the gradient for that single weight combines the gradients from all its uses; Rumelhart et al. (1986) already handled this by averaging those derivatives and moving all the copies together (1986, p. 535).
Where it appears.
- In convolution, the same filter sweeps the whole image: the operation amounts to multiplying by a matrix whose rows repeat the same coefficients, shifted (Witten et al., 2017, p. 440). That is why Convolutional network (LeNet) had about 340,000 connections but only about 60,000 trainable parameters (LeCun et al., 1998, secs. II.A–B).
- In recurrence, the same matrices are applied at every time step (Witten et al., 2017, p. 457); this is how Simple recurrent network (Elman) or LSTM read sequences of any length. Training them requires backpropagation through time.
- In the Transformer, the same weights process every position, and the input embedding matrix is reused before the output softmax (Vaswani et al., 2017, sec. 3.4).
- In siamese networks, two identical copies process two inputs in order to compare them: signatures in Bromley and colleagues (1993), faces in face verification (Witten et al., 2017, p. 524), sentences in Sentence-BERT (Reimers & Gurevych, 2019).
Why it works. Fewer parameters mean less data is needed, but the deeper reason is another: what the network learns in one place becomes available everywhere. Sharing weights is declaring a symmetry, stating in advance that two places in the input deserve the same treatment. It is an explicit inductive bias. LeCun and colleagues called translation invariance "a priori knowledge of the task," and Gary Marcus held it up as covert nativism; Buckner (2024) replies that it is a domain-general assumption, which saves resources where the world satisfies it and may get in the way where it does not (2024, pp. 20–21, 130) (see empiricism and nativism).