Picture the loss as a landscape whose height depends on the values of the parameters. The gradient is the vector of derivatives pointing in the direction of steepest ascent; gradient descent takes a step the opposite way and repeats. Each step subtracts from every parameter its derivative multiplied by a small constant, the learning rate.
Witten et al. (2017) give a toy example: if the loss were w² + 1, its derivative would be 2w; with w = 4 and a learning rate of 0.1, the first step leads to 4 − 0.1 × 8 = 3.2, then to 2.56, 2.048, and so on towards the minimum at 0 (2017, pp. 265–266). Too high a rate makes the search oscillate or overshoot; too low a rate makes progress painfully slow. And the method only guarantees a local minimum.
In practice the gradient is not computed on all the data at once but on a small batch drawn at random: this is mini-batch stochastic gradient descent (SGD), a compromise between accuracy and cost (Chollet, 2021, sec. 2.4.3). Variants such as momentum (a ball that rolls and gathers speed so as not to get stuck in a hollow), Adagrad or RMSProp adjust each step according to the history of past updates. What computes the gradient in a deep network is backpropagation; gradient descent is what uses it.
Chollet (2021) adds a caveat: our intuitions come from pictures in one or two dimensions, but a real network descends through a space of millions of dimensions, where those pictures can mislead (2021, sec. 2.4.3) (see high-dimensional space). Epistemologically, the method does not deduce the solution: it finds it by repeated local correction, never knowing whether the valley it ends up in is the deepest one.