Training a network does not mean looking at all the data at once and then correcting. Mini-batch stochastic gradient descent takes a small group of examples, computes how wrong the model is on average over them, nudges the weights slightly in the direction that reduces that error, and moves on to the next group (Chollet, 2021, sec. 2.4.3). That group is the batch. Each full pass through the training set is an epoch; the data are usually shuffled between epochs (Witten et al., 2017, p. 433).
Chollet (2021)'s example does the arithmetic: 60,000 images of handwritten digits (MNIST), batches of 128, and 5 epochs. That is 469 updates per epoch and 2,345 in total (2021, sec. 2.5). At the extremes, a batch the size of the whole dataset is classical batch gradient descent, and a batch of one example is pure stochastic gradient descent; mini-batches usually hold between two and several hundred examples (Witten et al., 2017, p. 433).
Language models stretch both notions. Their batches are measured in tokens: for the largest models of Kaplan et al. (2020), the optimal batch was roughly one to two million tokens (2020, sec. 1.1). And epochs nearly disappear: the corpus is so large that many parts are seen once or less. In GPT-3, Common Crawl was sampled less than once over the whole training run, while datasets judged to be of higher quality were seen 2 or 3 times (Brown et al., 2020, sec. 2.2).
A few epochs, fine; too many, overfitting: if the validation loss starts rising while the training loss keeps falling, the model is memorizing and it is time to stop (early stopping) (Chollet, 2021, sec. 4.1.4). The analogy with human study is tempting (rereading the same book five times), but misleading: the network does not "remember" having seen an example; it has only become slightly different after each batch.