A layer takes a vector (or a tensor) and returns another. The simplest is the dense or fully connected layer: each output is a unit connected to all inputs, and the whole computation is activation(W·x + b), where W is a weight matrix. Other layers impose structure: convolutional, recurrent, attention layers. Chollet (2021) calls them the LEGO bricks of deep learning (2021, sec. 3.6.1) and describes a network as a series of filters that progressively distill the information useful for the task (2021, sec. 1.1.4).
One distinguishes the input layer, the output layer, and the hidden layers, so called because they have no direct connection to the environment (Witten et al., 2017, p. 263); M. Mitchell (2019) suggests that "interior" would have been a better name (2019, ch. 2).
Counting layers is not trivial, and it is worth knowing before reading the "height" on a data sheet (see base stats). Some count every layer with weights; M. Mitchell (2019) defines depth as the number of hidden layers (2019, ch. 2). In transformers, blocks are counted: each "layer" of GPT-3 is a transformer block with attention, a feedforward (MLP) block network and normalizations, and GPT-3 has 96 (Brown et al., 2020, sec. 2.1). The data sheets usually take the figure given in the original paper.
Depth matters because layers compose: what one computes the next can reuse, which is why a deep network can be orders of magnitude more efficient than a shallow one at computing the same thing (Buckner, 2024, p. 51). The hierarchy invites talk of levels of abstraction (edges, shapes, objects), but what each level represents has to be checked case by case, for instance with probing techniques.