A neural network is a composition of simple functions. Each unit computes a weighted sum of its inputs and applies an activation function to it; units are organized in layers, and the output of one layer is the input of the next. Everything the network "knows" is in the weights of its connections, which are adjusted by backpropagation and gradient descent. A network without cycles is feedforward; if part of the output is fed back in at the next step, it is recurrent.
The Pokédex lineage runs from the McCulloch–Pitts neuron (McCulloch & Pitts, 1943) and the Perceptron (Rosenblatt, 1958) to the multilayer perceptron, and from there to nearly all modern species. With one hidden layer, a network can represent any expression of propositional logic: Witten et al. (2017) show how three units (an OR, a NAND and an AND) compute the XOR that a single perceptron cannot (2017, pp. 262–263). Turing had already speculated about "unorganized machines," randomly connected networks of neurons that might be trained (Buckner, 2024, p. 50).
The name is more metaphor than hypothesis. Witten et al. (2017) speak ironically of a graph "imaginatively termed" a network of neurons (2017, p. 133), and Chollet (2021) insists that deep learning models are not models of the brain (2021, sec. 1.1.4). In the 1980s people spoke of connectionist networks, because knowledge resides in the connections (M. Mitchell, 2019, ch. 2); under that name they entered the philosophy of mind, as an alternative to the thesis that thinking is symbol manipulation (see connectionism and GOFAI). Whether a network can exhibit systematicity without implementing a symbol system is the question raised by Fodor & Pylyshyn (1988), and it remains open.