The architecture is the blueprint; the trained weights, the inhabited building. It describes which kinds of layers there are, how many, in what order and with what connections: for instance, "a stack of 12 blocks of self-attention and feedforward (MLP) block networks with residual connections." Chollet (2021) gives it a precise sense: choosing an architecture restricts the hypothesis space to a specific series of operations, and training searches that space for good values of the weights (2021, sec. 3.6.2). That is why the architecture encodes the designer's assumptions about the problem (see inductive bias).
There are three levels that everyday language mixes up and that Elemedex keeps apart in each species' kind:
- Architecture: Transformer, LSTM, the convolutional network. A type, not an individual.
- Model: an architecture with trained weights, such as GPT-3 or BERT. When a model is released with open weights, both the weights and the architecture are shared (Alammar & Grootendorst, 2024, p. 30).
- System: a model inside a product with other parts (system prompt, tools, filters, search), such as ChatGPT (InstructGPT) or an agent.
The same architecture can yield very different models depending on data and scale: the authors of GPT-3 state that they used the same model and architecture as GPT-2, except for an attention pattern (Brown et al., 2020, sec. 2.1). And for some models only the behavior is known: the GPT-4 technical report discloses neither its architecture nor its size (OpenAI, 2023).
The philosopher will meet the word in two earlier traditions. In computing, the "von Neumann architecture" names the organization of a computer. In cognitive science, the cognitive architecture is the fixed structure of the mind on which processes run; the argument of Fodor & Pylyshyn (1988) is precisely that connectionism cannot offer a theory of cognitive architecture without implementing a symbol system. With transformers, the question is still alive (see systematicity).