The phrase comes from engineering and cybernetics: a device studied only through what goes in and what comes out. In machine learning it is used in two senses worth keeping apart.
The first is epistemic. Even with all of a network's weights in hand, it is hard to decide which ones mattered for a given decision (Buckner, 2024, p. 61). Buckner (2024) illustrates this by comparing Deep Blue with AlphaGo (2024, pp. 62–63): when Deep Blue made a famous mistake against Kasparov in 1997, the programmers traced it to a coding bug; when AlphaGo played its bizarre move 37 against Lee Sedol, nobody could say whether it was a blunder or a stroke of genius until the game moved on. Nor does a complete mathematical description of the network as a composite function help: redescribing what a model does as matrix multiplication does not explain its behavior (Millière & Buckner, 2024b, sec. 2.2).
The second is about access: the weights, data and training details of many commercial models are not released, so they can be studied only through their behavior. Millière and Buckner warn that this turns the most striking results into irreproducible anecdotes (Millière & Buckner, 2024b, sec. 3.3). In this Pokédex, the transparency stat drops for both reasons (see open weights).
There are two responses. One is to open the box with interpretability techniques. The other, defended by Rudin (2019), is not to use black boxes where the stakes are high: for judicial or medical decisions, prefer models that are interpretable from the start over approximate explanations of opaque ones.
The metaphor reopens an old debate. Behaviorism treated the mind as a black box, and Block (1981)'s Blockhead shows why behavior alone might not suffice to ascribe intelligence: how the behavior is produced matters. But the brain is opaque too, and people cannot always explain how they reached a decision (M. Mitchell, 2019, ch. 6). Are we demanding of machines a transparency we do not demand of ourselves?