Gato is a single 1.2-billion-parameter transformer decoder that DeepMind trained on 604 distinct tasks. The same weights play Atari, caption images, chat and stack blocks with a real robot arm, deciding from context whether what comes next is text, a torque or a button press (Reed et al., 2022, abstract). Its authors called it a generalist agent.
How it works
The idea is to treat everything as language. Texts are tokenized with a vocabulary of 32,000 subwords; images are cut into 16 × 16 patches as in the Vision Transformer (ViT); discrete values (buttons) become integers; and continuous ones (angles, torques) are compressed and discretized into 1024 bins (Reed et al., 2022, sec. 2.1). Each step of an episode is written as observation tokens, a separator and action tokens. The result is a flat sequence modeled by next-token prediction, like a language model, except that the loss is computed only on text and actions (2022, sec. 2.3).
The network has 24 layers and 2048-dimensional vectors, with a 1024-token window; it was trained for four days on a 16 × 16 TPU v3 slice (2022, sec. 2.3). There is no reinforcement learning: Gato imitates, in a purely supervised way, trajectories of expert agents filtered to keep those reaching at least 80% of the expert's return (2022, secs. 1, 3.1). To know which task it faces, it receives a demonstration of that task at the start of the prompt, instead of an identifier (2022, sec. 2.3). In use, each generated action is sent to the environment and the new observation is fed back in (2022, sec. 2.4).
What it can do
It exceeds half of the expert's score on more than 450 of its 604 tasks; it matches or beats the average human level on 23 Atari games; and on the real robot it stacks blocks of unseen shapes about as well as a specialized system (2022, secs. 4.1–4.2). Its dialogue and captions, by contrast, are "rudimentary," and the authors acknowledge that it often answers superficially or incorrectly (2022, sec. 4.3, fig. 7).
Generalist or collector?
The announcement set off a discussion about what being general means. Nando de Freitas, one of the authors, tweeted in May 2022: "It's all about scale now! The Game is Over!" (de Freitas, 2022). From the other side one could answer with Chollet (2019): piling up more tasks known in advance measures skills, not flexibility, because the system can practice each of them (2019, sec. I.3.3). The paper itself shows the limit: for genuinely new tasks it needed fine-tuning, because the window was too short to learn from demonstrations in the prompt (Reed et al., 2022, secs. 5.2, 8.2). Millière & Buckner (2024b) place it as the third kind of "agent" system: multimodal models trained end to end to output actions (2024b, sec. 3.1.2).
The p(doom) song gives it a line ("Gato, please don't let me go"), and its annotator remarks of DeepMind: "They were very early" (osmarks, n.d.).
Limits
It performs below the specialists it imitates; its size was capped by the need to control a robot in real time; and there is no web-scale control corpus comparable to text (Reed et al., 2022, secs. 1, 4.1, 8.1). It was never released.