AlphaStar is DeepMind's system for playing StarCraft II, Blizzard's real-time strategy game. In December 2018, a preliminary version beat Dario "TLO" Wünsch 5–0 and Grzegorz "MaNa" Komincz 5–0; the games were made public in January 2019 (AlphaStar team, 2019a; Vinyals et al. (2019), Methods: StarCraft Demonstration Evaluation). The final version, with a camera and a stricter action cap, reached Grandmaster level on Battle.net with all three races, above 99.8% of ranked players (Vinyals et al., 2019, Empirical Evaluation).
The problem
Chess and Go are turn-based games of perfect information. StarCraft runs in real time under a fog of war: you have to scout to find out what your opponent is doing. At each step there are some 10²⁶ possible actions (Vinyals et al., 2019, introduction), and a player must sustain macro (economy, production) and micro (fine control of each unit) at once, with no strategy better than all others: as in rock-paper-scissors, each one has its answer (AlphaStar team, 2019a). In 2017, DeepMind's first agents could not win a single game against the game's easiest built-in AI; the best one learned to lift off its Terran buildings and float them away so as not to lose before the 30-minute limit (Vinyals et al., 2017, sec. 4.4.1).
How it works
A Transformer reads the list of visible units with self-attention; a convolutional network reads the minimap. A deep LSTM keeps the memory of the game, and the action is assembled autoregressively: what to do, when to look again, whether to queue it, with which units (a pointer network picks them out of the list), and where (Vinyals et al., 2019, Learning Algorithm, Extended Data Fig. 3). It has 139 million parameters, of which 55 million are used at play time (-@vinyals2019, Methods: Architecture).
First comes imitation learning: predicting the actions in 971,000 replays by players with an MMR above 3,500, the top 22% (2019, Methods: Supervised Learning). That agent already beat 84% of human players (2019, Empirical Evaluation). Then comes reinforcement learning, with +1 for a win and −1 for a loss (-@vinyals2019, Methods: Reinforcement Learning), in a league with four agents per race: a main agent, which faces everyone and gives priority to the opponents it struggles with most; a main exploiter, which only looks for holes in the main agents; and two league exploiters, which look for weaknesses of the league as a whole. Each trained for 44 days on 32 TPUs; almost 900 players were created (2019, Learning Algorithm).
What changed
Compared with AlphaGo, tree search is gone: AlphaStar acts without simulating future moves (Vinyals et al., 2019, Methods: Related Work). Plain self-play becomes a league because it can run in circles (A beats B, B beats C, C beats A) without progress (-@vinyals2019, Learning Algorithm); the league held some three million such cycles involving at least one exploiter, and only about 200 among main agents (2019, Extended Data Fig. 8). And imitation does not stay at the start: a KL penalty ties it to the imitation policy throughout reinforcement learning. In ablations on a single map, with no human data the agent reached an Elo of 149, below the game's easiest bot (418); with all of it, 1540 (-@vinyals2019, Methods: Exploration and diversity, figs. 3E, 3J).
The games
The December games were Protoss versus Protoss on Catalyst LE, and TLO, a professional Zerg, had to play his off-race (AlphaStar team, 2019a). According to Ars Technica's report, in the fourth game against MaNa an all-Stalker army attacked his Immortals, Archons and Zealots from three sides, blinking damaged Stalkers to the rear while their shields recharged. In the live game, MaNa kept dropping two Immortals from a Warp Prism among AlphaStar's Probes and picking them up again before the defending Stalkers arrived, leaving them marching back and forth indecisively (Lee, 2019).
Limits
In December, AlphaStar saw everything visible on the map without moving the camera (AlphaStar team, 2019a; Canaan et al., 2019, sec. 2.4) and averaged some 280 actions per minute (APM), fewer than MaNa (390), though, as DeepMind clarified, with effective bursts sometimes higher than both players'. Live, MaNa beat a camera-based prototype trained, by DeepMind's account, for just seven days (AlphaStar team, 2019a). The Nature version moved a camera and made at most 22 actions every 5 seconds, but still targeted off-screen more accurately than a human (Vinyals et al., 2019, Methods: Camera View, APM Limits). The Battle.net evaluation does not measure how exploitable it is under repeated play (2019, Empirical Evaluation).