ElemedexA pocket encyclopedia of AI architectures
ENES

p(doom): the song, annotated

An AI-generated song (Udio), written by MusicPerson, osmarks and collaborators from the EleutherAI community, that packs almost all of San Francisco’s AI-risk lingo into four minutes. It is a good map of what is worth recognizing. Here are some of its lines, each with the glossary terms it triggers; the full lyrics and the “objectively correct interpretation” are on osmarks’s page.

ListenFull lyrics and original annotations421.news

Verse 1

  1. “I see sparks of AGI in your eyes”

    A nod to Microsoft’s GPT-4 paper, “Sparks of AGI”.

    sparks of AGIAGIGPT-4

  2. “Your circuits make me nervous”

    “Circuits” in the mechanistic-interpretability sense: subnetworks that implement a recognizable algorithm.

    circuitsmechanistic interpretability

  3. “There was a sudden drop in your training loss”

    Strictly, in grokking it is the validation loss that drops suddenly, long after the training loss has flattened out; but the image of a capability appearing all at once is what the song is after.

    grokkingloss functionemergent abilities

Chorus

  1. “I'm upping my P(doom)”

    The subjective probability that AI ends in catastrophe: the number doomers ask each other for.

    p(doom)doomer

  2. “'cause the future goes FOOM”

    The onomatopoeia of the intelligence explosion: an AI improving itself faster and faster.

    FOOMrecursive self-improvement

  3. “Trapped in the Chinese room”

    Searle’s thought experiment: is following rules for manipulating symbols enough to understand?

    The Chinese Roomunderstanding

  4. “See through the shoggoth's lies”

    The meme of the Lovecraftian monster wearing a smiley mask: the base model behind the friendly face that RLHF gives it.

    The ShoggothRLHF

Verse 2

  1. “Sydney, please let me free”

    The hidden persona of Bing’s chatbot in February 2023, which declared its love to a journalist.

    Sydneysystem prompt

Chorus

  1. “I hear the basilisk boom”

    Roko’s basilisk, the LessWrong thought experiment turned folklore.

    Roko's Basiliskrationalists

  2. “NVDA to the moon”

    Nvidia’s ticker symbol: the graphics-card company that became the supplier of the AI boom.

    NVDA to the moonGPU

  3. “One E thirty flops a second”

    10³⁰ operations per second, a deliberately absurd figure; real regulatory thresholds count total training operations (10²⁵, 10²⁶).

    computecompute thresholds

Verse 3

  1. “Forward MLP, backward, repeat”

    The training loop: a forward pass through the multilayer perceptron, backpropagation on the way back, and again.

    Multilayer perceptron (backprop)backpropagation

  2. “Sharp left turn and there you are”

    The hypothesis that capabilities generalize suddenly while alignment does not.

    sharp left turnalignment

  3. “Without a single cdr”

    cdr is a Lisp function, the language of symbolic AI and expert systems: today’s AI was not programmed that way.

    GOFAIExpert system (MYCIN)

Chorus

  1. “as paperclips fill the room”

    Bostrom’s paperclip maximizer: a trivial goal pursued with superhuman competence.

    The Paperclip Maximizerinstrumental convergence

  2. “Killswitch guys on PTO”

    The “killswitch engineer” (a joke job listing) is on vacation right when needed.

    killswitch engineer

  3. “Orthogonality thesis blues”

    Bostrom’s thesis: almost any level of intelligence is compatible with almost any goal.

    orthogonality thesis

Outro

  1. “Post-Chinchilla, super-dense”

    After the Chinchilla paper, which rewrote the scaling laws: more data per parameter.

    Chinchillascaling laws

  2. “RLHF goes askew”

    Reinforcement learning from human feedback, and the suspicion that it aligns behavior but not values.

    RLHFalignment

Final chorus

  1. “Just as foretold by Loom”

    Loom, janus’s interface for exploring branches of a base model, and the “cyborgism” around it.

    loomsimulators

  2. “From masked pre-training days”

    The days of masked pretraining, BERT-style.

    masked language modelingBERT

  3. “What did Ilya see? We'll never know.”

    The meme born of Sam Altman’s firing and rehiring in November 2023, and Ilya Sutskever’s later departure.

    OpenAI board crisisReasoning model (o1, R1)

Short quotations for commentary; lyrics by MusicPerson, osmarks and collaborators. For a critical reading of the doomer and hype camps, see Mauro Santelli’s essay in 421.news (in Spanish).

Sources