In academic philosophy, decision theory studies how to choose under uncertainty. On LessWrong, "decision theory" names something more specific: the program of finding the right rule for agents that can be simulated, predicted or copied, as an AI could be. It is part of MIRI's research agenda and a favorite topic of the rationalists.
The test case is Newcomb's problem, analyzed by Robert Nozick in 1969. A nearly infallible predictor has put a million dollars in an opaque box only if it predicted you would take that box alone; the other, transparent box holds a thousand. Do you take one box or both? Causal decision theory says both: your choice can no longer cause the contents. Evidential decision theory says one: choosing one is strong evidence that the million is there. Philosophers remain split: in the 2020 PhilPapers survey, 39% chose two boxes and 31% one. Wei Dai proposed updateless decision theory (UDT, 2009), Yudkowsky timeless decision theory (TDT, 2010), and Yudkowsky & Soares (2017) functional decision theory (FDT): decide as if choosing the output of the mathematical function that computes your decision, which the predictor also computes. FDT recommends one box.
From there comes acausal trade: agents that simulate each other could "cooperate" without communicating or interacting causally. Its sinister version is the acausal blackmail of Roko's Basilisk (LessWrong, 2022). For an AI the question is not exotic: it can be copied, and others can run its code to anticipate what it will do.
The academic reception was cool. Wolfgang Schwarz, who refereed the FDT paper for a philosophy journal, published his report in 2018: the theory is hard to make precise and, in clear cases, recommends choices that look irrational. Levinstein & Soares (2020) published a defense in the Journal of Philosophy. Is the rationality of a copyable agent the same as that of one that exists only once?