Which decision theory should we use? CDT? UDT? TDT? What exactly do we mean by a "better" decision theory?

To get some practice in answering this kind of question, lets look first at a simpler set of questions: Which play should I make in the game PSS? Paper? Stone? Scissors? What exactly do we mean by a better play in this game?

Bear with me on this. I think that a careful look at the process that game theorists went through in dealing with game-level questions may be very helpful in our current confusion about decision-theory-level questions.

The first obvious thing to notice about the PSS problem is that there is no universal "best" play in the game. Sometimes one play ("stone", say) works best; sometimes another play works better. It depends on what the other player does. So we make our first conceptual breakthrough. We realize we have been working on the wrong problem. It is not "which play produces the best results?". It is rather "which play produces the best expected results?" that we want to ask.

Well, we are still a bit puzzled by that new word "expected", so we hire consultants. One consultant, a Bayesian/MAXENT theorist tells us that the appropriate expectation is that the other player will play each of "paper", "stone", and "scissors" equally often. And hence that all plays on our part are equally good. The second consultant, a scientist, actually goes out and observes the other player. He comes back with the report that out of 100 PSS games, the other player will play "paper" 35 times, "stone" 32 times, and "scissors" 33 times. So the scientist recommends the play "scissors" as the best play. Our MAXENT consultant has no objection. "That choice is no worse than any other", says he.

So we adopt the strategy of always playing scissors, which works fine at first, but soon starts returning abysmal results. The MAXENT fellow is puzzled. "Do you think maybe the other guy found out about our strategy?" he asks. "Maybe he hired our scientist away from us. But how can we possibly keep our strategy secret if we use it more than once?" And this leads to our second conceptual breakthrough.

We realize that it is both impossible and unnecessary to keep our strategy secret (just as cryptographer knows that it is difficult and unnecessary to keep the encryption algorithm secret. But it is both possible and essential to keep the plays secret until they are actually made (just as a cryptographer keeps keys secret). Hence, we must have mixed strategies where the strategy is a probability distribution and a play is a one-point sample from that distribution.

Take a step back and think about this. Non-determinism of agents is an inevitable consequence of having multiple agents whose interests are not aligned (or more precisely, agents whose interests cannot be brought into alignment by a system of side payments). Lesson 1: Any decision theory intended to work in multi-agent situations must handle (i.e. model) non-determinism in other agents. Lesson 2: In many games, the best strategy is a mixed strategy.

Think some more. Agents whose interests are not aligned often should keep secrets from each other. Lesson 3: Decision theories must deal with secrecy. Lesson 4: Agents may lie to preserve secrets.

But how does game theory find the best mixed strategy? Here is where it gets weird. It turns out that, in some sense, it is not about "winning" at all. It is about equilibrium. Remember back when we were at the PSS stage where we thought that "Always play scissors" was a good strategy? What was wrong with this, of course, was that it induced the other player to switch his strategy toward "Always play stone" (assuming, of course, that he has a scientist on his consulting staff). And that shift on his part induces (assuming we have a scientist too) us to switch toward paper.

So, how is this motion brought to a halt? Well, there is one particular strategy you can choose that at least removes the motivation for the motion. There is one particular mixed strategy which makes your opponent not really care what he plays. And there is one particular mixed strategy that your opponent can play which makes you not really care what you play. So, if you both make each other indifferent, then neither of you has any particular incentive to stop making each other indifferent, so you both just stick to the strategy you are currently playing.

This is called Nash equilibrium. It also works on non-zero sum games where the two players' interests are not completely misaligned. The decision theory at the heart of Game Theory - the source of eight economics Nobel prizes so far - is not trying to "win". Instead, it is trying to stop the other player from squirming so much as he tries to win. Swear to God. That is the way it works.

Alright, in the last paragraph, I was leaning over backward to make it look weird. But the thing is, even though you no longer look like you are trying to win, you still actually do as well as possible, assuming both players are rational. Game theory works. It is the right decision theory for the kinds of decisions that fit into its model.

So, was this long parable useful in our current search for "the best decision theory"? I guess the answer to that must depend on exactly what you want a decision theory to accomplish. My intuition is that Lessons #1 through #4 above cannot be completely irrelevant. But I also think that there is a Lesson 5 that arises from the Nash equilibrium finale of this story. The lesson is: In any optimization problem with a multi-party optimization dynamics to it, you have to look for the fixpoints.

Here's a simplified version of your second counterexample:

Omega appears and asks you which colour you like better, red or blue. If you chose the same colour that Omega happens to like, you get a million dollars, otherwise zero.

Obviously, your decision in this ridiculous scenario depends on your prior for meeting Omegas who like red vs. Omegas who like blue. Likewise, in your original counterexample, your action in scenario 1 should depend on your prior for encountering scenario 1 vs. scenario 2.

So yeah, this is a pretty big flaw in UDT that I pointed out sometime ago on the workshop list, and then found out that Caspian nailed it even earlier in the comments to Nesov's original post on Counterfactual Mugging. The retort "just use priors" may or may not be satisfactory to you. It's certainly not completely satisfactory to me, so I'd like a decision theory that doesn't require anything beyond "local" descriptions of scenarios. Presumably, such a theory would win in Scenario 1 and lose in Scenario 2, which may or may not be what we want.

I don't think this counterexample is actually a counterexample. When you-simulation decides in Scenario 1, he has no knowledge of Scenario 2. Yes, if people respond in arbitrary and unexpected ways to your decisions, this sort of thing can easily be set up; but ultimately the best you can do is to maximize expected utility. If you lose due to Omega pulling such a move on you, that's due to your lack of knowledge and bad calibration as to his probable responses, not to a flaw in your decision theory. If you-simulation somehow knew what the result would be used for, he would choose taking that into account.

In the Iterated Prisoner's Dilemma, it is relatively trivial to prove that no strategy is "optimal" in the sense that it gets the best possible pay-out against all opponents. ... So no strategy is "optimal".

In order to reach this conclusion, you have to eliminate the possibility of making inferences about the other player's decision. When you factor in the probability of various other competing decision theories (which you can get either from that inference, or from a sensible prior like an Occamian one), this objection no longer applies, as there will be some set of strategies which is, on average, optimal.

Note that when you e.g. recognize the symmetry between your position and the other player's, that is an inference about the other's likely decision!

Or, you might not like Omega being able to make decisions about you based entirely on your sourcecode (or "ritual of cognition"), then how about this:in order for two decision theories to sensibly be described as "different", there must be some scenario in which they perform a different action (let's call this Scenario 1).

I'm not sure if there's some flaw in this reasoning ...

One flaw I see is this: in order to create a Scenario 1, you would have to arbitrarily reject from consideration some decision theories or otherwise predicate the result on choice-irrelevant details of the ritual of cognition.

The reason for this is that a decision theory can contain some element (as TDT, UDT, and folk decision theory) that recognizes the connection (in Drescher's terms, a subjunctive means-end link) between making decision A and getting less money. In order for that decision theory to nevertheless choose A, you would have to put some arbitrary restriction on its choices that unfairly disadvantages A against B from the beginning. This would make the comparison between the two irrelevant.

I'm interested in what decision theory is the best under the true distribution of future histories that lay ahead.

(Unrealistic) hypothetical scenarios that make A win over B don't interest me, except that I'm confident we can manufacture them - after all, the decision theories are fixed in advance, and then we just make up some arbitrary rules under which they fail (differently from one another).

I've just simulated you in Scenario 1. If you chose decision B, there's $1,000,000 in the left envelope. Otherwise it's empty. There's $1000 in the right envelope regardless.

If you want to make every decision theory fall into the trap, other than the most arbitrary and insane, simply have the S2 basis be:

If, during scenario 1, you recited the square ordinality primes (ie. 1st prime, 4th prime, 9th prime, etc.) up to the 100th prime, the envelope will contain $1 billion

Bentarm - you might be interested in my old post on The Limits of Moral Theory. There I discuss a related result in moral philosophy (due to Donald Regan): that there's no general way to specify what actions each individual ought to take, such that those agents who satisfy the theory (i.e. who do as they ought) thereby bring about the best (collectively possible) results. It's quite curious.

I have no doubt that TDT is an improvement on CDT, but in order for this to even make sense, we'd have to have some way of thinking about what sort of problem we want our decision theory to solve. Presumably the answer is "the sort of problems which you're actually likely to face in the real world".

If that's so, why do we spend so much time talking about Newcomb problems? Should we ban Omega from our decision theories?

the sort of problems which you're actually likely to face in the real world". Do we have a good formalism for what this means?"

Cellular automata represent a nice abstract model of the world - many of them exhibit key features like locality, reversibiity, massive parallelism, Occam's razor, universal computation, self-replication, 2lot, etc.

However - unlike the "real" world - we can model them exactly - and simulate and test in them freely. They make good universe-modelling material.