LESSWRONG

Chocolate Ice Cream After All?

3 points
expand_less
Upvote
expand_more
Downvote
pallas
Dec 9, 2013
78 comments

I have collected some thoughts on decision theory and am wondering whether they are any good, or whether I’m just thinking non-sense. I would really appreciate some critical feedback. Please be charitable in terms of language and writing style, as I am not a native English speaker and as this is the first time I am writing such an essay.

Overview

  • The classical notion of free will messes up our minds, especially in decision-theoretic problems. Once we come to see it as confused and reject it, we realize that our choices in some sense not only determine the future but also the past.
  • If determining the past conflicts with our intuitions of how time behaves, then we need to adapt our intuitions.
  • The A,B-Game shows us that, as far as the rejection of free will allows for it, it is in principle possible to choose our genes.
  • Screening off only applies if we consider our action to be independent of the variable of interest – at least in expectation.
  • When dealing with Newcomblike problems, we have to be clear about which forecasting powers are at work. Likewise, it turns out to be crucial to precisely point out which agent knows how much about the setting of the game.
  • In the standard version of Newcomb’s Soda, one should choose chocolate ice cream – unless the game were specified in a way that previous subjects did not (unlike us) know of any interdependence of soda and ice cream.
  • Variations of Newcomb’s Soda suggest that the evidential approach makes us better off.
  • The analysis of Newcomb’s Soda shows that its formulation fundamentally differs from the formulation of Solomon’s Problem.  
  • Given all study-subjects make persistent precommitments, a proper use of evidential reasoning suggests precommitting to take chocolate ice cream. This is why Newcomb’s Soda does not show that the evidential approach is dynamically inconsistent.
  • The tickle defense does not apply to the standard medical version of Solomon’s Problem. In versions where it applies, it does not tell us anything non-trivial.
  • Evidential reasoning seems to be a winning approach not only in Newcomb’s Problem, but also in Newcomb’s Soda and in the medical version of Solomon’s Problem. Therefore, we should consider a proper use of evidential reasoning as a potentially promising component when building the ultimate decision algorithm.

In the standard formulation of Newcomb’s Soda, the evidential approach suggests picking chocolate ice cream, since this makes it more probable that we will have been awarded the million dollars. Hence, it denies us the thousand dollars we actually could win if we only took vanilla ice cream. Admittedly, this may be counterintuitive. Common sense tells us that considering the thousand dollars, one could change the outcome, whereas one cannot change which type of soda one has drunk; therefore we have to make a decision that actually affects our outcome. Maybe the flaw in this kind of reasoning doesn’t pose a problem to our intuitions as long as we deal with a “causal-intuition-friendly” setting of numbers. So let’s consider various versions of this problem in order to thoroughly compare the  two competing algorithmical traits. Let’s find out which one actually wins and therefore should be implemented by rational agents.

In this post, I will discuss Newcomblike problems and conclude that the arguments presented support an evidential approach. Various decision problems have shown that plain evidential decision theory is not a winning strategy. I instead propose to include evidential reasoning in more elaborate decision theories, such as timeless decision theory or updateless decision theory, since they also need to come up with an answer in Newcomblike problems.
By looking at the strategies proposed in those problems, currently discussed decision theories produce outputs that can be grouped into evidential-like and causal-like. I am going to outline which of these two traits a winning decision theory must possess.
Let’s consider the following excerpt by Yudkowsky (2010) about the medical version of Solomon’s Problem:

“In the chewing-gum throat-abscess variant of Solomon’s Problem, the dominant action is chewing gum, which leaves you better off whether or not you have the CGTA gene; but choosing to chew gum is evidence for possessing the CGTA gene, although it cannot affect the presence or absence of CGTA in any way.”

In what follows, I am going to elaborate on why I believe this point (in the otherwise brilliant paper) needs to be reconsidered. Furthermore, I will explore possible objections and have a look at other decision problems that might be of interest to the discussion.
But before we discuss classical Newcomblike problems, let’s first have a look at the following thought experiment:

 

The school mark is already settled

Imagine you were going to school; it is the first day of the semester. Suppose you only care about getting the best marks. Now your math teacher tells you that he knows you very well and that this would be why he already wrote down the mark you will receive for the upcoming exam. To keep things simple, let’s cut down your options to “study as usual” and “don't study at all”. What are you going to do? Should you learn as if you didn’t know about the settled mark? Or should you not learn at all since the mark has already been written down?

This is a tricky question because the answer to it depends on your credence in the teacher’s forecasting power. Therefore let's consider the following two cases:

  1. Let's assume that the teacher is correct in 100% of the cases. Now we find ourselves in a problem that resembles Newcomb's Problem since our decision exactly determines the output of his prediction. Just as an agent that really wishes to win the most money should take only one box in Newcomb’s Problem, you should learn for the exams as if you didn't know that the marks are already settled. (EDIT: For the record, one can point out a structural (but not relevant) difference between the two problems: Here, the logical equivalences "learning" <--> "good mark" and "not learning" <--> "bad mark" are part of the game's assumptions, while the teacher predicts in which of these two worlds we live in. In Newcomb's Problem, Omega predicts the logical equivalences of taking boxes and payoffs.)
  2. Now let's consider a situation where we assume a teacher having no forecasting power at all. In such a scenario the student's future effort behaves independently of the settled marks, that is no matter what input the student provides, the output of the teacher will have been random. Therefore, if we find ourselves in such a situation we shouldn't study for the exam and enjoy the gained spare time.

(Of course we can also think of a case 3) where the teacher's prediction is wrong in 100% of all cases. Let’s specify “wrong” since marks usually don’t work in binaries, so let’s go with “wrong” as the complementary mark. For instance, the best mark corresponds to the worst, the second best to the second worst and so on. In such a case not learning at all and returning an empty exam sheet would determine receiving the best marks. However, this scenario won't be of big interest to us.)
This thought experiment suggests that a deterministic world does not necessarily imply fatalism, since in expectation the fatalist (who wouldn't feel obligated to learn because the marks are "already written down") would lose in cases where the teacher predicts other than random. Generally, we can say that – beside the case 2) – in all the other cases the learning behaviour of the student is relevant for receiving a good mark.
This thought experiment does not only make it clear that determinism does not imply fatalism, but it even shows that fatalists tend to lose once they stop investing ressources in desriable outcomes. This will be important in subsequent sections. Now let us get to the actual topic of this article which already has been mentioned as an aside: Newcomblike problems.

 

Newcomb’s Problem

The standard version of Newcomb’s Problem has been thoroughly discussed on Lesswrong. Many would agree that one-boxing is the correct solution, for one-boxing agents obtain a million dollars, while two-boxers only take home a thousand dollars. To clarify the structure of the problem: an agent chooses between two options, “AB“ and “B“. When relatively considered, the option B “costs” a thousand dollars because one would abandon transparent box A containing this amount of money. As we play with the predictor Omega, who has an almost 100%  forecasting power, our decision determines what past occured, that is we determine whether Omega put a million into box B or not. With determining I mean as much as “being compatible with”. Hence, choosing box B is compatible only with a past where Omega put a million into it.

 

Newcomb’s Problem’s Problem of Free Will

To many, Newcomb’s Problem seems counterintuitive. People tend to think: “We cannot change the past, as past events have already happened! So there’s nothing we can do about it. Still, somehow the agents that only choose B become rich. How is this possible?“
This uneasy feeling can be resolved by clarifing the notion of “free will”, i.e. by acknowledging that a world state X either logically implies (hard determinism) or probabilistically suggests (hard incompatibilism, stating that free will is impossible and complete determinism is false) another world state Y or a set of possible world states (Y1,Y2,Y3,..,Yn) – no matter if X precedes Y or vice versa. (Paul Almond has shown in his paper on decision theory – unfortunately his page has been down lately – that upholding this distinction does not affect the clarification of free will in decision-theoretic problems. Therefore, I chose to go with hard determinism.)

The fog will lift once we accept the above. Since our action is a subset of a particular world state, the action itself is also implied by preceding world states, that is once we know all the facts about a preceding world state we can derive facts about subsequent world states.
If we look more closely, we cannot really choose in a way that people used to think. Common sense tells us that we confront a “real choice” if our decision is not just determined by external factors and also not picked at random, but governed by our free will. But what could this third case even mean? Despite its intuitive usefulness, the classical notion of choice seems to be an ill-defined term since it requires a problematic notion of free will, that is to say one that ought to be non-random but also not determined at once.
This is why I want to suggest a new definition of choice: Choosing is the way agents execute what they were determined to by other world states. Choosing has nothing to do with “changing” what did or is going to happen. The only thing that actually changes is the perception of what did or is going to happen, since executions produce new data points that call for updates.
So unless we could use a “true” random generator (which would only be possible if we did not assume complete determinism to be true) in order to make decisions, what we are going to do is “planned” and determined by preceding and subsequent world states.
If I take box B, then this determines a past world state where Omega has put a million dollars into this box. If I take both box A and B, then this determines a past world state where Omega has left box B empty. Therefore, when it comes to deciding, taking actions that determine (or are compatible with) not only desirable future worlds, but also desirable past worlds are the ones that make us win.
One may object now that we aren’t “really“ determining the past, but we only determine our perception of it. That’s an interesting point. In the next section we are going to have a closer look on that. For now, I’d like to bring the underlying perception of time into question. Because once I choose only box B, it seems that the million dollars I receive is not just an illusion of my map but it is really out there. Admittedly the past seems unswayable, but this example shows that maybe our conventional perception of time is misleading as it conflicts with the notion of us choosing what happened in the past.
How come self-proclaimed deterministic non-fatalists in fact are fatalists when they deal with the past? I’d suggest to perceive time not as being divided into seperate caterogies like “stuff that has passed “ and “stuff that is about to happen“, but rather as one dimension where every dot is just as real as any other and where the manifestation of one particular dot restrictively determines the set of possible manifestations other dots could embody. It is crucial to note that such a dot would describe the whole world in three spatial dimensions, while subsets of world states could still behave independently.

Perceiving time without an inherent “arrow” is not new to science and philosophy, but still, readers of this post will probably need a compelling reason why this view would be more goal-tracking. Considering the Newcomb’s Problem a reason can be given: Intuitively, the past seems much more “settled” to us than the future. But it seems to me that this notion is confounded as we often know more about the past than we know about the future. This could tempt us to project this disbalance of knowledge onto the universe such that we perceive the past as settled and unswayable in contrast to a shapeable future. However, such a conventional set of intuitions conflicts strongly with us picking only one box. These intuitions would tell us that we cannot affect the content of the box; it is already filled or empty since it has been prepared in the now inaccessible past.

Changing the notion of time into one block would lead to “better” intuitions, because they directly suggested to choose one box, as this action is only compatible with a more desirable past. Therefore we might need to adapt our intution, so that the universe looks normal again. To illustrate the ideas discussed above and to put them into practice, I have constructed the following game:

 

The A,B-Game

You are confronted with Omega, a 100% correct predictor. In front of you, there are two buttons, A and B. You know that there are two kinds of agents. Agents with the gene G_A and agents with the gene G_B. Carriers of G_A are blessed with a life expectancy of 100 years whereas carriers of G_B die of cancer at the age of 40 on average. Suppose you are much younger than 40. Now Omega predicts that every agent who presses A is a carrier of G_A and every agent that presses B is a carrier of G_B. You can only press one button, which one should it be if you want to live for as long as possible?
People who prefer to live for a hundred years over forty years would press A. They would even pay a lot of money in order to be able to do so. Though one might say one cannot change or choose one’s genes. Now we need to be clear about which definition of choice we make use of. Assuming the conventional one, I would agree that one could not choose one’s genes, but for instance, when getting dressed, one could not choose one’s jeans either, as the conventional understanding of choice requires an empty notion of non-random, not determined free will that is not applicable. Once we use the definition I introduced above, we can say that we choose our jeans. Likewise, we can choose our genes in the A,B-Game. If we one-box in Newcomb’s Problem, we should also press A here, because the two problems are structurally identical (except for the labels “box” versus “gene”).
The notion of objective ambiguity of genes only stands if we believe in some sort of objective ambiguity about which choices will be made. When facing a correct predictor, those of us who believe in indeterministic objective ambiguity of choices have to bite the bullet that their genes would be objectively ambiguous. Such a model seems counterintuitive, but not contradictory. However, I don’t feel forced to adapt this indeterministic view.

Let us focus on the deterministic scenario again: In this case, our past already determined our choice, so there is only one way we will go and only one way we can go.
We don’t know whether we are determined to do A or B. By “choosing” the one action that is compatible only with the more desirable past, we are better off. Just as we don’t know in Newcomb’s Problem whether B is empty or not, we have to behave in a way such that it must have been filled already. From our perspective, with little knowledge about the past, our choice determines the manifestation of our map of the past. Apparently, this is exactly what we do when making choices about the future. Taking actions determines the manifestation of our map of the future. Although the future is already settled, we don’t know yet its exact manifestation. Therefore, from our perspective, it makes sense to act in ways that determine the most desirable futures. This does not automatically imply that some mysterious “change” is going to happen.
In both directions it feels like one would change the manifestation of other world states, but when we look more closely we cannot even spell out what that would mean. The word “change” only starts to become meaningful once we hypothetically compare our world with counterfactual ones (where we were not determined to do what we do in our world). In such a framework we could consistently claim that the content of box B “changes” depending on whether or not we choose only box B.

 

Screening off

Following this approach of determining one’s perception of the world, the question arises, whether every change in perception is actually goal-tracking. We can ask ourselves, whether an agent should avoid new information if she knew that the new information had negative news value. For instance, if an agent, being suspected of having lung cancer and awaiting the results of her lung biopsy, seeks actions that make more desirable past world states more likely, then she should figure out a way so that she doesn’t receive any mail, for instance by declaring an incorrect postal address. This naive approach obviously fails because of lack of proper use of Bayesian updating. The action ”avoiding to receive mail” screens off the desirable outcome so that once we know about this action we don’t learn anything about the biopsy in (the very probable) case that we don’t receive any mail.

In the A,B-Game, this doesn’t apply, since we believe Omega’s prediction to be true when it says that A necessarily belongs to G _A and B to G_B. Generally, we can distinguish the cases by clarifying existing independencies: In the lung cancer case where we simply don’t know better, we can assume that P(prevention|positive lab result)=P(prevention|negative lab result)=P(prevention). Hence, screening off applies. In the A,B-Game, we should believe that P(Press A|G_A)>P(Press A)=P(Press A|G_A or G_B). We obtain this relevant piece of information thanks to Omega’s forecasting power. Here, screening off does not apply.

Subsequently, one might object that the statement P(Press A|G_A)>P(Press A) leads to a conditional independence as well, at least in cases where not all the players that press A necessarily belong to G_A. Then you might be pressing A because of your reasoning R_1 which would screen off pressing A from G_A. A further objection could be that even if one could show a dependency between G_A and R_1, you might be choosing R_1 because of some meta-reasoning R_2 that again provides a reason not to press A. However, considering these objections more thoroughly, we realize that R_1 has to be congruent or at least evenly associated (in G_A as well as in G_B) with Pressing A. The same works for R_2. If this wasn’t the case, then we would be talking about another game, a game where we knew, for instance, that 90% of the G_A carriers choose button A (without thinking) because of the gene and 10% of the G_B carriers would choose button A because of some sort of evidential reasoning. Knowing this, choosing A out of evidential reasoning would be foolish, since we already know that only G_B carriers could do that. Once we know this, evidential reasoners would suggest not to press A (unless B offers an even worse outcome). So these further objections fail as well, as they implicitly change the structure of the discussed problem. We can conclude that  no screening off applies as long as an instance with forecasting power tells us that a particular action makes the desirable outcome likelier.
Now let’s have a look at an alteration of the A,B-Game in order to figure out whether screening-off might apply here.

 

A Weak Omega in The A,B-Game

Thinking about the A,B-Game, what happens if we decreased Omega’s forecasting power? Let’s assume now that Omega’s prediction is correct only in 90% of all cases. Should this fundamentally change our choice whether to press A or B because we only pressed A as a consequence of our reasoning?
To answer that, we need to be clear about why agents believe in Omega’s predictions. They believe in Omega’s prediction because they were correct so many times. This constitutes Omega’s strong forecasting power. As we saw above, screening off only applies if the predicting instance (Omega, or us reading a study) has no forecasting power at all.
In the A,B-Game, as well as in the original Newcomb’s Problem, we also have to take the predictions of a weaker Omega (with less forecasting power) into account, unless we face an Omega that happens to be right by chance (i.e. in 50% of the cases when considering a binary decision situation).

If, in the standard A,B-Game, we consider pressing A to be important, and if we were willing to spend a large  amount of money in order to be able to press A (suppose the button A would send a signal to cause a withdrawal from our bank account), then this amount should only gradually shrink once we decrease Omega’s forecasting power. The question now arises whether we also had to “choose” the better genes in the medical version of Solomon’s Problem and whether there might not be a fundamental difference between it and the original Newcomb’s Problem.

 

Newcomb’s versus Solomon’s Problem

In order to uphold this convenient distinction, people tell me that “you cannot change your genes” though that’s a bad argument since one could reply “according to your definition of change, you cannot change the content of box B either, still you choose one-boxing”. Further on, I quite often hear something like “in Newcomb’s Problem, we have to deal with Omega and that’s something completely different than just reading a study”. This – in contrast to the first – is a good point.

In order to accept the forecasting power of a 100% correct Omega, we already have to presume induction to be legitimate. Or else one could say: “Well, I see that Omega has been correct in 3^^^3 cases already, but why should I believe that it will be correct the next time?”. As sophisticated this may sound, such an agent would lose terribly. So how do we deal with studies then? Do they have any forecasting power at all? It seems that this again depends on the setting of the game. Just as Omega’s forecasting power can be set, the forecasting power of a study can be properly defined as well. It can be described by assigning values to the following two variables: its descriptive power and its inductive power. To settle them, we have to answer two questions: 1. How correct is the study's description of the population? 2. How representative is the population of the study to the future population of agents acting in knowledge of the study? Or in other words, to what degree can one consider the study subjects to be in one’s reference class in order to make true predictions about one’s behaviour and the outcome of the game? Once this is clear, we can then infer the forecasting power. How much forecasting power does the study have? Let’s assume that the study we deal with is correct in what it describes. Those who wish can use a discounting factor. However, this is not important for subsequent arguments and would only make it more complicated.

Considering the inductive power, it get’s more tricky. Omega’s predictions are defined to be correct. In contrast, the study’s predictions have not been tested. Therefore we are quite uncertain about the study’s forecasting power. It were 100% if and only if every factor involved was specified so that the total of them compel identical outcomes in the study and our game. Due to induction, we do have reason to assume a positive value of forecasting power. To identify its specific value (that discounts the forecasting power according to the specified conditions), we would need to settle every single factor that might be involved. So let’s keep it simple by applying a 100% forecasting power. As long as there is a positive value of forecasting power, the basic point of the subsequent arguments (that presume a 100% forecasting power) will also hold when discounted.
Thinking about the inductive power of the study, there still is one thing that we need to specify: It is not clear what exactly previous subjects of the study knew.

For instance in a case A), the subjects of the study knew nothing about the tendency of CGTA-carriers to chew gum. First, their genom was analyzed, then they had to decide whether or not to chew gum. In such a case, the subjects‘ knowledge is quite different from those who play the medical version of Solomon’s Problem. Therefore screening off applies. But does it apply to the same extent as in the avoiding-bad-news example mentioned above? That seems to be the case. In the avoiding-bad-news example, we assumed that there is no connection between the variables „lung cancer“ and „avoiding mail“. In Solomon’s Problem such an indepence can be settled as well. Then the variables „having the gene CGTA“ and „not chewing gum because of evidential reasoning“ are also assumed to be independent. Total screening off applies. Considering an evidential reasoner who knows that much, choosing not to chew gum would then be as irrational as declaring an incorrect postal address when awaiting biopsy results.

Now let us consider a case B) where the subjects were introduced to the game just as we were. Then they would know about the tendency of CGTA-carriers to chew gum, and they themselves might have used evidential reasoning. In this scenario, screening off does not apply. This is why not chewing gum would be the winning strategy.
One might say that of course the study-subjects did not know of anything and that we should assume case A) a priori. I only partially agree with that. The screening off can already be weakend if, for instance, the subjects knew why the study was conducted. Maybe there was anecdotal evidence about heredity of a tendency to chew gum, which was about to be confirmed properly.
Without further clarification, one can plausibly assume a probability distribution over various intermediate cases between A and B where screening off becomes gradually fainter when getting closer to B. Of course there might also be cases where anecdotal evidence leads astray, but in order to cancel out the argument above, anecdotal evidence needs to be equalized with in expectation knowing nothing at all. But since it seems to be better (even though not much) than knowing nothing, it is not a priori clear that we have to assume case A right away.  
So when compiling a medical version of Solomon’s Problem, it is important to be very clear about what the subjects of the study were aware of.

 

What about Newcomb’s Soda?

After exploring screening off and possible differences between Newcomb’s Problem and Solomon’s Problem (or rather between Omega and a study), let’s investigate those questions in another game. My favourite of all Newcomblike problems is called Newcomb’s Soda and was introduced in Yudkowsky (2010). Comparing Newcomb’s Soda with Solomon’s Problem, Yudkowsky writes:
“Newcomb’s Soda has the same structure as Solomon’s Problem, except that instead of the outcome stemming from genes you possessed since birth, the outcome stems from a soda you will drink shortly. Both factors are in no way affected by your action nor by your decision, but your action provides evidence about which genetic allele you inherited or which soda you drank.”

Is there any relevant difference in structure between the two games?
In the previous section, we saw that once we settle that the study-subjects in Solomon’s Problem don’t know of any connection between the gene and chewing gum, screening off applies and one has good reasons to chew gum. Likewise, the screening off only applies in Newcomb’s Soda if the subjects of the clinical test are completely unaware of any connection between the sodas and the ice creams. But is this really the case? Yudkowsky introduces the game as one big clinical test in which you are participating as a subject:

“You know that you will shortly be administered one of two sodas in a double-blind clinical test. After drinking your assigned soda, you will enter a room in which you find a chocolate ice cream and a vanilla ice cream. The first soda produces a strong but entirely subconscious desire for chocolate ice cream, and the second soda produces a strong subconscious desire for vanilla ice cream.”

This does not sound like previous subjects had no information about a connection between the sodas and the ice creams. Maybe you, and you alone, received those specific insights. If this were the case, it clearly had to be mentioned in the game’s definition, since this factor is crucial when it comes to decision-making. Considering a game where the agent herself is a study-subject, without further specification, she wouldn’t by default expect that other subjects knew less about the game than she did. Therefore let’s assume in the following that all the subjects in the clinical test knew that the sodas cause a subconscious desire for a specific flavor of ice cream.

 

Newcomb’s Soda in four variations

Let “C” be the causal approach which states that one has to choose vanilla ice cream in Newcomb’s Soda. C only takes the $1,000 of the vanilla ice cream into account since one still can change the variable “ice cream”, whereas the variable “soda” is already settled. Let “E” be the evidential approach which suggests that one has to choose chocolate or vanilla ice cream in Newcomb’s Soda – depending on the probabilities specified. E takes both the $1,000 of the vanilla ice cream and the $1,000,000 of the chocolate soda into account. In that case, one argument can outweigh the other.  

Let’s compile a series of examples. We denote “Ch” for chocolate, “V” for vanilla, “S” for soda and “I” for ice cream. In all versions Ch-S will receive $1,000,000 and V-I will receive $1,000 and P(Ch-S)=P(V-S)=0.5. Furthermore we settle that P(Ch-I|Ch-S)=P(V-I|V-S) and call this term “p” in every version so we don’t vary unnecessarily many parameters. As we are going to deal with large numbers, let’s assume a linear monetary value utility function.

Version 1: Let us assume a case where the sodas are dosed homeopathically, so that no effect on the choice of ice creams can be observed. Ch-S and V-S choose from Ch-I and V-I randomly so that p=P(V-I|Ch-S)=P(Ch-I|V-S)=0.5. Both C and E choose V-I and win 0.5 *$1,001,000 + 0.5*$1000=$501,000 in expectation. C only considers the ice cream whereas E considers the soda as well, though in this case the soda doesn’t change anything as the Ch-S are equally distributed over Ch-I and V-I.

Version 2: Here p=0.999999. Since P(Ch-S)=P(V-S)=0.5, one Ch-I in a million will have originated from V-S, whereas one V-I in a million will have originated from Ch-S. The other 999,999 Ch-I will have determined the desired past, Ch-S, due to their choice of Ch-I. So if we participated in this game a million times and tracked E that suggests choosing Ch-I each time, we overall could have expected to win 999,999*$1,000,000=$999,999,000,000. This is different to following C’s advice. As C tells us that we cannot affect which soda we have drunk we would choose V-I each time and could expect to win 1,000,000*$1,000+$1,000,000=$1,001,000,000 in total. The second outcome, which C is responsible for, is 999 times worse than the first (which was suggested by E). In this version, E clearly outperforms C in helping us to make the most money.

Version 3: Now we have p=1. This version is equivalent to the standard version of the A,B-Game. What would C do? It seems that C ought to maintain its view that we cannot affect the soda. Therefore, only considering the ice cream-part of the outcome, C will suggest choosing V-I. This seems to be absurd: C leaves us disappointed with $1,000, whereas E makes us millionaires every single time.
A C-defender might say: “Wait! Now you have changed the game. Now we are dealing with a probability of 1!” The response would be : “Interesting, I can make p get as close to 1 as I want as long as it isn’t 1 and the rules of the game and my conclusions would still remain. For instance, we can think of a number like 0.999…(100^^^^^100 nines in a row). So tell me why exactly the probability change of 0.000…(100^^^^^100 -1 zeros in a row)1 should make you switch to Ch-I? But wait, why would you – as a defender of C –  even consider Ch-I since it cannot affect your soda while it definitely prevents you from winning the $1,000 of the ice cream?”

The previous versions tried to exemplify why taking both arguments (the $1,000 and the $1,000,000) into account makes you better off at the one edge of the probability measure, whereas at the other edge, C and E produce the same outcomes. With a simple equation we can figure out for which p E would be indifferent about whether to choose Ch-I or V-I: solve(p*1,000,000=(1-p)*1,000,000+1,000,p). This gives us p=0.5005. So for 0.5005<p<=1 E does better than C and for 0<=p<=0.5005 E and C behave alike. Finally, let us consider the original version:  

Version 4: Here we deal with p=0.9. According to the above we could already deduce that deciding according to E makes us better off, but let’s have a closer look at it for the sake of completeness: In expectation, choosing V-I makes us win 0.1*$1,000,000+$1,000=$101,000, whereas Ch-I leaves us with 0.9*$1,000,000=$900,000 almost 9 times richer. After the insights above, it shouldn’t surprise us too much that E clearly does better than C in the original version of Newcomb’s Soda as well.

The variations above illustrated that C had to eat V-I even if 99.9999% of C-S choose C-I and 99.9999% of V-S eat V-I. If you played it a million times, in expectation C-I would win the million 999,999 times and V-I just once. Can we really be indifferent about that? Wasn’t it all about winning and losing? And who is winning here and who is losing?

 

Newcomb-Soda and Precommitments

Another excerpt from Yudkowsky (2010):
“An evidential agent would rather precommit to eating vanilla ice cream than precommit to eating chocolate, because such a precommitment made in advance of drinking the soda is not evidence about which soda will be assigned.”
At first sight this seems intuitive. But if we look at the probabilities more closely suddenly a problem arises: Let’s consider an agent that precommits (let’s assume a 100% persistent mechanism) one’s decision before a standard game (p=0.9) starts. Let’s assume that he precommits – as suggested above – to choose V-I. What credence should he assign to P(Ch-S|V-I)? Is it 0.5 as if he didn’t precommit at all or does something change? Basically, adding precommitments to the equation inhibits the effect of the sodas on the agent’s decision. Again, we have to be clear about which agents are affected by this newly introduced variable. If we were the only ones who can precommit 100% persistently, then our game fundamentally differs from the previous subjects’ one. If they didn’t precommit, we couldn’t presuppose a forecasting power anymore because the previous subjects decided according to the soda’s effect, whereas we now decide independently of that. In this case, E would suggest to precommit to V-I. However, this would constitute an entirely new game without any forecasting power. If all the agents of the study make persistent precommitments, then the forecasting power holds; the game doesn’t change fundamentally. Hence, the way previous subjects behaved remains crucial to our decision-making. Let’s now imagine that we were playing this game a million times. Each time we irrevocably precommit to V-I. In this case, if we consider ourselves to be sampled randomly among V-I, we can expect to originate from V-S 900,000 times. If we approach p to 1 we see that it gets desperately unlikely to originate from Ch-S once we precommit ourselves to V-I. So a rational agent following E should precommit Ch-I in advance of drinking the soda. Since E suggests Ch-I both during and before the game, this example doesn’t show that E would be dynamically inconsistent.

In the other game, where only we precommit persistently and the previous subjects don’t, picking V-I doesn’t make E dynamically inconsistent, as we would face another decision situation where no forecasting power applies. Of course we can also imagine intermediate cases. For instance one, where we make precommitments and the previous subjects were able to make them as well, but we don’t know whether they did. The more uncertain we get  about their precommitments, the more we approach the case where only we precommit while the forecasting power gradually weakens. Those cases are more complicated, but they do not show a dynamical inconsistency of E either.

 

The tickle defense in Newcomblike problems

In the last section I want to have a brief look at the tickle defense, which is sometimes used to defend evidential reasoning by offering a less controversial output. For instance, it states that in the medical version of Solomon’s Problem an evidential reasoner should chew gum, since she can rule out having the gene as long as she doesn’t feel an urge to chew gum. So chewing gum doesn’t make it likelier to have the gene since she already has ruled it out.

I believe that this argument fails since it changes the game. Suddenly, the gene doesn’t cause you to “choose chewing gum” anymore but to “feel an urge to choose chewing gum”. Though I admit, in such a game a conditional independence would screen off the action “not chewing gum” from “not having the gene” – no matter what the previous subjects of the study knew. This is why it would be more attractive to chew gum. However, I don’t see why this case should matter to us. In the original medical version of Solomon’s Problem we are dealing with another game where this particular kind of screening off does not apply. As the gene causes one to “choose chewing gum” we can only rule it out by not doing so. However, this conclusion has to be treated with caution. For one thing, depending on the numbers, one can only diminish the probability of the undesirable event of having the gene – not rule it out completely; for another thing, the diminishment only works if the previous subjects were not ignorant of a depedence of the gene and chewing gum – at least in expectation. Therefore the tickle defense only trivially applies to a special version of the medical Solomon’s Problem and fails to persuade proper evidential reasoners to do anything differently in the standard version. Depending on the specification of the previous subjects’ knowledge, an evidential reasoner would still chew or not chew gum.

3 points
expand_less
Upvote
expand_more
Downvote
pallas

Comments

sorted by
magical algorithm
Highlighting new comments since Today at 9:50 AM
Select new highlight date
Rendering 50/78 comments  show more
Viliam_Bur
3 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Too many complex ideas in one article. I think it would be better to treat them individually. If some former idea is invalid, then it does not make sense to discuss latter ideas which depend on the former one.

I don't get the idea of the A,B-Game.

You are confronted with Omega, a 100% correct predictor. In front of you, there are two buttons, A and B. You know that there are two kinds of agents. Agents with the gene GA and agents with the gene GB. Carriers of GA are blessed with a life expectancy of 100 years whereas carriers of GB die of cancer at the age of 40 on average. Suppose you are much younger than 40. Now Omega predicts that every agent who presses A is a carrier of GA and every agent that presses B is a carrier of GB.

I try imagining how the gene GB could make people consciously choose to shorten their lives (all other things being equal). Maybe it is a gene for suicide, depression, mental retardation, or perhaps dyslexia (the person presses the wrong button). But it wouldn't work for a random gene. It only works for a gene that specifically fits this scenario.

It is important to note that this Omega scenario presumes different things than the typical one used in the Newcomb’s Problem. In the Newcomb's case, we assume that Omega is able to predict one's thought processes and decisions. That's a very strong and unusual situation, but I can imagine it. However, in the A,B-Game we assume that a specific gene makes people presented with two options choose the worse one -- please note that I have not mentioned Omega in this sentence yet! So the claim is not that Omega is able to predict something, but that the gene can determine something, even in absence of the Omega. It's no longer about Omega's superior human-predicting powers; the Omega is there merely to explain the powers of the gene.

So I am not whether this says anything different than "if you choose not to commit suicide, you caused yourself to not have a certainly-suicide-causing gene". And even in that case, I think it has the direction of the causality wrong. By choosing not to commit suicide, I observe myself as not having a certainly-suicide-causing gene. We could call it a survivor bias or a non-suicidal-anthropic principle. It's kinda similar to the quantum suicide experiment, just instead of the quantum event I am using my genome.

If this argument is incorrect (and to me it seems it is), how much does the rest of the article fall apart?

3 points
expand_less
Upvote
expand_more
Downvote
pallas
5 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Thanks for the comment!

However, in the A,B-Game we assume that a specific gene makes people presented with two options choose the worse one -- please note that I have not mentioned Omega in this sentence yet! So the claim is not that Omega is able to predict something, but that the gene can determine something, even in absence of the Omega. It's no longer about Omega's superior human-predicting powers; the Omega is there merely to explain the powers of the gene.

I think there might be a misunderstanding. Although I don't believe it to be impossible that a gene causes you to think in specific ways, in the setting of the game such a mechanism is not required. You can also imagine a game where Omega predicts that those who pick a carrot out of a basket of vegetables are the ones that will die shortly of a heart attack. As long as we believe in Omega's forecasting power, its statements are relevant even if we cannot point at any underlying causal mechanisms. As long as the predicted situation is logically possible (here, all agents that pick the carrot die), we don't need to reject Omega's prediction just because such a compilation of events would be unlikely. Though we might call Omega's predictions into question. Still, as long as we believe in its forecasting power (after such a update), we have to take the prediction into account. Hence, the A,B-Game holds even if you don't know of any causal connection between the genes and the behaviour, we only need a credible Omega.

5 points
expand_less
Upvote
expand_more
Downvote
RichardKennaway
3 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Although I don't believe it to be impossible that a gene causes you to think in specific ways, in the setting of the game such a mechanism is not required.

It is required. If Omega is making true statements, they are (leaving aside those cases where someone is made aware of the prediction before choosing) true independently of Omega making them. That means that everyone with gene A makes choice A and everyone with gene B makes choice B. This strong entanglement implies the existence of some sort of causal connection, whether or not Omega exists.

More generally, I think that every one of those problems would be made clear by exhibiting the causal relationships that are being presumed to hold. Here is my attempt.

For the School Mark problem, the causal diagram I obtain from the description is one of these:

pupil's character ----> teacher's prediction ----> final mark
   |
   |
   V
studying ----> exam performance

or

pupil's character ----> teacher's prediction
   |
   |
   V
studying ----> exam performance ----> final mark

For the first of these, the teacher has waived the requirement of actually sitting the exam, and the student needn't bother. In the second, the pupil will not get the marks except by studying for and taking the exam. See also the decision problem I describe at the end of this comment.

For Newcomb we have:

person's qualities --> Omega's prediction --> contents of boxes
    |                                            |
    |                                            |
    V                                            V
person's decision --------------------------> payoff

(ETA: the second down arrow should go from "contents of boxes" to "payoff". Apparently Markdown's code mode isn't as code-modey as I expected.)

The hypotheses prevent us from performing surgery on this graph to model do(person's decision). The do() operator requires deleting all in-edges to the node operated on, making it causally independent of all of its non-descendants in the graph. The hypotheses of Newcomb stipulate that this cannot be done: every consideration you could possibly employ in making a decision is assumed to be already present in the personal qualities that Omega's prediction is based on.

A-B:

Unknown factors ---> Gene ---> Lifespan
   |
   |
   V
Choice

or:

Gene ---> Lifespan
  |
  |
  V
Choice

or both together.

Here, it may be unfortunate to discover oneself making choice B, but by the hypotheses of this problem, you have no choice. As with Newcomb, causal surgery is excluded by the problem. To the extent that your choice is causally independent of the given arrow, to that extent you can ignore lifespan in making your choice -- indeed, it is to that extent that you have a choice.

For Solomon's Problem (which, despite the great length of the article, you didn't set out) the diagram is:

charisma ----> overthrow
   |
   |
   V
commit adultery

This implies that while it may be unfortunate for Solomon to discover adulterous desires, he will not make himself worse off by acting on them. This differs from A-B because we are given some causal mechanisms, and know that they are not deterministic: an uncharismatic leader still has a choice to make about adultery, and to the extent that it is causally independent of the lack of charisma, it can be made, without regard to the likelihood of overthrow.

Similarly CGTA:

CGTA gene ----> throat abcesses
   |
   |
   V
chew gum

and the variant:

CGTA gene ----> throat abcesses
   |               ^
   |               |
   V               |
chew gum ---------/

(ETA: the arrow from "chew gun" to "throat abcesses" didn't come out very well.)

in which chewing gum is protective against throat abscesses, and positively to be recommended.

Newcomb's Soda:

soda assignment ---> $1M
    |
    |
    V
choice of ice cream ---> $1K

Here, your inclination to choose a flavour of ice-cream is informative about the $1M prize, but the causal mechanism is limited to experiencing a preference. If you would prefer $1K to a chocolate ice-cream then you can safely choose vanilla.

Finally, here's another decision problem I thought of. Unlike all of the above, it requires no sci-fi hypotheses, real-world examples exist everywhere, and correctly solving them is an important practical skill.

I want to catch a train in half an hour. I judge that this is enough time to get to the station, buy a ticket, and board the train. Based on a large number of similar experiences in the past, I can confidently predict that I will catch the train. Since I know I will catch the train, should I actually do anything to catch the train?

The general form of this problem can be applied to many others. I predict that I'm going to ace an upcoming exam. Should I study? I predict I'll win an upcoming tennis match. Should I train for it? I predict I'll complete a piece of contract work on time. Should I work on it? I predict that I will post this. Should I click the "Comment" button?

3 points
expand_less
Upvote
expand_more
Downvote
Beluga
1 point
expand_less
Upvote
expand_more
Downvote
4 years ago

For the School Mark problem, the causal diagram I obtain from the description is one of these:

diagram

or

diagram

For the first of these, the teacher has waived the requirement of actually sitting the exam, and the student needn't > bother. In the second, the pupil will not get the marks except by studying for and taking the exam. See also the decision problem I describe at the end of this comment.

I think it's clear that Pallas had the first diagram in mind, and his point was exactly that the rational thing to do is to study despite the fact that the mark has already been written down. I agree with this.

Think of the following three scenarios:

  • A: No prediction is made and the final grade is determined by the exam performance.
  • B: A perfect prediction is made and the final grade is determined by the exam performance.
  • C: A perfect prediction is made and the final grade is based on the prediction.

Clearly, in scenario A the student should study. You are saying that in scenario C, the rational thing to do is not studying. Therefore, you think that the rational decision differs between either A and B, or between B and C. Going from A to B, why should the existence of someone who predicts your decision (without you knowing the prediction!) affect which decision the rational one is? That the final mark is the same in B and C follows from the very definition of a "perfect prediction". Since each possible decision gives the same final mark in B and C, why should the rational decision differ?

In all three scenarios, the mapping from the set of possible decisions to the set of possible outcomes is identical -- and this mapping is arguably all you need to know in order to make the correct decision. ETA: "Possible" here means "subjectively seen as possible".

By deciding whether or not to learn, you can, from your subjective point of view, "choose" wheter you were determined to learn or not.

1 point
expand_less
Upvote
expand_more
Downvote
Viliam_Bur
2 points
expand_less
Upvote
expand_more
Downvote
4 years ago

You can also imagine a game where Omega predicts that those who pick a carrot out of a basket of vegetables are the ones that will die shortly of a heart attack.

Those who pick a carrot after hearing Omega's prediction, or without hearing the prediction? Those are two very different situations, and I am not sure which one you meant.

If some people even after hearing the Omega's prediction pick the carrot and then die of a heart attack, there must be something very special about them. They are suicidal, or strongly believe that Omega is wrong and want to prove it, or some other confusion.

If people who without hearing the Omega's prediction pick the carrot and die, that does not mean they would have also picked the carrot if they were warned in advance. So saying "we should also press A here" provides no actionable advice about how people should behave, because it only works for people who don't know it.

2 points
expand_less
Upvote
expand_more
Downvote
pallas
4 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Those who pick a carrot after hearing Omega's prediction, or without hearing the prediction? Those are two very different situations, and I am not sure which one you meant.

That's a good point. I agree with you that it is crucial to keep apart those two situations. This is exactly what I was trying to address considering Newcomb's Problem and Newcomb's Soda. What do the agents (previous study-subjects) know? It seems to me that the games aren't defined precise enough.
Once we specify a game in a way that all the agents hear Omega's prediction (like in Newcomb's Problem), the prediction provides actionable advice as all the agents belong to the same reference class. If we, and we alone, know about a prediction (whereas other agents don't) the situation is different and the actionable advice is not provided anymore, at least not to the same extent.
When I propose a game where Omega predicts whether people pick carrots or not and I don't specify that this only applies to those who don't know about the prediction then I would not assume prima facie that the prediction only applies to those who don't know about the prediction. Without further specification, I would assume that it applies to "people" which is a superset of "people who know of the prediction".

4 points
expand_less
Upvote
expand_more
Downvote
Khoth
1 point
expand_less
Upvote
expand_more
Downvote
4 years ago

If the scenario you describe is coherent, there has to be a causal mechanism, even if you don't know what it is. If Omega is a perfect predictor, he can't predict that carrot-choosers have heart attacks unless carrot-choosers have heart attacks.

1 point
expand_less
Upvote
expand_more
Downvote
pallas
3 points
expand_less
Upvote
expand_more
Downvote
4 years ago

I think I agree. But I would formulate it otherwise:

i) Omega's prediction are true. ii) Omega predicts that carrot-choosers have heart attacks.

c) Therefore, carrot-choosers have heart attacks.

As soon as you accept i), c) follows if we add ii). I don't know how you define "causal mechanism". But I can imagine a possible world where no biological mechanism connects carrot-choosing with heart attacks but where "accidentally" all the carrot-choosers have heart-attacks (Let's imagine running worlds on a computer countless times. One day we might observe such a freak world). Then c) would be true without there being some sort of "causal mechanism" (as you might define it I suppose?). If you say that in such a freak world carrot-choosing and heart attacks are causally connected, then I would agree that c) can only be true if there is a underlying causal mechanism.

3 points
expand_less
Upvote
expand_more
Downvote
V_V
3 points
expand_less
Upvote
expand_more
Downvote
4 years ago

It is important to note that this Omega scenario presumes different things than the typical one used in the Newcomb’s Problem. In the Newcomb's case, we assume that Omega is able to predict one's thought processes and decisions. That's a very strong and unusual situation, but I can imagine it. However, in the A,B-Game we assume that a specific gene makes people presented with two options choose the worse one -- please note that I have not mentioned Omega in this sentence yet! So the claim is not that Omega is able to predict something, but that the gene can determine something, even in absence of the Omega. It's no longer about Omega's superior human-predicting powers; the Omega is there merely to explain the powers of the gene.

I agree with your objection, but I think there is a way to fix this problem into a properly Newcomb-like form:
Genes GA and GB control your lifespan, but they don't directly affect your decision making processes, and cause no other observable effect.
Omega predicts in advance, maybe even before you are born, whether you will press the A button or the B button, and modifies you DNA to include gene GA if he predicts that you will press A, or GB if he predicts that you will press B.
This creates a positive correlation between GA and A, and between GB and B, without any direct causal relation.
In this scenario, evidential decision theory generally ( * ) chooses A.

The same fix can be applied to the Solomon's problem, Newcomb's soda, and similar problems. In all these cases, EDT chooses the "one-box" option, which is the correct answer by the same reasoning which can be used to show that one-boxing is the correct choice ( * ) for the standard Newcomb's problem.

I think that allowing the hidden variable to directly affect the agent's decision making process, as the original version of the problems above do, makes them ill-specified:
If the agent uses EDT, then the only way a hidden variable can affect their decision making process is by affecting their preferences, but then the "tickle defence" becomes valid: since the agent can observe their preferences, this screens off the hidden variable from any evidence gathered by performing an action, therefore EDT chooses the "two-box" option.
If the hidden variable, other other hand, affects whether the agent uses EDT or some corrupted version of it, then, the question "what should EDT do?" becomes potentially ill-posed: EDT presumes unbounded rationality, does that mean that EDT knows that it is EDT, or is it allowed to have uncertainty about it? I don't know the answer, but I smell a self-referential paradox lurking there.

( *
One-boxing is trivially the optimal solution to Newcomb's problem if Omega has perfect prediction accuracy.
However, if Omega's probability of error is epsilon > 0, even for a very small epsilon, telling the optimal solution isn't so trivial: it depends on the stochastic independence assumptions you make. )

3 points
expand_less
Upvote
expand_more
Downvote
Pentashagon
0 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Another possibility is that Omega presents the choice to very few people with the G_B gene in the first place; only to the ones who for some reason happen to choose B as Omega predicts. Likewise, Omega can't present the choice to every G_A carrier because some may select B.

0 points
expand_less
Upvote
expand_more
Downvote
SilentCal
2 points
expand_less
Upvote
expand_more
Downvote
4 years ago

My thoughts: 1) The failure of CDT is its modeling of the decision process as ineffable 'free will' upon which things in the past cannot depend. Deviation from CDT is justified only when such dependencies exist. 2) The assumption that your decision is predictable requires the existence of such a dependency. 3) If we postulate that no such dependency exists, either CDT wins or our postulates are contradictory.

In particular, in Newcomb's Soda, the assumptions that the soda flavor predicts the ice-cream flavor with high probability and that the assignment of soda (and the choice of subjects) is uncorrelated with subjects' decision theory require that we are exceptional in employing decision theory. If lots of subjects were using CDT or EDT, they would all be choosing ice cream independently of their soda, and we wouldn't see that correlation (except maybe by coincidence). So it doesn't have to be stated in the problem that other subjects aren't using evidential reasoning--it can be seen plainly from the axioms! To assume that they are reasoning as you are is to assume a contradiction.

The AB game is confusing because it flirts with contradiction. You act as if you're free to choose A or B according to your decision theory while simultaneously assuming that Omega can predict your choice perfectly. But in fact, the only way Omega can predict perfectly is by somehow interacting with your decision theory. He can either administer the game only to people whose decision theory matches their genes, or manipulate people's answers, or manipulate their genes. In the first case, EDTers will get a free gene test if they have G_A, but will not be miraculously healed if they have G_B. In the second case, you'll find yourself pressing 'B' if you have G_B no matter what you try to precommit to. In the third case only, you have legitimate reason to commit to 'A', because your predetermined decision has causal influence on your genes.

You might try to counter with the case where Omega ensures that all children who are born will answer in a way consistent with their genes, and both things are determined at conception. But if this is the case, then if you have G_B, you can't commit yourself to EDT no matter how hard you think about decision theory. This follows from the assumptions, and the only reason to think otherwise is if you still count free will among your premises.

2 points
expand_less
Upvote
expand_more
Downvote
pallas
3 points
expand_less
Upvote
expand_more
Downvote
4 years ago

If lots of subjects were using CDT or EDT, they would all be choosing ice cream independently of their soda, and we wouldn't see that correlation (except maybe by coincidence). So it doesn't have to be stated in the problem that other subjects aren't using evidential reasoning--it can be seen plainly from the axioms! To assume that they are reasoning as you are is to assume a contradiction.

If lots of subjects were using CDT or EDT, they would be choosing ice cream independently of their soda iff the soda has no influence on whether they argue according to CDT or EDT. It is no logical contradiction to say that the sodas might affect which decision theoretic intuitions a subject is going to have. As long as we don't specify what this subconscious desire for ice cream exactly means, it is thinkable that the sodas imperceptibly affect our decision algorithm. In such a case, most of the V-I people (the fraction originating from V-S) would be attracted to causal reasoning, whereas most of the Ch-I people (the fraction originating from Ch-S) would find the evidential approach compelling. One can say now that the sodas "obviously" do not affect one's decision theory, but this clearly had to be pointed out when introducing a "subconscious desire."
I agree that once it is specified that we are the only agents using decision theory, screening off applies. But the game is defined in a way that we are subjects of a study where all the subjects are rewarded with money:

(an excerpt of the definition in Yudkowsky (2010))

It so happens that all participants in the study who test the Chocolate Soda are rewarded with a million dollars after the study is over, while participants in the study who test the Vanilla Soda receive nothing. But subjects who actually eat vanilla ice cream receive an additional thousand dollars, while subjects who actually eat chocolate ice cream receive no additional payment.

After reading this, it is not a priori clear to me that I would be the only subject who knows about the money at stake. To the contrary, as one of many subjects I assume that I know as much as other subjects know about the setting. Once other subjects know about the money they probably also think about whether choosing Ch-I or V-I produces the better outcome. It seems to me that all the agents base their decision on some sort of intuition about which would be the correct decisional algorithm.

To sum up, I tend to assume that other agents play a decision theoretic game as well and that the soda might affect their decision theoretic intuitions. Even if we assigned a low prior to the event that the sodas affect the subject's decision algorithms, the derived reasoning would not be invalid but it's power would shrink in proportion to the prior. Finally, it is definetly not a contradictory statement to say that the soda affects how the subject's decide and that the subject's use CDT or EDT.

3 points
expand_less
Upvote
expand_more
Downvote
Khoth
2 points
expand_less
Upvote
expand_more
Downvote
4 years ago

Can you edit the post to include a "Continue reading..." cut rather than having the whole thing take up a dozen screenfuls of main page?

2 points
expand_less
Upvote
expand_more
Downvote