I agree that as you defined the problems, both have problems. But I don't agree that the problems are equal, for the reason stated earlier. Suppose someone says that the boundary is that 1,526,216,123,000,252 dust specks is exactly equal to 50 years of torture (in fact, it's likely to be some relatively low number like this rather than anything like a googleplex.) It is true that proving this would be a problem. But it is no particular problem that 1,526,216,123,000,251 dust specks would be preferable to the torture, while the torture would be preferable to 1,526,123,000,253 dust specks would be worse than the torture: the point is that the torture would differ from each of these values by an extremely tiny amount.

But suppose someone defines a qualitative boundary: 1,525,123 degrees of pain (given some sort of measure) has an intrinsically worse quality from 1,525,122 degrees, such that no amount of the latter can ever add up to the former. It seems to me that there is a problem which doesn't exist in the other case, namely that for a trillion people to suffer pain of 1,525,122 degrees for a trillion years is said to be preferable to one person suffering pain of 1,525,123 degrees for one year.

In other words: both positions have difficult to find boundaries, but one directly contradicts intuition in a way the other does not.

So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort...

Eliezer, your readiness to assume that all 'bad things' are on a continuous scale, linear or no, really surprises me. Put your enormous numbers away, they're not what people are taking umbrage at. Do you think that if a googol doesn't convince us, perhaps a googolplex will? Or maybe 3^^^3? If x and y are finite, there will always be a quantity of x that exceeds y, and vice versa. We get the maths, we just don't agree that the phenomena are comparable. Broken ankle? Stubbing your toe? Possibly, there is certainly more of a tangible link there, but you're still imposing your judgment on how the mind experiences and deals with discomfort on us all and calling it rationality. It isn't.

Put simply - a dust mote registers exactly zero on my torture scale, and torture registers fundamentally off the scale (not just off the top, off) on my dust mote scale.

You're asking how many biscuits equal one steak, and then when one says 'there is no number', accusing him of scope insensitivity.

Eliezer - the way question #1 is phrased, it is basically a choice between the following:

1. Be perceived as a hero, with certainty.

2. Be perceived as a hero with 90% probability, and continue not to be noticed with 10% probability.

This choice will be easy for most people. The expected 50 extra deaths are a reasonable sacrifice for the certainty of being perceived as a hero.

The way question #2 is phrased, it is similarly a choice between the following:

1. Be perceived as a villain, with certainty.

2. Not be noticed with 90% probability, and be perceived as a villain with 10% probability.

Again, the choice is obvious. Choose #2 to avoid being perceived as a villain.

If you argue that the above interpretations are then not altruistic, I think the "Repugnant Conclusion" link shows how futile it is to try to make actual "altruistic decisions".

I don't think even everyone going blind is a good excuse for torturing a man for fifty years. How are they going to look him in the eye when he gets out?

That's cold brother. Real cold....

The idea of an ethical discontinuity between something that can destroy a life (50 years of torture, or 1 year) and something that can't (1 minute of torture, a dust speck) has some intuitive plausibility, but...

Sorry, no. 'Torture' and 'dust speck' are not two different quantities of the same currency. I wouldn't even be confident trying to add up individual minutes of torture to equal one year. Humans do not experience the world like disinterested machines. They don't even experience a logarithmic progression of 'amount of discomfort.' 50 years of torture does things to the mind and body that one year (for 50 people) can never do. One year of torture does things one minute can never do. One minute of torture does things x dust specks in x people's eyes could never do. None of these things registers on each others' scales.

Cash, possessions, whatever, I'm with you and Eliezer. Pure human perception is different, even when you count neurons. And no, this isn't a blind irrational reaction to the key word 'torture'. This is how human beings work.

Something occurred to me reading through all this earlier. Do we put no weight on the fact that if you polled the 3^^^3 people and asked them whether they would all undergo one dust speck to save one person from 50 years of torture, they'd almost certainly all say yes? Who would say "no, look how many of us there are! Torture him!" I find this goes a long way to exploding the idea of 'cumulative discomfort'.

I agree, you have to just multiply.

But your math is an attempt to abstract human harm to numbers, and the problem I have (and I suspect others intuitively have) is that the abstraction is wrong. You've failed to understand the lessons of the science of happiness: we forget small painful things quickly. The cost of a speck in the eye, let us imagine, is 1 unit of harm. But that's only true in the moment the speck hits the eye. In the hour the speck hits the eye, because of the human ability to ignore or forget small pains, the extended cost is 0. This is why a googol of specks is better than torture, because a googol 0s...

The real problem is defining the boundary between momentary (transient) harm and extendable harm. This is a psychological math problem.

Consider these two facts about me:

(1) It is NOT CLEAR to me that saving 1 person with certainty is morally equivalent to saving 2 people when a fair coin lands heads in a one-off deal.

(2) It is CLEAR to me that saving 1000 people with p=.99 is morally better than saving 1 person with certainty.

Models are supposed to hew to the facts. Your model diverges from the facts of human moral judgments, and you respond by exhorting us to live up to your model.

Why should we do that?

Eliezer's point would have been valid, had he chosen almost anything other than momentary eye irritation. Even the momentary eye-irritation example would work if the eye irritation would lead to serious harm (e.g. eye inflammation and blindness) in a small proportion of those afflicted with the speck of dust. If the predicted outcome was millions of people going blind (and then you have to consider the resulting costs to society), then Eliezer is absolutely right: shut-up and do the math.

Eliezer, as I'm sure you know, not everything can be put on a linear scale. Momentary eye irritation is not the same thing as torture. Momentary eye irritations should be negligible in the moral calculus, even when multiplied by googleplex^^^googleplex. 50 years of torture could break someone's mind and lead to their destruction. You're usually right on the mark, but not this time.

Torture vs dust specks, let me see:

What would you choose for the next 50 days:

1. Removing one mililiter of the daily water intake of 100,000 people.
2. Removing 10 liters of the daily water intake of 1 person.

The consequence of choice 2 would be the death of one person.

Yudkowsky would choose 2, I would choose 1.

This is a question of threshold. Below certain thresholds things don't have much effect. So you cannot simply add up.

Another example:

1. Put 1 coin on the head of each of 1,000,000 people.
2. Put 100,000 coins on the head of one guy.

What do you choose? Can we add up the discomfort caused by the one coin on each of 1,000,000 people?

Roland, I'll take that bet.

The idea of an ethical discontinuity between something that can destroy a life (50 years of torture, or 1 year) and something that can't (1 minute of torture, a dust speck) has some intuitive plausibility, but ultimately I don't buy it. It very much seems like death must be in the same 'regime' as torture, but also that death is in the same regime as trivial harms, because people risk death for trivial benefit all the time - I imagine anyone here would drive across town for $100 or $500 or $1000, even though it's slightly more dangerous than staying at home. The life-destroying aspect means that the physical pain is only part (probably the smaller part) of the harm of prolonged torture, and that the badness of torture rises greater than linearly with duration, but doesn't necessarily make it incommensurable.

One can easily make an argument like the torture vs. dust specks argument to show that the Repugnant Conclusion is not only not repugnant, but certainly true.

More intuitively, if it weren't true, we could find some population of 10,000 persons at some high standard of living, such that it would be morally praiseworthy to save their lives at the cost of a googolplex galaxies filled with intelligent beings. Most people would immediately say that this is false, and so the Repugnant Conclusion is true.

Ben, you are right. Two people with dusty eyes is worse than one. But it isn't twice as worse. It's not even nearly twice as worse. On the other hand I would say that two people being tortured is almost twice as bad as one, but not quite. I'm sure I can't write down a formula for my utility function in terms of number of deaths, or dusty eyes, or tortures, but I know one thing: it is not linear. There's nothing inherently irrational about choosing a nonlinear utility function. So I will continue to prefer any number of dusty eyes to even one torture. I would also prefer a very large number of 1-day tortures to a singe 50-year one. (far far more than 365 * 50). Am I being irrational? How?

OK, my final response on the subject, which has had me unable to think about anything else all day. Thanks to all involved for helping me get my thoughts in order on this topic, and sorry for hijacking.

therefore burying the whole group in dust

You've forgotten the rules of the game. There's no 'burying everyone in dust.' You either have a speck of dust in your eye and blink it away, or you don't. And that's for every individual in the group. Playing with the numbers doesn't change the scenario much either.

My #1 complaint is that no-one seems bothered by things like this:

So we then double the number in set A while halving their discomfort.

Halving their discomfort? Care to go into some more depth on that? Would that be half as many neurons firing 'pain!'? Rods inserted half as deep? Thumbs only halfway screwed?

And this:

We can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until...

As far as I know, there is no reason why I should agree that you can do anything of the sort. You might be able to divide torture by N and get dust mote, and you might not. But you certainly can't take it for granted then tell me I'm irrational.

A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.

Maybe the strange notation has me confused, but I don't see the contradiction here. Consider sneezes. One human sneezing N times in a row, where N>2, seems at least super-exponentially worse than N humans each sneezing once (assuming that no noticeable consequences for any of this last beyond the day). In fact, if we all sneeze simultaneously that would be pretty cool.

This next part doesn't directly address the original question. But if 3^^^3 humans know that by getting a dust speck in their eye they helped save someone from torture, the vast majority would likely feel happy about this and we wind up with a mountain of increased utility from Dust Specks relative to No Pain. Whereas an average-human torture victim who learns that the torture served to prevent dust specks might try to kill you bare-handed.

Eliezer -- depends again on whether we're aggregating across individuals or within one individual. From a utilitarian perspective (see The Post That Is To Come for a non-utilitarian take), that's my big objection to the specks thing. Slapping each of 100 people once each is not the same as slapping one person 100 times. The first is a series of slaps. The second is a beating.

Honestly, I'm not sure if I'd have given the same answer to all of those questions w/o having heard of the dust specks dilemma. I feel like that world is a little too weird -- the thing that motivates me to think about those questions is the dust specks dilemma. They're not the sort of things practical reason ordinarily has to worry about, or that we can ordinarily expect to have well-developed intuitions about!

ultimately having no one drive at all would save the most lives

And ultimately no dust specks and no torture and lollipops all round would be great for everyone. Stick to the deal as presented. You have a choice to make. Speed is quantifiable. Death is very quantifiable. Pain - even physical pain - goes in the same category as love, sadness, confusion. They are abstract nouns because you cannot hold, measure or count them. Does N losing lottery tickets spread equally over N people equal one dead relative's worth of grief?

Reconsider my poll scenario: Wouldn't the opinions of 3^^^3 people, all willing to bear the brunt of a dust speck for you, sway your judgment one little bit? Are you that certain of your rationality? You are about to submit yourself to 50 years of torture, and you have 3^^^3 people screaming at you 'don't bother, it's okay, no single person has a problem with just blinking once, even those who would opt for torture in your place!' What do you reply? 'Stop being so damned irrational! Just insert the rods!'

How come these examples and subsequent narratives never mention the value of floors and diminishing returns? Is every life valued the same? If there was a monster or disease that will kill everyone in the world there is a floor involved. Choice 1 of saving 400 lives ensures that humanity continues (assuming 400 people are enough to re-populate the world). While have a 90% chance of saving 500 leaves 10% chance that humanity on earth ends. Would you agree that floors are important factors that do change the value of an optimal outcome when they are one time events? In other words the marginal utility of a life is diminishing in this example.

Whilst the your analysis of life-saving choices seems fairly uncontentious, I'm not entirely convinced that the arithmetic of different types of suffering add together the way you assume. It seems at least plausible to me that where dust motes are individual points, torture is a section of a contiuous line, and thus you can count the points, or you can measure the lengths of different lines, but no number of the former will add up to the latter.

Great New Theorem in color perception : adding together 10 peoples' perceptions of light pink is equivalent to one person's perception of dark red. This is demonstrable, as there is a continuous scale between pink and red.

Mr. Yudkowsky, I'm not sure the duration/intensity of the torture is the only bad thing relevant here. A friend of mine pointed out that a problem with 50 years of torture is that it permanently destroys someone's life. (I think it was in one of the "fake altruism" family of posts that you pointed out that belief of utility != utility of belief.) So the utility curve would be pretty flat for the first couple thousand dust specks, beginning to slope down in proportion to the pain through a few minutes of torture. After that, it would quickly become steeper as the torture began to materially alter the person tortured. Another factor to consider is the difference between pain during which you can do other things, and pain during which you can't. So the 50-year-torturee's (or even a 1-minute torturee's) life is effectively shortened in a way that even a 1,000,000-dust-speck person's life is not. So I'm not sure people aren't implicitly including those factors sometimes, when they get mad about torture. I'd rather five years of chronic back pain than five minutes of permanently soul-crushing torture.

You might argue that it's still irrational, but it's not as obvious as you make it out to be.

It seems a lot of people are willing to write off minimal discomfort and approximate that to 0 discomfort, I don't think thats fair at all.

If we are talking in terms of this 'discomfort' lets start out with two sets of K people out of a population of X >>> K people, with each set having the same 'discomfort' applied to them each, set A and set B. One set must bear with the discomfort, which set should we pick?.

Clearly at the start, both are defined to be the same. So we then double the number in set A while halving their discomfort.

One way to define an activity A to be 'half the discomfort' of B is to ask the average person how long will they take activity A for say... $100 and the same for B, if they are willing to take twice as long on A than B, lets call that half. There is no such thing as infinite discomfort because we are dealing with people here, double the number of people you double the discomfort.

Torturing two people for 50 years is twice as bad as torturing one person for fifty years, how do we work that out? well we have some non infinite of discomfort for "torturing for 50 years".

Eventually after many repetitions, we hit on some discomfort which 'suddenly' someone has arbitrary said is ~ 0 discomfort even though it is really some small discomfort. And since 0*K = 0, we can set the number of people in set A to be (X-K) (a bit unfair to have the guys in set B to be in set A too I think, give them a break) yet it will still be better to choose set A over set B. The sum of the discomforts is less in A than B.

Lets say that we are now that the discomfort of A is a speck of dust, and the discomfort of B is 50 years torture. Lets now crank up the discomfort of A, until we hit the precise point that you just about start to care about your discomfort (just before it changes from ~0 to some number). I reckon a stubbed toe would be a good point though I bet even more discomfort would be the true discontinuity point. K is now 1 person, and X is the entire human population of the planet. This is fine because its ~0*6.6 billion = 0, yet you take a stumble after you stub a toe, you lose .1 minutes of your life in a bad way, its not nice, if it was nice then the discomfort would be negative!

But we are all happy right? everyone stubs their toe, and a man (or woman) is saved from 50 years of torture. However, that comes to 1256 years of stubbed toes, that comes to a total of 1256 years of lost living, time spent wincing at your sore toe rather than looking the sky and so on. Is that still acceptable? double the people, 2512 years, still fine? keep on going cause to you (and people in general if your right), the discomfort is ~0.

Keep on doubling that number and watch those wasted pain filled years double and double. If you suddenly say 'ok thats enough, a trillion subbed toes is worse than 50 years torture!' then we simply double the number of people tortured, then half the population stub their toes for one guy, half for the other. Imagine the hundreds of thousands of years humanity has wasted with stubbed toes, and if you dont see that as bad you should wonder if your biased by scope insensitivity.

How about:

1. floor(3^(3^(3^(3 + sin(3^^^3))))) people are tortured for a day.

2. floor(3^(3^(3^(3 + cos(3^^^3))))) people are tortured for a day.

Choose.

Well, why are certain notations for large integers to be taken seriously but not others? Shut up and do trig!

(Mainly though I want to claim dibs to the name "a googolplex simultaneous sneezes")

I think the argument is misguided. Why? The choice is not only hypothetical but impossible. There is not the remotest possibility of a googolplex persons even existing.
So I'll tone it down to a more realistic "equation", then I'll argue that it's not an equation after all.
Then I'll admit that I'm lost, but so are you... =)
Let's assume 1e7 people experiencing pain of a certain intensity for one second vs. one persion experiencing equal pain for 1e7 seconds (approx. 19 years).
Let's assume that every person in question has an expectancy of, say, 63 years of painless life. Then my situation is eqivalent to either extending the painless life expectency of 1e7 people from 63y-1s to 63y or to extend it for one person from 54y to 63y.
According to the law of diminishing returns, the former is definitely much less valuable than the latter.
But how much so? How to quantify this?
I have no idea, but I claim that neither do you... =)

regards, frank

p.s.
I have a hunch that you couldn't fit enough people with specks in their eyes into the universe to make up for one 50-year-torture.

Eliezer's question for Paul is not particularly subtle, so I presume he won't mind if I give away where it is leading. If Paul says yes, there is some number of dust specks which add up to a toe stubbing, then Eliezer can ask if there is some number of toe stubbings that add up to a nipple piercing. If he says yes to this, he will ultimately have to admit that there is some number of dust specks which add up to 50 years of torture.

Rather than actually going down this road, however, perhaps it would be as well if those who wish to say that the dust specks are always preferable to the torture should the following facts:

1) Some people have a very good imagination. I could personally think of at least 100 gradations between a dust speck and a toe stubbing, 100 more between the toe stubbing and the nipple piercing, and as many as you like between the nipple piercing and the 50 years of torture.

2) Arguing about where to say no, the lesser pain can never add up to the slightly greater pain, would look a lot like creationists arguing about which transitional fossils are merely ape-like humans, and which are merely human-like apes. There is a point in the transitional fossils where the fossil is so intermediate that 50% of the creationists say that it is human, and 50% that it is an ape. Likewise, there will be a point where 50% of the Speckists say that dust specks can add up to this intermediate pain, but the intermediate pain can't add up to torture, and the other 50% will say that the intermediate pain can add up to torture, but the specks can't add up the intermediate pain. Do you really want to go down this path?

3) Is your intuition about the specks being preferable to the torture really greater than the intuition you violate by positing such an absolute division? Suppose we go down the path mentioned above, and at some point you say that specks can add up to pain X, but not to pain X+.00001 (a representation of the minute degree of greater pain in the next step if we choose a fine enough division). Do you really want to say that you prefer that a trillion persons (or a google, or a googleplex, etc) than that one person suffer pain X+.00001?

While writing this, Paul just answered no, the specks never add up to a toe stub. This actually suggests that he rounds down the speck to nothing; you don't even notice it. Remember however that originally Eliezer posited that you feel the irritation for a fraction of a second. So there is some pain there. However, Paul's answer to this question is simply a step down the path laid out above. I would like to see his answer to the above. Remember the (minimally) 100 gradations between the dust speck and the toe stub.

Tcpkac: wonderful intuition pump.

Gary: interesting -- my sense of the nipple piercing case is that yes, there's a number of unwilling nipple piercings that does add up to 50 years of torture. It might be a number larger than the earth can support, but it exists. I wonder why my intuition is different there. Is yours?

Unknown, there is nothing inherently illogical about the idea of qualitative transitions. My thesis is that a speck of dust in the eye is a meaningless inconvenience, that torture is agony, and that any amount of genuinely meaningless inconvenience is preferable to any amount of agony. If those terms can be given objective meanings, then a boundary exists and it is a coherent position.

I just said genuinely meaningless. This is because, in the real world, there is going to be some small but nonzero probability that the speck of dust causes a car crash, for example, and this will surely be considerably more likely than a positive effect. When very large numbers are involved, this will make the specks worse than the torture.

But the original scenario does not ask us to consider consequences, so we are being asked to express a preference on the basis of the intrinsic badness of the two options.

what matters is that there is crossover at some point

But there isn't necessarily one. That's the point - Eliezer is presuming that dust speck harm is additive and that enough of such harms will equal torture. This presumption does not seem to have a basis in rational argument.

I don't think even everyone going blind is a good excuse for torturing a man for fifty years. How are they going to look him in the eye when he gets out?

The problem is not that I'm afraid of multiplying probability by utility, but that Eliezer is not following his own advice - his utility function is too simple.