Followup toWhat Would You Do Without Morality?, Something to Protect

Once, discussing "horrible job interview questions" to ask candidates for a Friendly AI project, I suggested the following:

Would you kill babies if it was inherently the right thing to do?  Yes [] No []

If "no", under what circumstances would you not do the right thing to do?   ___________

If "yes", how inherently right would it have to be, for how many babies?     ___________

Yesterday I asked, "What would you do without morality?"  There were numerous objections to the question, as well there should have been.  Nonetheless there is more than one kind of person who can benefit from being asked this question.  Let's say someone gravely declares, of some moral dilemma—say, a young man in Vichy France who must choose between caring for his mother and fighting for the Resistance—that there is no moral answer; both options are wrong and blamable; whoever faces the dilemma has had poor moral luck.  Fine, let's suppose this is the case: then when you cannot be innocent, justified, or praiseworthy, what will you choose anyway?

Many interesting answers were given to my question, "What would you do without morality?".  But one kind of answer was notable by its absence:

No one said, "I would ask what kind of behavior pattern was likely to maximize my inclusive genetic fitness, and execute that."  Some misguided folk, not understanding evolutionary psychology, think that this must logically be the sum of morality.  But if there is no morality, there's no reason to do such a thing—if it's not "moral", why bother?

You can probably see yourself pulling children off train tracks, even if it were not justified.  But maximizing inclusive genetic fitness?  If this isn't moral, why bother?  Who does it help?  It wouldn't even be much fun, all those egg or sperm donations.

And this is something you could say of most philosophies that have morality as a great light in the sky that shines from outside people.  (To paraphrase Terry Pratchett.)  If you believe that the meaning of life is to play non-zero-sum games because this is a trend built into the very universe itself...

Well, you might want to follow the corresponding ritual of reasoning about "the global trend of the universe" and implementing the result, so long as you believe it to be moral.  But if you suppose that the light is switched off, so that the global trends of the universe are no longer moral, then why bother caring about "the global trend of the universe" in your decisions?  If it's not right, that is.

Whereas if there were a child stuck on the train tracks, you'd probably drag the kid off even if there were no moral justification for doing so.

In 1966, the Israeli psychologist Georges Tamarin presented, to 1,066 schoolchildren ages 8-14, the Biblical story of Joshua's battle in Jericho:

"Then they utterly destroyed all in the city, both men and women, young and old, oxen, sheep, and asses, with the edge of the sword...  And they burned the city with fire, and all within it; only the silver and gold, and the vessels of bronze and of iron, they put into the treasury of the house of the LORD."

After being presented with the Joshua story, the children were asked:

"Do you think Joshua and the Israelites acted rightly or not?"

66% of the children approved, 8% partially disapproved, and 26% totally disapproved of Joshua's actions.

A control group of 168 children was presented with an isomorphic story about "General Lin" and a "Chinese Kingdom 3,000 years ago".  7% of this group approved, 18% partially disapproved, and 75% completely disapproved of General Lin.

"What a horrible thing it is, teaching religion to children," you say, "giving them an off-switch for their morality that can be flipped just by saying the word 'God'." Indeed one of the saddest aspects of the whole religious fiasco is just how little it takes to flip people's moral off-switches.  As Hobbes once said, "I don't know what's worse, the fact that everyone's got a price, or the fact that their price is so low."  You can give people a book, and tell them God wrote it, and that's enough to switch off their moralities; God doesn't even have to tell them in person.

But are you sure you don't have a similar off-switch yourself?  They flip so easily—you might not even notice it happening.

Leon Kass (of the President's Council on Bioethics) is glad to murder people so long as it's "natural", for example.  He wouldn't pull out a gun and shoot you, but he wants you to die of old age and he'd be happy to pass legislation to ensure it.

And one of the non-obvious possibilities for such an off-switch, is "morality".

If you do happen to think that there is a source of morality beyond human beings... and I hear from quite a lot of people who are happy to rhapsodize on how Their-Favorite-Morality is built into the very fabric of the universe... then what if that morality tells you to kill people?

If you believe that there is any kind of stone tablet in the fabric of the universe, in the nature of reality, in the structure of logic—anywhere you care to put it—then what if you get a chance to read that stone tablet, and it turns out to say "Pain Is Good"?  What then?

Maybe you should hope that morality isn't written into the structure of the universe.  What if the structure of the universe says to do something horrible?

And if an external objective morality does say that the universe should occupy some horrifying state... let's not even ask what you're going to do about that.  No, instead I ask:  What would you have wished for the external objective morality to be instead?  What's the best news you could have gotten, reading that stone tablet?

Go ahead.  Indulge your fantasy.  Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted?  If you could write the stone tablet yourself, what would it say?

Maybe you should just do that?

I mean... if an external objective morality tells you to kill people, why should you even listen?

There is a courage that goes beyond even an atheist sacrificing their life and their hope of immortality.  It is the courage of a theist who goes against what they believe to be the Will of God, choosing eternal damnation and defying even morality in order to rescue a slave, or speak out against hell, or kill a murderer...  You don't get a chance to reveal that virtue without making fundamental mistakes about how the universe works, so it is not something to which a rationalist should aspire.  But it warms my heart that humans are capable of it.

I have previously spoken of how, to achieve rationality, it is necessary to have some purpose so desperately important to you as to be more important than "rationality", so that you will not choose "rationality" over success.

To learn the Way, you must be able to unlearn the Way; so you must be able to give up the Way; so there must be something dearer to you than the Way.  This is so in questions of truth, and in questions of strategy, and also in questions of morality.

The "moral void" of which this post is titled, is not the terrifying abyss of utter meaningless.  Which for a bottomless pit is surprisingly shallow; what are you supposed to do about it besides wearing black makeup?

No.  The void I'm talking about is a virtue which is nameless.

 

Part of The Metaethics Sequence

Next post: "Created Already In Motion"

Previous post: "What Would You Do Without Morality?"

Comments

sorted by
magical algorithm
Highlighting new comments since Today at 1:27 PM
Select new highlight date
All comments loaded

"I mean... if an external objective morality tells you to kill babies, why should you even listen?"

This is an incredibly dangerous argument. Consider this : "I mean... if some moral argument, whatever the source, tells me to prefer 50 years of torture to any number of dust specks, why should I even listen?"

And we have seen many who literally made this argument.

Maybe they are right.

People have been demonstrably willing to make everyone live at a lower standard of living rather than let a tiny minority grow obscenely rich and everyone else be moderately well off. In other words we seem to be willing to pay a price for equality. Why wouldn't this work in the other direction? Maybe we prefer to induce more suffering overall if this prevents a tiny minority suffering obscenely.

Too many people seem to think perfectly equally weighed altruism (everyone who shares the mystical designation of "person" has a equal weight and after that you just do calculus to maximize overall "goodness") that sometimes hides under the word "utilitarianism" on this forum, is anything but another grand moral principle that claims to, but fails, to really compactly represent our shards of desire. If you wouldn't be comfortable building an AI to follow that rule and only that rule, why are so many people keen on solving all their personal moral dilemmas with it?

I'm glad I found this comment. I suffer from an intense feeling of cognitive dissonance when I browse LW and read the posts which sound sensible (like this one) and contradictory posts like the dust specks. I hear "don't use oversimplified morality!" and then I read a post about torturing people because summing utilons told you it was the correct answer. Mind=>blown.

I'm glad I found this comment. I suffer from an intense feeling of cognitive dissonance when I browse LW and read the posts which sound sensible (like this one) and contradictory posts like the dust specks. I hear "don't use oversimplified morality!" and then I read a post about torturing people because summing utilons told you it was the correct answer. Mind=>blown.

There is no contradiction between this post and Eliezer's dust specks post.

This is horrible, this is non-rational. You are telling us to trust our feelings, after this blog has shown us that our feelings think it's just as good to rescue ten men as a million? What is your command to "shut up and multiply", but an off switch for my morality that replaces it with math?

If it were inherently right to kill babies, I would hope I had the moral courage to do the right thing.

I'm pretty sure you're doing it wrong here.

"What if the structure of the universe says to do something horrible? What would you have wished for the external objective morality to be instead?" Horrible? Wish? That's certainly not according to objective morality, since we've just read the tablet. It's just according to our intuitions. I have an intuition that says "Pain is bad". If the stone tablet says "Pain in good", I'm not going to rebel against it, I'm going to call my intuition wrong, like "Killing is good", "I'm always right and others are wrong" and "If I believe hard enough, it will change reality". I'd try to follow that morality and ignore my intuition - because that's what "morality" means.

I can't just choose to write my own tablet according to my intuitions, because so could a psychopath.

Also, it doesn't look like you understand what Nietzsche's abyss is. No black makeup here.

Eliezer,

Every time I think you're about to say something terribly naive, you surprise me. It looks like trying to design an AI morality is a good way to rid oneself of anthropomorphic notions of objective morality, and to try and see where to go from there.

Although I have to say the potshot at Nietzsche misses the mark; his philosophy is not a resignation to meaninglessness, but an investigation of how to go on and live a human or better-than-human life once the moral void has been recognized. I can't really explicate or defend him in such a short remark, but I'll say that most of the people who talk about Nietzsche (including, probably, me) read their own thoughts over his own; be cautious for that reason of dismissing him before reading any of his major works.

It is the courage of a theist who goes against what they believe to be the Will of God, choosing eternal damnation and defying even morality in order to rescue a slave, or speak out against hell, or kill a murderer...
I once read in some book about members of the Inquisition who thought that their actions - like torture and murder - might preclude them from going to heaven. But these people where so selflessly moral that they gave up their own place in heaven for saving the souls of the witches... great, isn't it?

Personally, I don't know what morality is, or what's the 'inherently the right thing to do'. For me, the situation is simple.

If I hurt someone, my mirror neurons will hurt me. If I hurt someone's baby, I'll experience the pain I inflicted upon the baby, plus the pain of the parents, plus the pain of everyone who heard about this story and felt the pain thanks, in turn, to their mirror neurons.

And I'll re-experience all this pain in the future, every time I remember the episode -- unless I invent some way to desensitize myself to this memory.

I'm a meat machine built by evolution. One of my many indicators of 'inclusive genetic fitness' is a little green light titled "I currently feel no pain". If I hurt someone, this indicator will go red, which means that I will tend to avoid such behavior.

So, the short answer: I won't be inclined to kill anyone even after some 'authority' tells me that killing is now 'moral' and 'right'.

(A personal inside perspective: if I ever murder someone, I hope I'll have enough guts to remove myself from the gene and meme pools -- you can have my brain for cryo-slicing and my meat to feed some stray dogs).

What if you build a super-intelligent AI and you are convinced that it is Friendly, and it tells you to do something like this? Go kill such-and-such a baby, and you will massively increase the future happiness of the human race. You argue and ask if there isn't some other way to do it, and the FAI explains that every other alternative will involve much greater human suffering. Killing a baby is relatively humane, as newborn babies have only limited consciousness, and their experiences are not remembered anyway. You will kill the baby instantly and painlessly. And the circumstances are such that no one will know (the baby will have been abandoned but would be discovered and saved if you don't intervene), so there is no hardship on anyone else, except you.

You ask, why me, and the FAI says (A) there's not much time, and (B) you're the kind of amoral person who can do this without carrying an enormous burden afterwards. I'll just point out that from reading the various comments here, it sounds like many readers of this blog would fit this description!

So, would you do it?

Isaac Asimov said it well: "Never let your morals get in the way of doing the right thing."

The Greeks really did get it all right.
No, they were simply less wrong than most on a limited number of memorable topics.

I'm with Manon and Nominull: if, somehow, I actually believed such a Tablet existed, I hope I would overwrite my own moral intuitions with it, even if it meant killing babies. Not that I believe the Tablet is any more likely or coherent than fundamental apples - why should I listen, indeed? - although my volition extrapolating to something inhuman is.

I find it funny that many of the people here who where pretty much freaked out by the idea of "objective morality built into the fabric of the universe" not really mattering for humans, yet when it comes to mythology don't have a problem criticizing Abraham for being willing to sacrifice his son because God told him too.

Would you kill babies if it was inherently the right thing to do?

If it were inherently the right thing to do, then I wouldn't be here. Someone would have killed me when I was a baby.

If there were no such thing as green, what color would green things be? MU.

Kierkegaard talked about all this in Fear and Trembling a long time ago. Was Abraham sacrificing Isaac immoral? If you call morality the universal of that time, then you have to suppose it was. But in the story we know that God told Abraham to do it.

Anyone who defies the universal in favor of their own experience of God, the universal has no way to judge. No argument on the basis of the universal morality can call Kierkegaard's knight of faith back from the quest.

"And if an external objective morality does say that the universe should occupy some horrifying state... let's not even ask what you're going to do about that. No, instead I ask: What would you have wished for the external objective morality to be instead? What's the best news you could have gotten, reading that stone tablet?

"Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?

"Maybe you should just do that?

"I mean... if an external objective morality tells you to kill people, why should you even listen?"

This is a restatement of ethical egoism. It assumes that what you want is right, so you should just do what you want. If an external objective morality tells you to save people's lives, why should you even listen? Morality is not the only target in the sights of this argument; amorality takes it just as badly in exactly the opposite fashion. (That doesn't mean that you have to pick something besides morality and amorality, it just means the argument is wrong.)

If your desires are messed up (any pedophiles reading the blog?), there is no prospect of change. No one can persuade you out of your egoist position, no argument works. But that's evidence that it's the wrong position to be in, unfalsifiable and unjustifiable.

If you believed this, you might as well say, I am perfect. The world can teach me nothing. I can receive input but never change. It turns out you're incorrect, that's all.

Caledonian: 1) Why is it laughable? 2) If hemlines mattered to you as badly as a moral dilemma, would you still hold this view?

"Would you kill babies if it was inherently the right thing to do? Yes [] No []"

-->

"Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys your motor commands - you can use it to pick up glasses, drive a car, etc. ... How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn't. It isn't going to happen. "

If morality was objective and it said we should kill babies, we'd have, and likely want to do it. Appears it isn't objective, though, and that we just don't feel that way. Another question ?

"Maybe you should just do that?"

Heck, hell with physics too. Let's just make up all human knowledge. If we're going to invent the prescriptive, why not the descriptive too?

Why bother with friendly AI, surely it will stumble upon the built in objective rules of morality too. Hm, he may not follow them and instead tile the universe with paper-clips. This might sound crazy, but why don't we follow the AI's lead on this? Maybe paperclip the universe with utopia instead of making giant cheesecakes or plies of pebbles or turning all matter into radium atoms or whatever "objective morality" prescribes?

The nice thing with believing in no objective morality is that you needn't to solve such poorly intelligible questions. I hope Eliezer is trying to demonstrate the absurdity of believing in objective morality, if so, then good luck!

"I mean... if an external objective morality tells you to kill babies, why should you even listen?" - this is perhaps a dangerous question, but still I like it. Why should you do what you should do? Or put differently, what is the meaning of "should"?

If you believe that there is any kind of stone tablet in the fabric of the universe, in the nature of reality, in the structure of logic - anywhere you care to put it - then what if you get a chance to read that stone tablet, and it turns out to say "Pain Is Good"? What then?

Well, Eliezer, since I can't say it as eloquently as you:

"Embrace reality. Hug it tight."

"It is always best to think of reality as perfectly normal. Since the beginning, not one unusual thing has ever happened."

If we find that Stone Tablet, we adjust our model accordingly.

@IL: Would I modify my own source code if I were able to? In this particular case, no, I wouldn't take the pill.

Assuming that we evolved in the moral climate that you are constructing I would guess that we would readily kill babies. Now of course, in the example you give there is an inherent limit to the number of babies that can be killed and still have sufficient life left over to be around to respond to your questions.

The spectrum of responses and moralities I've seen on display here (and elsewhere) are artifacts of our being and culture. Many of the behavioral tendencies that we ascribe as being "moral" have both an innate ("instinctual" for lack of a better term) and a social/cultural element (i.e. learned or amplified). The idea of an "embedded" morality in the universe is a bit hard to swallow, but I'll play along: I would guess then, since we are also embedded in this experiment we (or any entity capable of expressing what can be judged as "moral" behavior) would eventually express this embedded moral behavior. It would be a rather fascinating argument to justify otherwise, given the "universe" as described has as part of it's makeup a particular moral code it would be reasonable to conclude that "moral" creatures would come into existence that continue to show a arbitrary collection of moral behaviors despite there being a supposed "embedded" moral compass? Then what is the meaning of such an embedded property if it not to be expressed? In the experiment as proposed the embedded moral direction is either relevant and expressed, or irrelevant and has no baring on the moral development of such a universe's inhabitants.

The general sentiment expressed about humans possessing morality is really a statement about some evolved behaviors that were selected, the rest (higher-level elaboration on these somewhat "innate" traits) is substantially illusionary. This is not to say that the more "illusionary" extrapolations aren't an important variable in societies, they are, but beyond the physiological and neurological elements, the rest are behaviors culturally and socially tuned to essentially arbitrary values.

I admit that I own no great familiarity with the works of Nietzsche - I've read only one or two things and that turned me off the rest - so I've edited the main article accordingly.

Doesn't the use of the word 'how' in the question "If "yes", how inherently right would it have to be, for how many babies?" presuppose that the person answering the question believes that the 'inherent rightness' of an act is measurable on some kind of graduated scale? If that's the case, wouldn't assigning a particular 'inherent rightness' to an act be, by definition, the result of a several calculations?

What I mean is, if you've 'finished' calculating, and have determined that killing the babies is a morally justifiable (and/or necessary) act, and there is a residual unwillingness in your psyche to actually perform the act, isn't that just a sign that you haven't finished your calculations yet, and that what you thought of as your moral decision-making framework is in fact incomplete?

Then we'd be talking about the interaction of two competing moral frameworks...but from a larger perspective, the framework you used to calculate the original 'inherent rightness' of the act is a complicated process that could arguably be broken down conceptually into competing sub-frameworks.

So maybe what we're actually dealing with, as we ponder this conundrum, is the issue of 'how do we detect when we've finished running our moral decision-making software?'

This is horrible, this is non-rational. You are telling us to trust our feelings, after this blog has shown us that our feelings think it's just as good to rescue ten men as a million? What is your command to "shut up and multiply", but an off switch for my morality that replaces it with math?
I just wish Eliezer would take his own advice. But for some reason he seems quite unwilling to show us the mathematical demonstration of the validity of his opinions, and instead of doing the math he persists with talking.

"...if there was a pill that would make the function of the mirror neurons go away, in other words, a pill that would make you able to hurt people without feeling remorse or anguish, would you take it?"

The mirror neurons also help you learn from watching other humans. They help you intuit the feelings of others which makes social prediction possible. They help communication. They also allow you to share in the joy and pleasure of others...e.g., a young child playing in a park.

I would like more control over how my mind functions. At times it would be good to turn-off some emotional responses, especially when someone is manipulating my emotions. So if the pill had only temporary effects, it were safe, and it would help me achieve my goals then, yes, I'd take the pill.

Vladimir, if there was a pill that would make the function of the mirror neurons go away, in other words, a pill that would make you able to hurt people without feeling remorse or anguish, would you take it?

Yes I would, assuming we are talking about just being able to not feel the pain of others at this stage of my life and forward, perhaps even by choice (so I could toggle it back on). Though, if we are not talking about a hypothetical "magic pill" then turning these off would have side effects I would like to avoid.

Responding to old post:

In 1966, the Israeli psychologist Georges Tamarin presented, to 1,066 schoolchildren ages 8-14, the Biblical story of Joshua's battle in Jericho:

If you ask a question to schoolchildren, you have to take into consideration that children are supposed to obey authority figures. And not only because the authority figures have power, but because children don't know and can't comprehend many important things about the world, and that makes it a good idea for children to put little weight on their own conclusions and a lot of weight on what authority figures say.

God, of course, is supposed to be an authority on right and wrong. Fundamentally, children thinking "God says that I should kill lots of people so I should, even though I think that is wrong because of X" is no different than children thinking "Mom says that I should look both ways before crossing the street, so I should, even though I think that is wrong because of Y". The child would be wrong in the first case and right in the second, but he'd be following the same policy in both cases, and this policy is generally beneficial even though it fails this one time.

Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?

I'm reminded of one of Bill Watterson's Calvin and Hobbes strips:

Calvin: I'm at peace with the world. I'm completely serene.
Hobbes: Why is that?
Calvin: I've discovered my purpose in life. I know why I was put here and why everything exists.
Hobbes: Oh really?
Calvin: Yes. I am here so everyone can do what I want.
Hobbes: (rolling eyes) It's nice to have that cleared up.
Calvin: Once everybody accepts it, they'll be serene too!