The Costs of Rationality

32 points

expand_less expand_more

Mar 3, 2009

The word "rational" is overloaded with associations, so let me be clear: to me [here], more "rational" means better believing what is true, given one's limited info and analysis resources.

Rationality certainly can have instrumental advantages. There are plenty of situations where being more rational helps one achieve a wide range of goals. In those situtations, "winnners", i.e., those who better achieve their goals, should tend to be more rational. In such cases, we might even estimate someone's rationality by looking at his or her "residual" belief-mediated success, i.e., after explaining that success via other observable factors.

But note: we humans were designed in many ways not to be rational, because believing the truth often got in the way of achieving goals evolution had for us. So it is important for everyone who intends to seek truth to clearly understand: rationality has costs, not only in time and effort to achieve it, but also in conflicts with other common goals.

Yes, rationality might help you win that game or argument, get promoted, or win her heart. Or more rationality for you might hinder those outcomes. If what you really want is love, respect, beauty, inspiration, meaning, satisfaction, or success, as commonly understood, we just cannot assure you that rationality is your best approach toward those ends. In fact we often know it is not.

The truth may well be messy, ugly, or dispriting; knowing it make you less popular, loved, or successful. These are actually pretty likely outcomes in many identifiable situations. You may think you want to know the truth no matter what, but how sure can you really be of that? Maybe you just like the heroic image of someone who wants the truth no matter what; or maybe you only really want to know the truth if it is the bright shining glory you hope for.

Be warned; the truth just is what it is. If just knowing the truth is not reward enough, perhaps you'd be better off not knowing. Before you join us in this quixotic quest, ask yourself: do you really want to be generally rational, on all topics? Or might you be better off limiting your rationality to the usual practical topics where rationality is respected and welcomed?

32 points

expand_less expand_more

Comments

sorted by

Highlighting new comments since Today at 10:27 AM

All comments loaded

22 points

expand_less expand_more

This parallels a discussion I've had numerous times in the field of computer games. I've had any number of artists / scripters / managers say that what a computer game needs is not a realistic physics engine, but a cinematic physics engine. They don't want it to be right, they want it to be pretty.

But, you'll find that "cinematic style" isn't consistent, and if you start from that basis, you won't be able to make boring, every-day events look realistic, and you'll have to add special-case patch-upon-patch and you'll never get it right in the end. The cinematic stuff will look right, but nothing else will.

If you start with a rigidly-correct physics engine (or at least, within current state-of-the-art) you'll find it MUCH easier to layer cinematic effects on top when asked for. Its usually far simpler than the other way around.

In an analogous way, I find that rationality makes it far easier for one to achieve one's goals, EVEN WHEN SAID GOALS ARE NON-RATIONAL. Now, that may mean that the rational thing to do in some cases is to lie to people about your beliefs, or to present yourself in a non-natural way. If you end up being uncomfortable with that, then one needs to reassess what, exactly, one's goals are, and what you are willing to do to achieve them. This may not be easy, but its far simpler than going the route of ignorance and emotionally-driven actions and then trying to put your life back together when you don't end up where you thought you would.

22 points

expand_less expand_more

6 points

expand_less expand_more

You'll need to clarify what you mean by "non-rational goals".

6 points

expand_less expand_more

2 points

expand_less expand_more

Yes, I suppose I should. By a non-rational goal I meant a goal that was not necessarily to my benefit, or the benefit of the world, a goal with a negative net sum worth. Things like poisoning a reservoir or marrying someone who will make your life miserable.

2 points

expand_less expand_more

2 points

expand_less expand_more

You decided to try achieving that "non-rational" goal, so it must be to your benefit (at least, you must believe so).

An example that I usually give at this point is as follows. Is it physically possible that in the next 30 seconds I'll open the window and jump out? Can I do it? Since I don't want to do it, I won't do it, and therefore it can not happen in reality. The concept of trying to do something you'll never want to do is not in reality either.

2 points

expand_less expand_more

4 points

expand_less expand_more

This is a plausible claim, but do you have concrete details, proposed mechanisms, or examples from your own or others lives to back it up? "I find that rationality makes it far easier" is a promising-sounding claim, and it'd be nice to know the causes of your belief.

4 points

expand_less expand_more

8 points

expand_less expand_more

I always made a distinction between rationality and truth-seeking. Rationality is only intelligible when in the context of a goal (whether that goal be rational or irrational). Now, if one acts rationally, given their information set, will chose the best plan-of-action towards succeeding their goal. Part of being rational is knowing which goals will maximize their utility function.

My definition of truth-seeking is basically Robin's definition of "rational." I find it hard to imagine a time where truth-seeking is incompatible with acting rationally (the way I defined it). Can anyone think of an example?

8 points

expand_less expand_more

mark_spottswood

7 points

expand_less expand_more

Pwno said: I find it hard to imagine a time where truth-seeking is incompatible with acting rationally (the way I defined it). Can anyone think of an example?

The classic example would invoke the placebo effect. Believing that medical care is likely to be successful can actually make it more successful; believing that it is likely to fail might vitiate the placebo effect. So, if you are taking a treatment with the goal of getting better, and that treatment is not very good (but it is the best available option), then it is better from a rationalist goal-seeking perspective to have an incorrectly high assessment of the treatment's possibility of success.

This generalizes more broadly to other areas of life where confidence is key. When dating, or going to a job interview, confidence can sometimes make the difference between success and failure. So it can pay, in such scenarios, to be wrong (so long as you are wrong in the right way).

It turns out that we are, in fact, generally optimized to make precisely this mistake. Far more people think they are above average in most domains than hold the opposite view. Likewise, people regularly place a high degree of trust in treatments with a very low probability of success, and we have many social mechanisms that try and encourage such behavior. It might be "irrational" under your usage to try and help these people form more accurate beliefs.

7 points

expand_less expand_more

4 points

expand_less expand_more

Well, sure. Repeating other posts - but one of the most common examples is when an agent's beliefs are displayed to other agents. Imagine that all your associates think that there is a Christian god. This group includes all your prospective friends and mates. Do you tell them you are an agnostic/atheist - and that their views are not supported by the evidence? No, of course not! However, you had better not lie to them either - since most humans lie so poorly. The best thing to do is probably to believe their nonsense yourself.

4 points

expand_less expand_more

6 points

expand_less expand_more

Tim, that's an excellent argument for why rationality isn't always the winning strategy in real life. People have been saying this sort of thing all week, but it was your "most humans lie so poorly" comment that really made it click for me, especially in the context of evolutionary psychology.

I'd really like to hear one of the "rationalists should always win" people address this objection.

6 points

expand_less expand_more

9 points

expand_less expand_more

We're talking about at least two different notions of the word "rational":

Robin Hanson used the definition at the top of this post, regarding believing the truth. There are social/evolutionary costs to that, partly because humans lie poorly.
The causal decision theorists' definition that Eliezer Yudkowsky was annoyed by. CDT defines rationality to be a specific method of deciding what action to take, even though this leads to two-boxing (losing) Newcomb's problem. Yudkowsky's objection, summarized by the slogan "Rationalists should WIN." was NOT a definition. It is a quality of his informal concept of rationality which the CDT definition failed to capture.

The claim "rationalists should always win" comes from taking Yudkowsky's slogan as a definition of rationality. If that is the definition that you are using, then the claim is tautological.

Please note that I don't endorse this misreading of Yudkowsky's post, I'm just trying to answer your question.

9 points

expand_less expand_more

9 points

expand_less expand_more

Thanks, John.

As you say, defining rationality as winning and then saying rationalists always win is a tautology. But aside from your two definitions, there's a third definition: the common definition of rationality as basing decisions on evidence, Bayes, and logic. So as I see it, supporters of "rationalists always win" need to do one of the following:

Show that the winning definition is the same as the Bayes/logic/evidence definition. Tim's counterexample of the religious believer who's a poor liar makes me doubt this is possible.
Stop using "rationality" to refer to things like the Twelve Virtues and Bayesian techniques, since these virtues and techniques sometimes lose and are therefore not always rational.
Abandon "rationalists always win" in favor of Robin's "rationalists always seek the truth". I think that definition is sufficient to demonstrate that a rationalist should one-box on Newcombe's problem anyway. After all, if it's true that one boxing is the better result, a seeker of truth should realize that and decide to one-box.

9 points

expand_less expand_more

4 points

expand_less expand_more

There are no supporters of "rationalists always win" – the slogan is "rationalists should win". Long-term / on-average, it's rational to expect a high correlation between rationality and success.

[1] – I'd bet that the rationalist strategy fares well against other heuristics; let's devise a good test. There may always be an effective upper-bound to the returns to increasing rationality in any community, but reality is dangerous – I'd expect rationalists to fair better.

[2] – Winning or losing one 'round' isn't sufficient grounds to declare a strategy, or particular decisions, as being non-rational. Buying lottery tickets isn't rational because some people win. And sometimes, winning isn't possible.

[3] – I like "rationalists always seek the truth" but would add "... but they don't seek all truths."

4 points

expand_less expand_more

5 points

expand_less expand_more

You realize, of course, that under this policy everyone stays Christian forever.

5 points

expand_less expand_more

2 points

expand_less expand_more

Indeed - religion is persistent. Of course in the real world you would find that isolated communities would arise, where "belief mutations" could arise without them being severely punished by the crowd.

2 points

expand_less expand_more

2 points

expand_less expand_more

I like to distinguish information-theoretic rationality from decision-theoretic rationality. (But these are rather long terms.) Often on this blog it's unclear which is meant (although you and Robin did make it clear.)

2 points

expand_less expand_more

7 points

expand_less expand_more

Willful stupidity is often easier and more profitable in the short run, but you just might be picking up pennies in front of a steamroller.

I think it's best to go out of your way to believe the truth, even though you won't always succeed. I'm very suspicious when tempted to do otherwise, it's usually for most unwise or unhealthy reasons. There are exceptions, but they're much rarer than we'd like to think.

7 points

expand_less expand_more

6 points

expand_less expand_more

Learning many true facts that are not Fun and are morally irrelevant (e.g. learning as many digits of pi as you can by spending your whole life on the activity), because this way you can avoid thinking about facts that are much less certain, shouldn't be considered rational. Rationality intrinsically needs to serve a purpose, the necessity for this is implicit even in apparently goal-neutral definitions like the one Robin gave in the post.

Another problem, of course, is that you don't know the cost of irrationality if you are irrational.

6 points

expand_less expand_more

6 points

expand_less expand_more

Are commitment mechanisms rational?

A malicious genius is considering whether to dose the dashing protagonist with a toxin. The toxin is known to be invariably fatal unless counteracted, and the malicious genius has the only antidote. The antagonist knows that the protagonist will face a choice: Either open a specific locked box containing, among other things, the antidote - surviving, but furthering the antagonist's wicked plan, or refuse to open the box, dying, and foiling the plan.

We analyze this as an extensive form game: The antagonist has a choice to dose or not to dose. If dose, then protagonist gets a choice, to die or not to die.

If only the protagonist was not so very very rational! Because the protagonist is known to be very very rational, the antagonist knows that the protagonist will choose to live, and thereby further the antagonist's plan.

A commitment mechanism, then, is the protagonist (rationally) sabotaging their rationality before the antagonist has an opportunity to dose. The "irrational revenge circuit" will revenge harm at any cost, even an irrationally high cost. Even antagonists step carefully around people with revenge circuits installed. (Yes, evolution has already installed some of these.)

6 points

expand_less expand_more

9 points

expand_less expand_more

A decision theory that doesn't need to go through the motions of making a commitment outside the cognitive algorithm is superior. Act as if you have made a commitment in all the situations where you benefit from having made the commitment. Actually make commitment only if it's necessary to signal the resulting decision.

(Off-point:) The protagonist may well be rational about sacrificing his life, if he cares about stopping the antagonist's plan more.

9 points

expand_less expand_more

7 points

expand_less expand_more

I've always wondered if there are any documented instances of someone unscrewing his steering wheel and tossing it out during a game of chicken.

7 points

expand_less expand_more

5 points

expand_less expand_more

You seem to be taking the opposite tack as in this video, where rationality was best for everyone no matter their cause.

5 points

expand_less expand_more

4 points

expand_less expand_more

A rational belief isn't necessarily correct or true. Rational beliefs are justified, in that they logically follow from premises that are accepted as true. In the case of probabilistic statements, a rational strategy is one that maximizes the chance of being correct or otherwise reaching a defined goal state. It doesn't have to work or be correct in any ultimate sense to be rational.

If I play the lottery and win, playing the lottery turned out to be a way to get lots of money. It doesn't mean that playing the lottery was a rational strategy. If I make a reasonable investment and improbable misfortune strikes, losing the money, that doesn't mean that the investment wasn't rational.

4 points

expand_less expand_more

3 points

expand_less expand_more

This has no bearing on the point above. In essence you're just rephrasing Robin's definition, "better believing what is true, given one's limited info and analysis resources." The disposition best-calculated to lead to true beliefs will not produce true beliefs in every instance, because true beliefs will not always be justified by available evidence.

So what?

3 points

expand_less expand_more

4 points

expand_less expand_more

Places where rationality* is not welcome:

Churches, political parties, Congress, family reunions, dates, cable news, bureaucracy, casinos... . *Of course rationality might dictate deception- but I take it lying confers some cost on the liar.

Please list the rest. Also, who here is involved with any of the things on the list? Am I wrong to include something and if not how do you deal with being rational in a place that discourages it.

4 points

expand_less expand_more

4 points

expand_less expand_more

Re: The word "rational" is overloaded with associations, so let me be clear: to me, more "rational" means better believing what is true, given one's limited info and analysis resources.

Ouch! A meta discussion, perhaps - but why define "rational" that way? Isn't the following much more standard?

"In economics, sociology, and political science, a decision or situation is often called rational if it is in some sense optimal, and individuals or organizations are often called rational if they tend to act somehow optimally in pursuit of their goals. [...] In this concept of "rationality", the individual's goals or motives are taken for granted and not made subject to criticism, ethical or otherwise. Thus rationality simply refers to the success of goal attainment, whatever those goals may be."

http://en.wikipedia.org/wiki/Rationality

4 points

expand_less expand_more

2 points

expand_less expand_more

Indeed - that was my first thought, but I was waiting till I figured out a good way of stating it. RH's definition of 'rational' seems to go against the usual definition presented above, while EY's seems to embrace it.

2 points

expand_less expand_more

Eliezer_Yudkowsky

10 points

expand_less expand_more

My definition differs from the one in Wikipedia because I require that your goals not call for any particular ritual of cognition. When you care more about winning then about any particular way of thinking - and "winning" is not defined in such a way as to require in advance any particular method of thinking - then you are pursuing rationality.

This, in turn, ends up implying epistemic rationality: if the definition of "winning" doesn't require believing false things, then you can generally expect to do better (on average) by believing true things than false things - certainly in real life, despite various elaborate philosophical thought experiments designed from omniscient truth-believing third-person standpoints.

Conversely you can start with the definition of rational belief as accuracy-seeking, and get to pragmatics via "That which can be destroyed by the truth should be" and the notion of rational policies as those which you would retain even given an epistemically rational prediction of their consequences.

10 points

expand_less expand_more

6 points

expand_less expand_more

For most people, most of the things they want do in fact prefer some ways of thinking, so your definition requires us to consider a counterfactual pretty far from ordinary experience. In contrast, defining in terms of accuracy-seeking is simple and accessible. If this site is going to use the word "rational" a lot, we'd better have a simple clear definition or we'll be arguing this definitional stuff endlessly.

6 points

expand_less expand_more

Eliezer_Yudkowsky

3 points

expand_less expand_more

I usually define "rationality" as accuracy-seeking whenever decisional considerations do not enter. These days I sometimes also use the phrase "epistemic rationality".

It would indeed be more complicated if we began conducting the meta-argument that (a) an ideal Bayesian not faced with various vengeful gods inspecting its algorithm should not decide to rewrite its memories to something calibrated away from what it originally believed to be accurate, or that (b) human beings ought to seek accuracy in a life well-lived according to goals that include both explicit truth-seeking and other goals not about truth.

But unless I'm specifically focused on this argument, I usually go so far as to talk as if it resolves in favor of epistemic accuracy, that is, that pragmatic rationality is unified with epistemic rationality rather than implying two different disciplines. If truth is a bad idea, it's not clear what the reader is doing on Less Wrong, and indeed, the "pragmatic" reader who somehow knows that it's a good idea to be ignorant, will at once flee as far as possible...

3 points

expand_less expand_more

6 points

expand_less expand_more

You started off using the word "rationality" on this blog/forum, and though I had misgivings, I tried to continue with your language. But most of the discussion of this post seems to be distracted by my having tried to clarify that in the introductory sentence. I predict we won't be able to get past this, and so from now on I will revert to my usual policy of avoiding overloaded words like "rationality."

6 points

expand_less expand_more

mark_spottswood

4 points

expand_less expand_more

Eliezer said: This, in turn, ends up implying epistemic rationality: if the definition of "winning" doesn't require believing false things, then you can generally expect to do better (on average) by believing true things than false things - certainly in real life, despite various elaborate philosophical thought experiments designed from omniscient truth-believing third-person standpoints.

--

I think this is overstated. Why should we only care what works "generally," rather than what works well in specific subdomains? If rationality means whatever helps you win, than overconfidence will often be rational. (Examples: placebo effect, dating, job interviews, etc.) I think you need to either decide that your definition of rationality does not always require a preference for true beliefs, or else revise the definition.

It also might be worthwhile, for the sake of clarity, to just avoid the word "rationality" altogether in future conversations. It seems to be at risk of becoming an essentially contested concept, particularly because everyone wants to be able to claim that their own preferred cognitive procedures are "rational." Why not just talk about whether a particular cognitive ritual is "goal-optimizing" when we want to talk about Eliezer-rationality, while saving the term "truth-optimizing" (or some variant) for epistemic-rationality?

4 points

expand_less expand_more

Eliezer_Yudkowsky

4 points

expand_less expand_more

Maybe "truth-seeking" versus "winning", if there's a direct appeal to one and not the other. But I am generally willing to rescue the word "rationality".

4 points

expand_less expand_more

mark_spottswood

4 points

expand_less expand_more

Sorry -- I meant, but did not make clear, that the word "rationality" should be avoided only when the conversation involves the clash between "winning" and "truth seeking." Otherwise, things tend to bog down in arguments about the map, when we should be talking about the territory.

4 points

expand_less expand_more

2 points

expand_less expand_more

I agree – in contexts where 'truth seeking' and 'winning' are different, we should qualify references to 'rationality'.

2 points

expand_less expand_more

3 points

expand_less expand_more

Regarding "rationalists should win" - that still leaves us with the problem of distinguishing between someone who won because he was rational and someone who was irrational but won because of sheer dumb luck.

For example, buying lottery tickets is (almost always) a negative EV proposition - but some people do win the lottery. Was it irrational for lottery winners to have bought those specific tickets, which did indeed win?

Given a sufficiently large sample, the most spectacular successes are going to be those who pursued opportunities with the highest possible payoff regardless of the potential downside or even the expected value... for every spectacular success, there are probably several times as many spectacular failures.

3 points

expand_less expand_more

5 points

expand_less expand_more

A common example of where rationality and truth-seeking come into conflict is the case where organisms display their beliefs - and have difficulty misrepresenting them. In such cases, it may thus benefit them to believe falsehoods for reasons associated with signalling their beliefs to others:

"Definitely on all fronts is has become imperative not to bristle with hostility every time you encounter a stranger. Instead observe him, find out what he might be. Behave to him with politeness, pretending that you like him more than you do - at least while you find out how he might be of use to you. Wash before you go to talk to him so as to conceal your tribal odour and take great care not to let on that you notice his own, foul as it may be. Talk about human brotherhood. In the end don't even just pretend that you like him (he begins to see through that); instead, really like him. It pays."

Discriminating Nepotism - as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.359.

5 points

expand_less expand_more

Eliezer_Yudkowsky

3 points

expand_less expand_more

The truth is not beautiful... it stands on cloven hooves... it is covered in coagulated blood... but it is the truth.

3 points

expand_less expand_more

5 points

expand_less expand_more

Eek, now there's Transhuman babyeaters! I see it also says "Man needs lies like children need toys." :-)

5 points

expand_less expand_more

Eliezer_Yudkowsky

3 points

expand_less expand_more

Ew. Didn't know that was where it came from, just saw the demotivator.

3 points

expand_less expand_more

2 points

expand_less expand_more

Query: Need the quest for the truth necessarily be quixotic? Tilting at windmills would be an example of delusional activity. Isn't the quixotic then the opposite of the rational?

2 points

expand_less expand_more

2 points

expand_less expand_more

As I see it, rationality is much more about choosing the right things to use one's success for than it is about achieving success (in the conventional sense). Hopefully it also helps with the latter, but it may well be that rationality is detrimental to people's pursuit of various myopic, egoist, and parochial goals that they have, but that they would reject or downgrade in importance if they were more rational.

2 points

expand_less expand_more

2 points

expand_less expand_more

It may be possible to have it both ways, to know rationality without having it interfere with achieving happiness and other goals.

For me, rationality is a topic of interest, but I don't make a religion out of it. I cultivate a sort of Zen attitude towards raaitonalit, trying not to grasp it too tightly. I am curious to know what the rational truth is, but I'm willing and, with the right frame of mind, able to ignore it.

I can be aware at some level that my emotional feelings are technically irrational and reflect untrue beliefs, but so what. They're true enough. They're as true as everyone else's. That's good enough for me. I can still embrace them wholeheartedly.

Now how about overconfidence. The truth is that I am either lucky or unlucky on this issue, depending on how you look at it. I am not very overconfident. Modesty comes naturally to me. Asserting my opinions makes me uncomfortable. When someone else says I'm wrong, I take it very much to heart. This is just my personality. When I was younger I was more assertive, but as I've gotten older I find that my uncertainties have grown. Probably learning the rational truth on these matters has contributed to the change, but it is one which has come naturally to me.

Most people are different, but they can still know that their overconfidence is irrational and mistaken, without particularly acting any less confident. After all, they know that the other guy's confidence is just as inflated, so they are equally as justified in flaunting their excellence as anyone else.

Can one really use rationality like this? Listen to Lewis Carroll: "The question is, which is to be the master - that's all."

2 points

expand_less expand_more