I don't know how well you know this person, so my advice may be unnecessary. But your post gives me the impression that you need to be much more careful about speculating on how her mind works. I think that it's a red flag when you write first that

I can see the pattern in the words coming out of her lips, but I can't understand the mind behind on an empathic level. I can imagine myself into the shoes of baby-eating aliens and the Lady 3rd Kiritsugu, but I cannot imagine what it is like to be her.

. . . and then proceed to make apparently confident declarations about how her mind works, such as

I now realize that the whole essence of her philosophy was her belief that she had deceived herself, and the possibility that her estimates of other people were actually accurate, threatened the Dark Side Epistemology that she had built around beliefs such as "I benefit from believing people are nicer than they actually are."

She has taken the old idol off its throne, and replaced it with an explicit worship of the Dark Side Epistemology that was once invented to defend the idol; she worships her own attempt at self-deception. The attempt failed, but she is honestly unaware of this.

As you yourself have observed, we largely understand other people by taking a portion of our own black-box mind, plugging in a few explicit settings (such as beliefs or experiences), letting the model run for a bit, and seeing what pops out. In particular, to understand how another person makes judgments, we collect their evinced beliefs, try to twiddle some dials until our model expresses the same beliefs, and then let it run for a bit. We then try to peer into the model as best we can, getting as good a picture of its inner workings as introspection allows us. We then take this picture as our hypothesis about how the other person thinks.

But the first quote above is strong evidence that your mind works differently from hers in some highly relevant respects. Therefore, you should be highly skeptical that what is going on in her mind resembles what it took to make the model of her in your own mind match her utterances. But you give me the impression that you haven't been sufficiently skeptical of the match between her mind and your model of it. I think that this has led you astray on several points.

For example, based on what you've written, I don't think that you're using the right model to understand what was going on in her mind when she said, "I believe that people are nicer than they really are." You were led to this confusion because she was not using the word "believe" in the way that you, and your model of her, do. You are using "belief" to mean a feature of a model of how the world is. But that, I expect, is not what she meant. Thus, your remarks here --

I tried to explain that if you say, "People are bad," that means you believe people are bad, and if you say, "I believe people are nice", that means you believe you believe people are nice. So saying "People are bad and I believe people are nice" means you believe people are bad but you believe you believe people are nice.

-- were irrelevant because they do not apply to the sense of the word "believe" that she was using.

For what it's worth, in my model of her, when she said "I believe that people are nicer than they really are," she meant, "When I reflect on my emotional attitude towards people, I see that this attitude is of the sort that, in the absence of its actual cause, could have been caused by a falsely high belief (in your sense) about peoples' niceness."

The actual cause for her emotional attitude is perhaps her "religion". Or perhaps it is something else. Perhaps she has no idea what the actual cause is, or perhaps she thinks she does, but she doesn't really. But none of this implies that she was attributing to herself the belief that people are nicer than she actually believes them to be (where, here, I'm using "belief" in your sense.)

Her utterance seems analogous to someone who walks out of an optometrist's office after having his pupils dilated and says, "Because of those drops the optometrist gave me, I believe the sun is brighter than it really is." If we heard this, we shouldn't conclude that he believes something contradictory, or that he has incorrect beliefs about his beliefs. His word "belief" in this case probably does not mean "best guess about how things really are." Rather, it's a clumsy way to say that some qualities of his experience of the world are as if he had a certain belief (in the sense normally understood). He does not mean to imply that he has any wrong beliefs (in the conventional sense). It would be a mistake to say that his subjective experience of the light is in any way erroneous. After all, it accurately reflects the fact that he had those drops put in his eyes.

Similarly, your interlocutor's statement that she "believes" that people are nicer than they really are referred to a particular quality of her emotional attitude towards them, not to a belief (in your sense) about how they are. In particular, it didn't imply any expectation about how they would behave. That, I expect, is why she was initially taken aback when you asked, "So, are you consistently surprised when people undershoot your expectations?" The problem wasn't, as you appear to think, that she had prevented her own mind from drawing obvious conclusions. The problem was that you (because of her confusing wording) were speaking of her so-called "belief" as though it were a belief in the normal sense, something that should lead to certain expectations about other peoples' actions. But I expect that it wasn't any such thing, notwithstanding her unfortunate choice of words.

I'll go one step further and defend belief in belief, infinitely regressed. ;-) As you point out, the placebo effect here is simply the expectation of a positive result -- and it applies equally at any level of recursion here.

Humans only need a convincing argument for predicting a positive result, not a rational proof of that prediction! Once the positive result is expected, we get positive emotions activated every time we think of anything linked to that result, leading to self-fulfilling prophecies on every level.

This being the case, one might question whether it's rational to disbelieve in belief, if you have nothing equally beneficial to replace it with.

When it comes to external results, sure, it makes sense to have greater prediction accuracy. But for interior events -- like confidence, creativity, self-esteem, etc. -- biasing one's predictions positively is a significant advantage, as it stabilizes what would otherwise be an unstable system of runaway feedback loops.

People whose systems are negatively biased, on the other hand, can get seriously stuck. They basically hit one little setback and become paralyzed because of runaway negative self-fulfilling prophecy.

(I've been such a person myself, and I've worked with/on many of them. Indeed, it was noticing that other, far less "rational" and "intelligent" individuals were much more confident, calm, and successful than I was, that led me to start seriously investigating the nature of mind and beliefs in the first place, and to begin noting the distinctions between people I dubbed "naturally successful" and those I considered "naturally struggling".)

Consistent consciously intended self-deception may be hard. But our minds are designed to produce self-deceptions all the time without us noticing. Just don't look behind the curtain and "let it be", "go with the flow" etc. and you can be as self-deceived as most folks.

If I had been talking to the person you were talking to, I might have said something like this:

Why are you deceiving yourself into believing Orthodox Judaism as opposed to something else? If you, in fact, are deriving a benefit from deceiving yourself, while at the same time being aware that you are deceiving yourself, then why haven't you optimized your deceptions into something other than an off-the-shelf religion by now? Have you ever really asked yourself the question: "What is the set of things that I would derive the most benefit from falsely believing?" Now if you really think you can make your life better by deceiving yourself, and you haven't really thought carefully about what the exact set of things about which you would be better off deceiving yourself is, then it would seem unlikely that you've actually got the optimal set of self-deceptions in your brain. In particular, this means that it's probably a bad idea to deceive yourself into thinking that your present set of self deceptions is optimal, so please don't do that.

OK, now do you agree that finding the optimal set of self deceptions is a good idea? OK, good, but I have to give you one very important warning. If you actually want to have the optimal set of self deceptions, you'd better not deceive yourself at all while you are constructing this set of self deceptions, or you'll probably get it wrong, because if, for example, you are currently sub-optimally deceiving yourself into believing that it is good to believe X, then you may end up deceiving yourself into actually believing X, even if that's a bad idea. So don't self deceive while you're trying to figure out what to deceive yourself of.

Therefore, to the extent that you are in control of your self deceptions, (which you do seem to be) the first step toward getting the best set of self deceptions is to disable them all and begin a process of sincere inquiry as to what beliefs it is a good idea to have.

And hopefully, at the end of the process of sincere inquiry, they discover the best set of self deceptions happens to be empty. And if they don't, if they actually thought it through with the highest epistemic standards, and even considered epistemic arguments such as honesty being one's last defence, slashed tires, and all that.... Well, I'd be pretty surprised, but if I were actually shown that argument, and it actually did conform to the highest epistemic standards.... Maybe, provided it's more likely that the argument was actually that good, as opposed to my just being deceived, I'd even concede.

Disclaimer: I don't actually expect this to work with high confidence, because this sort of person might not actually be able to do a sincere inquiry. Regardless, if this sort of thought got stuck in their head, it could at least increase their cognitive dissonance, which might be a step on the road to recovery.

I await the eager defenses of belief in belief in the comments, but I wonder if anyone would care to jump ahead of the game and defend belief in belief in belief? Might as well go ahead and get it over with.

My boyfriend was once feeling a bit tired and unmotivated for a few months (probably mild depression), and he also wanted to stop eating dairy for ethical reasons. He felt that his illness was partly mentally generated. He decided that he was allergic to dairy, and that dairy was causing his illness. Then he stopped eating dairy and felt better!

He told me all this, and also told me that he usually believes he is actually allergic to dairy, and it is hard to remember that he is not. When someone asks how he knows he is allergic to dairy, he says something plausible and false ("The doctor ran blood tests") and believes it if he doesn't stop and think too much.

He believes he is not allergic to dairy, but he believes he believes he is allergic to dairy? Belief-in-belief. But he recognizes this and explained it to me -- so that's a belief-in-belief-in-belief? But it helped him get over his mental illness and stop eating dairy... that's winning.

In general I would say a belief-in-belief is useful if you decide some behaviors are desirable, but some false model of the world better motivates you to behave properly. Belief-in-belief-in-belief is useful if you know too much to think both "Z is true" and "I believe not-Z". Then you tell yourself you have a belief-in-belief.

Disclaimer: This is weird to me and I don't really understand how he pulls it off.

"I believe that people are nicer than they really are." That part made me ponder. Because, actually, it's something I believe, too. So I froze for a while, and looked at that belief. Do I have Escher loops in my belief networks ? Well, maybe, I'm far from being a perfect bayesian, but I can't allow myself to stop here.

My first justification for that thought was : I don't refer to the same thing in the two parts of the sentence. A bit like "sound" can refer to acoustic vibrations, or to a perception, and if you switch from one to the other into the same sentence, you can make a sentence that seems self-contradicting but is still valid.

People is a vast group. Nice or not is a characteristic of a person. So, to attribute "niceness" to people in general, you've to make an aggregate value. There are many ways to make an aggregate value, for example, mean and median. So that sentence could mean something like "I believe the median people to be nicer than the average people" (implying a minority of very un-nice people who drive the average backward, but don't change the median).

But then I thought "Hey, stop. You're trying to find excuses here. That's not really what you meant with that sentence, or you would have said it clearly. Don't find yourself excuses, just face the fact you were doing knots with your believes."

So I tried to dig where this knot could came from. And I think I found it, and it's linked to the first excuse, but not as simple. The problem comes from the fact that I use different algorithms to evaluate the niceness of a single person I'm interacting with (be it a friend/family, or just a passerby asking me "what time is it ?") and to evaluate the overall nicest of "people" in general.

When I evaluate niceness of people in general, I think about the horrors of history, about the Milgram experiment, about the crimes the news love to report about, the scary statistics about the number of husbands who hit their wives, ... And also about the "heroes", those who did risk their lives to hide unknown Jews during WW2, those who did run in a house of fire to save their neighbor. That gives me a mitigated image of people in general, neither very nice nor very un-nice, able of the best and of the worst.

When I evaluate niceness in one single person interacting with me, I tend more to recall my own interactions with individual persons. And in those, I had a few bad memories (like being assaulted to steal my money and cell phone once) but mostly good memories, be it by luck or by selective memory, most of the interactions with others I can remember were mostly positive. So when I interact with a new individual person, I assume there is a huge chance of that person being "nice", even if I have a more mitigated view of humans in general.

That probably comes from deeper, evolutionnary psychology reasons : the individual your interact with are your tribe, they are friendly. People in general are other tribes, not so friendly. But I'm not well versed enough into evolutionnary psychology to go further on that line. But anyway that's, I think, where the contradiction comes from. It may be partly justified by the fact that the median is higher the average, if it really is (which I've no factual evidence of, only a vague feeling). But it mostly comes from just using to different algorithms, which should, in a well-calibrated brain, lead to the same result, but which for many reasons (all the biases, imperfect knowledge, ...) just don't.

"I wish I could believe that no one could possibly believe in belief in belief in belief..."

You wish you could believe Eliezer? Is this a dliberate stroke of irony or a subconcious hint at the fact that you do have an empathic understanding of the thought processes behind tailoring your own beliefs?

Hrm... While on the one hand I can look at her position and basically react with a "your mind is entirely alien to me", on the other hand, I can actually imagine being in that state.

That does NOT mean, of course, that it is a reasonable state to be in, but it does seem to be the sort of state that my mind can support.

I guess the basic key is that human minds aren't necessarally naturally consistent. So we can end up in actual inconsistent states. Including states a bit confused about consistency itself.

A bit more of a personal example would be a state I sometimes recall having been in in the past, and have certainly seen in others, would be when one might say something like, oh, I dunno, "and scientifically, the universe is about 13.7 billion years old and earth is about 4.5 billion years old" and of course, the world was created about 6000 years ago."

As near as I can tell, happens is that we almost imagine the "scientific world" and the "religious world" as parallel universes that... are actually the same one, so mentally we keep track of it by keeping track of different things.

The way this works is someone might manage to end up in a state that they completely fail to really face the question of "okay, but if you rewind time a bit, will you see 6000 years ago the universe poofing into existence, or can you go farther back, etc? ie, what ACTUALLY happened in ACTUAL REALITY?"

Then, when facing that question, all sorts of Escher mentality stuff starts forming as a defense. But what I think initially happens, at least in part, is sort of mentally tracking those as being about different subjects, rather than contradictory statements about the same thing. So that one will end up, with "science glasses", visualizing prehistoric humans doing stuff tens of thousands of years ago, while etc etc etc...

At least, that's my own, partly introspective model of what's going on here, of how people can end up in these states.

I am so much a one-level person that my sense of social insincerity has atrophied.

Rational straight man syndrome. So much a truth-finder you forget how to not speak the truth.

"And so humanity's token guardians of sanity (motto: "pooping your deranged little party since Epicurus") must now fight the active worship of self-deception - the worship of the supposed benefits of faith, in place of God."

• As I keep saying, helping people to overcome biases (such as the above) is a lot easier if there are psychologically viable places for people to jump to once they've overcome their bias.

You should have spent much more of your time in this debate convincing your tangled friend that, if she were to abandon her religious belief (or belief in belief, or whatever), she would still be able to feel good about herself and good about life; that life would still be a happy meaningful place to be.

Maybe she has a massive internal guilt complex and thinks of herself as a bad person, and she thinks that only religion can help her with this. Maybe she is frightened that atheism will lead to nihilism.

Why destroy placebo effects? According to some stuff Robin Hanson points to, it seems that most of medicine might consist of placebos. Aren't you fighting what wins in favor of the truth?

As a theist, I don't believe in God because I perceive some positive benefit from that belief. My experiences and perceptions point to the existence of God. Of course those experiences and perceptions may be inaccurate and are subject to my own interpretations, so I can't claim that my beliefs are rational. I accept on an intellectual level that my belief could be wrong. This doesn't seem to enable me to stop believing.

However, I am involved in a religious community because there are positive benefits -- chiefly that of being able to compare notes with other people who share my irrational belief in God and my desire to do good work in the world. I can see that there might be positive benefits in religious communities for non-theists, though I don't really see the point.

Voltaire, using rationalist arguments, concluded that “if God did not exist, it would be necessary to invent him”. So could it be that adhering to facts in all situations is essentially an irrational position?

Consider the following statements:

1) Rational humans (unlike rational AI) should aim to be happy.

2) Rational humans should not believe fanciful notions unsupported by empirical evidence.

3) Empirical studies (e.g. http://www.lifesitenews.com/ldn/2008/mar/08031807.html) suggest that humans who believe in such notions are more likely to be happier.

The consequence of the above statements seems to be that a rational human should reject rationality.

Does anyone see flaws in this reasoning?

I know some people who are like the woman you describe, my own folks might be like that to some extent. I became atheist pretty early on. So I'm not sure that adults who believe in belief are likely to be passing that along to their kids, if they even try. In my case, I put on a show for a while, but when I stopped it was no big deal.

If these people are able to agree with a scientific worldview and not be obstructionist on things like stem cell, but simply want to add "and I believe there is a god" to the end of it, fine. Seems like a natural step towards the end of belief in god entirely.

To further illustrate the point that self-deception isn't easy: if believe you're shy, you can't just make yourself believe you're not shy.

Maybe you can make yourself believe that you believe that you're not shy, but I don't think you'll reap many benefits from placebo effect - you'll still get nervous when you want to speak up or go talk to a girl you don't know or whatnot. You can't argue yourself logically into self-confidence.

It sounds to me that she simply is using a different definition of "to believe". If she says "I believe people are nicer than they are," I think she means something like, "I choose to act as if people are nicer than they really are, because it is consonant with my sense of morality to do so." It's choosing to give people the benefit of the doubt, knowing they probably don't deserve it.

Placebo effects from 'belief in (false) beliefs' only work as long as self-deception is maintainable.

I think the point at which self-deception ceases to work is when you can consciously be aware of it breaking your causal models of the world. Highly intelligent people, or anyone for that matter, cannot continue to deceive themselves into believing in god or unregulated markets, or whatever complex concept take your pick, if you explicitly show how it breaks a model they cannot disagree with. Controversial topics of the day like belief in god, public policy, etc. are not single data points under contention, but tangled balls of causation that must be dealt with in a somewhat parallel fashion--to see the big picture and say, wait a minute that cannot fit unless this, and this, and this, and finally reach a dead end and have to relinquish their starting belief. The more abstract or the more complex a concept is, the easier it is to deceive yourself of its falsehood.

The limits to working memory plays a role here, and if we are to truly be less wrong, we not only have to overcome biases, but we need to amplify our rational intelligence by using tools designed for these specific purposes. What if beliefs such as 'a personal god exists' were as hard to believe in as 'the sky is green'? What if it was explicitly laid out in front of someone that they absolutely could not hold a belief in something because of all the cascading links it breaks in their world model that is confirmed to be 'reality'.

I want to work on such tools.