0 points

expand_less expand_more

In reply to:

I think you should have posted just the links to the questions and the instructions on which link to follow without explaining what the experiment is about.

That would be certainly a reasonable way to do it, however I suppose that the LW audience is well informed about the biases and able to guess what it is all about even without being explicitly told. Meanwhile I wanted to provide some explanation why I am doing this, and to publish the tested hypothesis in advance (as a measure of honesty).

The effect seems robust enough to survive when people know that they are being tested for it.

See comment in full context

An Anchoring Experiment

12 points

expand_less expand_more

Apr 1, 2011

The experiment is closed, for the results look here.

In recent discussion I have expressed an opinion that anchoring may, for some quantitative questions, cause the answer to lie further away from the correct value than the anchor itself. For concreteness, let's suppose that the correct value of a quantity Q is x, and the subject is asked whether Q is greater or lower than y, y > x. My hypothesis is that the anchor moves the subject's probability distribution up as a whole, including the part which already has been lying above y. Therefore the subjects will positively answer the question "Is Q > y ?" more often than their guess would exceed y if they were just asked to estimate the value of Q with no anchor given. One commenter apparently disagreed. I thought it may be interesting to resolve the disagreement experimentally. (More generally, I would like to see how well LW audience fights the standard biases, and if this experiment turns out successful - which means the number of respondents be greater than, say, five - I would think about posting more of this kind.)

How to participate:

The experiment has two parts.

First, toss a coin to decide whether you belong to the biased group I or the control group II for the first question. If you belong to the group I, look at a comment linked below, which will give you a question of form "is Q is greater or lower than y", where y is either significantly lower or significantly greater than the correct value of Q. The comment has a form of a typical LW poll. If you belong to the group II, look at different linked comment which asks "what is the value of Q", and then give your estimate in a subcomment there.

The second part is completely analogical to the first one, only with a different question. If you have participated in the first part within the group I, take part in the group II for the second part, and vice versa. Try to eliminate the irrelevant biases: switch on the anti-kibitzer before looking on the group I questions to avoid being influenced by the votes of others. Don't read the subcomments of the group II questions before writing down your own.

The hypothesis is that the percentage of the group I respondents answering incorrectly will be greater than the percentage of the group II respondents estimating on the incorrect side of the anchor.

First part: Question for the group I. Question for the group II.

Second part: Question for the group I. Question for the group II.

12 points

expand_less expand_more

Comments

sorted by

Highlighting new comments since Today at 1:08 AM

All comments loaded

3 points

expand_less expand_more

Uh, some people's answers are short enough that their numbers are showing up in the 'Recent Comments' section. Please make a buffer in your comments prior to the answer if it's short (I don't think spaces work, but I'm not sure.) so this doesn't happen.

3 points

expand_less expand_more

1 points

expand_less expand_more

Thanks. I have been already given a notice about that and have included a similar instruction in the relevant comments.

1 points

expand_less expand_more

1 points

expand_less expand_more

Your sample buffer is one character too short.

1 points

expand_less expand_more

1 points

expand_less expand_more

Fixed.

1 points

expand_less expand_more

rhollerith_dot_com

2 points

expand_less expand_more

My coinflip put me in group II, but since I was reading /comments/ I saw one of the anchors intended for group I, so I decided that for me to continue would bias the results.

In addition I was primed (anchored? terminology?) by the answer of another subject in group II, and this time the problem could not be blamed on my having been reading /comments/.

Nice try, but I doubt the method of collecting responses you have chosen captures the aspects of reality you wish to investigate. Maybe you could redo the collection of responses by having respondents email their responses. (ADDED: rot13ing the response like muflax did works too.) If you do redo the collection of responses, I humbly suggest that you hide the instructions given to group I from group II -- e.g. by putting them behind a hyperlink to another page.

2 points

expand_less expand_more

1 points

expand_less expand_more

Thanks for feedback. I know that it is difficult to avoid seeing the comments of others. The problem with e-mail is that it is marginally less convenient to do, and I was afraid that people would have not enough motivation to respond.

As for hiding the instructions, I don't see a strong reason. I suppose priming is a strong enough effect to persist even if people know they are being tested for it.

1 points

expand_less expand_more

rhollerith_dot_com

1 points

expand_less expand_more

As for hiding the instructions, I don't see a strong reason.

If the coin flip puts me in group II, does not my inadvertently seeing the question intended for group I (e.g., "Is the population of the Central African Republic lower or greater than 20 million?") de facto reassign me to group I? In other words, after I had been exposed to the anchor (20 million) would not my continued participation in the control group have undermined the purpose of the control group?

ADDED.

I suppose priming is a strong enough effect to persist even if people know they are being tested for it.

I tend to agree, but that's not it.

1 points

expand_less expand_more

0 points

expand_less expand_more

after I had been exposed to the anchor, would not my continued participation in the control group have undermined the purpose of the control group?

Agreed. It would have been better to always answer the group-II question first, whichever one you were assigned to. (That is, switch the part 1 group II question with the part 2 group I question.)

0 points

expand_less expand_more

1 points

expand_less expand_more

We should vote down the group II questions as well.

1 points

expand_less expand_more

1 points

expand_less expand_more

I suggest upvoting the main post and downvoting the group II question you answered.

Edit: At least as a stopgap until prase makes a proper karma balance comment.

1 points

expand_less expand_more

1 points

expand_less expand_more

Err, I think I screwed this up. I thought if you were in group I for the first part, you'd also be in group I for the second part. I still think this would be more intuitive (just reverse the questions in part 2).

1 points

expand_less expand_more

0 points

expand_less expand_more

From a quick eyeballing of the results so far it does seem that the anchors are having the expected effect, even though we know that we are being anchored.

0 points

expand_less expand_more

0 points

expand_less expand_more

Anti-kibitzer is acting odd in my Google Chrome browser - it's not bolding links and all the comments are shown in full, rather than truncated to a single line.

0 points

expand_less expand_more

0 points

expand_less expand_more

I think you should have posted just the links to the questions and the instructions on which link to follow without explaining what the experiment is about.

0 points

expand_less expand_more

0 points

expand_less expand_more

That would be certainly a reasonable way to do it, however I suppose that the LW audience is well informed about the biases and able to guess what it is all about even without being explicitly told. Meanwhile I wanted to provide some explanation why I am doing this, and to publish the tested hypothesis in advance (as a measure of honesty).

The effect seems robust enough to survive when people know that they are being tested for it.

0 points

expand_less expand_more