Easy Predictor Tests

11 points

expand_less expand_more

Jan 21, 2010

A fun game you can play on LessWrong is to stop just as you are about to click "comment" and make a prediction for how much karma your comment will receive within the next week. This will provide some quick feedback about how well your karma predictors are working. This exercise will let you know if something is broken. A simpler version is to pick from these three distinct outcomes: Positive karma, 0 karma, negative karma.

What other predictors are this easy to test? Likely candidates match one or more of the following criteria:

Something we do on a regular (probably daily) basis
An action that has a clear starting point
Produces quick, quantifiable feedback (e.g. karma, which is a basic number)
An action that is extremely malleable so we can take our feedback, make quick adjustments, and run through the whole process again
An ulterior goal other than merely testing our predictors so we don't get bored (e.g. commenting at LessWrong, which offers communication and learning as ulterior goals)
Something with a "sticky" history so we can get a good glimpse of our progress over time

A more difficult challenge is predicting karma on your top level posts. My predictors in this area tend to be way off the mark. For this post, my guess is between +4 and +20. Reasoning: I don't see how it could get over 20 unless it gets promoted and the concept surprises readers; 4 seems like a solid guess for "interesting, uncontroversial, but not groundbreaking."

Update: As of January 29, this post is at +10.

11 points

expand_less expand_more

Comments

sorted by

Highlighting new comments since Today at 12:10 PM

Rendering 50/66 comments show more

5 points

expand_less expand_more

A simpler version is to pick from these three distinct outcomes: Positive karma, 0 karma, negative karma.

Personally, if I expect a comment or post to get negative karma, I don't make it. Perhaps that's too conservative on my part, but it's the case.

5 points

expand_less expand_more

5 points

expand_less expand_more

Voted down for reasons suggested by thomblake: it's often important to make a comment that you expect to get negative karma.

5 points

expand_less expand_more

5 points

expand_less expand_more

I make plenty of comments for which I expect negative karma. It's not too hard to shield unpopular views in language that will soften the karma hit, but sometimes it's better to state things plainly.

For a while, I had a rough handle on how various factors influenced the karma of a comment (like a link to a previous Lw post, or a relevant XKCD) but never crunched any numbers.

5 points

expand_less expand_more

6 points

expand_less expand_more

My intuition is that if you are about to make a comment for which you expect negative karma, nearly all of the time you should instead choose one of: a) send the comment in a private communication b) expand your comment with a more clear explanation of where you're coming from c) make your comment into a top-level post (or posts if the idea needs introduction) complete with your arguments and reasoning for all to learn from without the miscommunication perils of quick comments d) refrain from commenting

Sure, I'd bet there are exceptions, but making plenty of comments for which you expect negative karma? Strikes me as off. Pros: you can comment faster and more often since you don't need to thoroughly explain yourself, and since you don't need to prune as many threads. Cons: negative karma presumably means readers would rather have not read what you wrote, which is a bad sign unless maybe it's a topic that you feel will actually improve their lives even without sufficient explanation, even when after reading it they decide they wish they hadn't.

That last bit? Sure, maybe it happens, sometimes... maybe.

6 points

expand_less expand_more

3 points

expand_less expand_more

Cons: negative karma presumably means readers would rather have not read what you wrote, which is a bad sign unless maybe it's a topic that you feel will actually improve their lives even without sufficient explanation, even when after reading it they decide they wish they hadn't.

Not necessarily. You can, for example, make comments that you expect to improve the lives of a majority despite the displeasure of a majority.

3 points

expand_less expand_more

4 points

expand_less expand_more

I make plenty of comments for which I expect negative karma.

You are not well-calibrated.

4 points

expand_less expand_more

2 points

expand_less expand_more

I've been mulling over asking for a data set of posts to do some stats on. I've noticed that short pithy statements can get a lot of karma, and would like to see some stats of karma vs comment length.

2 points

expand_less expand_more

3 points

expand_less expand_more

I suspect that short, concise posts and long, thought-out ones both get higher karma than ones that fall in between.

3 points

expand_less expand_more

1 point

expand_less expand_more

I make plenty of comments for which I expect negative karma. It's not too hard to shield unpopular views in language that will soften the karma hit, but sometimes it's better to state things plainly.

I tend to share your philosophy. It is also not too hard to offset karma costs by making a few extra posts.

Interestingly, I am often surprised when posts that I am expecting to be a karma sacrifice turn out not to be. As you suggest, extra time phrasing potentially unpopular views probably makes some difference. Even stating things plainly can be done elegantly.

1 point

expand_less expand_more

3 points

expand_less expand_more

I made a comment recently that got a lot of karma, and I must confess, I did expect it. I printed out the thread and read it all before making it to make sure it wasn't already made, and I was quite proud of it, so I was really pleased people liked it.

3 points

expand_less expand_more

3 points

expand_less expand_more

Somewhat offtopic, but I'd like to see someone writing a GreaseMonkey script which hides the name of the commenter and the current score level on all comments, so you're not being influenced by the status of the commenter and/or the current score level on that comment. The commenter name could be seen with a mouseover so you can reply to it though.

3 points

expand_less expand_more

3 points

expand_less expand_more

You seem to be describing Marcello's anti-kibitzer.

3 points

expand_less expand_more

1 point

expand_less expand_more

I've been using that for a while. It's, not so much surprising as interesting, how much it changes one's outlook.

It doesn't deal very well with the "Recent Comments" sidebar, I added a .5s delay to wait while the AJAX loads. That's not quite satisfactory, but I don't know how to tell a GreaseMonkey script to wait for a specific AJAX query to load.

EDIT: OK, now I know. It would be helpful if LessWrong could be upgraded to a more recent version of the Prototype framework.

1 point

expand_less expand_more

1 point

expand_less expand_more

Cool idea. If there were an easy way to hide the scores on comments, you could wait a week to make the predictions and then immediately see if your predictions are correct. This would reduce the interval between making the prediction and seeing if it's correct, which presumably would provide better training and feedback.

1 point

expand_less expand_more