"If you believe A => B, then you have to ask yourself: which do I believe more? A, or not B?"

-- Hal Daume III

To rip off Steven Kaas for a moment, I wonder if rather than saying "Study shows existence of psychic powers" it might be better to say "Nonexistence of psychic powers shows a study was wrong".

Paper criticizing the statistical analysis here:

http://www.ruudwetzels.com/articles/Wagenmakersetal_subm.pdf

From the conclusion:

In eight out of nine studies, Bem reported evidence in favor of precognition. As we have argued above, this evidence may well be illusory; in several experiments it is evident that Bem’s Exploration Method should have resulted in a correction of the statistical results. Also, we have provided an alternative, Bayesian reanalysis of Bem’s experiments; this alternative analysis demonstrated that the statistical evidence was, if anything, slightly in favor of the null hypothesis. One can argue about the relative merits of classical t-tests versus Bayesian t-tests, but this is not our goal; instead, we want to point out that the two tests yield very different conclusions, something that casts doubt on the conclusiveness of the statistical findings.

what sort of p-values would you need to see in that paper in order to believe with, say, 50% probability that the effect measured is real?

P-values won't do it. Psi experiments consistently have high sounding significance because the trials are so large. Mass replication is what will make me believe.

I would need considerably more than one study. That said, I think it is really good news this is getting published in a real journal. Parapsychologists have been publishing interesting results for years at strong enough levels that the publication bias would have to be really high to explain it. On the recommendation of someone here I read Outside the Gates over the summer which makes a moderately convincing case something weird is going on. I don't assign nearly the same credence to the results that the author does but it did convince me that mainstream science should be looking at it. At worst this will help force psychology to confront publication bias and some of the statistical issues that plague the field generally. And at least we should see some non-parapsychologists attempting to replicate this.

I've been meaning to write a book review.

Part of the issue is that when we do see events that look like psy they aren't ever at the p-values that would be conclusive. If there is something like psy it isn't that strong so you need replication.

It would also be good to get the studies out of the hands of the New Age crazies and into the hands of some reductionists who could go to work theorizing. Though of course the most likely explanation remains publication bias/fraud/methodological issues.

I'll look over the study later tonight. Thanks for posting it.

One experiment can't convince my of psi, the chance that they might have done the experiment badly in some way puts an upper bound on how much evidence it can provide. Only multiple independent verifications could do it, and even then I'd still feel massively confused.

His name is Bem by the way, as opposed to Bern (although I've seen that typo in multiple places).

This has been discussed a few places on LW already. See for example here.

I skimmed the paper's section on randomness and random number generators. I don't think they're making mistakes there.

what sort of p-values would you need to see in that paper in order to believe with, say, 50% probability that the effect measured is real?

A lot more than one study. But the other issue is, if the measured effect is real, the correct response is, "I see that I am confused," not, "They must have psychic powers!" Unexpected stuff happening in an experiment is a rather more probable prior than people having psychic powers. Moreover, you'd need very precise experiments to figure out exactly what's going on. This experiment may in fact show more about the invalidity of priming - if priming works in either direction, and people don't have psychic powers, then something other than the priming is probably responsible for their reactions.

It is a very, very common error in psychology to take limited experimental evidence with "statistical significance" and then infer extremely complicated and precise notions of human psychology from those, despite the fact that such precise explanations would require vastly more evidence. (A hypothetical example would be: Men respond to red letters at a faster rate than women, as compared with blue letters. Therefore, the color red must have had some specific importance in hunting during our evolutionary past. The latter observation is an arbitrarily privileged hypothesis with no real evidenciary support over thousands of other hypotheses. I've seen more absurd jumps, as well as people simply ignoring evidence that did not conform to their prior theory.)

More critique at the James Randi forums.