"The Journal of Real Effects"

13 points

Mar 5, 2012

Luke's recent post mentioned that The Lancet has a policy encouraging the advance registration of clinical trials, while mine examined an apparent case study of data-peeking and on-the-fly transformation of studies. But how much variation is there across journals on such dimensions? Are there journals that buck the standards of their fields (demanding registration, p=0.01 rather than p=0.05 where the latter is typical in the field, advance specification of statistical analyses and subject numbers, etc)? What are some of the standouts? Are there fields without any such?

I wonder if there is a niche for a new open-access journal, along the lines of PLoS, with standards strict enough to reliably exclude false-positives. Some possible titles:

The Journal of Real Effects
(Settled) Science
Probably True
Journal of Non-Null Results, Really
Too Good to Be False
_________________?

13 points

expand_less expand_more

CarlShulman

Comments

sorted by

Highlighting new comments since Today at 1:01 AM

All comments loaded

Vladimir_Nesov

8 points

expand_less expand_more

6 years ago

"Journal" might be a wrong concept, this service looks like something that could be decoupled from publishing.

8 points

expand_less expand_more

Viliam_Bur

4 points

expand_less expand_more

6 years ago

I imagine it would not contain articles, only information about reliability of statements. Such as "result R published in article A was replicated successfully, here are the relevant data". So when someone reads an article, they could come to this service to check its reliability.

How should such thing be organized?

4 points

expand_less expand_more

CarlShulman

3 points

expand_less expand_more

6 years ago

Wikis, Consumer Reports, Cochrane Reports, etc. The virtue of journals would be to reduce switching costs for academia.

3 points

expand_less expand_more

gwern

4 points

expand_less expand_more

6 years ago

I know there are already journals which require pre-registration; perhaps one could investigate how successful they have been. (We've known about these issues for a long time, but they haven't been fixed - which suggests incentive problems....)

4 points

expand_less expand_more

CarlShulman

1 points

expand_less expand_more

6 years ago

Yes, I'm curious about the barriers if anyone knows more.

1 points

expand_less expand_more

CasioTheSane

0 points

expand_less expand_more

5 years ago

Would one get banned from this journal for registering the testing of 100 only slightly different hypotheses at p=0.01?

0 points

expand_less expand_more