How to Tell Good Studies from Bad? Bet on Them

It is funny they were worried about whether they just got lucky with their result, then did the prediction market thing, and then didn&#x27;t worry whether they just got lucky with <i>that</i> result! (At least the article didn&#x27;t, perhaps the researchers did.) So here are some amateur stats, please check my work.<p>This article says that the prediction market correctly predicted 71% of the replication results of 44 studies, or 31 correct.<p>Assume the studies have a 50% chance of being replicable. Then a random coin would predict a mean of 22 correct with a std dev of sqrt(0.5 * 0.5 * 44) = 3.3. This sample has a z score of 2.72, which means there&#x27;s a probability of 0.003264 (0.3%) of the random chance approach being correct 71% or better. So the result seems pretty significant. (Changing the assumed 50% to other values makes the probability even more extreme.)

This is a good answer from the incentives angle - how to motivate people to check whether studies are good or bad. On the object level, the answer is surprisingly simple: actually read the thing. The press is full of stories where a journalist rephrased another journalist&#x27;s story about a press release about a study, and when you actually go to the study, it says something subtly different. When I see those, I try to jump out of the journalist&#x27;s summary and get to a PDF as fast as possible, guess what the caveat is going to be, then check for it.

&quot;The beauty of the market is that we allow people to be Bayesian&quot; [...] &quot;People come in with some prior belief, but they can also follow prices to see what other people believe and may update their beliefs accordingly [...] participants in the market could focus their bets on the studies they felt most sure of, and as a result, rough guesses didn’t skew the averages as much.&quot;<p>It certainly isn&#x27;t a fool-proof method of increasing accuracy, and it does favor popularity of a theory over other factors, but overall it&#x27;s probably a nice layer of data to consider adding to the mix.

The problem with this is that it seems likely to be biased against unexpected results or results that contradict the dominant theory. The old saying, &quot;Science advances one funeral at a time,&quot; has a lot of truth in it.

I really like the idea at the end of using prediction markets to figure out which studies should be challenged by attempting replication.

I&#x27;ve often though there should be a similar mechanism for solving disagreements in the workplace.

The cost of a contract doesn&#x27;t represent whether the result is reproducible or not, it predicts the <i>probability</i> that it&#x27;s reproducible.<p>So what do they mean when they say it correctly predicted the outcome? Are they just saying the odds fell on the same side as the reproduction indicated?<p>If so, that seems arbitrary. If the cutoff for a p-value is 0.05, then shouldn&#x27;t we say that any contract selling for less than $95 predicts a reproduction failure?

So in other words, a bunch of people guessing which hypotheses were true was more accurate than actual scientific studies of them (71% vs 39%). Great.

&gt;With a p-value [of 0.01], the result hardly screamed “false positive” like a barely significant one of, say, 0.05 might.<p>Is 0.01 that low for such a crazy finding? Let&#x27;s say you believe that it has a probability of 1 in 10,000. And that result really seemed really really unlikely. 1 in 10,000 might be generous. Then, after this study, the probability that it&#x27;s true is 1%.

Prediction markets are great tools in general. Unfortunately, incentives are usually against implementing them. Experts are easier to control.

isn&#x27;t that just how options or event prediction exchange &#x2F; markets work?