Grad Student Who Took Down Reinhart and Rogoff Explains Why They're Wrong

I have argued in the past that data and code need to be submitted with academic papers: <a href="http://blog.jgc.org/2013/04/the-importance-of-open-code.html" rel="nofollow">http://blog.jgc.org/2013/04/the-importance-of-open-code.html</a> This is just the latest example of why, IMHO, code needs to be open.

For me this was the 'tl;dr' quote:<p>"It would be absurd to think that governments never have to worry about their level of indebtedness. The aim of our paper was much more narrowly focused. We show that, contrary to R&#38;R, there is no definitive threshold for the public debt/GDP ratio, beyond which countries will invariably suffer a major decline in GDP growth."

ErSo, there are a few things I don't quite understand. In the original study, Renhart had erroneously averaged 7 numbers together to get a slightly negative growth. The 'correct' result is obtained from averaging eight numbers together, resulting in slightly positive growth. What is unclear to me, is how are either of these numbers considered to be very significant? With so few samples in the average, I would draw the conclusion that both measurements agree with one another and the error bars for their measured result are quite a bit larger than they seem to be suggesting. Is this common in economics?

Seems like a great argument for more transparency. This guy wasted a lot of time trying to guess what they had done. Given that publishing all the supporting materials is approximately free, perhaps journals could start requiring a git link that contains data, code, and paper drafts.

From Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-access journal) comes a set of general suggestions<p>Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020<p><a href="http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full" rel="nofollow">http://www.frontiersin.org/Computational_Neuroscience/10.338...</a><p>on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. It appears that the discipline of economics research needs help with data openness too.<p>"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."

It's not related to the article, but rather the site hosting it.<p>I noticed more than one fishy looking 3rd party domain loading while the page downloaded, so went into Inspector to see what was up. There are one or more resources loaded from each of the following domains, in many cases including javascript...<p>2mdn.net, scorecardresearch.com, bizographics.com, tynt.com, optimizely.com, google.com, sail-horizon.com, facebook.com, 247realmedia.com,akamaihd.net, vizu.com, pubmatic.com, imrworldwide.com, advertising.com, googlesyndication.com, doubleclick.net, chartbeat.com, sharethrough.com, fbcdn.net, skimresources.com, gstatic.com, stumbleupon.com, tynt.com, adadvisor.net, youtube.com, shareth.ru, agkn.com, yimg.com<p>'shareth.ru' seemed particularly suspect, until I realized it was probably sharethrough.com trying to be cute.<p>There are so many domains being trusted here the drive-bys could have drive-bys.

Its fascinating that part of the problem in the Reinhart and Rogoff paper comes down to an Excel formula error <a href="http://nymag.com/daily/intelligencer/2013/04/grad-student-who-shook-global-austerity-movement.html" rel="nofollow">http://nymag.com/daily/intelligencer/2013/04/grad-student-wh...</a>

Journals shouldn't publish papers without all the supporting data and computer code necessary to reproduce the result.<p>This seems obvious, but established academics have a vested interest in not sharing. Sharing data and code opens their work up to impeachment (as is seen here), and it gives others a jumping off point to extend their work.<p>Both of these side effects of sharing code and data are bad for the careers of successful academics, but good for literally everyone else in the world (including, ironically, these academics). It should be a no brainer, but then again, who referees these papers? The very people who have the most to lose by such a change.<p>It will take a lot of clamoring from the outside to bring about a world where scientific work is considered not legitimate without full documentation of the experiment performed.

This was an interesting and well written followup but I'm disappointed in Herndon's decision to publish it on Business Insider. BI is generally full of blogspam and has interstitial ads.

More debate over 20 data points with no explanatory power one way or the other (as Yglesias pointed out). This is not uncommon in macro. Another recent case also centered on work by R&#38;R<p><a href="http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-that-strong-growth-has.html" rel="nofollow">http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-th...</a><p>(one of a host of commentary pieces on both sides... but essentially none in the middle)

I think this method of averaging should hereby be called the Reinhart-Rogoff Average, RRA for short.<p>I should program it into my business intelligence software, it's gotta be useful for something--like for cooking the books.

This whole situation makes me think: is there a static analysis tool for excel? Seems like missing a few rows in an excel calculation could be flagable.

That grad student won't be getting the big job offers in industry, great way to destroy your career by biting the hand that feeds you.<p>(Sarcastic, but it's real life)