I've thought about this problem before. I'm a neuroscientist, and I would often like to try out other people's modeling techniques on my data, but since the vast majority of published papers do not have corresponding published implementations, I have to implement the algorithms myself, and hope I'm doing the same thing the paper did, and hope they did the same thing the paper says. There is some pedagogical value to this, since it ensures a good understanding of how the algorithms work, but often this pedagogical value is limited. In many cases the main innovation in the paper is a better optimization algorithm and not a different way of framing the problem.<p>There is little incentive to publish your code. Refusing to give away your implementation does not in any way constrain your ability to publish, and giving away your code has only minimal benefits for your career. On the other hand, it's risky, since someone might find a bug in your implementation that changes your results. Additionally, a competing lab might show their algorithm is better than yours, or worse, improve your algorithm and publish a higher impact paper based on it, which might affect the profile of your publication. Finally, publishing your code means you have to package it in a way that is usable by others. Given these facts, it is not entirely surprising that most people would not publish their code.<p>Since science is pretty decentralized, it's hard to achieve the kind of large-scale change in behavior you'd need to make code sharing standard. The only people who could simply decide that people should publish their code and make it happen as a consequence are the funding agencies (e.g. NSF), who have only recently begun thinking about data sharing and have yet to make code sharing a priority.<p>One thing I'd like to see happen is a "viral" AGPL-like license for scientific code. Code would be freely redistributable, but if it is used in a commercial product <i>or publication</i>, the license should require that the modified work be freely redistributed. Given the choice between building on publicly available code or writing everything from scratch, I feel that most researchers would choose the former, even if it meant releasing their code as well.
Good article. Bad title for wider audiences though. I expected a faux philosophy essay from a narrow minded empiricist. Instead I got an interesting article about software publication within linguistics.
Bad tittle: a better one would be "Empiricism Should Not Be a Matter of Faith", because our current 'scientific method' is very, very broken.
I don't exactly remember where (I think it was a project trying to reproduce results by pharmaceutical research) but I remember that this project could only reproduce results <i>half the time</i> on papers which included 'double blind' experiences, less than that when the paper didn't have 'double blind' experiences!
Think about it when you'll hear about the new 'discovery' :-( :-(
From a practical standpoint, the IPyton Notebook [1] seems to be a good starting place to enable sharing of software and reproducing results.<p>[1] <a href="http://ipython.org/notebook.html" rel="nofollow">http://ipython.org/notebook.html</a>