The Big Data Brain Drain: Why Science is in Trouble

This is so true. I'm in a Ph.D. program and everyone around me is wasting so much time by reinventing the wheel every time they need to code something. So I spend my time making libraries to help them out, but then I get scolded because that's time that's not going directly toward getting publications. And few people use my code because they don't trust software as up to the scientific standard unless (a) they spent thousands of dollars on it, a la MATLAB, or (b) they wrote it themselves and, e.g., take a mean by manually iterating over an array, "just to make sure" the mean is calculated correctly. Ugh. It doesn't matter how many tests I an point them to. I can't wait to get out of here and work somewhere where coding is appreciated, where I can actually get paid, and where I have some choice as to which state I live in.

I&#x27;m a software developer working with big data, but I think the premise of this article (&quot;the ability to effectively process data is superseding other more classical modes of research&quot;) is simply false.<p>The example problem domain (&quot;automated language translation&quot;) is actually a stellar counter-example to the claim. Has anyone actually tried to use Google Translate for anything sophisticated? It&#x27;s still truly horrible, by human standards. The field needs more research and deeper conceptual understanding, not less.<p>There may be some problems that can be solved by throwing software&#x2F;hardware&#x2F;data at them, but I don&#x27;t think this is a good paradigm for the big unsolved problems in general.

It&#x27;s a nice article, but it overlooks one fact: There needs to be a brain drain out of academia, because academia can&#x27;t absorb more than a fraction of its own production of talent.

Honestly,<p>The situation is simpler.<p>Academia has become abjectly miserable and abusive in its practices. It no longer offers good but low-paid jobs for smart non-conformists, it just offers it&#x27;s special brand of misery based on some long-past promise of this.<p>Given this, only the mediocre stay (and compounding it, anyone who stays has no reason to be better than mediocre). And that is a huge, huge loss to the whole project of the development of human knowledge, something that has a long history in Western society.

Technical work, including indispensable scientific software development, tends to be considered of low academic value in academia. This is an ingrained attitude. I very recently left after having heard &quot;oh, you&#x27;re the technical guy&quot; once too often from other academics.<p>Here&#x27;s an example. The Globus Online grid ftp service web page intended for users adopts an overtly apologetic tone [1]. Users of this service are promised freedom from &quot;low-value IT considerations and processes&quot;--considerations and processes that the Globus Online team has humbly sought to undertake on their behalf. I have to laugh at the claim that there is &quot;No need to involve your IT admin—all you need is Globus Online.&quot; The message is that information technology is of low academic value--unless you happen to have been one of the authors of publications that came out of the Globus Online project. If not, your career is sidelined.<p>Software development, system administration, network administration and desktop support have become somewhat specialized in the past 30 years, but in the minds of some principal investigators and academic administrators, these very different activities are conflated. An expert in numerical methods, computational fluid dynamics and dynamic downscaling methods for climate assessment models is a seasoned web developer with a portfolio, fluent in jQuery, underscore, backbone, responsive websites with bootstrap, CSS3, HTML5, PostgreSQL, PostGIS, the Google maps API, Cartodb visualizations, as well as an Android developer conversant with the SPen library for the Galaxy Note 10.1. It&#x27;s as much effort to stay current technically as it is to keep up in the scientific literature.<p>There are faint signs of improvement. On January 14th, the NSF revised the biosketch format by changing the <i>Publications</i> section to <i>Products</i> [2]. &quot;This change makes clear that products may include, but are not limited to, publications, data sets, software, patents, and copyrights.&quot; The previous biosketch format was awkward for software developers, inventors and producers of data sets. <p>Recently, a number of prominent computer scientists, and scientific software developers affiliated with the Climate Code Foundation [3], published a Science Code Manifesto [4]. The manifesto includes the recommendation that &quot;software contributions must be included in systems of scientific assessment, credit, and recognition.&quot; Software developers in the digital humanities may wish to add their names to the list of signatories.<p>Whether these developments reflect a broader understanding that software developers ought to enjoy greater recognition and opportunity for advancement in academia that they do currently remains to be seen. Greater career advancement opportunity for software developers, inventors and data set producers working in academia might do something to address the Ph.D. overproduction problem.<p>But these developments were too little and too late for me. I left.<p>[1] <a href="https://www.globusonline.org/forusers/" rel="nofollow">https:&#x2F;&#x2F;www.globusonline.org&#x2F;forusers&#x2F;</a><p>[2] <a href="http://www.nsf.gov/pubs/2013/nsf13004/nsf13004.jsp" rel="nofollow">http:&#x2F;&#x2F;www.nsf.gov&#x2F;pubs&#x2F;2013&#x2F;nsf13004&#x2F;nsf13004.jsp</a><p>[3] <a href="http://climatecode.org/" rel="nofollow">http:&#x2F;&#x2F;climatecode.org&#x2F;</a><p>[4] <a href="http://sciencecodemanifesto.org" rel="nofollow">http:&#x2F;&#x2F;sciencecodemanifesto.org</a>

HN really amazes me sometimes. I posted the exact same article with the exact same title 12 days ago [0] and didn&#x27;t receive much traction. In fact, the only comment on the submission read<p><pre><code> this should have landed on the front page... </code></pre> Anyways, I wonder if there is any explanation of this phenomena.<p>[0] <a href="https://news.ycombinator.com/item?id=6623501" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6623501</a>

Nicely written. A couple of things, somewhat independent of the economic issue: the &quot;big data - dumb analysis&quot; trend is bound to change, it&#x27;s a matter of time till volume is not an advantage anymore. And the fact that &quot;academia&quot; is by no means a uniform discipline, some fields can only move ahead with the gathering of more data, while other fields need new testable hypotheses more.

Compensation is a huge issue. Unlike pure biologists, computational scientists have plenty of job opportunities outside of academia and pharma.<p>As a staff programmer at a prestigious institution, I was making about $50k. I left for a large company and a few short years on I&#x27;m at around $200k.

&quot;I have some serious doubts about whether the project will be able to attract a sufficient pool of applicants for these positions.&quot;<p>Really?? Certainly, academia has its inefficiencies, but if there is an area of academia that has an undersupply of PhD&#x27;s, please correct me!

My basic line of argument is fairly simple. If you work at a university and your salary is paid via taxes you have an obligation to make your work available to the taxpayers.<p>This means the software you write should be open sourced and the papers and articles you write should be freely available.<p>I&#x27;ll use the basic idea of this article as more fodder for that argument (it&#x27;s even more important now because...Big Data ZOMG) in the future :)<p>I don&#x27;t fully agree with the thought that writing software and papers are exclusive. With a little creativity you can get a paper or two out of most academic software development projects (granted they won&#x27;t usually be the A-level &quot;this advances my discipline&quot; kind of papers but one can&#x27;t only write those anyways imo).

Re underpaid researchers:<p>The part that is always unclear to me is: there is lot of evidence that research&#x2F;university budgets are growing at a high rate. Where does all that money go to. It doesn&#x27;t seem to go to the people who actually do science.<p>That seems to be the fundamental problem here.

This article has some incredible insights which have gone right past me. Thanks for sharing. I&#x27;m currently studying for a Masters degree which is centered on visualization techniques for exploring and analyzing large datasets.<p>I have found myself thinking many times that a position in the industry, where I can use my teaching, data processing and analysis skills to further some business goal, seems like a much more preferable option than sitting around writing research papers and applying for grants all day. Not to mention that academia pays less and has worse overtime conditions than <i>any</i> industry job I could concievably get.<p>This article really nails the key issues for why I am feeling this reluctance towards an acedemic career.

I am an example of this trend, having dropped out of a physics PhD program to pursue programming, and feeling great about the decision. There are two points which are not made by the author and which I think deserve to be mentioned:<p>1.) Scientific research is <i>hard</i>. It is very frustrating to spend all day working on something you can&#x27;t be sure is going to even work. On the other hand, programming is easy, it&#x27;s mostly monkey work. Sure there are places where you have to be clever and think things out carefully, but at the end of the day, when you write code it just feels like you have so much more to show than when you do research.<p>2.) In research, you need to get grant money to do anything, because research is expensive. So much time is then spent writing grant proposals, and even when you get money you still can&#x27;t do everything you want because it&#x27;s just too expensive and&#x2F;or time consuming. I&#x27;m not talking about LHC money here, but just the standard money for a professor running a lab in a university.<p>In programming, you can work on (I&#x27;d estimate) 95+% of problems with nothing more than your computer and some old fashioned hard work. There are so many good, free to use open source libraries out there, making it pretty easy to jump into whatever field is interesting to you. Best of all, when you finish a project, there is no need to spend weeks writing a paper about it or any of the nonsense associated with that, you can just publish your code.

<i>&quot;Simple models and a lot of data trump more elaborate models based on less data...&quot;<p>If we make the leap and assume that this insight can be at least partially extended to fields beyond natural language processing, what we can expect is a situation in which domain knowledge is increasingly trumped by &quot;mere&quot; data-mining skills.</i><p>This is a great point. For many years, domain knowlege was merely such experience ex ante. In otherwords, the barrier to domain knowlege was access to data. In lieu of this, perhaps theory to estimate it. As data becomes larger, more free, and more amenable to analysis by a larger group of talent the &quot;domain knowledge&quot; itself as a barrier to entry (prestige, effectiveness) seemingly declines. Whle this is in theory good (more access, more analysis), there is probably a corralary that we should expect turf-wars and restriction to access, as those previously in positions of privledge fight to retain their status as &quot;keepers of the keys&quot;.

&gt; We always come across reports of how much talented Indians are and are conquering the world in the field of technology &amp; business.<p>Um, I don&#x27;t mean to be offensive, but I have never heard that.

My biggest worry is that it&#x27;s not science that is in trouble, it&#x27;s humanity as a whole that is in trouble.<p>The drain towards industry is to solve industry&#x27;s problems: make a profit. And yes, these can be extremely interesting problems, and hard ones too.<p>But what would our current world look like if the scientists from the last 500 years had bent their minds to solve the problems of merchants? (and don&#x27;t get me wrong, some of the problems merchants had evolved great solutions for mankind)

I&#x27;m an example of this trend. I trained for a decade in big data for neuroimaging. Then I realized I was working long hours with little upside beyond my own curiosity. I certainly wasn&#x27;t changing the world in any tangible way. I had originally gone into neuroscience to understand my family&#x27;s history of mental illness. Ten years later I was helping no one - not even myself.<p>I&#x27;m not convinced Big Science is in trouble though. Those who have the motivation and talent to stay in the academy will continue to do so. Yes, outstanding people will be lost but Science can progress from their contributions to commercial efforts. Geoff Hinton ends up at Google but we&#x27;ll keep hearing from him the best use of his talents - the improvements to products we all use every day at a massive scale.

Scientists in actual science are seriously underpaid, but there&#x27;s a case of the Teacher-Executive Problem that&#x27;s going to make it hard to fix that (<a href="http://michaelochurch.wordpress.com/2013/11/03/software-engineer-salaries-arent-inflated-at-least-not-for-the-99/" rel="nofollow">http:&#x2F;&#x2F;michaelochurch.wordpress.com&#x2F;2013&#x2F;11&#x2F;03&#x2F;software-engi...</a>). The more useful you are, the more need there is for many of you (individual &quot;rock stars&quot; are overrated in the real world) and the greater the implicit multiplier on improving your salary. Increasing scientific pay is has a bigger cost load than increasing executive pay because we actually need scientists (or teachers, in the original formulation of TEP) in significant numbers.<p>Society has reached a point where the academic route means practically begging for a job that won&#x27;t even pay for a house, while the startup lottery offers, at least, a chance. (And finance, better yet, offers a high likelihood of being well-off.)

I haven&#x27;t read Wolfram&#x27;s <i>A New Kind of Science</i>, but something tells me that this is a one page summary of it?