After few years of research in DL I learned not to trust any single paper at all. 99% of DL papers are not scientific but rather 'hey guys, look this trick is our new awesome discovery'.<p>Also I learned to communicate with other teams and exchange with ideas proved by practices - this really helps.<p>To improve, I would suggest to publish whole setup, all the parameters used, the programming code, and publish either all the data or a reference to a large free data set (no MNIST anymore in papers, please).
Anecdotally, my master thesis on natural language processing was supposed to consist of first reproducing the results of an influential paper (back then) and then hopefully improving upon it by extending the model used.<p>The paper made it seem like they had been using a standard PCFG parser (which circulated in the research community at the time) to achieve their results. It turned out they hadn't and instead had written a custom one and in fact their results were not reproducible using the standard parser.<p>What was meant to be a timesaver in terms of engineering (using a standard parser instead of writing your own) turned out to be a massive time sink. It also turned out that by using a custom parser they had unintentionally diverted from a vanilla PCFG (probabilistic context free grammar), or in other words, some implementation details had led to a departure from the assumed underlying theoretical model.
A lot of research depends on software written by researchers and yet writing software is not really supported or incentivised by academia. Organizations like the software sustainability institute in the UK (<a href="http://software.ac.uk" rel="nofollow">http://software.ac.uk</a>) are lobbying to change this, with some success, but I guess it takes a long time to effect cultural change.
Unfortunately CS is not as rigorous as some other academic fields.<p>I've been to 100+ colloquia in the physics dept at Cornell and I have never been at one that I felt was a waste of time or that the person should not belong there.<p>The CS department colloquium is a different story: yes I got to see Geoff Hinton before he became a celebrity but maybe half of the talks are awful.
Ah yes, the story of my master's ML thesis.<p>I had to select features from multiple papers in order to try and select the best ones with classification results to prove it.<p>A few problems included:<p>- Incomplete/unavailable datasets (404 on some copyright pictures)<p>- Features consisted on Math formulas and text descriptions (no code whatsoever)<p>- Classifier names only (which framework did you use? parameter values?)<p>In the end i couldn't contribute as well, got instructions to save my work in a private repo despite being funded by an EU academical scholarship.
Agreed. The tooling around deep learning is not as mature as the tooling around software development. There is a fair amount of engineering and grunt work needed to even get started, let alone build on others' research. A few problems from top of mind:
- Setup: Installing DL frameworks, Nvidia drivers and CUDA is an exercise in dependency hell. Trying to run someone's project, which has different dependencies than what you have is difficult to get right. Docker images [1] and nvidia-docker make this simple, but are still not the norm.
- Reproducibility: This is big as Denny mentions. Folks still use Github for sharing code. But DL pipelines need versioning of more than just code. It's code, environment, parameters, data and results.
- Sharing and collaboration: I've noticed that most collaboration on deep learning research, unlike software, happens only when the folks are co-located (e.g. part of the same school or company). This likely links back to reproducibility, but there are not many good tools for effective collaboration currently IMHO.
[1] <a href="https://github.com/floydhub/dl-docker" rel="nofollow">https://github.com/floydhub/dl-docker</a> (Disclaimer: I created this)
Sorry to knock the post author off his high horse, but "just like you wouldn’t want a highly trained surgeon spending several hours a day inputting patient data from paper forms." Highly trained surgeons _do_ spend several hours a day doing tedious paperwork.<p>As a researcher, I expect 50-90% of my time to be slogging through organizational and preparatory work.
I agree with the central thesis: engineering is a huge bottleneck. I work for a FinTech company that is building novel machine learning models and this is our experience.<p>We've had a few machine learning experts working here for a couple of years, but recently brought in a software engineer with a passion for machine learning. He was able to, within a few months, streamline the data acquisition pipeline to the point where we could iterate on a new models in about 30 minutes, down from days. He accomplished this not just with better data but by building efficient in-memory data structures. It saves literally days of time per iteration because of disk I/O.<p>Before his work the training data versus the data we used in production had minor differences. Each new release required intensive manual verification to make sure that our model worked. Now we have much more certainty that the two match up.<p>Looking down on engineering problems is like a famous architect looking down on structural engineers. You're not gonna have a very good skyscraper if your foundation is shaky and ad-hoc.
coming from the machine learning research community, i am in awe of the availability and relative ease-of-use of the deep learning frameworks. Rarely can i find comparison code in ML that i didn't have to bug the author for, or try and implement myself based on the paper alone. in short DL is on a much better path than the author realizes. Perhaps we have to thank the github/bitbucket era. The real problem with DL is that until there is a more robust theory(if even possible, the bane and boon of ML is the complexity of the models), much of the application research will sorta be a form of digital alchemy.
Just yesterday, browsing the forum of kaggle, I thought that we may need a GitHub for Deep Learning...<p>So that you could link your code to a dataset, have it automatically run, and show the result...<p>Not sure if it is worth the time...
i was playing around with the emotion data set on kaggle using tensorflow. depending on the seed, i was getting between 58 and 60% accuracy on the hold out test set (what you submit against).<p>i thought i came up with a good set of hyper parameters using aws gpu instances (python 2.7). i wanted to visualize some of the outputs so I copied the code to my machine and ran under python 3.5 (windows) and only got 57% accuracy. these swings in accuracy are huge