I know there are a lot of ML researchers and practitioners here - and I have unfortunately only a very shallow experience with reading ML papers, more so with the recent output.<p>But, I have to ask, how do you get a feel that the content actually looks correct, and not just only quack? The improvements are usually in the 1% range, from old models to new models, and the models are complex. More often than not also lacking code, implementation / experiment procedures, etc.<p>Basically, I have no idea if the paper is reproducible, if the results are cherry picked from hundreds / thousands of runs, if the paper is just cleverly disguised BS with pumped up numbers to get grants, and so on.<p>As it is right now, I can only rely on expert assurance from those that peer review these papers - but even then, in the back of my mind, I'm wondering if they've had time to rigorously review a paper. The output of ML / AI papers these days is staggering, and the systems are so complex that I'd be impressed if some single post. doc or researcher would have time to reproduce results.
Can we apply the steps to reading source code?<p>Here's my take on adopting it to reading code:<p>1. Read readme if available, read the list of source files to get a sense of how the project is modularized. Identify the entry point. Identify type of program from main entry point: is it a server, a CLI, or a graphical app?<p>2. Run call graph analysis tool if you have it, so you can study callgraph tree starting from main entry point. Read just the function names and start making notes of how the execution works at various levels, e.g does it read input then enter an infinite loop, does it wait on network packets, does it use update/render loop, etc. Also make note of whether a function is trivial/non-trivial based on quick glance at the code.<p>3. Ignore the trivial ones, and read the non-trivial ones in detail. Make note of the algorithm, data structures, and dependencies.
My spouse has a rule for science papers: First, look at the pictures. She figures that people will put a lot of effort into their graphs and diagrams telling a good story.
How do i find all the interesting papers?<p>Like i like to read about things like:
ML, Scaling, Filesystem, Databases, Algorithms etc.<p>I do get a lot of input through hn, friends, youtube, blogs but i'm not getting my papers from direct sources. I don't have anything like nature or so laying around either.
Is there a text that will explain the difference between a paper, an article, a manuscript, a monograph and all the other words often used to describe different kinds of written scientific material?
This reminds me of How to Read a Book[1], which is also a great read.<p>[1] <a href="https://en.m.wikipedia.org/wiki/How_to_Read_a_Book" rel="nofollow">https://en.m.wikipedia.org/wiki/How_to_Read_a_Book</a>
There's a 2016 updated version here : <a href="https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf" rel="nofollow">https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/...</a>
Should the link update?