TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Problems of Reproducibility in Computer Science

24 pointsby mkpankovover 10 years ago

5 comments

nkurzover 10 years ago
I've been particularly bothered by the way that almost all "real numbers" in CS papers attempt to explain the advantages of one algorithm over another based on small performance differences of two partially optimized implementations, compiled with a single compiler with arbitrary options, and then tested on a single older machine. The few that go further merely repeat the exercise with an even older compiler an an even older machine with a forgotten architecture. While it's a step up from simply counting pseudo-instructions and memory accesses, and while there is room for theoretical analysis of asymptotic behaviour, it's painful how much is read into the particular tea leaves of one or two data points.
malux85over 10 years ago
Recently I was asked to help judge the 2014 iGEM competition which includes software entries in the field of biology, I made a special point to the other judges that we should give special consideration to those teams who included instructions or configurations for setting up their environments - too much of the software entered wouldn't compile (or run) when checked out of SC!
michaelochurchover 10 years ago
One of the trade-offs we&#x27;ve continually made in computing is between general performance and <i>legibility of</i> performance. Caches (i.e. the memory hierarchy) is one example of that, but we also have pipelining and ILP. Moving up into software, we see this with garbage collection and (more dangerously) with databases. PostgreSQL is great at taking complex queries (many, many joins) and turning them into things that (often) can be very quickly executed. Of course, there&#x27;s no silver bullet. If your join is a full Cartesian product over two or more large tables, you&#x27;re screwed. If you assumed that the table was set up with certain &quot;obvious&quot; indexes and it wasn&#x27;t set up that way, you can get burned on that.<p>We&#x27;ve built systems that are very fast, but almost no one knows <i>why</i> they are fast and when those speed advantages go against us (which is quite rare) it&#x27;s often unclear why. For the most part, all of this has been a huge win. It might make reasoning about performance from first principles ugly, if not impossible, but the economics favor it. That said, we&#x27;re at the point where the only way to reason about performance in many real-world systems is empirically, because there&#x27;s just so much going on in terms of memory hierarchy, network traffic, etc. that is hard to predict in advance. Combine this with the myriad variables, some in the physical world, that can influence performance... and what we end up with is that results that seem meaningful (30 percent faster!) might be artifacts of differences in configuration.<p>That said, I don&#x27;t think the term <i>computer science</i> was coined with performance measurement in mind but, all that said, the field is not really a science. It&#x27;s a mix of mathematics, microeconomics, design and engineering... with a lot of science involved, but most often not <i>scientific</i> under the original, strict definition of the word. Science is about investigating the natural world and finding the right answers; CS is about <i>making</i> answers. And they&#x27;re both extremely important, but fundamentally different.<p>Also, just to underscore how difficult CS <i>is</i>, I tend to think of it as a branch of mathematics that lives on an unfair playing field. Reasoning about <i>arbitrary</i> code, in even the most trivial ways, is known to be impossible (Turing, Church, Rice) and much of CS is contending with the obscenity of arbitrary programs (i.e. often having to quantify over the space of programs) and, typically, that means limiting what can be written while still retaining as much power as is needed.
评论 #8774628 未加载
BruceIVover 10 years ago
Even if there&#x27;s no publicly published code, any credible CS researcher will give you access to their code and tests for research purposes if you ask (excepting the rare case where they include someone else&#x27;s IP), for exactly this reason of reproducibility.<p>Secondly, testing in a number of different environments is part of the nature of scientific reproducibility, to check if the claimed result is robust or simply an artifact of the experimental setup, so releasing more detailed specs of the hardware used is not so essential.
评论 #8774384 未加载
scottpiperover 10 years ago
Moyix&#x27;s Panda will get you most of what you want: <a href="http://moyix.blogspot.com/2014/12/reproducible-malware-analyses-for-all.html" rel="nofollow">http:&#x2F;&#x2F;moyix.blogspot.com&#x2F;2014&#x2F;12&#x2F;reproducible-malware-analy...</a>