Is Peer Review a Good Idea? (2020)

I&#x27;ve been in academic research for 22 years. Based on my experience, I&#x27;ll go out on a limb and say that even single-blind reviews are to be discarded.<p>This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.<p>Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.<p>Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.

It would be a much better idea if double-blind peer review were a default. See &quot;Nobel and Novice: Author Prominence Affects Peer Review&quot;: <a href="https:&#x2F;&#x2F;papers.ssrn.com&#x2F;sol3&#x2F;papers.cfm?abstract_id=4190976" rel="nofollow">https:&#x2F;&#x2F;papers.ssrn.com&#x2F;sol3&#x2F;papers.cfm?abstract_id=4190976</a>

I think ML, at least when I was involved in it, had a much more sane way of approaching publication than other fields (I came from a biology background).<p>Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know &quot;hey, at least I can expect some degree of quality given that it was published in this or another magazine&quot; specially as a student.<p>But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.<p>So I&#x27;m all for abolishing &quot;pre-publication peer review&quot;.

As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don&#x27;t want to invite 4Chan into my daily work.<p>I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won&#x27;t lead to a 4Chan&#x2F;Twitter mob coming for them.<p>According to the ACL&#x27;s 2019 survey [1], &quot;female respondents were less likely to support public review than male respondents&quot;. I&#x27;d be weary of implementing any changed that would make academia even more hostile to women and minorities.<p>[1] <a href="http:&#x2F;&#x2F;acl2019pcblog.fileli.unipi.it&#x2F;wp-content&#x2F;uploads&#x2F;2019&#x2F;07&#x2F;ReportACL2019ReviewingSurvey.pdf" rel="nofollow">http:&#x2F;&#x2F;acl2019pcblog.fileli.unipi.it&#x2F;wp-content&#x2F;uploads&#x2F;2019...</a>

&gt; I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified.<p>This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].<p>This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?<p>[1] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;about" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;about</a><p>[2] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;help&#x2F;submit" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;help&#x2F;submit</a>

The standard should be reproducibility, not peer-reviewed. Reproducibility is open loop, peer-review only is very susceptible to closed loop.

One of the best uses of Reddit is to find a thread where someone well educated is taking apart an academic paper.<p>You will be amazed what you didn&#x27;t spot&#x2F;assumed and learn a lot.

Having been on both sides of the peer review process in my field (Theoretical Physics), I say we should completely abolish it. Just publish the papers on the preprint archives. Some will be bad, some will be good, and you will know the impact of a paper after 10 years, nothing different from how it is now. Currently the very big majority of the effort in the reviews (on both sides of the process) is wasted time.

Off-topic comment: it surprised me that The <i>British</i> Journal for the Philosophy of Science is published by The University of <i>Chicago</i> Press.

While knowing who the authors are can cause bias, hiding behind anonymization (as a reviewer) can as well. I think publishing the reviews (and who reviews) would help a lot (some fields do this). Science is really a dialog. The authors are putting forth part of the dialog and the reviewers are reflecting on it and giving their opinions. Hopefully publishing names next to reviews would minimize crappy reviews (often by the most senior folks). However the other problem is there are not enough qualified reviewers to go around and I could see many opting out of putting their name next to a review. But, we&#x27;re likely better off.<p>There also often isn&#x27;t a clear cut line between &quot;good&quot; and &quot;bad&quot;. Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.

having worked in academic publishing, i&#x27;ve concluded that peer review would perform best if it were embedded in the very act of reading (so post-publication, non-formalized kind of peer review) instead of having it be a formalized process riddled with issues and cracks as it is currently.<p>it would enable faster and more seamless communication, academics wouldn&#x27;t be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).<p>sadly my impression is that it&#x27;s simply too entrenched in the publishing process -- and publisher&#x27;s raison d&#x27;etre to a large extent -- for the publishers to relinquish control of it

I like the basic principle of peer review, but not how it&#x27;s done. I understand why it was done that way for so long, but we have the technology to do better now.<p>I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.<p>Just publish your papers publicly with a comments section. If there&#x27;s problems with it people will tear it apart. Source their work and let the world help you improve your work.

Doesn&#x27;t the proposed solution already exist? Authors are allowed to publish their papers in preprint repositories before submitting to a journal.

The way Journal of Open Source (JOSS) - <a href="https:&#x2F;&#x2F;joss.theoj.org" rel="nofollow">https:&#x2F;&#x2F;joss.theoj.org</a> - does the reviewing process is good. Everything is out in the open, reviewers and authors names, reviews, responses, discussions etc. The whole process is out there for everyone to see.

The main issue with critical and complex code is that most if not all of your peers who are going to review your code are actually hackers..........