Goodhart’s Law: Are Academic Metrics Being Gamed?

Glad to see this important piece here (disclosure: I am one of the editors of The Gradient).<p><a href="https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;document&#x2F;5089308" rel="nofollow">https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;document&#x2F;5089308</a>, from RCIS 2009 (Beel and Gipp) noted that &quot;Google Scholar seems to be more suitable for searching standard literature than for gems or articles by authors advancing a view different from the mainstream.&quot;<p>Unrelated, but interesting: scraping Google Scholar is remarkably annoying if you want to actually use the data. The easiest way (in my experience) seems to be regex hacking on the BibTeX files, but this seems truly broken.

Something seems a bit wrong with the graph &quot;Publication rate by career length&quot; -- should the y-axis be &quot;Average number of published papers <i>per year</i>&quot;? (I can&#x27;t imagine that someone whose first paper was in the 1950s only published 1 additional paper in the next 30 years)

Fantastic over view of current trends in academia. There is truly a huge bias in the research publications. I think blogging is a better way to put forth your ideas and research rather than getting a publication in some cases.

Excellent article. A relevant link would be the San Francisco Declaration on Research Assessment (DORA) <a href="https:&#x2F;&#x2F;sfdora.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sfdora.org&#x2F;</a><p>Back to the article, lots of gems, like:<p>&gt;Today&#x27;s researchers can publish not only in an ever-growing number of traditional venues, such as conferences and journals, but also in electronic preprint repositories and in mega-journals that offer rapid publication times.<p>Did I just read a very normal point of view of a researcher putting on equal footing electronic preprint repositories and mega-journals?<p>The BOAI Open Access preachers surely can&#x27;t believe their eyes :) Heresy! (No researcher was involved in the BOAI flawed definition of green OA as archiving and gold OA as publishing.)

Do people benefit from gaming the system? Then it surely is being gamed. And they do. Funding and tenure depend on these metrics.<p>&gt; Overwhelmed by the volume of submissions, editors at these journals may choose safety over risk and select papers written by only well-known, experienced researchers.<p>There is a bit of a &quot;circle jerk&quot; in this process: if you know the right people, you can get better reviews. In return, you review their papers or requests favorably. That also leads to repeating authors.

Yes. But a good department will know you for who you are. Plenty of great science takes a long time do do, and it is known it&#x27;s hard to get funding for long-term monitoring experiments. Most grant money is for new and innovative ideas and no one is pushing out one of those every year. And if you are then you&#x27;re not doing 90% of the work on any of them.

They should have limited the analysis to the most popular journals. There are tons more journals nowadays because its so easy to run one - but it s more important to know what’s happening at the well known ones. The lesser know are largely ignored

Tough call: Goodhart&#x27;s Law says yes, metrics will be gamed, but Betteridge&#x27;s law says the answer is no.

No. We&#x27;re all being gamed. Academics were just nearer the front of the queue.

I wish there was a Google n grams for Google scholar...