The mathematics of science’s broken reward system

187 pointsby seycombiover 8 years ago

13 comments

infinity0over 8 years ago

The challenge will be to fix this whilst keeping the precise, mechanical and quantitative spirit of science alive.The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. Yet the scientific culture of today is falling into that trap itself.The article criticises metric-driven incentives, but it is not metrics (the general concept) that are at fault. It is the choice of metric, and the meta-analysis of this that is lacking. These choices are themselves often backed up by non-scientific vague arguments, of the similar sort that scientists often criticise other fields as depending upon.We must certainly not conclude from these studies that quantitative analysis is itself what is at fault. I know the article doesn't explicitly say this, but it hints at it - using suggestive phrasing like "disdained sociological studies" and referring to all "metric incentives" as a single group - and it is a point I have seen made by many non-scientists. That is, using these flaws as a straw man to attack the very qualities of what has made science so successful and useful.To improve the situation, we must reject these straw-man arguments against science, and develop better methods that are more quantitive, over a broader spectrum of what is being analysed, and that are more self-critical.

评论 #12980077 未加载

评论 #12978774 未加载

评论 #12977627 未加载

评论 #12980803 未加载

评论 #12980369 未加载

评论 #12980697 未加载

评论 #12979411 未加载

评论 #12978640 未加载

评论 #12977763 未加载

评论 #12978382 未加载

hedgewover 8 years ago

Outside of fields with direct penalties for invalid science (think aviation, nuclear reactors, sinking boats, collapsing bridges), we're just relying on the integrity and honor of researchers. On an individual level it's much more effective to cut corners and publish trendy research than it is to make valid science.The situation is only made worse by the fact that universities reward this; because they too are rewarded more for high-profile research than they are for scientific validity.My guess is that most studies in softer sciences are simply erroneous. Many social sciences have already been shown to be unreliable by the replication crisis.My personal experience comes from reviewing hundreds of published studies that evaluated the applied effectiveness of machine learning models. Half had significant statistical errors.

评论 #12978092 未加载

randomsearchover 8 years ago

I agree that governments should stop using metrics to evaluate science in this way, and instead qualitatively review research output.It's obvious that citation counts are open to manipulation (e.g. through self-citation and encouraging citations from others via various mechanisms) but there's also a very large factor that can't be easily quantified: an enormous amount of good will has been lost in the system.Many scientists who are interested mainly in satisfying their curiosity and contributing to society resent the top-down mismanagement of science in the UK and US. They see a system where those who play political games do well, but the smartest, most dedicated, and passionate researchers are often sidelined or ignored because they do not spend the requisite time playing the political system to artificially boost their reputations. Many times I've seen very talented academics with huge potential leaving academia as they were passed up for promotion, left on a temporary contract indefinitely, or neglected in other ways.Strangely enough, I think that if you don't have pressure to climb the ladder (e.g. a family), then the current system offers great opportunities to do outstanding science. If you choose to stay low on the ladder and spend the vast majority of your time actually focusing on research problems, you can get far more actual research done than those chasing promotion or esteem. And because so many people are focused on their citation count, or some political game, the competition isn't as strong as it should be - you can make real advances if you quietly focus and leave the politicians to fight each other.

lbhnactover 8 years ago

I've always wanted a good word for the 'science of studying science'.Never really got past 'Scientology', and that probably won't stick.What's a better term than the vapid 'Metascience', or the clunky 'scientometrics'.How about 'Superscience'? That would be an awesome Doctorate to hang on the wall.

评论 #12977524 未加载

评论 #12977751 未加载

评论 #12977520 未加载

评论 #12977502 未加载

评论 #12979183 未加载

cuantosover 8 years ago

You can have science without academia and visa versa. Perhaps it should say that academia has a broken reward system.

nonbelover 8 years ago

This is presented as if it is a new problem. It stems back to the 1940s with the adoption of NHST (with its arbitrary "statistical significance" metric) first by educational researchers, then by psychologists, followed by spreading like a cancer throughout the research community from there.Things have only been getting worse and worse as the "old guard" in each field retires/dies leaving behind only people trained to think rejecting a strawman hypothesis according to an arbitrary metric, then concluding something about your hypothesis, counts as science.

hannobover 8 years ago

The irony is that this is published in Nature, which undoubtedly is a huge contributor to questionable research practices by publishing spectacular findings and ignoring negative results and replications.

评论 #12978726 未加载

BurningFrogover 8 years ago

Where does this broken system come from?Did anybody construct it, or has it just emerged out of the surrounding world?Is it the same across the planet?

评论 #12981064 未加载

Balgairover 8 years ago

Science has a lot going wrong with it.Here are some I can list: P-values, the h-index (article discusses), the funding crunches and (US specific) the cyclic nature of NIH funding, the massively skewed incentives of publish or perish, the entire idea of trying to assign value to discoveries (article discusses), the 2-body problem of family life and science (at least the Germans expect you to have no life and are clear on that), the work-life non-balance, the relatively very poor pay, the hyper-competition and sabotage (in some fields, labs, or universities), the total ignorance of doctors/politicians that take your work and apply it in appalling ways, the entire racket of scientific publishing, etc.As always, we should point out to newcomers to this discussion of one of the most read scientific paper EVER by John P. Ioannidis "Why Most Published Research Findings Are False" [0]. If you don't have access and don't want to use Sci-Hub or ICanHazPDF, here is a youtube of John discussing his work [1]. You can use the google to really jump into a rabbit hole here. Essentially, flip a coin, heads the paper is right, tails the paper is wrong, it really is that bad these days.Ok, so lets just scrap the whole thing then, right?Here is why not:<a href="https://www.youtube.com/watch?v=_Px5sZyxFYc" rel="nofollow">https://www.youtube.com/watch?v=_Px5sZyxFYc</a><a href="https://www.youtube.com/watch?v=XcPuRaSEq1I" rel="nofollow">https://www.youtube.com/watch?v=XcPuRaSEq1I</a><a href="https://www.youtube.com/watch?v=5EL_bLOK8_A" rel="nofollow">https://www.youtube.com/watch?v=5EL_bLOK8_A</a><a href="https://www.youtube.com/watch?v=qXWYSdijCGU" rel="nofollow">https://www.youtube.com/watch?v=qXWYSdijCGU</a><a href="https://www.youtube.com/watch?v=uBh2LxTW0s0" rel="nofollow">https://www.youtube.com/watch?v=uBh2LxTW0s0</a> (a good showing of why science is right and helps folks)<a href="https://www.youtube.com/watch?v=gs0JQRT6TpY" rel="nofollow">https://www.youtube.com/watch?v=gs0JQRT6TpY</a>So, yeah, science is a mess right now. But if we scientists don't step up and make it not a mess, those scummy money-grubbing scammer dirtbags and their ilk are more than happy to take up the slack. These fucking asshats are going to kill people like you poor uninformed cousins and their kids and take all their money because they think homeopathy is right and the moon landing was faked.So, you young scientist that looks at this pile of garbage that is modern academia, do not despair! Fight the good fight! Yes, you may end up shirtless and ridiculed by the older scientists. But you have to do what you think is right because the rest of this world is depending upon you! Even for that tiny little bullcrap paper you are getting out just to graduate, that matters too. Be in the mud, be in the arena, fight for truth![0]<a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124" rel="nofollow">http://journals.plos.org/plosmedicine/article?id=10.1371/jou...</a>[1]<a href="https://www.youtube.com/watch?v=KOZAV9AvIQE" rel="nofollow">https://www.youtube.com/watch?v=KOZAV9AvIQE</a>

评论 #12981720 未加载

评论 #12979552 未加载

foolrushover 8 years ago

Science hasn't existed forever, and Foucault already smashed to bits of any sort of "evolution with direction" account of science in The Order of Things.The most disturbing thing appears to be that many folks aren't aware of Foucault's work.

AlphaWeaverover 8 years ago

Interesting, even if a bit meta.

jsprogrammerover 8 years ago

Couldn't this model be used to detect and alert on bad actors?

knownover 8 years ago

<a href="https://en.wikipedia.org/wiki/Economic_mobility" rel="nofollow">https://en.wikipedia.org/wiki/Economic_mobility</a> != <a href="https://en.wikipedia.org/wiki/Social_mobility" rel="nofollow">https://en.wikipedia.org/wiki/Social_mobility</a>