Comment re-pasted from pubpeer [1], as this article contains claims which amy be disingenuous due to data handling.<p>"This article was reviewed on the error bar podcast episode 25: <a href="https://theerrorbar.com/?e=25" rel="nofollow">https://theerrorbar.com/?e=25</a> & story 106: <a href="https://theerrorbar.com/?s=106" rel="nofollow">https://theerrorbar.com/?s=106</a>. The review was done in the light of newspaper headlines about the article. The slightly-edited transcript is copied here:<p>headline: BRAIN DOESN'T SLOW DOWN UNTIL 60<p>tweet: analysis of 1.2 million responses on the 'implicit association test' finds that adults' mental processing speed is totally good & constant until age 60. that's good news for all adults over age 35. until you look at the data.<p>description: in a story likely comforting to middle aged scientists & journalists, the Guardian & Independent reported last week that, Good News: the brain does not slow down until we are aged 60. what did they mean?<p>the result comes from a quite complicated analysis of an enormously huge set of data. a social psychology project in the USA has collected nearly 2 million responses from people who were asked to categorise things on a computer screen as, for example, 'good' versus 'bad' or as 'black' versus 'white'. they did this 120 times each.<p>in general, the results of this so-called 'implicit association test' are able to reveal people's unconscious attitudes - in this example, that white things are good & black things are bad. this test is a classic & it relies on people responding as quickly as possible.<p>in the new study, researchers in Germany downloaded 1.8 million test responses from the free online database. they analysed how people's responses changed as they age, from 10 to 80 years old. using some sophisticated (& unnecessarily complicated) analyses, the researchers reported that, while the total time spent doing the task changes a lot over life, one conclusion is that, in fact, our 'mental processing speed' remains good & strong until age 60.<p>phew!<p>evaluation: does mental speed stay high until age 60?<p>no.<p>this is not a hard one. with a total of nearly 1.2 million responses, all openly available online & clearly-presented in this paper, it is a fantastically-simple task to fact-check this claim. all we need to do is look at the data.<p>i've annotated the image to accompany this story on the error bar website. what the image shows is a gradual & continuous increase in mental speed between age 10 to a peak at about 35, then a gradual & continuous decrease to age 80. it's true that the speed of change seems to be different at different ages, but the strong claim that mental speed is 'high & constant until age 60' is contradicted by just looking at the data.<p>how did it come to this? the error bar dug deeper.<p>first, this is a massive dataset - which is great - but quite a lot of data were removed, for example anyone performing the task without making errors was excluded. that's a bit odd, if the best responders are removed.<p>second, the task itself is quite subjective - is something good or bad? or is something black of white? - & although a well-studied task, it's just one task, so conclusions about mental speed in general must be extremely limited.<p>third, the data are not a good measure of mental processing speed. to test mental speed, we need volunteers to be fully-focussed & making responses as quickly as possible. but the authors included 'response times' as long as nearly 6 seconds, the average times were around 1 second & there were some massive - and unexplained - differences between men's & women's responses.<p>fourth, one claim in the report is that the 'merely mechanical' processes of seeing the stimuli on screen & making the keyboard press response take as long as half a second in the oldest adults. this is a pure fiction - the study provides no sensible measurement of these sensory & motor processing speeds.<p>fifth, & amusingly, the authors chose a particular kind of deep, machine-learning analysis because they said it was "not possible" to analyse all the data in a different, simpler, but more computationally-expensive way. but the new analysis only took 32 hours to run on a computer, which makes me wonder how long the full analysis would take & why was there such a rush to get it done?<p>sixth & critically, the conclusion that mental processing speed is high and constant until age 60 relies only on the authors' choice of how to put a best-fit line on the data. the analysis used requires all the people to be divided into two or more different age groups. for error bar nerds: the data were fit with three straight lines for three age groups. but age is a continuous variable, so why did the authors not just fit a single, continuous curve?<p>well, listeners, i downloaded the data, sharpened Occam's razor & fit that single curve. it provides a very slightly worse explanation of the data, but that's what you would expected, as it is also a much less-complex explanation."<p>[1]