The problem with most of these is not adjusting for cohort changes. For instance, in the SAT example, the author writes:<p>> In the 1980s, the Reagan administration seized on a report called A Nation at Risk, which claimed that the US was on the verge of collapse due to its falling SAT scores.<p>Suppose that low-income individuals start to take the SAT in 1980 whereas they didn't in 1970. The <i>wrong</i> way to analyze SAT scores is to evaluate:<p>sum over cohorts P(SAT Score | cohort, Y)<p>where Y is the year. For instance, you might compare the total average score in 1980 vs. 1970. Doing so will show a decrease in SAT score because of the increase in low-income individuals taking the SAT, <i>not</i> because the high-income individuals are doing worse. (This assumes that low-income people have less access to SAT training materials, and those training materials affect the score).<p>The correct way is to only compare scores <i>within a cohort</i>:<p>P(SAT Score | cohort, 1980) > P(SAT Score | cohort, 1970)<p>That is, did the same cohort do better in 1980 vs. 1970?<p>(There might <i>still</i> be some differences between the cohorts in 1980 vs. 1970. Maybe the low-income individuals who took it in 1970 had high confidence in school, whereas the 1980s kids were from a broader background.)