There is a deeper and largely overlooked problem with sports judging, especially in the sports with artistic component (like gymnastics or figure skating) – Goodhart's effects. Goodhart's effect stands for "when the metric becomes a target it stops being a good metric". As soon as you make measurement part of the reward system, it changes the whole system.<p>Take figure skating as an example. Since 2006, ISU has introduced a similar judging system - IJS – that uses points for elements and an increasingly complex code of rules on how to assign those points. Each element has a so-called "base value" and judges determine GOE value in the range from -5 to +5 for each element. The computer averages that and takes trimmed mean to produce the final points for the element.<p>Those base values and points are logically assigned by the complexity of elements. More complex elements cost more. Great skaters can perform more complex elements. Sounds logical, right?<p>What has happened, though, is that using these points as a target has changed the way how skaters train. Why spend time learning "easy" elements that give little points? All time must be spent now learning "complex" elements that give more points. In a way, before, IJS skaters were chasing overall greatness; now, they chase points.<p>"Skater's greatness" – whatever that means – is hard to define, let alone measure. The ability to perform complex jumps was a consequence of it, not a cause. Using points as a metric changes the whole system of training, and a new generation of skaters can perform insane quads (often because of pre-puberty body sizes, though), but no one would call them great skaters.<p>I'm not even starting with how these points affected artistry. "Program components" (PCS) that should measure artistry are practically a joke now. You basically can multiply jump scores by some constant and get PCS. [1] All skating programs look the same now and are very predictable because of points maximization.<p>Back to AI. The rules for figure skating become more and more complex each year in order to duct-tape the constantly emerging issues. Many judges admit that complexity is so high that the human brain can't even possibly apply them in real time and has to resort to cognitive shortcuts. There are some early flirtations with using AI for judging figure skating [2], but nothing serious yet.<p>As outlined in the "Categorized Variants of Goodhart's Law" [3], one of the major reasons of this effect in sports is not just the fact that elements != greatness, but the fact that there is no shared definition of what is a True Goal of this sport.
What exactly judging system try to measure? What would constitute the best skater?<p>Where there is no explicitly stated True Goal, any proxy variable used as metrics (like "ability to perform complex elements") will be far from good and will be distorting the system even further.<p>Eventually, this boils down to the structure and decision-making in the international federation that governs this sport.<p>And these things cannot be solved by AI. In a way, using AI is a yak-shaving for ever-increasing rules of an inherently wrong judging system.<p>[1] <a href="http://libjournals.unca.edu/OJS/index.php/mas/article/view/23" rel="nofollow">http://libjournals.unca.edu/OJS/index.php/mas/article/view/2...</a><p>[2] <a href="https://www.youtube.com/watch?v=pq_mv1eyZyA" rel="nofollow">https://www.youtube.com/watch?v=pq_mv1eyZyA</a><p>[3] <a href="https://arxiv.org/abs/1803.04585" rel="nofollow">https://arxiv.org/abs/1803.04585</a>