Author here. I think our approach to AI benchmarks might be too human-centric. We keep creating harder and harder problems that humans can solve (like expert-level math in FrontierMath), using human intelligence as the gold standard.<p>But maybe we need simpler examples that demonstrate fundamentally different ways of processing information. The dice prediction isn't important - what matters is finding clean examples where all information is visible, but humans are cognitively limited in processing it, regardless of time or expertise.<p>It's about moving beyond human performance as our primary reference point for measuring AI capabilities.