This is based on users choosing the better from 2 models at a time, and calculating an ELO rating from who-beats-who.<p>BYOT - bring your own tests style.<p>Gives a better picture of real-world performance and more robust against contamination.<p>They collected over 6000 and 1500 votes for Mixtral-8x7B and Gemini Pro.<p>While ELO ratings are widely used to rank performance in Chess or among sports teams, here's a disclaimer by the makers of the leaderboard:<p>---<p>> Please note Arena is a "live eval" and pretty much a sampling process to estimate models capability.<p>> That's why we show the confidence intervals through bootstrapping. Statistically, these models (e.g., GPT-3.5, Mixtral, Gemini Pro) are very close and only looking at their ranking can be misleading.<p><a href="https://twitter.com/lmsysorg/status/1735729398672716114" rel="nofollow noreferrer">https://twitter.com/lmsysorg/status/1735729398672716114</a><p><a href="https://twitter.com/lmsysorg/status/1735751052287226059" rel="nofollow noreferrer">https://twitter.com/lmsysorg/status/1735751052287226059</a>