Building an AI that watches rugby

I want AIs that clean my apartment while I watch rugby, not AIs that watch rugby while I clean the apartment. ;)<p>In seriousness, this is a cool project and show how sophisticated analysis LLMs can do in a plug and play manner. They may not always be the best solution but a fantastic baseline that can be deployed and adapted to a usecase in less than an hour.

Better title: &quot;LLM OCR on Rugby screenshots to read score and clock&quot;

The moment I started reading this, I got reminded of this recent study: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;html&#x2F;2503.10212v1" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;html&#x2F;2503.10212v1</a><p>The scope is a bit different. The study uses an LLM to interpret pose estimation data and describe the behavior in each frame. The output is text which can be used to create embeddings of behavior. As someone who works in ethology, that&#x27;s a clever (but maybe expensive) idea.<p>I think the author could use something similar. With multi-person pose estimation models.

Reading the scoreboard from a TV screen and selling that data is restricted in many jurisdictions. This work is pretty naive I think.

Why the focus on scorekeeping? I feel like an AI model is overkill here, when you have text-based sources readily available such as news apps, Twitter feeds, and apps such as Livescore which would be easier and cheaper to scrape. They probably cover more matches that aren&#x27;t televised too.<p>I&#x27;d be curious to see what useful insights could be gleamed from the match commentary. You have the main commentator giving play-by-play objective reporting and then a &#x27;colour&#x27; commentator giving some subjective analysis during breaks in play. I bet there&#x27;s a lot of interesting ways this could be used.

I love that as soon as he writes,<p>&gt; The plan was simple.<p>You know you&#x27;re in for a funny read.<p>More seriously though, the JSON example from a vision language model is interesting but does not take into account how much extrapolation (hallucination) the model will insert over time.<p>For instance, even if not visible in the image, your VLM will probably start inserting details (such as the color of the team&#x27;s jersey) based on knowing the team&#x27;s three-letter identifier.<p>So the reliability of the system will go down over time, and it probably compounds if you&#x27;re using some of that info to feed further steps in the loop.

So Rugby is missing a lot of data beside the scoreline, so they created an AI that can extract the scoreline.

I wrote a similar script that used a TV tuner during the last World Cup. Since I had an ATSC source, I was able to just pull the CTA-708 captions directly and with little delay.

My observation is that watching Rugby on TV is no the same as watching a Rugby match. You&#x27;re watching something where choices have been made around what you&#x27;re to see, so your model is already restricted in what it can see.<p>You really need to take a &#x27;full pitch&#x27; feed directly from the venue, rather than what is broadcast.

I&#x27;m not a rugger bugger but every 5 seconds doesn&#x27;t really seem like often enough to be taking screenshots. In soccer anyway, a lot can happen in 5 seconds.

I don&#x27;t quite get how diffing frames allows you to find the scores.<p>TFA mentions comparing a frame with and without - but how do you generate that frame without? If you can already do it, what&#x27;s useful about doing that?

Does this mean there&#x27;s probably AI that&#x27;s already watching high profile football (soccer) matches?

&gt; Sending a full-resolution screenshot every five seconds gets expensive fast.<p>For now.

Why does yolo not work?

aw. I thought this would be about an AI cat that makes wrong commentaries that you can make pointless arguments against. There should be one.

I want AI that does my job while I watch rugby

&gt; We can’t hire analysts to watch every match and enter data manually.<p>I&#x27;m surprised there&#x27;s not enough fans willing to do that if you could gamify it.

TL;DR: It extracts the score from the video and gets text from the commentary in the audio.<p>I was hoping for more.