Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]!<p>We still have a long way to go for the grand prize -- we'll be back next year. Also got some new stuff in the works for 2025.<p>Watch for the official ARC Prize 2024 paper coming Dec 6. We're going to be overviewing all the new AI reasoning code and approaches open sourced via the competition [3].<p>[1] <a href="https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/" rel="nofollow">https://deepmind.google/discover/blog/ai-solves-imo-problems...</a><p>[2] <a href="https://github.com/ekinakyurek/marc">https://github.com/ekinakyurek/marc</a><p>[3] <a href="https://x.com/arcprize" rel="nofollow">https://x.com/arcprize</a>
Test-Time Training is incredibly powerful. Most recently, it has been shown that Self-Attention can in fact be viewed through the lens of test-time training, with a kernel-smoother "learning" from context. Simply replacing that with more powerful models than a kernel-smoother result in very capable and scalable models!<p><a href="https://arxiv.org/abs/2407.04620" rel="nofollow">https://arxiv.org/abs/2407.04620</a>