Long context windows are IMO, “AGI enough.”<p>100M context window means it can probably store everything you’ve ever told it for years.<p>Couple this with multimodal capabilities, like a robot encoding vision and audio into tokens, you can get autonomous assistants than learn your house/habits/chores really quickly.
It should be benchmarked against something like RULER[1]<p>1: <a href="https://github.com/hsiehjackson/RULER">https://github.com/hsiehjackson/RULER</a> (RULER: What’s the Real Context Size of Your Long-Context Language Models)
Context windows are becoming larger and larger, and I anticipate more research focusing on this trend. Could this signal the eventual demise of RAG? Only time will tell.
I recently experimented with RAG and the limitations are often surprising (<a href="https://www.lycee.ai/blog/rag-fastapi-postgresql-pgvector" rel="nofollow">https://www.lycee.ai/blog/rag-fastapi-postgresql-pgvector</a>). I wonder if we will see some of the same limitations for long context LLM. In context learning is probably a form of semantic / lexical cues based arithmetic.
I was wondering how they could afford 8000 H100’s, but I guess I accidentally skipped over this part:<p>> We’ve raised a total of $465M, including a recent investment of $320 million from new investors Eric Schmidt, Jane Street, Sequoia, Atlassian, among others, and existing investors Nat Friedman & Daniel Gross, Elad Gil, and CapitalG.<p>Yeah, I guess that'd do it. Who are these people and how'd they convince them to invest that much?