Can we RAG the whole web?

This is exactly what <a href="https:&#x2F;&#x2F;www.perplexity.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.perplexity.ai&#x2F;</a> is trying to do. Maybe not &quot;RAGing&quot; the entire internet, but sure using the mapping between natural language query to their own (probably) vector database which contains &quot;source of truth&quot; from the internet.<p>The way how they build that database and what models they use for text tokenization, embeddings generation and ranking at &quot;internet&quot; scale is the secret sauce that enabled them to raise more than $165M to date.<p>For sure this is where the internet search will be in a couple of years and that&#x27;s why Google got really concerned when original ChatGPT was released. That said, don&#x27;t assume Google is not already working on something similar. In fact, the main theme of their Google Next conference was about LLMs and RAG.

Is connecting a search engine to an LLM not technically a RAG for the whole web?

Cool idea. This is a decentralized RAG approach and useful for individual site, e.g. those from Wordpress. How do you find the site that you want to &quot;RAG&quot; on, though? Individual domains can be vast, e.g. Google itself.

Well, there&#x27;s nothing new under the sun. The whatever cooperation model you may have come up with, it has been invented again, and again, and again.<p>Before you invent a new protocol, look at Semantic Web (RDF et al), and Google Microformats, and...

I think we need a search engine that has an API. Doesn&#x27;t Kagi has an API?

FIYDRI^: The core idea discussed in this post is less about RAG and more about sharing web content in packages that are easier for crawlers to access - including an experiment that uses downloadable SQLite databases for that.<p>^ For If You Didn&#x27;t Read It

this is exa&#x27;s mission: <a href="https:&#x2F;&#x2F;exa.ai">https:&#x2F;&#x2F;exa.ai</a>

I&#x27;ve been using Kagi&#x27;s &quot;Quick answer&quot; more and more these days, which I guess is a form of &quot;index the whole web&quot; RAG.<p>Here&#x27;s their blog article for it: <a href="https:&#x2F;&#x2F;help.kagi.com&#x2F;kagi&#x2F;ai&#x2F;quick-answer.html" rel="nofollow">https:&#x2F;&#x2F;help.kagi.com&#x2F;kagi&#x2F;ai&#x2F;quick-answer.html</a> You have to fire up your bullshit detector when looking at the results, but I find it saves a good 3&#x2F;4 clicks on average.

&quot;RAG, or Retrieval-Augmented Generation, is a method where a language model such as ChatGPT first searches for useful information in a large database and then uses this information to improve its responses.&quot;

Aren&#x27;t the LLM&#x27;s already trained on the whole web? no need to RAG, in theory.