Hmmm... checking against external sources is an interesting idea -- but using Google as a source of ground truth is a little bit tricky, given how often these days Google itself is spitting up confabulated AI-generated crud (or other low-quality stuff).
For those interested in using search-augmented "reasoning", I implemented something similar in Emerging Trajectories[1], an open source package that forecasts geopolitical and economic events. We extract facts[2] from various websites (Google searches, news articles, RSS feeds) and have the LLM generate a hypothesis on a metric.<p>We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3].<p>[1] <a href="https://emergingtrajectories.com/" rel="nofollow">https://emergingtrajectories.com/</a><p>[2] Sample code: <a href="https://github.com/wgryc/emerging-trajectories/blob/main/emergingtrajectories/factsforecaster.py">https://github.com/wgryc/emerging-trajectories/blob/main/eme...</a><p>[3] <a href="https://emergingtrajectories.com/a/statement/28" rel="nofollow">https://emergingtrajectories.com/a/statement/28</a>