Cool. To be totally honest, I'd feel a lot better if we could just extend robots.txt especially given that we finally have a standard for it with IETF RFC 9309: <a href="https://datatracker.ietf.org/doc/html/rfc9309" rel="nofollow">https://datatracker.ietf.org/doc/html/rfc9309</a><p>Also might be worth looking at llms.txt (<a href="https://github.com/AnswerDotAI/llms-txt">https://github.com/AnswerDotAI/llms-txt</a>), though it's aimed to be more of an agent-friendly sitemap.xml alternative.