I'm Jed, the author. I wrote this guide up because I'm working on an AI search startup using agents, and have been asked by other founders a few times why their content doesn't show up well with AI search engines like ours or Perplexity or ChatGPT with search, or with agent computer-use like the new OpenAI Operator.<p>We're ingesting around 30m to 50m pages a day so have learned a lot about what blocks AI crawlers and agents. And I wanted to share some guidelines on how to make sure your startup or documentation or other content shows up as well as possible for agent use.