1. Verify AI crawler access
AI search engines use dedicated crawlers to read your pages. If your robots.txt blocks them, your content is invisible — regardless of how well it ranks on Google. Start by checking that OpenAI's GPTBot, Anthropic's ClaudeBot, and Google's Google-Extended have access to your key pages.
Also create an llms.txt file at your domain root — a Markdown summary of your site that AI models use as a discovery map. Unlike robots.txt (which controls access), llms.txt tells AI what your site is about and which pages matter most.
- Verify robots.txt allows GPTBot, ClaudeBot, and Google-Extended on key pages
- Create and publish llms.txt at yoursite.com/llms.txt
- Run the free LLMs.txt Checker to validate structure and best practices