docs
AI features and your website
Google's own guidance on AI Overviews and AI Mode. Short version: the same fundamentals apply, and no special AI files or markup are required.
Topic
AI crawlers and access
Which bots visit your site, what they can read, and how to allow or block them.
AnswerVerdict: Watch
Does llms.txt matter? What the data says
Lots of plugins and tools now generate an llms.txt file. Here's what it is, what the evidence shows, and our verdict.
PlaybookVerdict: Do it now
Headless sites and AI crawlers: is your site an empty page to ChatGPT?
Most AI crawlers don't run JavaScript. If your headless or app-style site builds content in the browser, they may see nothing at all. Here's how to check in five minutes and what to fix.
Resources
docs
Overview of OpenAI crawlers
The official list of OpenAI's user agents (GPTBot, OAI-SearchBot, ChatGPT-User) and what each one does. Use it to write robots.txt rules on purpose.
study
Why your site is invisible to ChatGPT and Claude
AI crawlers don't execute JavaScript, so client-rendered pages can look empty to them. Required reading for anyone running a headless or app-style site.