What this checker looks at
- Crawler access. Your robots.txt file tells each crawler what it may read. We check the rules for the crawlers that feed ChatGPT search, Claude, Perplexity, Google and Bing.
- Readable text. Many AI-built apps send an almost empty page and fill it in with JavaScript. A crawler that doesn’t run JavaScript finds nothing to quote.
- Search settings. A
noindextag or a site-wide block keeps you out of search results, and out of the answers built on them. - Structured data. JSON-LD on your homepage states your app’s name, logo and website in a form search engines read without guessing.
- llms.txt, if you have one. It’s optional, more on that below.
Search crawlers and training crawlers are different
Each AI company runs separate crawlers for different jobs, and robots.txt treats each one on its own:
- OpenAI:
OAI-SearchBotfinds pages for ChatGPT search, andGPTBotcollects training data. OpenAI’s crawler docs say sites that block OAI-SearchBot won’t be shown in ChatGPT search answers. - Anthropic:
Claude-SearchBotindexes pages for Claude’s search, andClaudeBotcollects training data, according to Anthropic’s help center. - Perplexity:
PerplexityBotfinds and links websites in its answers and isn’t used for model training, says Perplexity’s documentation. - Google: AI Overviews and AI Mode use the regular
Googlebot.Google-Extendedonly controls Gemini training and grounding, and Google’s crawler overview says it doesn’t affect Search.
So you can block training and still be found. Block GPTBot and ClaudeBot if you don’t want your pages used for training, and keep the search crawlers allowed.
Do you need llms.txt?
llms.txt is a short Markdown file that lists your most useful pages for AI tools. It’s a proposal from 2024, not a standard, and Google’s guide to AI features in Search says you don’t need new machine-readable files to appear in AI Overviews or AI Mode. It’s cheap to add and some coding assistants use it, so it can’t hurt, but it won’t get you cited on its own.
Allowed in, but still not mentioned?
Access is only the first step. Assistants quote pages that answer a question clearly and sites that other people mention. Our guide on getting found in ChatGPT and other assistants covers the rest.