Free AI crawler checker
Can ChatGPT, Claude, Perplexity and Gemini read your website? Check which AI crawlers your robots.txt allows or blocks in seconds. Free, no sign-up.
The 15 AI crawlers we check
Each AI company runs different crawlers for different jobs. Blocking a training crawler keeps your content out of future models; blocking a search or user-request crawler can keep you out of today’s answers.
| User agent | Company | Product | Used for |
|---|---|---|---|
| GPTBot | OpenAI | ChatGPT | Training |
| OAI-SearchBot | OpenAI | ChatGPT search | Search |
| ChatGPT-User | OpenAI | ChatGPT | User requests |
| ClaudeBot | Anthropic | Claude | Training |
| Claude-SearchBot | Anthropic | Claude search | Search |
| Claude-User | Anthropic | Claude | User requests |
| PerplexityBot | Perplexity | Perplexity | Search |
| Perplexity-User | Perplexity | Perplexity | User requests |
| Google-Extended | Gemini | Training | |
| Bingbot | Microsoft | Bing and Copilot | Search |
| Applebot-Extended | Apple | Apple Intelligence | Training |
| meta-externalagent | Meta | Meta AI | Training |
| Amazonbot | Amazon | Alexa and Rufus | Search |
| CCBot | Common Crawl | Open web dataset | Training |
| Bytespider | ByteDance | Doubao | Training |
How to allow AI search crawlers
To let AI search and user-request crawlers read your site while keeping private areas closed, add groups like this to your robots.txt and keep your existing rules for everything else:
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Disallow: /account/
Then test again here, and confirm the pages you want cited are in your sitemap. For the full picture, including page structure, schema and content gaps, see Marketing & Technical Analysis or read the glossary entry on AI crawlers.
Questions
What does the AI crawler checker test?
It reads your public robots.txt and applies the same rules the crawlers do to your homepage for 15 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Common Crawl and ByteDance. It also checks whether you publish an llms.txt file and which sitemaps robots.txt declares.
Should I block AI crawlers?
It depends on the crawler. Search and user-request crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot fetch pages to answer questions and cite sources, so blocking them can keep your brand out of answers. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect data for future models; many brands allow them, others block them for licensing reasons.
Does allowing crawlers mean AI will recommend my brand?
No. Access is a precondition, not a result. Whether AI systems mention, recommend and cite you depends on your content, third-party sources and competitors. A free Digraph report shows how AI actually describes your brand today.
What does "Allowed, some paths excluded" mean?
The crawler can read your homepage, but your robots.txt disallows at least one path for it, such as an account area or a pricing section. Check that the excluded paths do not include pages you want AI answers to cite.
Is llms.txt required?
No. llms.txt is a proposed file that summarises your most useful pages for AI systems. Support varies between providers, so treat it as a helpful addition to a crawlable site and a complete sitemap, not a replacement.
Put this into practice for your brand.
Digraph tracks how ChatGPT, Gemini, Perplexity and 6 other AI systems describe your brand, finds what to fix and plans the next move with our team.