New
AI Readiness Checker
Score how well a page can be read, understood and cited by AI search and assistants.
How it works
AI Crawler Checker
Find out which AI bots can read your site. Check a live robots.txt or paste your own, test any page path, and generate rules to block AI training while staying visible in AI search.
AI crawler access
Check a site or paste a robots.txt
You'll see whether each AI crawler can reach the path, and which rule decided it.
| Crawler | Purpose | Access | Decided by |
|---|
- 01
Enter your domain to fetch its robots.txt, or paste the file.
- 02
Optionally enter a page path to test. Each AI crawler shows allowed or blocked, with the rule that decided it.
- 03
Use the generator to create rules for your policy and add them to your robots.txt.
About this tool
Training crawlers, AI search crawlers and assistant fetches
AI companies use different crawlers for different jobs. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so assistants can cite and link to them. User-triggered fetchers such as ChatGPT-User and Claude-User load a page because someone asked an assistant about it. Blocking them all removes your site from AI answers, not just from training data.
This checker reads robots.txt the way the standard (RFC 9309) and Google describe: it finds the group for each crawler's token, falls back to the * group, and applies the most specific matching rule, with Allow winning ties. Some tokens, such as Google-Extended and Applebot-Extended, aren't separate crawlers but control how content already crawled for search is used for AI. robots.txt is a request, not an enforcement mechanism; well-behaved crawlers follow it.
22 AI crawlers
OpenAI, Anthropic, Google, Apple, Perplexity, Meta, Common Crawl and more, grouped by purpose.
Standards-based matching
RFC 9309 groups, wildcards, $ anchors and longest-match precedence.
Test any path
See the exact rule and line that allows or blocks each crawler.
Rule generator
Block AI training only, block all AI, or allow all, ready to paste.
FAQ
Frequently asked questions
How do I block ChatGPT from training on my site?
Add “User-agent: GPTBot” followed by “Disallow: /” to robots.txt. To stay in ChatGPT search results, don't block OAI-SearchBot.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls use in Gemini models and grounding. Googlebot and Search indexing aren't affected.
Will blocking AI crawlers remove my content from existing models?
No. robots.txt only affects future crawling. Content already collected may remain in existing training data.
Do all AI bots respect robots.txt?
Major AI companies say their crawlers do, with some exceptions for user-initiated fetches. Less reputable scrapers may ignore it; for those you need server or firewall rules.
What happens when I fetch a URL?
The address is sent to our fetching service, which downloads only that public page (or its robots.txt, sitemap or llms.txt) and returns it to your browser. Nothing is stored. Private network addresses and non-web ports are refused. You can always paste content instead.
Missing something?
Tell us the tool you wish existed.
Instant.tools grows from real requests. Describe the task you're stuck on and we'll consider building it next.