AI crawlability checker
Paste any URL. We read that site’s robots.txt and check the page against every major AI crawler, then show which are allowed, which are blocked, and the exact rule that decides it.
| Crawler | Used for | Status | Rule that applies |
|---|
This checks the rules a site declares. It does not prove that a crawler visited, indexed or cited the page.
How to use this
- Enter the full URL of a page you care about, such as a product or pricing page.
- Run the check. If your browser cannot read the file, paste your robots.txt instead.
- Read the status for each crawler and the rule that applies.
- Download the report to share with your developer.
What to do next
- Decide which AI uses you want to allow: search and answers, training, or both.
- Update robots.txt with the AI robots.txt builder, then run this check again.
- Remember that robots.txt is a request. Check your server logs to see which bots actually visit.
Common questions
What does this checker test?
It reads the robots.txt file for the domain you enter and applies the standard matching rules (the most specific Allow or Disallow wins) for each AI crawler, for the exact path you entered.
Does "Allowed" mean my page will be cited?
No. It only means the site does not ask that crawler to stay away. Whether a page is fetched, indexed or cited depends on the assistant.
Why would I block some AI crawlers?
Some companies allow search and answer crawlers such as OAI-SearchBot or PerplexityBot, which can send visitors, but block training crawlers such as GPTBot. It is a business choice.
Why could the tool not read my robots.txt?
Browsers block reading files from other sites unless the site allows it. When that happens, open your robots.txt, copy the text and paste it into the tool.
More free tools
Track this every day.
Seeqly asks your buyer questions on every AI assistant daily, with citations and share of voice. ChatGPT, Google AI Overviews, Gemini, Claude, Perplexity and Grok. Try Basic free for 7 days.