AI Bot
Checker

Two things, for each of 14 top AI crawlers. First, it parses your robots.txt file and checks whether the bot is blocked by a Disallow rule, either named directly or caught by a wildcard. Second, it sends a live request to your homepage identifying as that bot and compares the response with a normal browser request. If the browser gets HTTP 200 and the bot gets a 403, a 429, or a dropped connection, something at the server, WAF or CDN level is rejecting that bot.

Requests presenting that bot's user agent are being turned away before they reach your page, while normal browser traffic gets through. That is a server, firewall or CDN rule rather than anything in your robots.txt, which means you usually cannot fix it yourself. The tool writes a message you can send to your hosting company asking them to confirm whether the real crawler is allowed and to unblock it if not.

It can be, and the tool is upfront about this. Our test requests come from our own servers, not from OpenAI's or Anthropic's networks. Some security systems verify crawlers by IP address and only reject impostors while letting the real, verified bot through. From the outside those two cases look identical. That is why the generated message asks your host a precise question: is the genuine crawler, coming from the provider's published IP ranges, allowed to crawl the site? Their answer settles it either way.

Allow the search and assistant crawlers, decide for yourself on the training-only ones. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot decide whether your business can appear in ChatGPT, Claude and Perplexity answers, and AI assistants are now a real source of customers. Pure training crawlers like CCBot and Bytespider bring no direct traffic, so blocking those is a legitimate choice. Blanket-blocking everything makes you invisible to AI search.

Three usual suspects. Some SEO and security plugins add AI bot blocks as a default or one-click setting. Some hosts ship a managed robots.txt template with AI crawlers pre-blocked. And some site owners copied a "block all AI bots" snippet from a blog post in 2023 and forgot about it. Check your SEO plugin settings first, then ask your host whether they manage your robots.txt.

The big ones do. OpenAI, Google, Anthropic and Apple all document and honour their robots.txt tokens. Some training crawlers have a worse reputation, with Bytespider the best-known example of a bot widely reported to ignore robots.txt. That is exactly why this tool also runs the live server test: robots.txt tells you what you have asked bots to do, the live test tells you what your server actually enforces.

A means every AI search and assistant crawler can reach your site. C means only training-focused bots are blocked, which is a reasonable policy choice. D means one or two important search or assistant crawlers are blocked. F means several of the crawlers that power AI search answers cannot reach your site, and your business is unlikely to be cited by ChatGPT, Claude or Perplexity.

Results for each domain are cached for 15 minutes. That keeps repeat checks instant and stops the tool being used to bombard a site with requests, but it also means a change you made seconds ago will not show up straight away. The notice above your results tells you when the check actually ran and how long until a fresh one is available. Wait that out, run the check again, and you will see the effect of your change.

Yes. It makes around 15 ordinary GET requests to your homepage and robots.txt, deliberately staggered so no more than three arrive at once, with a pause between each batch. That is gentler than a single visitor browsing your site. It does not log in, post data, or probe for vulnerabilities, and it has no effect on your search rankings. Results are also cached per domain for 15 minutes, so repeat checks do not touch your site at all.