AI Crawler
Checker

Four signals. First, it parses your robots.txt file and checks each of the 14 top AI crawlers against it, whether the bot is named directly or caught by a wildcard. Second, it sends a live request to your homepage identifying as that bot and compares the response with a normal browser request. If the browser gets HTTP 200 and the bot gets a 403, a 429, or a dropped connection, something at the server, WAF or CDN level is rejecting that bot. Third, it sends one more request as a neutral bot nobody has heard of, which is what separates a rule aimed at GPTBot by name from a host that filters all unrecognised bots. Fourth, it reads your <meta name="robots"> tag and X-Robots-Tag header for noindex and nosnippet, because reaching a page and being allowed to quote it are different things.

Requests presenting that bot's user agent are being turned away before they reach your page, while normal browser traffic gets through. That is a server, firewall or CDN rule rather than anything in your robots.txt, which means you usually cannot fix it yourself. The tool writes a message you can send to your hosting company asking them to confirm whether the real crawler is allowed and to unblock it if not.

It can be, and the tool is upfront about this. Our test requests come from our own servers, not from OpenAI's or Anthropic's networks. Some security systems verify crawlers by IP address and only reject impostors while letting the real, verified bot through. From the outside those two cases look identical. That is why the generated message asks your host a precise question: is the genuine crawler, coming from the provider's published IP ranges, allowed to crawl the site? Their answer settles it either way.

Allow the search and assistant crawlers, decide for yourself on the training-only ones. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot decide whether your business can appear in ChatGPT, Claude and Perplexity answers, and AI assistants are now a real source of customers. Pure training crawlers like CCBot and Bytespider bring no direct traffic, so blocking those is a legitimate choice. Blanket-blocking everything makes you invisible to AI search.

Three usual suspects. Some SEO and security plugins add AI bot blocks as a default or one-click setting. Some hosts ship a managed robots.txt template with AI crawlers pre-blocked. And some site owners copied a "block all AI bots" snippet from a blog post in 2023 and forgot about it. Check your SEO plugin settings first, then ask your host whether they manage your robots.txt.

The big ones do. OpenAI, Google, Anthropic and Apple all document and honour their robots.txt tokens. Some training crawlers have a worse reputation, with Bytespider the best-known example of a bot widely reported to ignore robots.txt. That is exactly why this tool also runs the live server test: robots.txt tells you what you have asked bots to do, the live test tells you what your server actually enforces.

Those are two different questions and they can disagree. The crawler checks ask whether a bot can reach your homepage. The noindex and nosnippet check asks what your page tells it to do once it arrives. A page every AI crawler can fetch is still absent from AI answers if it carries noindex, and cannot be quoted in one if it carries nosnippet (or its equivalent, max-snippet:0), because a quote is exactly the text extract that directive forbids. We read both the <meta name="robots"> tag in your page and the X-Robots-Tag response header, because either can carry them. If you did not put them there deliberately, look at the per-page settings in your SEO plugin, at a staging flag that survived going live, or at a server rule adding the header.

No, deliberately. The grade scores crawler access, which is what the bot-by-bot results measure, so folding a different kind of problem into the same letter would make it mean two things at once. An indexability finding is reported separately, in its own section and in the summary line under your grade. In practice treat it as at least as serious as a blocked crawler: a noindex page is invisible to AI search however many bots can reach it.

A means every AI search and assistant crawler can reach your site. C means only training-focused bots are blocked, which is a reasonable policy choice. D means one or two important search or assistant crawlers are blocked. F means several of the crawlers that power AI search answers cannot reach your site, and your business is unlikely to be cited by ChatGPT, Claude or Perplexity.

Results for each domain are cached for 15 minutes. That keeps repeat checks instant and stops the tool being used to bombard a site with requests, but it also means a change you made seconds ago will not show up straight away. The notice above your results tells you when the check actually ran and how long until a fresh one is available. Wait that out, run the check again, and you will see the effect of your change.

Yes. It makes around 15 ordinary GET requests to your homepage and robots.txt, deliberately staggered so no more than three arrive at once, with a pause between each batch. That is gentler than a single visitor browsing your site. It does not log in, post data, or probe for vulnerabilities, and it has no effect on your search rankings. Results are also cached per domain for 15 minutes, so repeat checks do not touch your site at all.