Robots.txt
Checker
Analyse any site's robots.txt file. See parsed rules, test if specific URLs are blocked, check sitemap references, and spot common mistakes.
Check Robots.txt
Enter a domain name to fetch and analyse its robots.txt file
Analysis
Test URL Path
Parsed Rules
What Does the Robots.txt Checker Do?
This tool fetches any site's robots.txt file and breaks it down into clear, readable rules. You see every user-agent directive, which paths are allowed, which are blocked, and which sitemaps are declared. It also flags common mistakes: wildcard rules that accidentally block CSS or JavaScript, missing sitemap references, and directives that conflict with each other.
The built-in URL path tester lets you check whether a specific page would be blocked or allowed by the current rules. Type a path, pick a user-agent, and get an instant answer.
Who Is This Tool For?
SEO professionals debugging indexing problems. If Google isn't indexing a page that should be visible, a misconfigured robots.txt rule is often the cause. Developers use it before launching a new site to verify crawl rules. Site owners use it to check whether their WordPress robots.txt is helping or hurting their search visibility.
Frequently Asked Questions
A common cause is your robots.txt file accidentally blocking those pages. This tool fetches and parses your robots.txt, showing all crawl rules grouped by user-agent. You can test specific page URLs to see whether Googlebot is allowed or blocked. Also check for noindex meta tags on the page itself using the Meta Tag Checker.
If your staging site is publicly accessible, Google may be crawling and indexing it, which creates duplicate content problems. Enter your staging domain here to verify that Googlebot is blocked. A properly configured robots.txt should have Disallow: / for all user-agents on staging and development environments.
Your robots.txt should disallow crawlers from admin paths like /wp-admin/ or /admin/. Enter your site URL here to check whether those paths are blocked for Googlebot. If they're not, add a disallow rule. For pages already indexed, you'll also need to request removal through Google Search Console.
Yes. A Disallow: / directive for Googlebot blocks all crawling. Google may still index URLs if other sites link to them, but it can't read the page content, resulting in thin or missing listings. This tool flags a full-site block as an error if detected. Note that robots.txt controls crawling, not indexing. To fully prevent indexing, you also need a noindex meta tag.
Blocking CSS and JS files prevents Google from rendering your pages properly, which hurts rankings. Google needs to see your site the way users see it. This was common practice years ago but is now actively harmful. This tool lets you test specific file paths (like /assets/css/style.css) against your crawl rules to check whether they're allowed for Googlebot.
Robots.txt is a public file on every website. Enter any domain here and it fetches and parses their robots.txt, showing all user-agent rules, disallowed paths, sitemap references, and crawl-delay directives. Useful for understanding how competitors manage search engine access to different sections of their site.
The Sitemap: line tells search engines where to find your XML sitemap. It's the simplest way to ensure crawlers discover your sitemap without having to submit it manually in Search Console. You can list multiple sitemaps. The URL must be absolute (starting with https://). This tool shows whether your robots.txt includes a sitemap reference.
Want to understand robots.txt directives and avoid common crawling mistakes? Read the full Robots.txt Checker guide.
Need Your Site Crawled Properly?
All 365i hosting includes a properly configured robots.txt, XML sitemaps, and Cloudflare CDN for fast, discoverable websites.
