ROBOTS.TXT
TESTER
Enter a URL to fetch the site’s robots.txt and test whether that exact path is allowed or blocked for Googlebot. We parse every group, show the rule that decides the verdict, and list the declared sitemaps, the same check Google Search Console’s retired robots.txt tester used to do.
What is a Robots.txt Tester?
A robots.txt tester is a free tool that fetches a website’s robots.txt file and checks whether a given URL is allowed or blocked for a search engine crawler. It parses the crawl rules into groups and applies Google’s longest-rule-wins matching, so you can confirm a page is crawlable and see exactly which rule decides it.
How does a Robots.txt Tester work?
- 01We fetch robots.txt from the root of the domain in your URL.
- 02The rules are parsed into groups and matched with Google’s longest-rule-wins logic.
- 03We show whether the path is allowed for Googlebot and which rule decided it.
Frequently asked questions
How does Google decide which robots.txt rule applies?
Google does not use rule order. It evaluates every Allow and Disallow rule in the group that matches the crawler, and the most specific rule wins, measured by the number of characters in the rule path. If an Allow and a Disallow are equally specific, the Allow wins. This tool applies the same longest-match logic, so its verdict matches Googlebot’s.
Does robots.txt stop a page being indexed?
No. Robots.txt only controls crawling, not indexing. A URL blocked in robots.txt can still appear in Google if other pages link to it, just without a description, because Google never fetched it. To keep a page out of the index, allow crawling and use a noindex meta tag or header instead, the opposite of what most people assume.
Why is my page blocked when I did not intend it?
The usual causes are a broad Disallow rule (like Disallow: / left over from staging), a rule that matches more paths than expected because of a missing trailing slash, or a wildcard. This tool shows the exact rule that matched, so you can see which line to fix. A misconfigured robots.txt can silently deindex a whole site.