Robots.txt Tester
Inspect crawler rules and test whether a path is allowed
Sitemaps
Test a path
Rule groups
User-agent:
Robots.txt Tester — Complete Guide & Instructions
MakeCard Robots.txt Tester fetches the live robots.txt file of any public website and breaks it down into readable rule groups. You can see every User-agent block with its Allow and Disallow directives, all declared sitemap URLs, and then test a specific path against a chosen crawler name to get an instant ALLOWED or BLOCKED verdict. It is made for SEO professionals, developers, and site owners debugging crawl issues.
How to Use Robots.txt Tester
- 1 Type a domain such as example.com into the "Domain" field.
- 2 Click "Fetch robots.txt" — the tool retrieves the live file and displays the raw contents.
- 3 Review the parsed rule groups: each User-agent block lists its Disallow and Allow rules, color-coded red and green.
- 4 Check the Sitemaps section for every sitemap URL declared in the file.
- 5 Under "Test a path", enter a user-agent (e.g. * or Googlebot) and a path (e.g. /admin/panel) to see whether it is ALLOWED or BLOCKED, with the matching rule shown.
Key Features & Capabilities
- Live fetch of the real robots.txt file from any public domain.
- Parsed view of every User-agent group with its Allow and Disallow rules.
- Sitemap declarations extracted and listed separately.
- Path tester that applies the longest-match rule, exactly like Googlebot does.
- Clear ALLOWED / BLOCKED verdict showing the exact directive that matched.
- Works for wildcard (*) and named crawlers such as Googlebot or Bingbot.
Supported Formats & Specifications
| Parameter | Specification |
|---|---|
| Input | Domain name (e.g. example.com) |
| Parsed Output | User-agent groups, Allow/Disallow rules, Sitemap URLs, raw file text |
| Path Test | User-agent + URL path, longest-match rule evaluation |
| Fetch Mode | Server-side request retrieves the live file (browsers cannot fetch cross-origin robots.txt) |
| Verdicts | ALLOWED or BLOCKED with the matching directive displayed |
Practical Everyday Use Cases
- Confirming a newly added Disallow rule actually blocks the intended section before it goes live.
- Checking whether a staging or admin path is accidentally exposed to crawlers.
- Finding the XML sitemap URLs a competitor or your own site declares.
- Debugging why Google Search Console reports a page as "blocked by robots.txt".
- Verifying that AI or specific named bots are handled by dedicated rule groups.
Pro Tips for Best Results
- Robots.txt uses longest-match precedence — a more specific rule beats a shorter one regardless of order.
- A Disallow with an empty path means nothing is blocked for that user-agent group.
- Test with both * and the exact bot name; a named group overrides the wildcard group entirely.
- Remember robots.txt is advisory — it controls crawling, not indexing or access security.
Common Problems & Solutions
The domain may not publish one, or it may block automated fetches. Check https://domain.com/robots.txt manually in a browser.
A page can be disallowed from indexing by a noindex meta tag or X-Robots-Tag header instead — robots.txt is only one layer.
Subdomains have their own robots.txt — www.example.com and blog.example.com are checked separately.
Privacy & Security
100% Client-Side Execution
The domain you enter is sent to our server, which fetches that site's public robots.txt file on your behalf. Path tests run locally in your browser; we do not log or store the domains you check.
Frequently Asked Questions
Everything you need to know about using Robots.txt Tester
Enter your domain in the tool, fetch the live file, then type a user-agent and path under "Test a path". The verdict shows whether that crawler may access the URL and which rule decided it.
It blocks the matching user-agent group from crawling every path on the site. A Disallow with no value (empty path) blocks nothing.
Not reliably. Robots.txt stops crawling, but a page can still appear in results via inbound links — use a noindex meta tag if you need guaranteed removal from the index.
The asterisk is a wildcard meaning "all crawlers". Rules in that group apply to every bot that does not have its own dedicated group in the file.
Related FreeToolsBazaar Tools
Ready to Use Robots.txt Tester?
100% free, browser-based, and private. No installation, registration, or file upload to external servers required.
Start Using Robots.txt Tester