Site Files Checker
robots.txt, sitemap.xml, ads.txt, llms.txt, security.txt and AI crawler rules
Site Files Checker — Complete Guide & Instructions
Probe all the standard well-known files on a domain in one shot — robots.txt, sitemap, ads.txt and app-ads.txt, llms.txt, security.txt, PWA manifest, service worker and favicon — and see which AI crawlers the robots.txt mentions.
How to Use Site Files Checker
- 1 Enter a domain name.
- 2 Click Check — every well-known path is probed.
- 3 See status and size per file, plus the AI crawler table.
Key Features & Capabilities
- Ten well-known files probed with status and size.
- AI crawler access table — GPTBot, ClaudeBot, PerplexityBot and more.
- Single-domain run completes in seconds.