Robots.txt Tester

Test robots.txt rules the way crawlers actually read them

Robots.txt looks simple, but the matching rules are full of traps: only one user-agent group applies to each crawler, the longest matching pattern wins, wildcards and $ anchors change everything, and an innocent-looking rule can block your whole site. Since Google retired its official robots.txt Tester, there’s been no easy way to check rules before they cost you traffic. This free robots.txt tester fetches your live file, parses it under RFC 9309 — the same specification Googlebot follows — and shows you exactly which rule allows or blocks each URL, down to the line number.

Why test against AI crawlers too?

GPTBot, ClaudeBot, PerplexityBot, and CCBot now account for a meaningful share of crawl traffic, and your robots.txt is the primary control over whether AI systems can read your content. A common mistake: adding a dedicated User-agent: GPTBot group means GPTBot stops reading your * rules entirely — sites accidentally open (or close) far more than they intended. The access matrix makes those interactions visible at a glance. Manage your policy with Block AI Bots, generate the file with the Robots.txt Generator, and check how AI already sees your brand with the AI Visibility Checker.

Common robots.txt mistakes this tool catches

• Rules before any User-agent line — silently ignored by every crawler.
• Noindex in robots.txt — unsupported by Google since 2019.
• Crawl-delay for Googlebot — Google ignores it entirely.
• Files over 500 KB — Google stops parsing at 500 KiB; later rules don’t exist.
• 5xx errors on robots.txt — Google treats the whole site as temporarily disallowed.