Home › Free SEO tools › robots.txt Tester
robots.txt Tester
Paste your robots.txt and test any URL against Googlebot, Bingbot, GPTBot and other crawlers. See the exact rule that allows or blocks it, plus common mistakes.
Results for all crawlers
Checks
FAQ
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it. Use a noindex tag, and let Google crawl the page so it can see that tag.
How does Google choose between Allow and Disallow?
Google uses the most specific rule, which is the longest matching path. If an Allow and a Disallow are the same length, Allow wins. This tester follows the same logic.
Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking GPTBot or ClaudeBot keeps your content out of their training data, but blocking search-focused bots such as OAI-SearchBot or PerplexityBot can also remove you from their answers.
More free tools
SERP Snippet Preview & Meta Length Checker
See how your title and description look in Google, in English or Arabic.
Free toolSchema Markup Generator
Create FAQ, LocalBusiness and Article JSON-LD in seconds.
Free toolhreflang Tag Generator
Build hreflang tags for Arabic and English sites without mistakes.
Free toolArabic & English Word Counter
Count words, characters, sentences and reading time in Arabic or English.
Free toolKeyword Density Checker
Find the most used words and phrases, and check your target keyword.