Robots.txt Tester
Test whether Googlebot, Bingbot or AI bots may crawl a URL according to the site's robots.txt.
How to use the Robots.txt Tester
- 1Enter the page URL to testFor example yoursite.com/shop/product-1.
- 2Pick a botGooglebot is the most important; try GPTBot to see if AI crawlers may read your site.
- 3TestSee Allowed or Blocked and the rule that decided it.
What is the Robots.txt Tester?
One wrong line in robots.txt can hide a whole website from Google. This robots.txt tester reads the live robots.txt file of any website and tells you whether a specific page is allowed or blocked for Googlebot, Googlebot-Image, Bingbot, AI crawlers like GPTBot and ClaudeBot, SEO bots like AhrefsBot and SemrushBot, or any other bot.
The result shows the exact rule and line number that decided it, highlights that line in the file, and lists the sitemaps declared in robots.txt. It follows Google's rules: the most specific user-agent group wins, and among matching rules the longest path wins, with Allow winning a tie. The * and $ wildcards are supported.
Enter any page URL and pick a bot. The tool reads the live robots.txt and shows Allowed or Blocked with the exact rule.
Robots.txt rules in one minute
| Line | Meaning |
|---|---|
| User-agent: * | The rules below apply to all bots |
| Disallow: /admin/ | Do not crawl anything starting with /admin/ |
| Allow: /admin/public/ | But this part may be crawled |
| Disallow: /*?s= | Do not crawl URLs that contain ?s= |
| Disallow: /*.pdf$ | Do not crawl URLs ending in .pdf |
| Sitemap: https://site.com/sitemap.xml | Where the sitemap is |
Create a new file with the Robots.txt Generator.
Blocked is not the same as noindex
Robots.txt stops crawling, not indexing. A blocked page can still appear in Google (without a description) if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.
Should you block AI bots?
Blocking GPTBot, ClaudeBot or Google-Extended stops those companies from using your content to train AI, but it can also mean your brand is mentioned less in AI answers. Decide based on your business: publishers often block, while shops and service businesses usually allow.
Questions about the Robots.txt Tester
Is the robots.txt tester free?
Yes, free with no sign-up.
What if a site has no robots.txt?
Then every bot may crawl every page. That is fine for most small sites, but adding a Sitemap line helps.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay; Bing and some other bots follow it.
Is it safe to use?
Yes. We only read public information such as a web page, DNS or registration record. Nothing is changed on the site you check.
Does it work on mobile?
Yes, it works in Chrome, Safari and other modern browsers on Android, iPhone, Windows and Mac.
This tool is also called
- robots.txt tester
- robots txt checker
- is my page blocked by robots.txt
- googlebot crawl test
- robots.txt validator
Last updated: November 2026 · Made by the FaisalitebTools team in Bangladesh
Do real keyword research with Semrush and Ahrefs
Free checkers show the basics. Semrush and Ahrefs show keywords, competitors and backlinks so you can rank, at group-buy prices. Pay with bKash, Nagad, Rocket or card and get access in minutes.
Semrush$2.45/30 Days
Ahrefs$8.60/30 Days
Ubersuggest$2.70/30 Days
Se Ranking$2.05/30 Days
AnswerThePublic$2.87/30 Days
Brightlocal$3.69/30 Days
