Free Tools Guides Combo Packages Single Tools Custom Package Registration Login
All Tools
Loading tools...
All 65 tools with prices & limits
  1. Home
  2. Free Tools
  3. SEO & marketing tools

Robots.txt Tester

Test whether Googlebot, Bingbot or AI bots may crawl a URL according to the site's robots.txt.

Free, no sign-up Live check
Loading tool…

How to use the Robots.txt Tester

  1. 1
    Enter the page URL to testFor example yoursite.com/shop/product-1.
  2. 2
    Pick a botGooglebot is the most important; try GPTBot to see if AI crawlers may read your site.
  3. 3
    TestSee Allowed or Blocked and the rule that decided it.

What is the Robots.txt Tester?

One wrong line in robots.txt can hide a whole website from Google. This robots.txt tester reads the live robots.txt file of any website and tells you whether a specific page is allowed or blocked for Googlebot, Googlebot-Image, Bingbot, AI crawlers like GPTBot and ClaudeBot, SEO bots like AhrefsBot and SemrushBot, or any other bot.

The result shows the exact rule and line number that decided it, highlights that line in the file, and lists the sitemaps declared in robots.txt. It follows Google's rules: the most specific user-agent group wins, and among matching rules the longest path wins, with Allow winning a tie. The * and $ wildcards are supported.

Enter any page URL and pick a bot. The tool reads the live robots.txt and shows Allowed or Blocked with the exact rule.

Robots.txt rules in one minute

LineMeaning
User-agent: *The rules below apply to all bots
Disallow: /admin/Do not crawl anything starting with /admin/
Allow: /admin/public/But this part may be crawled
Disallow: /*?s=Do not crawl URLs that contain ?s=
Disallow: /*.pdf$Do not crawl URLs ending in .pdf
Sitemap: https://site.com/sitemap.xmlWhere the sitemap is

Create a new file with the Robots.txt Generator.

Blocked is not the same as noindex

Robots.txt stops crawling, not indexing. A blocked page can still appear in Google (without a description) if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.

Should you block AI bots?

Blocking GPTBot, ClaudeBot or Google-Extended stops those companies from using your content to train AI, but it can also mean your brand is mentioned less in AI answers. Decide based on your business: publishers often block, while shops and service businesses usually allow.

Questions about the Robots.txt Tester

Is the robots.txt tester free?

Yes, free with no sign-up.

What if a site has no robots.txt?

Then every bot may crawl every page. That is fine for most small sites, but adding a Sitemap line helps.

Does Google follow Crawl-delay?

No. Google ignores Crawl-delay; Bing and some other bots follow it.

Is it safe to use?

Yes. We only read public information such as a web page, DNS or registration record. Nothing is changed on the site you check.

Does it work on mobile?

Yes, it works in Chrome, Safari and other modern browsers on Android, iPhone, Windows and Mac.

This tool is also called

  • robots.txt tester
  • robots txt checker
  • is my page blocked by robots.txt
  • googlebot crawl test
  • robots.txt validator

Last updated: November 2026 · Made by the FaisalitebTools team in Bangladesh

Do real keyword research with Semrush and Ahrefs

Free checkers show the basics. Semrush and Ahrefs show keywords, competitors and backlinks so you can rank, at group-buy prices. Pay with bKash, Nagad, Rocket or card and get access in minutes.

See all premium tools