Robots.txt & Sitemap Checker
Fetch a site’s robots.txt and XML sitemap, count sitemap URLs, and test whether a path is allowed for Googlebot or other crawlers.
Test a URL path
About the Robots.txt & Sitemap Checker
A single wrong line in robots.txt can hide an entire website from Google, and a broken sitemap slows down how fast new pages get indexed. This checker downloads your live robots.txt, highlights risky rules, follows the Sitemap lines (or tries /sitemap.xml) and tells you whether the sitemap is valid XML and how many URLs it lists.
The built-in tester applies the same matching rules search engines use, including * wildcards, $ end anchors and the longest-match rule, so you can check any path for Googlebot, Bingbot, GPTBot or other crawlers before you publish changes.
How to use the Robots.txt & Sitemap Checker — step-by-step guide
How to use the Robots.txt & Sitemap Checker
- 1Enter your website address.
- 2Press Check to fetch robots.txt and the sitemap.
- 3Review warnings, the robots file and sitemap URL count.
- 4Use the tester to see if a path is allowed for a crawler.
Key features
Frequently asked questions
Does Disallow remove a page from Google?
No. It only stops crawling. A blocked URL can still appear in results if other sites link to it. Use a noindex tag (and allow crawling) to remove it.
How can I block AI crawlers?
Add groups such as “User-agent: GPTBot” followed by “Disallow: /”. Then test GPTBot here to confirm it is blocked.
Is the Robots.txt & Sitemap Checker free?
Yes. The Robots.txt & Sitemap Checker on Apex Tool Station is 100% free with no signup and no watermark. Browser tools have no usage limits.
Do you store the websites or domains I check?
No. Our server makes the request, sends the result back to your browser and keeps nothing. Private and internal network addresses are blocked for safety.
Does it work on mobile?
Yes. The tool works on any modern browser on Android, iPhone, Windows, Mac and Linux.