robots.txt and sitemap.xml checker.
A domain is enough: the tool reads the robots.txt, spots important sections blocked for Google, finds the declared sitemaps and counts the URLs they contain.
Reads robots.txt, then the declared sitemaps or /sitemap.xml and /sitemap_index.xml by default. Up to 4 sitemap files are analyzed.
Free, no sign-up. Nothing is stored: the page is analyzed, then forgotten.
The two files that decide what Google crawls
The robots.txt tells bots what they may crawl. A single misplaced line, often forgotten after going live, can block a whole site, its CSS and JavaScript files, or an entire section of the shop. The sitemap.xml lists the pages to index: it speeds up the discovery of new URLs and lets you track their indexing in Search Console.
This tool checks both at once. It reads each rule group per user-agent, flags important paths blocked for Googlebot, finds the declared sitemaps, follows a sitemap index, counts URLs and detects the most frequent format errors: invalid XML, an HTML page returned instead of the file, URLs from another domain.
It is the first check to run when a client site is not getting indexed or suddenly loses traffic after a redesign.
How to read the results
- Blocking (red): Disallow on the whole site, on CSS or JS, or robots.txt returning a server error. Google pauses or degrades crawling.
- To check (orange): a section of the site blocked, malformed lines, a sitemap in error. Often intentional, sometimes not.
- Info: no sitemap declared, Crawl-delay ignored by Google, missing lastmod.
- Sitemaps: the type (URL list or index), the URL count and the status of each file. Beyond 50,000 URLs or 50 MB, split into several files.
- No robots.txt is not an error: everything is crawlable. But the sitemap declaration is then missing.
Frequently asked questions
What is the robots.txt file for?
It tells search engine bots which parts of the site they must not crawl, per user-agent group, and declares where the sitemaps are. It does not de-index a page that is already known: that requires a noindex meta robots, and the page must remain crawlable so Google can read it.
Which robots.txt mistakes are the most serious?
A Disallow: / that blocks the whole site, often inherited from a staging environment, blocking CSS and JavaScript files which prevents Google from rendering pages correctly, and a file returning a 5xx server error, because Google then stops crawling the whole site.
Should the sitemap contain every page of the site?
It should contain the pages you want indexed, and only those: no noindex, redirected, erroring or robots-blocked pages. A clean sitemap lets you compare, in Search Console, the number of pages submitted with the number indexed.
What if the site has no sitemap?
Most CMSs generate one automatically: WordPress with an SEO plugin, Shopify, Webflow, Wix and PrestaShop natively. Then declare its address in the robots.txt with a Sitemap line and submit it in Search Console.
How do I know which sitemap pages are actually indexed?
The Search Console indexing report shows it for each submitted sitemap. GoMySeo connects to Search Console and cross-checks that data with the site crawl to spot pages that are listed but never indexed, and explain why.
The other free tools
Four other quick checks, no account needed, for a client’s site or your own.
Pages not indexed, traffic dropping after a redesign?
GoMySeo cross-checks the site crawl and your Search Console data to find the cause, page by page.