Free survey
Starting surveyrobots.txt, then XML sitemaps, then a live page sample. Typical scans finish in 8–40 seconds.
Critical
Following the sitemap URL never reaches a terminal response.
Crawlers stop. The URL will not be indexed in a useful form.
/a 301 → /b 301 → /a.
Sitemapper follows redirects hop-by-hop and records a loop when a URL repeats.
curl -I --max-redirs 0 each hop.
Break the cycle. Sitemap URLs should be the canonical, non-redirecting form when possible.
Sitemapper