Robots and sitemap checker
Check crawl rules and sitemap documents for a public page.
This check runs on the server. Input and results are not saved as execution history; hosted allowances apply.
About Robots and sitemap checker
Check whether robots.txt allows a page and inspect the site’s declared sitemap documents.
When to use it
Catch a blocked documentation page or broken sitemap before a release.
Page URL → robots decision, matched line, sitemap URLs and issuesHow to use it
- Enter a page URL and crawler user agent.
- Inspect the matching robots rule and crawl decision.
- Review sitemap status, entry counts and common problems.
Check crawl access before publishing
Enter a page URL, rather than just the domain, so the robots decision applies to the path you care about. The report shows the matching rule, line and crawler name. A server error leaves the decision unknown because crawlers should defer while robots.txt is unavailable. An ordinary missing robots file applies no restrictions.
The checker reads up to two declared sitemaps, or tries /sitemap.xml when none are declared. URL sets list page locations and sitemap indexes list child documents without crawling them. Review response errors, malformed XML and invalid locations. Allowing a page and publishing a sitemap help discovery but cannot guarantee indexing or ranking.
Limits & privacy
Public HTTP and HTTPS only with private network protection on every hop. Ten seconds total, 256 KiB per document, at most two sitemaps and 1,000 entries per sitemap. XML DTDs and compressed sitemaps are unsupported. Parsing is limited to 5,000 elements and 64 nesting levels. Sitemap indexes are listed without recursive crawling. No page indexing or search ranking guarantee.
Use this tool from the API reference or connect through the SDK, CLI, SSH, or MCP. Hosted calls share your plan allowances.