CrawlControlBot

Crawl Control is a free scanner that tells a site owner which bots can currently reach their site, and what changes for them on 15 September 2026. When someone scans a domain, we fetch a small number of its pages.

Our requests identify themselves as:

CrawlControlBot/0.1 (+https://cc.routecraft.link/bot)

What we fetch, per scan:

• robots.txt, and sitemap.xml if it is small
• the homepage, and at most 14 further pages linked from it
• the homepage once more per crawler user-agent we test for

We obey the limits you would expect: a timeout and a size cap on every request, a redirect limit, and a hard cap on how many pages one scan touches. We do not crawl on a schedule — we only fetch when someone runs a scan. Results are cached for 24 hours, so repeated scans of the same domain do not repeat the fetches.

To keep us out entirely, disallow CrawlControlBot in your robots.txt:

User-agent: CrawlControlBot
Disallow: /