CiteSiteBot

The agent string

CiteSiteBot/0.1 (+https://citesite.tech/bot; SEO and AI visibility analysis)

Why it arrived

Someone asked for it. CiteSiteBot does not crawl the web looking for sites; every request it makes is the direct result of a person entering a domain — either a customer analysing a site they operate, or a visitor running one of the free tools. There is no discovery queue and no schedule of its own.

That has a practical consequence worth knowing: a visit is evidence that somebody typed your address into a form. It is not evidence of anything else.

What it does

• Reads /robots.txt first, and obeys it. A path disallowed for CiteSiteBot or for * is not fetched.
• Fetches a small, fixed number of pages — 5 for the instant check on the home page, 14 for the llms.txt generator, and a low double-figure budget for a full analysis. There is no configuration that makes it more.
• Pauses between requests to the same host, so a run is a trickle rather than a burst.
• Requests HTML, gives up on anything over 2 MB, and times out after twelve seconds.
• Stays on the host it was given. It does not follow links off your domain, and it re-checks every redirect rather than following a chain blindly.
• Reads the response and nothing else. It runs no JavaScript, submits no forms, sets no cookies, and never attempts a URL behind a login.

How to slow it down or stop it

To keep it out of part of your site:

User-agent: CiteSiteBot
Disallow: /private

To shut it out entirely:

User-agent: CiteSiteBot
Disallow: /

Both take effect on the next request — nothing is cached between runs. Note the consequence, since it is the whole reason anyone uses this product: with the second rule in place, CiteSite can no longer report anything about your site, including to you. If the traffic is a customer of ours analysing a domain they own, a rule here is how that stops.

Rate limiting or blocking us at the edge works too, and we would rather you did that than tolerated traffic you did not want. If you do, we would appreciate an email so we can find out what went wrong at our end.

What it is not

CiteSiteBot does not collect training data. Nothing it fetches trains a model, ours or anyone else’s. Page text is used to produce a report for the person who asked for it, and the AI engines this product measures are queried through their own APIs about what they already know — they are never handed a crawl.

If you are working out which crawlers to allow in general, the AI crawler checker reads your robots.txt and reports every major agent separately, including the distinction between the ones that decide whether you can be cited and the ones that only collect training data.

Contact

A wrong fetch, a rule we failed to honour, or a request to be excluded permanently: contact@citetech.site. Include a log line if you have one — the agent string above is enough to find the run.