Answer-engine crawlers are not blocked unintentionally
Have you decided, deliberately, which AI crawlers may access your site?
| Pass | A deliberate, documented position exists, and retrieval crawlers are permitted if citation visibility is a goal. |
|---|---|
| Advisory | No explicit rules, so all crawlers are permitted by default without a decision having been made. |
| Fail | Retrieval crawlers are blocked while the business simultaneously expects to appear in AI answers. |
Why it matters
Blocking training crawlers and blocking retrieval crawlers are two entirely different decisions with different consequences, and they are routinely conflated. Blocking a retrieval crawler removes you from live answers that would have cited and linked you. Blocking a training crawler does not. Many sites have blocked both by copying a snippet without understanding the distinction — and many CDNs now block them by default.
How we inspect it
We enumerate the agent groups in robots.txt and identify which answer-engine and AI crawlers are permitted or blocked, distinguishing training crawlers from retrieval crawlers that fetch pages to cite in live answers. We report the position; we do not judge it.
Source: robots.txt agent-group analysis plus CDN bot-rule inspection where declared
Remedy
- 1List every AI-related agent currently blocked and classify each as training or retrieval.
- 2Decide explicitly whether citation visibility is worth content ingestion for your business model.
- 3Implement the decision in robots.txt and check your CDN or bot-mitigation layer is not overriding it.
- 4Document the decision and its date so it is revisited deliberately rather than inherited.
Verification
Recheck re-reads agent groups and confirms the implemented position matches the declared intent.
- Usually fixed by
- Owner
- Effort
- Small · 0.5–2 hrs
- Est. impact
- medium
- Typical cost
- NZ$120–400