Free tool · robots.txt
Are you blocking the bots that cite you?
Two different decisions get made with one pasted rule. This reads your robots.txt agent by agent and tells you which of them you actually made.
01Why it matters
Blocking a training crawler and blocking a citation crawler are not the same choice.
Answer-time agents like OAI-SearchBot, Claude-User and PerplexityBot fetch a page while an assistant is composing its reply. Blocking one removes you from that assistant’s answers immediately, no matter how good the page is.
Training crawlers like GPTBot and Google-Extended absorb pages into future models. Blocking those is a defensible choice with a slow, diffuse cost, and plenty of publishers make it on purpose.
The two got conflated the moment “block the AI bots” snippets started circulating. Most sites running one pasted a rule meaning the second and caught the first with it, which is why this check separates them into two columns rather than giving you a score.
02Questions
- Which crawlers should I allow?
- Separate the question. If you want to appear in AI answers, the answer-time agents have to be allowed: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot and bingbot. Whether you allow training crawlers is a genuinely open choice about your content being absorbed into future models.
- I have no robots.txt. Is that bad?
- Not for this. No robots.txt means no rules, and no rules means every crawler is allowed. It is the permissive default, so nothing here is blocking you.
- Does allowing GPTBot mean ChatGPT will cite me?
- No. GPTBot collects training data; OAI-SearchBot and ChatGPT-User are the ones that fetch pages at answer time. Allowing the first without the second is a common and costly mix-up, which is why they are listed apart.
- Does this check anything beyond robots.txt?
- No, deliberately. Crawler policy is one of four things that decide whether you can be cited, alongside rendering, structure and authority. The full visibility check covers all four.
Next
Access is one of four preconditions.
Letting the crawlers in is necessary and not sufficient. The full check scores rendering, structured data, answer readiness and authority signals as well, with every finding traced to a line in your HTML.