What is Answer-Time Crawler?
- Answer-Time Crawler
- An answer-time crawler is a user agent that fetches a web page while an AI assistant is composing a response, so the page can be quoted or cited in that answer. OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot and bingbot are answer-time crawlers, and blocking one removes a site from that assistant's answers immediately.
01 In practice
Answer-time crawlers are distinct from training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot, which collect content to train future models. Blocking a training crawler keeps a site out of a model released months or years later, which is a defensible choice with a slow and diffuse cost.
Conflating the two is the single most common and most expensive robots.txt mistake. Many sites pasted a snippet intended to opt out of training and caught the answer-time agents in the same rule, removing themselves from live answers without ever deciding to.
The work behind it
Tell the models what you are.
Markup that tells models what your business is, on pages that render without JavaScript.