What is robots.txt?
- robots.txt
- robots.txt is a plain-text file at the root of a domain that tells automated clients which parts of a site they may request. For AI visibility it is decisive, because the rules in it determine whether answer-time crawlers can fetch a page in order to cite it, and whether training crawlers can absorb it into future models.
01 In practice
The file expresses two independent decisions that are easy to make as one by accident. Allowing training crawlers while blocking answer-time agents removes a site from live answers; blocking training crawlers while allowing answer-time agents keeps it citable today without contributing to tomorrow's models.
Absence is permissive. A site with no robots.txt allows everything, so a missing file is never the cause of an AI visibility problem, though it is often assumed to be.
02 Related terms
The terms next to this one.
The work behind it
Tell the models what you are.
Markup that tells models what your business is, on pages that render without JavaScript.