llms.txt vs robots.txt: two files, opposite jobs.
robots.txt and llms.txt sit at the root of a domain and do opposite jobs. robots.txt is an established convention that tells automated clients which paths they may not fetch, and every major crawler honours it. llms.txt is a proposed convention that offers a markdown map of the pages worth reading, intended to help a model find a site's substance without parsing navigation and boilerplate. robots.txt restricts and is respected; llms.txt suggests and is not yet guaranteed to be read by any engine.
01Side by side
Where llms.txt and robots.txt part company.
| Dimension | llms.txt | robots.txt |
|---|---|---|
| What the file does | Lists the pages worth reading, with a one-line description of each. | Lists the paths crawlers may not fetch, per user agent. |
| Status | Proposed convention, introduced in 2024. Adoption is partial and voluntary. | Established since 1994 and honoured by every major crawler. |
| Format | Markdown: an H1, a blockquote summary, then linked sections. | Plain text: User-agent, Disallow, Allow and Sitemap directives. |
| Effect of getting it wrong | Nothing happens. A malformed or ignored llms.txt costs the effort of writing it. | A single mistaken Disallow can remove a site from AI answers and search results at once. |
| Who reads it | No engine guarantees it will be read. Some tools and agents consume it. | Every well-behaved crawler, including GPTBot, ClaudeBot, PerplexityBot and Googlebot. |
| Is it a security control | No. It is a reading list. | No. It is a request, not an access control, and it publicly names the paths listed in it. |
| Where the real leverage is | Marginal today. Cheap to publish and possibly useful later. | Substantial today. Whether answer-time crawlers are allowed decides whether citation is possible at all. |
02The overlap
Only the location is shared
Both files live at the root of a domain, are plain text, and are fetched without authentication. That is the extent of the similarity, and treating them as two versions of the same idea is how sites end up publishing a careful llms.txt while a stray Disallow keeps every answer-time crawler out.
The order of operations follows from that. robots.txt decides whether a crawler may fetch anything at all; llms.txt only matters to a crawler that has already been let in. Publishing the second while the first is blocking is writing a reading list for a visitor who is locked out.
The one habit worth keeping from robots.txt is the Sitemap directive. It has been the reliable way to hand a crawler a complete map of a site for two decades, and it is read today by engines that have made no commitment about llms.txt.
03Which you need
The answer depends on what is broken.
01
Fix robots.txt first, always
Answer-time agents such as OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot fetch pages while an assistant is composing a reply. Blocking one removes the site from that assistant's answers outright, and most sites that block them did it by accident with a single pasted rule meant for training crawlers.
02
Publish llms.txt because it is nearly free
An hour of work, no risk of breaking anything, and a plausible payoff if adoption grows. It is worth doing on those terms and not worth claiming credit for, because no engine has committed to reading it.
03
Do not let llms.txt stand in for structure
A file listing the good pages does not make those pages parseable, server-rendered or quotable. Sites that publish one and change nothing else have added a document about their content rather than fixing the content.
04Questions
llms.txt and robots.txt, asked plainly.
- Do I need both files?
- You need robots.txt to be correct, which may mean having no file at all, since an absent robots.txt allows everything. llms.txt is optional and low-cost, so publishing it is reasonable as long as nothing is being sequenced behind it.
- Does ChatGPT read llms.txt?
- OpenAI has made no commitment to reading llms.txt, and neither has Anthropic, Google or Perplexity. Anyone claiming a specific engine honours it is claiming more than any engine has published.
- Can robots.txt block AI without blocking Google?
- Yes, and the useful distinction is finer than that. Training crawlers such as GPTBot, ClaudeBot and Google-Extended can be blocked separately from answer-time agents such as OAI-SearchBot and Claude-User, which are the ones that make citation possible.
- Will llms.txt hurt me if engines ignore it?
- No. An unread file has no effect on anything, which is the whole basis for publishing it: the downside is bounded at the time spent writing it.
- What should go in llms.txt?
- The pages that carry the substance, each with a line saying what it covers, and a short note on what the organisation actually is. Dumping every URL in the sitemap defeats the purpose, since the file exists to distinguish signal from the rest of the site.
Stop comparing, start checking
Tell the models what you are.
Markup that tells models what your business is, on pages that render without JavaScript.