What Is llms.txt and Do You Need It?

llms.txt is a plain Markdown file placed at the root of a website, at /llms.txt, that gives a language model a short description of the site and a curated list of its most useful pages. It was proposed by Jeremy Howard of Answer.AI in September 2024 and is described at llmstxt.org. The short answer to "do I need it?": it is useful for documentation, APIs and products that people use through AI assistants, it takes little time to make — and it does not change your Google rankings, because Google has said that Search ignores it.

What llms.txt is and what it looks like

A web page is built for people: menus, banners, scripts and cookie notices surround the actual content. Extracting the useful part is extra work for a model, and its context window is limited. llms.txt solves this by offering a hand-picked map in a format models read easily. The specification defines the order of blocks; only the first one is required:

1. An H1 with the project or site name — the first line of the file.
2. A blockquote with a short summary — one to three sentences with the key facts.
3. Optional details — paragraphs or lists explaining how to use the file, terms or limitations.
4. H2 sections with lists of links, each a Markdown link optionally followed by a colon and a note.

# Example Bakery

> Artisan bakery in Kyiv: sourdough bread, pastries and delivery
> across the city from 8:00 to 20:00.

Orders are accepted on the website and by phone.

## Catalogue

- [Sourdough bread](https://example.com/bread): Ingredients, weight and prices
- [Pastries](https://example.com/pastry): Croissants, buns, seasonal menu

## Delivery

- [Delivery](https://example.com/delivery): Zones, cost and timing

## Optional

- [Our story](https://example.com/about)

A section titled Optional has a special meaning: a tool short on context may skip it. The specification also suggests serving clean Markdown versions of pages at the same address with .md appended, but that part is optional and most sites link to ordinary HTML pages.

llms-full.txt and other variants

llms-full.txt is a convention popularised by documentation platforms: instead of links, one file contains the full text of the documentation, so a tool can load everything at once. It is not part of the original proposal, which describes expanded files called llms-ctx.txt and llms-ctx-full.txt. A full-text file makes sense for documentation, API references and knowledge bases. For a shop or a service website, llms.txt alone is usually enough. Watch the size: a multi-megabyte file will not fit into the context of many models.

Who actually reads llms.txt

This is where most marketing claims fall apart, so here is what is publicly known:

• Google Search does not use it. Google's guide to generative AI features in Search says you do not need to create special machine-readable or "AI text" files to appear, and that Search ignores them — adding one neither helps nor harms. We covered the wording and its context in llms.txt: Google says it does not read it.
• No major AI search provider has publicly committed to reading other sites' llms.txt files when choosing sources for answers.
• Chrome's Lighthouse does check it. Its experimental agentic browsing category includes an llms.txt audit. That measures how ready a site is for AI agents that browse it; it is not a ranking factor, and a failed check is not an SEO problem.
• Where it works predictably: coding assistants, documentation helpers and agents that you point at a URL yourself. That is why documentation platforms generate the file automatically and why many developer docs publish one.

llms.txt vs robots.txt

Controlling AI crawlers is a robots.txt job

If you want to allow or block AI bots, llms.txt cannot help. That is done in robots.txt, and the major operators split their bots by task:

• Search bots for assistants — OAI-SearchBot, Claude-SearchBot, PerplexityBot. Block them and the assistant cannot find or cite your pages.
• Training bots — GPTBot, ClaudeBot, CCBot and others. Blocking them keeps your content out of future training data and does not directly remove you from search-based answers.
• Control tokens — Google-Extended controls use of your content for Gemini models; it does not affect Google Search or AI Overviews.

A bot follows only the group with its own name, so once you add a separate group for GPTBot, the rules under User-agent: * no longer apply to it. Firewalls and CDNs can also block AI bots regardless of robots.txt. Check your file with the robots.txt checker and see whether AI search actually finds and cites your site with the AI visibility checker. The rules themselves are explained in how to check robots.txt.

How to create and publish llms.txt

1. Pick the pages that matter — usually 10 to 50: product or service pages, pricing, documentation, contacts. Do not dump the whole sitemap.
2. Generate a draft with the llms.txt generator, which builds the file from your site and validates the structure.
3. Write the summary by hand. One to three plain sentences: what the site is, for whom, where.
4. Upload the file to the site root so it opens at https://your-domain/llms.txt with status 200, in UTF-8, as text/plain or text/markdown.
5. Keep it current. A file pointing at deleted pages is worse than no file. Regenerate it when the structure of the site changes.

Common mistakes

• Pasting the whole sitemap. The point of the file is selection. Hundreds of links with no notes tell a model nothing that the sitemap does not.
• Links that redirect or return 404. Use final, canonical URLs and check them after every site migration.
• Marketing copy in the summary. "The best service on the market" is useless to a model. Facts are useful: what you do, for whom, in which region, at what price level.
• Serving an HTML page instead of the file. Some CMSs route unknown paths to a 404 template or the home page with status 200. Open the address in a browser and make sure you see plain text.
• Expecting it to block bots. It does not. Access rules live only in robots.txt and in your firewall or CDN settings.

So, do you need it?

• Yes, if you have technical documentation, an API or a product that developers use through AI assistants.
• Why not, if your CMS or documentation platform generates it automatically and it costs nothing to maintain.
• No, if the goal is Google rankings or AI Overviews. Nobody has shown an effect there, and Google says it ignores the file.

What does drive visibility in AI answers is the same foundation as ordinary search: pages that are indexed and eligible for snippets, content that answers questions directly, and mentions on other sites. See what is known about AI Overviews.