llms.txt sounds like “robots.txt for the AI era,” which is why half the internet shipped one over a weekend in 2024. The reality is more interesting: it's useful for some things, irrelevant for others, and Google has already said on the record it won't use it. Here's what it actually is, how to write it well, and who reads it today — so you decide on data, not FOMO.
What llms.txt is (and isn't)
llms.txt is a Markdown text file you place at your domain root (yourdomain.com/llms.txt) to tell a language model which pages on your site matter and what each one is about. Jeremy Howard — co-founder of Answer.AI and fast.ai — proposed it in September 2024, and the spec lives at llmstxt.org.
The idea is simple. The web is built for humans and browsers: menus, banners and JavaScript an LLM has to chew through. llms.txt hands it a clean Markdown map — the format every model already reads without translating — with your key pages and a line of context for each.
How to write it: the exact format
The spec is deliberately minimal. A valid llms.txt carries, in this order:
- An H1 with the project or site name. It's the only required section.
- A blockquote (a line starting with
>) with a short summary of what the site is. - Optional prose with extra context, no headings.
- H2 sections with link lists, where each link carries a short description after a colon.
- A special H2 section called “Optional”: anything under it can be skipped when a shorter context is needed.
There's an optional companion, /llms-full.txt, that packs the site's full content into a single Markdown document for models that want to swallow it whole. Rule of thumb: aim for 10-20 high-value evergreen pages, not your entire sitemap. An llms.txt with 400 links isn't a map, it's another maze.
llms.txt vs robots.txt vs schema: not the same thing
| File | What it's for | Who's in charge |
|---|---|---|
| robots.txt | Allow or block each crawler's access | Access control |
| llms.txt | Say which pages matter and what they're about | Context and priority |
| Schema (JSON-LD) | Declare what entity you are and how your data connects | Entity identity |
The three complement each other, they don't compete. robots.txt lets the crawler in — without that you don't exist; schema tells it who you are; and llms.txt gives it the index of where to start. Shipping llms.txt with a closed robots.txt is hanging a “read this” sign on a locked door.
Who actually uses it today
Here you have to be honest, because half the guides sell llms.txt like gold and the other half like smoke. The real picture in mid-2026:
- Perplexity: retrieves it and uses it to prioritize which pages it looks at. The one that pays it most attention among AI search engines.
- Anthropic (Claude), Cursor, Mintlify and several dev tools have officially supported it since January 2026.
- OpenAI / ChatGPT: no official confirmation, but teams that publish llms.txt are observed to see correlated changes in their SearchGPT citation patterns.
- Google: NO. In July 2025 Gary Illyes confirmed it doesn't support it and won't, and John Mueller compared it to the old meta keywords tag — the one ignored until it died.
And a stat that cools the hype: in an SE Ranking study of 300,000 domains, adoption was around 10% — one site in ten — after a year and a half of conversation. In another count of over 500 million AI-bot visits across 90 days, only 408 went for llms.txt directly. Today it's mostly read by dev tools and agents (Cursor, Claude Code, Copilot, MCP servers), not so much by consumer AI search.
When it's worth it (and when it's a waste of time)
It's not a question of fashion, it's cost-benefit. Shipping a decent llms.txt costs little — an afternoon, not a project — and it doesn't hurt you. The question is priority.
- Yes, now: if you have technical docs, a library, an API or a product that devs and agents consult. There llms.txt pays off today, not in the future.
- Yes, but no rush: if you sell Perplexity-first or want to cover the cheap flank of AI visibility. Low cost, real if modest upside.
- Not your priority: if your traffic depends on Google. Before touching llms.txt, let the crawler in, sort out your schema and front-load answers. That moves the needle; llms.txt is the cherry.
How to set it up without messing it up
- Pick 10-20 evergreen URLs that really represent what you do. Quality over inventory.
- Write the H1 with your name, the blockquote with one line on what you solve, and group links under thematic H2s with an honest description per link.
- Upload it to the root:
yourdomain.com/llms.txt. Check it by opening it in the browser. - First verify your robots.txt lets the AI crawlers through. Without that, nobody reads the map.
- Maintain it: when your key pages change, update it. An outdated map is worse than no map.
And measure it like everything in GEO: if you publish llms.txt, watch whether your citations in Perplexity and ChatGPT shift over the following weeks. If you don't measure, you won't know if it worked — and in GEO there's no official dashboard, you build your own.