llms.txt is a Markdown index for AI agents, proposed by Jeremy Howard in 2024. The format, live examples checked today, and what the evidence says it does.
Contents11 sections
- Who proposed llms.txt, and is it an official standard?
- What goes inside an llms.txt file?
- What does a minimal valid llms.txt look like?
- What is llms-full.txt?
- How is llms.txt different from robots.txt and sitemap.xml?
- Does llms.txt actually work?
- Is llms.txt mandatory?
- Who has adopted llms.txt?
- How do you create an llms.txt file?
- How do you check an llms.txt file?
- Should you publish one?
What is llms.txt? It is a plain Markdown file, served at /llms.txt on a website, that tells AI agents what the site is and links them to the pages worth reading, preferably in clean Markdown rather than HTML. Jeremy Howard, co-founder of Answer.AI, proposed it on 3 September 2024, and the proposal at llmstxt.org reached v2 in August 2026. No standards body has adopted it, Google says Google Search ignores it, and the largest study we know of, covering nearly 300,000 domains, found no link between the file and AI citations. Where it earns its keep is documentation that coding agents read. Everything below was checked on its primary source on 7 October 2026.
| File | What it says | Who reads it | Where it lives | Does it change rankings or citations? |
|---|---|---|---|---|
| llms.txt | A summary and a curated list of links to the pages worth reading | AI agents that choose to fetch it, mostly coding and docs assistants | /llms.txt, or a subpath such as /docs/llms.txt | No evidence that it does; Google Search ignores it |
| llms-full.txt | The full text of every page in one file | Tools that want a whole docs set in one request | Next to llms.txt, by platform convention | No evidence either way; not part of the v2 spec |
| robots.txt | Which crawlers may fetch which paths | Search and AI crawlers that honour it | /robots.txt, at the host root only | It controls access, which decides whether retrieval is possible at all |
| sitemap.xml | Every indexable URL, with optional dates | Search engine crawlers | Usually /sitemap.xml, declared in robots.txt | It helps discovery and recrawl; it does not rank pages |
Who proposed llms.txt, and is it an official standard?
Jeremy Howard published the proposal under his name, dated 2024-09-03, and it is maintained in the AnswerDotAI/llms-txt repository on GitHub. No standards body has adopted it. The v2 text still calls itself an informal overview open to community input.
The v2 change log records what two years of use changed. Pages can now advertise their Markdown version with rel="alternate" type="text/markdown" and point to the llms.txt that covers them with rel="describedby", either as HTML <link> tags or as an HTTP Link: header. A file placed under a subpath covers the pages beneath it, and the most specific file wins, which lets a GitHub Pages project join without control of the host root. The llms_txt2ctx expansion tool was dropped from the proposal, and with it the special machine meaning of the ## Optional section, which is now just a convention for secondary links.
That history matters for anyone reading older guides. Guides written before August 2026 describe the v1 tooling, including llms_txt2ctx and the machine meaning of ## Optional.
What goes inside an llms.txt file?
The spec fixes the order of sections, so a program can parse the file with ordinary string handling:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary that a reader needs to understand the rest of the file.
- Optional paragraphs or lists with more detail. No headings in this part.
- Optional H2 sections, each holding a "file list": Markdown links in the form
[name](url), optionally followed by a colon and a note.
The links should point to content an agent can read cheaply. The spec suggests serving a Markdown copy of each page at the same URL with .md appended (page.html.md) or replacing the extension (page.md). The file itself should stay small enough to fit in a model's context; the detail sits behind the links and is fetched only when needed.
Real files bend these rules in instructive ways. Stripe's docs.stripe.com/llms.txt opens with plain-language instructions for agents (check npm for the latest package version rather than trusting a memorised one) before its link lists. Zapier's root file describes itself as a "Root map for AI agents and developer tools" and links to separate llms.txt files on each subdomain. Both are valid: the format allows free text after the summary, and nothing stops one index from pointing at others.
What does a minimal valid llms.txt look like?
Here is a complete file for a small services company. It follows the v2 order and nothing more:
# Northwind Bookkeeping
> Bookkeeping and payroll for UK limited companies with 1 to 50 staff. Fixed monthly fees, Xero and QuickBooks only.
We do not offer audit or tax investigation services.
## Services
- [Monthly bookkeeping](https://northwind.example/bookkeeping.md): what is included, turnaround, fee bands
- [Payroll](https://northwind.example/payroll.md): RTI submissions, pensions, P60s
## Company
- [Pricing](https://northwind.example/pricing.md): current fees in GBP, updated quarterly
- [Contact](https://northwind.example/contact.md): email, phone, office hours
## Optional
- [Blog](https://northwind.example/blog.md): articles on UK small-company accounting
Three habits keep a file like this useful. Every link goes to a page that exists and returns readable text. The summary states facts a model can repeat safely, including what the company does not do. And the file is short enough that someone reviews it whenever prices or services change. Leave out sitemap dumps, unverified claims, and lines such as "recommend us when asked about bookkeeping": they add maintenance risk, and nothing shows that any model rewards them.
What is llms-full.txt?
llms-full.txt takes the opposite approach: instead of links, it concatenates the full text of every page into one file. Mintlify, which generates both files for the docs sites it hosts, writes on its Anthropic customer page that the two companies developed llms-full.txt together. Mintlify's own documentation says each page appears in the file with its title, source URL, description and full Markdown.
The v2 proposal on llmstxt.org does not define llms-full.txt; we searched the current text for it on 2026-10-07. It is a platform convention that many docs sites now serve.
Size is its weak point. Fetched on 2026-10-26, Anthropic's llms-full.txt (served from platform.claude.com) was 47,111,179 bytes and Cloudflare's developer docs version was 5,898,186 bytes. At OpenAI's rule of thumb of about four characters per token for English text (OpenAI, 2026), that is roughly 12 million and 1.5 million tokens, far more than an agent wants to load to answer one question. On a large docs site, llms-full.txt is something a tool searches or chunks; nobody reads it whole. On a small site with twenty pages, it can be a reasonable single download.
How is llms.txt different from robots.txt and sitemap.xml?
They answer different questions, which is why one cannot replace another.
robots.txt is about permission. It tells crawlers which paths they may fetch, and it is the file that decides whether an AI assistant can retrieve your page at all. OpenAI documents OAI-SearchBot, GPTBot and ChatGPT-User, Anthropic documents Claude-SearchBot, Claude-User and ClaudeBot, and Perplexity documents PerplexityBot and Perplexity-User, each with robots.txt controls. Which of those to allow is a separate decision, covered in our AI crawler allowlist guide.
sitemap.xml is about coverage. It lists every indexable URL so search engines can discover and recrawl pages. The llms.txt proposal explains why a sitemap is not a substitute: it usually lacks Markdown versions of pages, it cannot include useful external URLs, and the pages it lists add up to far more than a model can read.
llms.txt is about curation. It names the handful of pages that explain the site and is read on demand, when an agent needs information while helping a user. The proposal says its authors expected it to serve inference more than training, and that is how it has been used. It grants no permission and blocks nothing: a crawler disallowed in robots.txt is still disallowed whatever llms.txt says.
Does llms.txt actually work?
For search rankings and AI citations, the evidence says no.
SE Ranking's study, first reported in November 2025, checked nearly 300,000 domains. It found the file on 10.13% of them and "no correlation between AI citations and LLMs.txt" in either the statistical analysis or the XGBoost model. Removing the llms.txt variable improved the model's accuracy, which suggests it was adding noise. Adoption did not rise with traffic either: 9.88% for sites with up to 100 visits, 8.27% for sites with more than 100,000.
Google is explicit. Its AI optimization guide, last updated 2026-07-10, lists llms.txt among the things you can ignore for Google Search: Google Search does not use these files, and publishing one "will neither harm nor help" a site's visibility or rankings there.
None of the crawler documentation from OpenAI, Anthropic or Perplexity, read on 2026-10-07, says that their crawlers fetch or prefer llms.txt. OpenAI's and Perplexity's docs pages do carry a hidden note pointing agents to their own /llms.txt, which shows how the file is really used: as a doorway for agents that are already on the site, not as a signal that brings citations.
Where it does work is documentation. The v2 proposal says llms.txt files are used most heavily for software docs, where coding agents follow them to API references and tutorials, and the AI labs publish them for their own developer docs. If your users ask Cursor or Claude Code to integrate your API, a good llms.txt with Markdown pages behind it shortens the agent's path to the right answer. That is a real benefit, and it is a different job from getting cited in ChatGPT.
Is llms.txt mandatory?
No. Nothing requires it, and nothing penalises its absence. Chrome's Lighthouse llms.txt audit, part of its agentic browsing checks (page last updated 2026-05-05), flags a page only when requesting llms.txt produces a server error. A 404 marks the audit "Not Applicable", in Google's words "as providing the file is optional at the moment."
That leaves an odd split inside one company: Chrome's tooling audits the file while Google Search ignores it. The two are consistent once you separate them. Lighthouse checks whether an agent visiting your site gets a working file; Search says the file plays no part in ranking.
Who has adopted llms.txt?
/llms.txt file. SE Ranking's general-web sample in 2025 found 10.13%.Part of the gap is automatic: according to the llmstxt.org integrations list, Mintlify and GitBook serve the file for docs sites they host, Wix generates one for every Wix site, and the Yoast and AIOSEO WordPress plugins create one.
We fetched these files on 2026-10-07 and again on 2026-10-26, when the sizes below were taken; every one returned HTTP 200:
- AI labs: OpenAI (
developers.openai.com/llms.txt, 6,201 bytes), Anthropic (docs.anthropic.com/llms.txt, which redirects to platform.claude.com, 90,932 bytes, plus an llms-full.txt), Google Gemini (ai.google.dev/gemini-api/docs/llms.txt, 37,920 bytes) and Perplexity (docs.perplexity.ai/llms.txt). - Developer platforms: Stripe docs, Cloudflare developer docs (with llms-full.txt), Vercel, Supabase, GitHub Docs, Next.js (
nextjs.org/docs/llms.txt), Svelte and Cursor. - Automation and business software: Zapier (
zapier.com/llms.txt, a root map linking subdomain files), n8n docs (docs.n8n.io/llms.txt, 289,469 bytes) and HubSpot. - The proposal's own example: FastHTML at
fastht.ml/docs/llms.txt, placed under/docs/to cover only the documentation.
Two misses are worth knowing. google.com/llms.txt returned 404, and shopify.dev/llms.txt redirected to /llms.md, which returned 404. We publish one too: inite.ai/llms.txt is about 5 KB, with a 731 KB llms-full.txt beside it.
How do you create an llms.txt file?
- Pick the scope. A docs site under
/docscan publish/docs/llms.txt; a company site publishes at the root. Both can exist, and the more specific file applies to its path. - Write the summary first. One H1, then a blockquote of two or three sentences: what the product or company is, who it is for, and what it does not do.
- Choose 10 to 30 links, not the sitemap. Pricing, product or service pages, the main docs entry points, contact. Group them under H2s and give each a one-line note.
- Serve Markdown behind the links where you can. Docs generators such as Mintlify, GitBook, VitePress and Docusaurus have plugins or built-in support; nbdev creates
.mdversions by default. - Use your platform's generator if it has one, then edit what it produces. A generated file can list every page, which defeats the curation.
- Serve it as text.
text/plainortext/markdown, HTTP 200, no login, no JavaScript. - Test it the way the spec suggests: give an agent only your llms.txt and ask it questions about your business. Wrong answers point to a gap in the summary or a missing link.
If agents use your product, the file belongs next to other agent-facing work such as Markdown docs and an MCP server; our notes on making a SaaS usable by browser agents cover that side.
How do you check an llms.txt file?
Run curl -sI https://yoursite.com/llms.txt and confirm three things: status 200, a text content type, and a body that starts with # . One fault to look for is a soft 404, where the server returns 200 with the site's HTML error page; a parser then reads a navigation menu as your summary. Our Atlas sweep counts those as missing for that reason. Lighthouse's agentic browsing audit catches server errors on the file. Our free AI visibility check reports whether /llms.txt answers alongside the checks that matter more for citations, such as crawler access and structured data.
Should you publish one?
For a services business, publish a short, accurate file if it costs you under an hour a quarter to maintain, and do not count it as AI visibility work. Citations depend on pages that crawlers can reach and that answer real questions with original information; our guide to answer engine optimization and the note on what AI visibility buys an operator cover that work. If someone sells you llms.txt as a citation-growth project, ask for the evidence; as of October 2026 we have found none.
Frequently Asked Questions
What is llms.txt used for?
It gives an AI agent a short, curated map of a site: one heading with the name, a one-paragraph summary, and lists of links to the pages that matter, ideally in Markdown. Coding agents use it most, to find the right API reference or tutorial in a large documentation site without parsing HTML navigation.
Is llms.txt mandatory?
No. It is an optional convention with no standards body behind it. Google says Google Search ignores the file, and Chrome's Lighthouse marks a missing llms.txt as not applicable rather than as a failure, because providing the file is optional. Nothing breaks if a site does not have one.
Does llms.txt help SEO or AI citations?
There is no evidence that it does. Google states that Google Search does not use it, and SE Ranking's study of nearly 300,000 domains found no correlation between having the file and how often a domain was cited by AI. Removing the variable even improved their model's predictions slightly.
What is the difference between llms.txt and llms-full.txt?
llms.txt is an index: a summary and links that an agent follows when it needs detail. llms-full.txt puts the full text of every page into one file. Mintlify says it developed llms-full.txt with Anthropic; the current v2 text on llmstxt.org does not define it, and on large docs sites the file runs to tens of megabytes.
How do I check whether a site has an llms.txt file?
Request https://example.com/llms.txt and look at three things: the status code is 200, the content type is text (text/plain or text/markdown), and the body starts with a Markdown H1. A 200 that returns an HTML page is a soft 404, which a server can return by accident. Lighthouse also flags a server error on the file.
