What is llms.txt?
llms.txt is a simple text file in Markdown format placed in the root directory of a website – accessible at yourdomain.com/llms.txt. It contains structured information about the website specifically prepared for Large Language Models (LLMs) like ChatGPT, Claude and Perplexity.
The idea is simple: when an AI system crawls a website or an AI agent visits it, a compact well-structured overview could help it grasp the site's context more quickly. Whether and how strongly this effect occurs in practice has not yet been officially demonstrated.
In short: llms.txt is a voluntary, machine-readable description of your website – written in a format that AI systems can process well. The standard is still young and its effect has not been conclusively proven.
Where does the standard come from?
The llms.txt specification was proposed on 3 September 2024 by Jeremy Howard – co-founder of fast.ai. The page on llmstxt.org explicitly calls it a "proposal".
The standard is deliberately simple: no complex syntax, no new file format, no special parser needed. Plain Markdown that anyone can read and write.
Important to know: llms.txt is not an official W3C standard and not an RFC. It is a community proposal without binding support from the major AI platforms.
Current State in 2026
Since the standard appeared, quite a bit has happened – mainly in terms of adoption, less so in terms of demonstrated effect.
Google rejects llms.txt
Google has taken a clear position multiple times. In June 2025, John Mueller (Search Relations Lead at Google) wrote on Bluesky: "FWIW no AI system currently uses llms.txt". Gary Illyes (Search Advocate at Google) said in July 2025 that Google does not support llms.txt and has no plans to do so.
In May 2026, Google published a guide to optimizing for generative AI in Google Search. Its "Mythbusting" section says about llms.txt: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search." This applies to Google Search, not to other AI services.
Chrome Lighthouse includes an llms.txt audit
A different signal comes from Chrome development: Since version 13.3 (May 2026), Lighthouse has a new experimental category called "Agentic Browsing". According to the Chrome documentation, one audit in it checks whether a machine-readable summary is present at the domain root.
An experimental check is not proof of the file's effect, but it shows interest in the Chrome ecosystem.
Studies: no measurable effect on AI citations
Several studies find no demonstrable effect:
- SE Ranking (November 2025) analysed around 300,000 domains. llms.txt was found on 10.13% of domains. No correlation with citation frequency in AI responses could be demonstrated. GPTBot fetches the file rarely, with no apparent connection to citations.
- Peec AI (reported by Search Engine Land, January 2026) found among nearly 18,000 citations only 6 (0.03 %) that referred to llms.txt files.
- Common Crawl (analysis of the July 2026 crawl, published August 2026) analysed 584,107 files. Result: 68.27% are template-generated, only 49.90% follow the full specification, and 22.56% contain no links at all.
Summary of the current state: Files are being created in large numbers – mainly through CMS automation. That AI systems systematically evaluate them is not demonstrated. The core message remains: The file is easy to implement; its effect is not proven.
llms.txt vs. robots.txt – the difference
| Feature | robots.txt | llms.txt |
|---|---|---|
| Purpose | Tells crawlers what they must NOT crawl | Describes to AI what the website IS |
| Format | Own keyword-based format | Markdown |
| Audience | All web crawlers | Specifically LLMs and AI agents |
| Content | Rules and restrictions | Descriptions and links |
| Standard | Official RFC standard (RFC 9309) | Community proposal (September 2024) |
| Adoption | Almost every website | Growing, often auto-generated |
| Demonstrated effect | Yes | Not yet demonstrated |
Both files are complementary. robots.txt controls access and is not an access control mechanism for llms.txt – anyone wanting to block AI crawlers still needs to do so in robots.txt.
Who reads llms.txt?
That is the key question – and the answer is sobering. None of the major AI providers has officially confirmed that llms.txt is used as a signal for answer generation or ranking.
- Anthropic / Claude – The Anthropic developer documentation has its own llms.txt (platform.claude.com/llms.txt). Documented support for evaluating third-party llms.txt files does not exist.
- Perplexity AI – no officially confirmed support
- OpenAI / ChatGPT – no officially confirmed support; GPTBot fetches llms.txt files rarely, with no apparent connection to citations (SE Ranking, November 2025)
- Google – has explicitly rejected llms.txt: John Mueller (June 2025), Gary Illyes (July 2025), Google's AI search optimisation guide (May 2026)
- AI agents – Chrome Lighthouse includes a check for its presence in the experimental "Agentic Browsing" category.
CMS and Plugin Support
More and more content management systems and SEO plugins generate llms.txt files automatically. This explains the rapidly growing adoption – and is at the same time a pointer to the quality problems found in many files.
- Yoast SEO – supports llms.txt since June 2025, free of charge, auto-generated and updated by default, with manual mode and preview. Available for WordPress and Shopify (Yoast SEO for Shopify).
- All in One SEO – generates large numbers of files according to Common Crawl (around 73,000 in the July 2026 crawl)
- Rank Math – has its own documentation on llms.txt (rankmath.com/kb/llms-txt/)
- Wix – the single largest generator: around 241,000 files (41.34% of all files found in the July 2026 crawl). Wix files often include an MCP API endpoint.
Important: You should always review automatically generated files. Common Crawl found many files without a summary, without link descriptions and with unfilled placeholders. A poor llms.txt is no advantage over none at all.
Structure and syntax
An llms.txt file is plain Markdown. According to the specification (llmstxt.org) there is a clear structure:
- H1 (required): Name of the project or website – this is the only mandatory element.
- Blockquote (optional): A short summary directly below the H1.
- Paragraphs without headings (optional): More detailed contextual information.
- H2 sections with link lists (optional): The heart of the specification. Each entry is a Markdown link in the format
[Name](URL), optionally followed by a colon and a short note. - "Optional" section (optional): An H2 titled "Optional" contains secondary links that agents can skip when context is limited.
The file lives at /llms.txt in the root directory or at a sub-path (e.g. /docs/llms.txt). If multiple matching files exist, the most specific one applies.
What the specification does not include
In the literature and in many examples you will find sections like "About us", "Target audience", "Contact", "Language" or "Last update". This does not correspond to the specification – which does not provide for freely named sections without link lists. Such fields do no harm, but anyone wanting to remain spec-compliant should stick to H1, blockquote, optional prose and then H2 link lists.
Ready-made examples for different website types
Example: SaaS tool / web application
Example: Local business
Example: Blog / content website
When does it make sense?
llms.txt is not equally useful for every website. No benefit has been demonstrated anywhere; it fits best on sites with many sub-pages.
Well suited
- Developer documentation and knowledge bases – structured overviews of many sub-pages help agents navigate. Anthropic uses llms.txt precisely this way.
- Sites specifically intended to be used by AI agents – anyone offering an API, an MCP tool or a product for developers can use llms.txt as an entry point for agents.
- Complex websites with many sub-pages – when the main areas are not immediately obvious from the domain name.
Less suited
- When robots.txt, Schema.org markup and fast load times are not yet in order – these fundamentals should come first.
- For simple business websites without many sub-pages – the effort is best treated as an optional extra.
- As a substitute for good content – an llms.txt changes nothing about the quality of what AI systems find on your pages.
Common mistakes
Common Crawl found typical problems in llms.txt files in the July 2026 crawl:
- HTML delivered instead of plain text – many single-page apps respond to unknown paths with the app shell. This produces an HTTP 200 status with HTML content instead of the text file (soft-404).
- robots.txt content served at the llms.txt path – web server misconfiguration
- Unfilled placeholders – auto-generated files with generic text such as "[Insert company name]"
- Link lists without descriptions – bare URL dumps without context are of little help to AI systems. Only 32.94% of analysed files described their links.
- Manipulative or promotional content – Common Crawl found files with attempts to steer AI responses in a particular direction. This undermines trust.
- Outdated content – an llms.txt that no longer matches the current website is worse than none at all.
Step by step: create and deploy llms.txt
Step 1: Create the file
Create a new text file named llms.txt. Use UTF-8 encoding without BOM. Write the content in Markdown format following the pattern above – adapted to your website.
Tip: The llmshub.de generator automatically crawls your website and fills in all fields – you only need to review and download.
Step 2: Place it in the root directory
The file must be accessible at the main URL of the domain: https://yourdomain.com/llms.txt. For most web servers this means placing the file in the root directory – the same directory where index.html lives.
Step 3: Check the Content-Type and status
The web server should deliver the file as text/plain and respond with HTTP 200. This is normally automatic for .txt files. To verify:
If the server responds with HTTP 200 but delivers HTML (typical for single-page apps), this is a soft-404. In this case the server must be configured to deliver the file correctly.
Step 4: Link from robots.txt (optional)
You can add a comment at the end of robots.txt – this is not an officially supported format but does no harm:
Note: A standardised directive for llms.txt in robots.txt does not exist. More important is that the file is accessible at the correct URL.
llms-full.txt – a widespread convention
In practice you will often also find a file named llms-full.txt. This contains the complete content of the most important pages in a form optimised for LLMs – as a single readable file.
Important: llms-full.txt is not part of the official specification, but a convention that has spread in practice. The Anthropic developer documentation uses it, for example (platform.claude.com/llms-full.txt). For most normal websites it is a nice-to-have, not a requirement.
Checklist
- llms.txt created in the root directory of the domain
- File accessible at yourdomain.com/llms.txt (HTTP 200, not a soft-404)
- Content-Type is text/plain, not text/html
- Required: H1 with the name of the website
- Recommended: blockquote with a short summary
- Recommended: H2 sections with link lists and descriptions
- Saved as UTF-8 without BOM
- No sensitive or internal information included
- No manipulative or promotional content
- For auto-generated files: content reviewed – no placeholders
- Content kept up to date regularly
Create your llms.txt now
With the free llmshub.de generator you can create your llms.txt in seconds – automatically or manually.
Go to the generator →