llms.txt: What It Is, Why It Matters and How to Create It

robots.txt tells search engine crawlers where not to go. llms.txt is meant to explain to AI systems who you are and what you offer – a community standard with no officially demonstrated effect, but easy to implement.

What is llms.txt?

llms.txt is a simple text file in Markdown format placed in the root directory of a website – accessible at yourdomain.com/llms.txt. It contains structured information about the website specifically prepared for Large Language Models (LLMs) like ChatGPT, Claude and Perplexity.

The idea is simple: when an AI system crawls a website or an AI agent visits it, a compact well-structured overview could help it grasp the site's context more quickly. Whether and how strongly this effect occurs in practice has not yet been officially demonstrated.

In short: llms.txt is a voluntary, machine-readable description of your website – written in a format that AI systems can process well. The standard is still young and its effect has not been conclusively proven.

Where does the standard come from?

The llms.txt specification was proposed on 3 September 2024 by Jeremy Howard – co-founder of fast.ai. The page on llmstxt.org explicitly calls it a "proposal".

The standard is deliberately simple: no complex syntax, no new file format, no special parser needed. Plain Markdown that anyone can read and write.

Important to know: llms.txt is not an official W3C standard and not an RFC. It is a community proposal without binding support from the major AI platforms.

Current State in 2026

Since the standard appeared, quite a bit has happened – mainly in terms of adoption, less so in terms of demonstrated effect.

Google rejects llms.txt

Google has taken a clear position multiple times. In June 2025, John Mueller (Search Relations Lead at Google) wrote on Bluesky: "FWIW no AI system currently uses llms.txt". Gary Illyes (Search Advocate at Google) said in July 2025 that Google does not support llms.txt and has no plans to do so.

In May 2026, Google published a guide to optimizing for generative AI in Google Search. Its "Mythbusting" section says about llms.txt: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search." This applies to Google Search, not to other AI services.

Chrome Lighthouse includes an llms.txt audit

A different signal comes from Chrome development: Since version 13.3 (May 2026), Lighthouse has a new experimental category called "Agentic Browsing". According to the Chrome documentation, one audit in it checks whether a machine-readable summary is present at the domain root.

An experimental check is not proof of the file's effect, but it shows interest in the Chrome ecosystem.

Studies: no measurable effect on AI citations

Several studies find no demonstrable effect:

  • SE Ranking (November 2025) analysed around 300,000 domains. llms.txt was found on 10.13% of domains. No correlation with citation frequency in AI responses could be demonstrated. GPTBot fetches the file rarely, with no apparent connection to citations.
  • Peec AI (reported by Search Engine Land, January 2026) found among nearly 18,000 citations only 6 (0.03 %) that referred to llms.txt files.
  • Common Crawl (analysis of the July 2026 crawl, published August 2026) analysed 584,107 files. Result: 68.27% are template-generated, only 49.90% follow the full specification, and 22.56% contain no links at all.

Summary of the current state: Files are being created in large numbers – mainly through CMS automation. That AI systems systematically evaluate them is not demonstrated. The core message remains: The file is easy to implement; its effect is not proven.

llms.txt vs. robots.txt – the difference

Featurerobots.txtllms.txt
PurposeTells crawlers what they must NOT crawlDescribes to AI what the website IS
FormatOwn keyword-based formatMarkdown
AudienceAll web crawlersSpecifically LLMs and AI agents
ContentRules and restrictionsDescriptions and links
StandardOfficial RFC standard (RFC 9309)Community proposal (September 2024)
AdoptionAlmost every websiteGrowing, often auto-generated
Demonstrated effectYesNot yet demonstrated

Both files are complementary. robots.txt controls access and is not an access control mechanism for llms.txt – anyone wanting to block AI crawlers still needs to do so in robots.txt.

Who reads llms.txt?

That is the key question – and the answer is sobering. None of the major AI providers has officially confirmed that llms.txt is used as a signal for answer generation or ranking.

  • Anthropic / Claude – The Anthropic developer documentation has its own llms.txt (platform.claude.com/llms.txt). Documented support for evaluating third-party llms.txt files does not exist.
  • Perplexity AI – no officially confirmed support
  • OpenAI / ChatGPT – no officially confirmed support; GPTBot fetches llms.txt files rarely, with no apparent connection to citations (SE Ranking, November 2025)
  • Google – has explicitly rejected llms.txt: John Mueller (June 2025), Gary Illyes (July 2025), Google's AI search optimisation guide (May 2026)
  • AI agents – Chrome Lighthouse includes a check for its presence in the experimental "Agentic Browsing" category.

CMS and Plugin Support

More and more content management systems and SEO plugins generate llms.txt files automatically. This explains the rapidly growing adoption – and is at the same time a pointer to the quality problems found in many files.

  • Yoast SEO – supports llms.txt since June 2025, free of charge, auto-generated and updated by default, with manual mode and preview. Available for WordPress and Shopify (Yoast SEO for Shopify).
  • All in One SEO – generates large numbers of files according to Common Crawl (around 73,000 in the July 2026 crawl)
  • Rank Math – has its own documentation on llms.txt (rankmath.com/kb/llms-txt/)
  • Wix – the single largest generator: around 241,000 files (41.34% of all files found in the July 2026 crawl). Wix files often include an MCP API endpoint.

Important: You should always review automatically generated files. Common Crawl found many files without a summary, without link descriptions and with unfilled placeholders. A poor llms.txt is no advantage over none at all.

Structure and syntax

An llms.txt file is plain Markdown. According to the specification (llmstxt.org) there is a clear structure:

  • H1 (required): Name of the project or website – this is the only mandatory element.
  • Blockquote (optional): A short summary directly below the H1.
  • Paragraphs without headings (optional): More detailed contextual information.
  • H2 sections with link lists (optional): The heart of the specification. Each entry is a Markdown link in the format [Name](URL), optionally followed by a colon and a short note.
  • "Optional" section (optional): An H2 titled "Optional" contains secondary links that agents can skip when context is limited.

The file lives at /llms.txt in the root directory or at a sub-path (e.g. /docs/llms.txt). If multiple matching files exist, the most specific one applies.

# Name of the website or organisation > Short summary: what the website does, who it is for, > what the main offer is. One to three sentences. Optional prose with further background information, without a heading. ## Key pages - [About us](https://example.com/about/): Who we are and what drives us - [Products](https://example.com/products/): Our full range of offerings - [Blog](https://example.com/blog/): Latest articles on our topic ## Optional - [Imprint](https://example.com/imprint/): Legal information

What the specification does not include

In the literature and in many examples you will find sections like "About us", "Target audience", "Contact", "Language" or "Last update". This does not correspond to the specification – which does not provide for freely named sections without link lists. Such fields do no harm, but anyone wanting to remain spec-compliant should stick to H1, blockquote, optional prose and then H2 link lists.

Ready-made examples for different website types

Example: SaaS tool / web application

# AI-Ready Check > Free online tool that checks whether a website is technically > optimised for AI search engines like ChatGPT, Claude and Perplexity. > Score from 0–100 across five categories: robots.txt/llms.txt, > Schema.org, content structure, metadata quality and AI crawler > accessibility. Free, no registration required, instant results. For website owners, SEO professionals and web developers. ## Tools and guides - [Run AI-Ready Check](https://ai-ready-check.de/): Analyse your website for AI visibility - [Guides on GEO and AI visibility](https://ai-ready-check.de/guides/): Explanations and best practices - [Free tools](https://ai-ready-check.de/tools/): All available tools at a glance ## Optional - [Imprint](https://ai-ready-check.de/imprint/): Legal information

Example: Local business

# Bakery Müller Hamm > Traditional bakery in Hamm since 1987. Daily fresh bread, > rolls, cakes and pastries made to traditional recipes without > artificial additives. Breakfast roll service and catering. ## Products and information - [Bread and rolls](https://bakery-mueller.de/range/): Daily freshly baked selection - [Cakes and pastries](https://bakery-mueller.de/cakes/): Made to order, including for events - [Catering](https://bakery-mueller.de/catering/): Breakfast roll service and events - [Contact and opening hours](https://bakery-mueller.de/contact/): Address, phone, hours ## Optional - [Imprint](https://bakery-mueller.de/imprint/): Legal information

Example: Blog / content website

# Tech Blog Example > Independent blog on AI, web development and digital trends. > Since 2020, new articles weekly. All content under CC BY 4.0 – > use with attribution permitted. ## Content - [Latest articles](https://techblog-example.com/blog/): All posts in chronological order - [AI and machine learning](https://techblog-example.com/ai/): Articles on artificial intelligence - [Web development](https://techblog-example.com/webdev/): HTML, CSS, JavaScript, Node.js - [RSS feed](https://techblog-example.com/feed.xml): Subscribe for new articles ## Optional - [About the author](https://techblog-example.com/about/): Max Mustermann, software developer - [Imprint](https://techblog-example.com/imprint/): Legal information

When does it make sense?

llms.txt is not equally useful for every website. No benefit has been demonstrated anywhere; it fits best on sites with many sub-pages.

Well suited

  • Developer documentation and knowledge bases – structured overviews of many sub-pages help agents navigate. Anthropic uses llms.txt precisely this way.
  • Sites specifically intended to be used by AI agents – anyone offering an API, an MCP tool or a product for developers can use llms.txt as an entry point for agents.
  • Complex websites with many sub-pages – when the main areas are not immediately obvious from the domain name.

Less suited

  • When robots.txt, Schema.org markup and fast load times are not yet in order – these fundamentals should come first.
  • For simple business websites without many sub-pages – the effort is best treated as an optional extra.
  • As a substitute for good content – an llms.txt changes nothing about the quality of what AI systems find on your pages.

Common mistakes

Common Crawl found typical problems in llms.txt files in the July 2026 crawl:

  • HTML delivered instead of plain text – many single-page apps respond to unknown paths with the app shell. This produces an HTTP 200 status with HTML content instead of the text file (soft-404).
  • robots.txt content served at the llms.txt path – web server misconfiguration
  • Unfilled placeholders – auto-generated files with generic text such as "[Insert company name]"
  • Link lists without descriptions – bare URL dumps without context are of little help to AI systems. Only 32.94% of analysed files described their links.
  • Manipulative or promotional content – Common Crawl found files with attempts to steer AI responses in a particular direction. This undermines trust.
  • Outdated content – an llms.txt that no longer matches the current website is worse than none at all.

Step by step: create and deploy llms.txt

Step 1: Create the file

Create a new text file named llms.txt. Use UTF-8 encoding without BOM. Write the content in Markdown format following the pattern above – adapted to your website.

Tip: The llmshub.de generator automatically crawls your website and fills in all fields – you only need to review and download.

Step 2: Place it in the root directory

The file must be accessible at the main URL of the domain: https://yourdomain.com/llms.txt. For most web servers this means placing the file in the root directory – the same directory where index.html lives.

Step 3: Check the Content-Type and status

The web server should deliver the file as text/plain and respond with HTTP 200. This is normally automatic for .txt files. To verify:

curl -I https://yourdomain.com/llms.txt # Expected output: HTTP/2 200 content-type: text/plain; charset=utf-8

If the server responds with HTTP 200 but delivers HTML (typical for single-page apps), this is a soft-404. In this case the server must be configured to deliver the file correctly.

Step 4: Link from robots.txt (optional)

You can add a comment at the end of robots.txt – this is not an officially supported format but does no harm:

User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml # llms.txt for AI agents (unofficial) # LLMs: https://yourdomain.com/llms.txt

Note: A standardised directive for llms.txt in robots.txt does not exist. More important is that the file is accessible at the correct URL.

llms-full.txt – a widespread convention

In practice you will often also find a file named llms-full.txt. This contains the complete content of the most important pages in a form optimised for LLMs – as a single readable file.

Important: llms-full.txt is not part of the official specification, but a convention that has spread in practice. The Anthropic developer documentation uses it, for example (platform.claude.com/llms-full.txt). For most normal websites it is a nice-to-have, not a requirement.

Checklist

  • llms.txt created in the root directory of the domain
  • File accessible at yourdomain.com/llms.txt (HTTP 200, not a soft-404)
  • Content-Type is text/plain, not text/html
  • Required: H1 with the name of the website
  • Recommended: blockquote with a short summary
  • Recommended: H2 sections with link lists and descriptions
  • Saved as UTF-8 without BOM
  • No sensitive or internal information included
  • No manipulative or promotional content
  • For auto-generated files: content reviewed – no placeholders
  • Content kept up to date regularly

Create your llms.txt now

With the free llmshub.de generator you can create your llms.txt in seconds – automatically or manually.

Go to the generator →

Frequently asked questions about llms.txt

Is llms.txt required for good AI visibility? +

No – llms.txt is not required. For AI visibility, robots.txt configuration, Schema.org and fast load times are the fundamentals that should come first. llms.txt is an easy additional step whose effect has not yet been proven.

How often should I update llms.txt? +

Whenever there are major changes to the website – new products, new offerings, new main sections. A quarterly review makes sense. The file should reflect the actual current state of the website. An outdated llms.txt is worse than none at all.

Can llms.txt have negative effects? +

The risk is low – but not zero. Problematic would be an llms.txt with false, misleading or manipulative content. Also: do not include sensitive or internal data, since the file is publicly accessible. Always review auto-generated files, as Common Crawl found isolated prompt injection attempts in llms.txt files.

What is the difference between llms.txt and sitemap.xml? +

sitemap.xml lists all URLs of the website and helps crawlers find all pages. llms.txt describes selected content and the context of the website in natural language with links to the most important areas. Both files serve different purposes and complement each other.

Do I need llms.txt if I already have good Schema.org markup? +

Schema.org and llms.txt are complementary. Schema.org delivers machine-readable structured data for individual pages. llms.txt provides a high-level overview of the entire website with links to the most important sections. llms.txt is an optional additional step.

My CMS generates llms.txt automatically – is that enough? +

Many CMSs and plugins (Yoast, All in One SEO, Wix) generate llms.txt files automatically. This saves time, but quality varies considerably. Common Crawl found many auto-generated files with no link descriptions, unfilled placeholders and missing summaries. Always review and adjust the content after automatic generation.