What is llms.txt?
llms.txt is a simple text file in Markdown format located in the root directory of a website – accessible at yourdomain.com/llms.txt. It contains structured information about the website, specifically prepared for Large Language Models (LLMs) like ChatGPT, Claude and Perplexity.
The idea behind it: when an AI system crawls a website, a compact well-structured overview might help it grasp context more quickly. Whether and how strongly this effect occurs in practice has not been officially proven to date.
In short: llms.txt is a voluntary, machine-readable description of your website – written in a format that AI systems can process well. The standard is still young and its effect has not been conclusively demonstrated.
Where does the standard come from?
The llms.txt specification was proposed in 2024 by Jeremy Howard – co-founder of fast.ai and one of the most influential AI researchers of recent years. The idea quickly gained traction in the AI and web community. Anthropic (the maker of Claude) has its own llms.txt on anthropic.com – this shows some interest, but is not proof that foreign llms.txt files are actively evaluated.
The standard is deliberately kept simple: no complex syntax, no new file format, no special parser needed. Plain Markdown that anyone can read and write.
Important: llms.txt is not an official W3C standard and not an RFC. It is a community proposal without binding support from major AI platforms.
llms.txt vs. robots.txt – the difference
| Feature | robots.txt | llms.txt |
|---|---|---|
| Purpose | Tells crawlers what NOT to crawl | Describes to AI what the website IS |
| Format | Custom keyword format | Markdown |
| Audience | All web crawlers | Specifically LLMs and AI systems |
| Content | Rules and restrictions | Descriptions and context |
| Standard | Official RFC standard | Community proposal (2024) |
| Adoption | Almost every website | Still rare – growing |
| Proven effect | Yes | Not yet demonstrated |
The two files are not mutually exclusive. robots.txt controls access, llms.txt provides context – provided AI systems evaluate it.
Who reads llms.txt?
This is the key question – and the answer is sobering. None of the major AI providers have officially confirmed that llms.txt is used as a signal for answer generation or ranking.
- Anthropic / Claude – has its own llms.txt on anthropic.com, but has not announced any documented support for evaluating third-party llms.txt files
- Perplexity AI – no officially confirmed support
- OpenAI / ChatGPT – no officially confirmed support; GPTBot access to llms.txt files was not found in independent log analyses
- Google – has explicitly rejected llms.txt (John Mueller, June 2025)
- AI agents – for individual software agents that search the web, a structured llms.txt is plausibly useful, even if this has not yet been systematically measured
Current state of research: Independent studies (SE Ranking Nov. 2025, Search Engine Land Jan. 2026, Trakkr March 2026) found no measurable effect of llms.txt on AI citation rates. The implementation effort is low and the risk is zero – but the effect remains speculative. llms.txt makes sense once the technical foundations (robots.txt, Schema.org) are already in place.
Structure and syntax
An llms.txt file is pure Markdown. There is a recommended structure:
Required and optional fields
There are no required fields – llms.txt is an open standard. Recommended at minimum: name/title of the website, a short description and contact information. Everything else is optional.
Ready-made examples for different website types
Example: SaaS tool / web application
Example: Local business
Example: Blog / content website
Step-by-step: create and deploy llms.txt
Step 1: Create the file
Create a new text file named llms.txt. Use UTF-8 encoding without BOM. Write the content in Markdown format following the template above – adapted to your website.
Tip: The llmshub.de generator crawls your website automatically and fills in all fields – you just review and download.
Step 2: Place it in the root directory
The file must be accessible at the main URL of the domain: https://yourdomain.com/llms.txt. For most web servers this means: place the file in the root directory – the same directory where index.html lives.
Step 3: Check the Content-Type
The web server should serve the file as text/plain. For .txt files this is normally automatic. To verify:
Step 4: Reference in robots.txt (optional)
You can add a comment at the end of robots.txt – this is not an officially supported format but does no harm:
Note: A standardised directive for llms.txt in robots.txt does not exist. Google Search Console reports the LLMs: line as a syntax error – those who want to avoid this should leave out this comment. More important is that the file is accessible at the correct URL.
llms-full.txt – the extended variant
Alongside llms.txt there is an optional extension: llms-full.txt. This file contains the complete content of the most important pages in a form optimised for LLMs – as a single readable file.
For most normal websites llms-full.txt is a nice-to-have, not a requirement. It makes particular sense for documentation sites, knowledge bases or extensive guides.
Checklist
- llms.txt created in the root directory of the domain
- File accessible at yourdomain.com/llms.txt (HTTP 200)
- Content-Type is text/plain
- Minimum content: name, description, contact
- Markdown format correct (headings with ##)
- Saved as UTF-8 without BOM
- Content kept up to date regularly
- No sensitive or internal information included
Create your llms.txt now
Use the free llmshub.de generator to create your llms.txt in seconds – automatically or manually.
Go to generator →