llms.txt
llms.txt is a proposed AI search standard — a markdown file at the root of a domain that provides curated metadata about the site to AI systems — respected by Anthropic, Perplexity, and OpenAI's GPTBot but explicitly declined by Google as of 2026-05-15.
Definition
llms.txt is a plain markdown file placed at https://example.com/llms.txt that contains curated metadata about a site intended for consumption by AI systems. Proposed by Jeremy Howard (Answer.ai) in mid-2024, it aims to provide the same role for AI engines that robots.txt provides for traditional crawlers.
Standard format
# Site Name
> One- or two-sentence factual description.
## Docs
- [Title](URL): brief description
- [Title](URL): brief description
## Examples
- [Title](URL): brief description
## Optional
- [Title](URL): brief description
The file is markdown by design — readable to both humans and LLMs without a separate parsing layer.
Adoption status (2026)
| Platform | Status | Notes |
|---|---|---|
| Anthropic (Claude) | READS | Explicit support, documentation references it |
| Perplexity | READS | Incorporates into source ranking |
| OpenAI (ChatGPT, GPTBot) | RESPECTS | Bot honors site hints |
| Google (AIO + Gemini) | DECLINED | 2026-05-15 AI Optimization Guide states machine-readable AI files are not needed |
Strategic call (post-2026-05-15)
Google declining the standard does not kill it. Three of four major AI search engines still respect llms.txt. The practical guidance:
- Google-only target: skip llms.txt. Follow Google's AI Optimization Guide as written.
- Multi-platform target: ship llms.txt anyway. Cost is one static file (under 5KB typically). Benefit applies to Claude, Perplexity, and ChatGPT.
Common pitfalls
- Marketing copy. llms.txt is for factual metadata. Lines like "the world's leading X" reduce trust without improving citation.
- Exhaustive URL dumps. Curate the 10-30 highest-leverage URLs, not every page on the site.
- Forgetting to update. llms.txt should be regenerated whenever priority content changes.
See /llms-txt-guide for the deep dive and /tools/llms-txt-generator for the free generator tool.
Frequently asked
Is llms.txt worth shipping if Google declined the standard?
Yes, for multi-platform AI search targets. Google declining only affects Google AI Overview and Gemini ranking. Claude, Perplexity, and ChatGPT (via GPTBot) all still respect llms.txt. Cost is near-zero (one static file). Skip only if your strategy is Google-AI-only.
Where should llms.txt live on my site?
Root of the domain: https://yoursite.com/llms.txt. Same convention as robots.txt. Sub-path locations (e.g., /docs/llms.txt) are not part of the standard and will not be discovered by AI systems.
Does llms.txt replace robots.txt or sitemap.xml?
No. They serve different purposes. robots.txt controls crawl access (what bots can fetch). sitemap.xml is an exhaustive URL inventory for indexers. llms.txt is curated metadata for AI systems — the priority pages you want them to focus on. All three layers complement each other.
Related
Stop guessing where you rank in AI search
Citare measures citation rate and share of voice across ChatGPT, Google AI Overview, Gemini, Claude, and Perplexity — weekly, for your priority queries. Free forever tier.