- llms.txt is a Markdown file placed at the root of your domain (example.com/llms.txt) that tells AI which content on your site to read first.
- Standard proposed by Jeremy Howard in September 2024 via llmstxt.org. Adopted by Anthropic, Stripe, Cloudflare, Mintlify, Fern.
- Truth 2026: the big AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) rarely fetch. IDE agents (Cursor, Windsurf, Claude Code, Copilot) consult it systematically.
- Verdict: low cost (2 hours of work) high optional bet. Not yet a measurable SEO lever, but the infrastructure that will pay off when the standard becomes mainstream.
You have installed schema.org, optimized your content, populated your sameAs. You read a thread llms.txt is the next revolution. And you wonder if you really need to spend time on this.
Short answer: yes, but not for the reasons you were sold. Most guides on the subject oversell llms.txt as an SEO must-have. The 2026 reality is more nuanced and much more interesting. Here’s the guide that separates what works today, what will work tomorrow, and what will probably never work.
On the program: the exact spec (in five minutes), a copy-and-paste llms.txt skeleton adapted to your type of site, the honest verdict on the real adoption of AI crawlers in May 2026, the five errors that make an llms.txt useless, and the case study of our own file on NewsIA.
What is llms.txt and why it is controversial
llms.txt is a proposed open standard for telling language models which parts of your site to read first. The name is deliberately modeled on robots.txt : a single file, at the root of the domain, conventional and easy to find.
It was proposed in September 2024 by Jeremy Howard, founder of fast.ai and one of the most recognized public figures in the AI ecosystem. The official spec is on llmstxt.org and is on one page.
The principle is simple. Instead of letting AIs crawl your site like a human (which they do poorly, because they have no cookies, no JavaScript, no patience), you provide them with an annotated map: this is who I am, this is my reference content, this is how to classify it. It’s the AI equivalent of a book summary.
Why the debate? Because in May 2026, the real adoption of AI crawlers remains partial. According to data from Presenc.AI, 10.13% of sites have published an llms.txt, mainly in tech, cybersecurity and SaaSSoftware-as-a-Service, software billed for use and hosted by the publisher.. SaaS devs (Stripe, Anthropic, Cloudflare, Mintlify, Fern) are the first adopters. The media, e-commerce and consumer sectors are lagging behind.
llms.txt vs robots.txt vs sitemap.xml: which is used for what
Three files, three distinct roles. This is the most common confusion, and it leads to shaky implementations.
| File | Logic | Format | Target | Optional? |
|---|---|---|---|---|
| robots.txt | Exclusion | Guidelines User-agent / Disallow / Allow |
All crawlers (Google, Bing, AI) | Recommended |
| sitemap.xml | Exhaustive discovery | Normalized XML | Engine crawlers (Google, Bing) | Recommended |
| llms.txt | Curation | Structured Markdown (H1, blockquote, lists) | AI agents and IDE | Optional, growing |
robots.txt says, “Here’s what you can and can’t see.” sitemap.xml says: “here is everything that exists, with dates”. llms.txt says: “Here’s the main thing, here’s what I want you to remember, and here’s the order you should read it in.”
The three are complementary. None replaces the others. robots.txt and sitemap.xml remain essential for classic SEO. llms.txt comes on top, targeting a different audience.
The honest verdict: who is really reading your llms.txt in 2026
This is the section that other guides studiously avoid. However, it is the most useful.
Analyzes of server logs from sites that have implemented llms.txt since 2024 show a clear pattern:
- GPTBot (OpenAI) fetche occasionallywithout proven regularity.
- ClaudeBot (Anthropic) don’t fetch it almost never in automatic crawl mode.
- Google-Extended (Gemini) don’t fetch it not in the available public logs.
- PerplexityBot don’t fetch it not neither, despite the communication around the adoption by Perplexity.
Raw conclusion: in May 2026, llms.txt is not an SEO lever/GEOGenerative Engine Optimization, the optimization of content to be cited by AI response engines (ChatGPT, Perplexity, Gemini). measurable to have your brand cited in responses from ChatGPT, Perplexity, Gemini or Claude Search. The majority of content cited by these engines comes from their traditional index (Google, Bing) which does not consult llms.txt.
BUTand this is the pivot that the hyped guides miss, there is a public which consults llms.txt systematically: IDE agents and AI-assisted development tools.
- Cursor, Windsurfing, Claude Code, GitHub Copilot, Cline, Help : all get /llms.txt and /llms-full.txt when given a documentation site like contextThe context window: the maximum number of tokens that a model can process in a single request (eg: 200k, 1M)..
- Mintlify And Fernthe two leading doc generators for SaaS, automatically generate llms.txt for their clients.
- Anthropic released its llms.txt to help agents understand Claude.
The real game of llms.txt in 2026 is therefore not general AI indexing, it is developer experience. If your site has technical documentation accessed by developers who use an AI-assisted IDE, llms.txt makes those tools better. If you are a mainstream media outlet, the immediate ROI is lower, but the bet on the future remains rational.
Syntax: 3 rules, a Markdown format
llms.txt is a Markdown file. No JSON, no XML, no YAML. Pure Markdown.
The official spec imposes three elements in this exact order:
- An H1 with the name of the project or site. This is the only mandatory section.
- A blockquote (
>) with a short summary of the project, containing key information to understand what follows. - H2 sections organizing lists of links in the format
- [Titre](URL) : description en une ligne.
Minimum valid example:
# NewsIA
> Le magazine français de l'intelligence artificielle. Actualités, guides pratiques, fiches modèles et cartographie de l'écosystème IA français.
## Guides phares
- [Comment être visible dans ChatGPT en 2026](https://newsia.fr/guides/comment-etre-visible-chatgpt-geo-2026) : guide complet sur le GEO
- [Schema.org pour le GEO](https://newsia.fr/guides/schema-org-geo-guide-complet-2026) : les 7 schémas qui font citer votre marque
That’s all. You can test it, serve it to your root, and the standard is met. The rest is useful ornamentation.
Three important nuances:
- No marketing speak. THE LLMLarge Language Model, a language model trained on billions of parameters to generate and understand text. read your descriptions literally. A phrase like “the revolutionary solution that changes everything” is ignored or penalized. Prefer “complete guide to GEO, 4800 words, 7 diagrams with copy and paste examples”.
- One line per link. Standard Markdown, no nested paragraphs.
- Absolute URLs. No relative paths
/pagealwayshttps://exemple.com/page.
Anatomy of a walking llms.txt: the ideal skeleton
Here is a complete skeleton, ready to adapt to your site, derived from best practices observed at Anthropic, Stripe, Cloudflare and Mintlify.
# Nom de votre projet ou marque
> Phrase de résumé claire qui explique en une ligne ce que fait le site et à qui il s'adresse. Cette phrase est le contexte que l'IA utilise pour interpréter tout le reste.
Paragraphe optionnel d'introduction (2 à 4 phrases) avec les conditions de citation, la langue principale, la fréquence de mise à jour. Mentionnez ici votre licence si pertinent (Creative Commons, copyright).
## Actualités
- [Toutes les news](https://exemple.com/news) : actualités du domaine décryptées au quotidien
- [Flux RSS](https://exemple.com/rss.xml) : syndication des derniers articles
## Guides et ressources de référence
- [Guide phare 1](https://exemple.com/guides/guide-1) : description précise du contenu, 3000 mots, niveau intermédiaire
- [Guide phare 2](https://exemple.com/guides/guide-2) : description précise du contenu, 5000 mots, niveau avancé
## Produits, services ou outils
- [Produit principal](https://exemple.com/produit) : fonctionnalités, tarif, pour qui c'est conçu
- [Comparatif outils](https://exemple.com/outils) : tests et avis indépendants
## À propos
- [À propos](https://exemple.com/a-propos) : mission, équipe, principes éditoriaux
- [Contact](https://exemple.com/contact) : hello@exemple.com
- [Mentions légales](https://exemple.com/mentions-legales)
## Conditions de citation
- Citations libres avec attribution et lien retour vers la page source
- Pas de reproduction intégrale sans autorisation
- Sitemap général : https://exemple.com/sitemap.xml
Three principles for writing descriptions:
- Be factual and specific. “Complete guide 4800 words with 6 examples of copy and paste code” beats “the best guide on the subject”.
- Indicate the level and format when it is useful: beginner, intermediate, advanced, step-by-step tutorial, comparison, case study.
- 10 to 30 links maximum in the short version. Beyond that, switch to llms-full.txt.
llms.txt vs llms-full.txt: do you need both
Two files, two complementary uses.
llms.txt is the short, curated, editorial version. This is your AI business card. 10 to 30 links to your best content, with human descriptions. This is what an IDE agent reads first to understand your site.
llms-full.txt (unofficial but conventional) is the exhaustive version. All the canonical content of the site, with short descriptions, in a denser format. It’s the AI equivalent of an XML sitemap: less human readable but useful for agents who want a complete view.
Practical rule:
- Site < 50 pages: a neat llms.txt is enough. llms-full.txt does nothing for you.
- Site 50 to 500 pages: curated llms.txt + dynamically generated llms-full.txt.
- Site > 500 pages (SaaS documentation, e-commerce): both are essential, and llms-full.txt must be automatically generated from your database.
If you are starting, do llms.txt first. You add llms-full.txt only when you really need it.
The iceberg: what Google sees vs. what an IDE agent reads
llms.txt as a regular text file
- Indexable like a text page (Google can crawl it)
- No direct SEO bonus
- Not used by Googlebot or Bingbot robots to cite your content
- Often ignored in favor of your sitemap.xml
An annotated map to understand the site
- Cursor, Windsurf, Claude Code automatically retrieve it when you point them to the doc
- The IDE uses descriptions to decide which pages to load in context
- The section names (News, Guides, Products) structure the agent’s understanding of the site.
- The agent can combine llms.txt + loaded page + RAGRetrieval-Augmented Generation, a technique that combines an LLM with a documentary base to anchor its answers in sources. to respond with increased precision
If your target includes developers or power users who use an AI-assisted IDE, llms.txt changes their experience immediately. For the general public today, it is rather an infrastructure ready for tomorrow.
NewsIA in case of study: decryption of our own llms.txt
NewsIA publishes its llms.txt at newsia.fr/llms.txt. Here’s what it does and why, to serve as a concrete model.
Structure used:
- H1 “NewsIA”
- Blockquote : editorial positioning phrase (French AI media, decrypted news, guides, model sheets, cartography)
- Contextual paragraph specifying that NewsIA is an independent French editorial media, that all content is in French, and authorizing quotes with backlinks
- Sections H2 : News (news + RSS + sitemap news), Guides and resources, Flagship guides to cite as a priority (the section that makes the difference), AI models, Ecosystem mapping, About, Citation conditions
The choice that makes the difference: the section “Key guides to cite as a priority”. It lists 8 reference contents (GEO guide, ChatGPT/Claude/Gemini/Mistral comparison, best free AI, top Claude Skills, etc.) with a short but factual description of each. It is this section that an IDE agent consults first to decide which page to load.
Three lessons that can be transposed:
- Cure aggressively. Not all your content, just your centerpieces. Twenty well-chosen links are better than a hundred diluted links.
- Describe concretely. “Complete Generative Engine Optimization method for AIs to cite a company” gives the agent more leverage than “the best GEO guide”.
- To update. When a new flagship guide comes out, he joins the section. When a guide gets old, he leaves it. The file should reflect your current editorial.
To go further on the overall strategy, see our complete guide on GEO (Generative Engine Optimization), which details how llms.txt fits with the rest of the system (robots.txt, schema.org, sameAs).
The 5 errors that make your llms.txt useless
Five common pitfalls that hyped guides forget to mention.
-
Too many links. Twenty is enough. One hundred dilutes the signal and drowns out important pages. Better an llms.txt with ten excellent links than one with a hundred average links.
-
Marketing descriptions. “The leading solution”, “the French reference”, “our recognized know-how”: LLMs ignore this vocabulary. Prefer factual descriptions: type of content, approximate length, level, language, date updated.
-
Relative URLs.
/guide-1doesn’t work. All URLs must be absolute, withhttps://and the complete domain. Without this, the agent cannot fetch them correctly. -
File never updated. An llms.txt that is from 2024 and does not mention your flagship content from 2026 sends an abandoned site signal. To be updated at least once a quarter, or with each new pillar content.
-
Forget it
Cache-Control. llms.txt is not a static file: it must be able to be updated and properly invalidate caches. Serve it with a headerCache-Control: public, max-age=3600, s-maxage=3600(1 hour) or shorter depending on your update frequency.
Three concrete actions.
Check. Open votredomaine.com/llms.txt in the browser. You should see the Markdown served in Content-Type: text/plain; charset=utf-8. No redirects, no 404s, no wrapping HTML. The most used community validator is llmstxt.org/validator, which checks the syntax against the spec.
Submit. There is no official submission anywhere as of May 2026. Agents and IDE discover the file by testing the conventional URL. The only useful action: mention it in your secondary sitemaps, in your footer, and share it publicly (X, LinkedIn) to signal its existence to developers.
Measure. Monitor your server logs to identify who is fetching /llms.txt. Interesting user-agents to filter: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ChatGPT-User, Anthropic-User-Agent, plus IDE agents (cursor, windsurf, etc.). If you see fetches, compare the pages visited next: this is proof that the file is directing the agent.
For direct marketing ROI, give up measuring it in 2026. The impact on AI citations cannot be isolated. Instead, measure IDE usage if you have a developer audience, and include llms.txt in your long-term infrastructure.
5-step action plan
Step 1
Define your positioning in one sentence
Before writing the file, write the blockquote. A sentence that says who you are, for whom, and what you publish. This is the context that the AI will use to interpret everything else.
Editorial~30 minutes
Step 2
List your 10 to 20 key pieces of content
Not the whole site. Your pillar guides, your reference comparisons, your key product pages. For each, write a one-line factual description: type, length, level, subject.
Curation~1 hour
Step 3
Serve it at the root
Create the route `/llms.txt` which returns the Markdown as `text/plain`. If you are in Next.js, this is a route APIApplication Programming Interface, the programmatic access channel to a service or model.. If you are in WordPress, a plugin or mu-plugins. If you are static, a file in public/.
Technical~30 minutes
Step 4
Validate and report
Pass the file into llmstxt.org/validator. Mention it in your footer or on a “For AI” page. Post once publicly (X, LinkedIn) that your llms.txt exists: this increases the chances that an AI crawler will discover it.
Distribution~15 minutes
Step 5
Maintain and evolve
Update with each new pillar content. Monitor server logs for AI and IDE agent fetches. When your site exceeds 50 pages, add llms-full.txt in addition, dynamically generated.
Maintenancecontinuous
What to remember
llms.txt is not the GEO magic wand you were sold. It’s more subtle and more sustainable: a low-cost bet (two hours of work) on an infrastructure that already pays today for a specific audience (IDE agents and developers on ChatGPT) and which will pay much more broadly tomorrow, when the big AI crawlers finally follow.
The right reflex in May 2026: implement a clean and factual llms.txt, serve it properly, update it. Without expecting a measurable effect on ChatGPT or Perplexity citations in the short term. With the conviction that the sites that get started now will be the ones that the AI will recognize first when the standard is mainstream.
The rest of the system matters much more: a specific entity graph (schema.org Organization + @id + sameAs), structured content with FAQPage and answer-first, off-site mentions that confirm your brand. For details, read our complete schema.org guide to GEO, which details the schemas that really make a brand name, and our complete guide to GEO, which places llms.txt in the overall strategy.
For a hotel and restaurant sector case where brand consistency is particularly important, see our GEO hotels and restaurants 2026 guide.
And if you want to see a real, factual, up-to-date llms.txt, check out ours. No marketing speak, just the links and descriptions that help an AI understand NewsIA. This is exactly the model you want to copy.

