LLMS.TXT GUIDE
What is llms.txt, and do you need one?
llms.txt is a Markdown file at yoursite.com/llms.txt that gives AI agents a short, curated map of your site: a name, a one-line summary and links to your most useful pages. Jeremy Howard proposed it in September 2024. It is not an official standard, and Google Search ignores it. Digimetrics' llms.txt Generator builds one in about a minute.
- Free for 7 days
- Every tool unlocked
- No card needed
Updated 23 October 2026 · Digimetrics editorial team
The short answer
- llms.txt is a proposal, not a ratified standard. It was published by Jeremy Howard of Answer.AI on September 3, 2024 and revised as v2 in 2026.
- Only one part is required: an H1 with the site or project name. A summary blockquote, notes and H2 sections of links are optional but recommended.
- It is used mostly at inference time, when an agent needs to look something up, not for training.
- Google says Google Search does not use llms.txt and it will neither help nor harm your rankings. Its value is with other AI tools and agents.
- It takes minutes to make and costs almost nothing to maintain, so for most sites it is a cheap, low-risk addition rather than a priority fix.
What is llms.txt used for?
llms.txt helps an AI agent find the right information on your site quickly, without wading through navigation, ads and scripts. It is a guided starting point, read on demand when an agent is helping a user.
The proposal's reasoning is practical. Web pages are built for people, so converting them to clean text is hard and imprecise, and context windows are still too small for most websites in full. A short file of expert-level pointers saves the agent time and tokens. The author notes that llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references, but the same structure works for a business outlining its products and policies.
Adoption is real if uneven. According to llmstxt.org, thousands of sites publish one, documentation platforms generate them automatically, Chrome's Lighthouse audits sites for one in its agentic browsing checks, and OpenAI, Anthropic and Google's Gemini team publish llms.txt files for their own developer docs. Wix, Yoast SEO and AIOSEO can generate one too.
llms.txt format and example
An llms.txt file is plain Markdown in a fixed order: an H1 name (the only required part), a blockquote summary, optional notes, and H2 sections containing lists of links with short descriptions.
The spec lists the sections in this order: an optional byte-order mark, the H1, the blockquote, any non-heading paragraphs or lists, then H2-delimited file lists. Each list item is a Markdown link in the form [name](url), optionally followed by a colon and notes. A section named Optional is a convention for links an agent can skip when it needs a shorter context.
Here is a small example for a fictional local business, following the spec. Each bullet below is one line of the file:
- # Riverside Dental
- > Family dental practice in Riverside offering checkups, cleanings, implants and emergency appointments. Open six days a week.
- ## Services
- - [Dental implants](https://example.com/implants.md): Procedure, recovery time and financing options
- - [Emergency dentist](https://example.com/emergency.md): Same-day appointments and what counts as an emergency
- ## Practical
- - [Pricing and insurance](https://example.com/pricing.md): Accepted plans and self-pay prices
- - [Contact and hours](https://example.com/contact.md): Address, phone and opening hours
- ## Optional
- - [Our team](https://example.com/team.md): Dentist bios and qualifications
| Part | Required? | What to put in it |
|---|---|---|
| H1 | Yes, the only required section | Site or project name |
| Blockquote | Recommended | One or two sentences with the key facts needed to understand the rest |
| Notes (paragraphs, lists) | Optional | How to interpret the files; no headings allowed here |
| H2 file lists | Optional, usually used | Markdown links to detailed pages, each with a short description |
| Optional section | Convention | Secondary links an agent can skip |
The proposal also suggests offering a clean Markdown copy of important pages at the same URL with .md added, and linking to llms.txt with rel="describedby".
llms.txt vs robots.txt vs sitemap.xml
robots.txt says what bots may access, sitemap.xml lists every indexable page, and llms.txt is a short, curated guide for AI agents. They do different jobs and work together.
The proposal is explicit that robots.txt and llms.txt have different purposes: robots.txt tells automated tools what access is acceptable, while llms.txt is used on demand when an agent needs information. It also explains why a sitemap is not a substitute: sitemaps rarely point to LLM-friendly versions of pages, never include helpful external links, and usually cover far more content than fits in a context window.
| File | Audience | Purpose | Controls access? |
|---|---|---|---|
| robots.txt | All crawlers, including AI bots | Allow or disallow crawling by path and user agent | Yes, for bots that respect it |
| sitemap.xml | Search engines | List every indexable page for discovery | No |
| llms.txt | AI agents and LLM tools | Curated summary and key links for answering questions | No |
| llms-full.txt | AI agents needing depth | Detailed text version of key content in one file (a common companion convention) | No |
| AGENTS.md | AI coding agents | Build, test and code-style instructions for a code repository | No |
Does llms.txt work? Does Google use llms.txt?
Google says no: Google Search does not use llms.txt, and having one will neither help nor harm your Search rankings or AI Overview visibility. Other AI tools and agents do read it, especially for documentation.
Google's guide to optimizing for generative AI lists llms.txt under things you can ignore for Google Search, and adds that it is completely fine to create one for other services that use it. Its AI features page says you do not need new machine-readable or AI text files to appear in AI Overviews or AI Mode.
So the honest answer to “is llms.txt worth it?” depends on your audience. If you publish documentation, an API or a product agents need to understand, the evidence of use is strongest. For a typical business site, think of it as a low-cost courtesy to AI agents: it takes minutes and will not hurt you, but it will not move your Google rankings. Fix crawler access and page quality first; our AI SEO checklist puts it in that order.
OpenAI and Perplexity document their search crawlers' requirements in terms of robots.txt access, not llms.txt. See which AI crawlers to allow.
llms-full.txt vs llms.txt: which do you need?
llms.txt is the short index; llms-full.txt is a longer companion that puts the detailed content itself in one file. Publish llms.txt first, and add llms-full.txt if agents need depth without following links.
The llmstxt.org proposal defines llms.txt itself; llms-full.txt is a widely used companion convention, popular with documentation platforms, rather than part of the core spec. The trade-off is size: llms.txt stays small enough to fit in context, while the full version can be large.
Digimetrics' llms.txt Generator produces both from one website address. In the official tutorial's sample (fictional brand Brightpath), the check found no llms.txt yet and AI bots allowed, then listed four key pages with plain descriptions, ready to download. A follow-up AI Readiness Check in the tutorial passed 8 of 9 checks (89%); the one fix left was adding llms-full.txt.
llms.txt best practices
Keep it short, factual and current: plain language, a description on every link, links to clean pages, and a test with a real agent.
These follow the proposal's own guidelines plus what we see go wrong most often: For where llms.txt fits in the wider discipline, see what generative engine optimization is and how to rank in Perplexity.
- Use concise, clear language and avoid unexplained jargon, as the proposal recommends.
- Describe every link. “Pricing” alone is weak; “Pricing: plans, limits and what the free tier includes” tells the agent when to follow it.
- Link to content, not to menus. Pages with the facts, ideally clean Markdown versions.
- State the facts people ask about in the summary: what you do, where, for whom.
- Do not list everything. It is a curated guide, not a sitemap.
- Keep it in sync with your site. A link to a retired page or an old price misleads agents.
- Validate the structure by checking the H1 is the first heading and every list item is a proper Markdown link.
- Pair it with structured data. The proposal notes llms.txt can reference your structured data; generate Organization markup with the Schema Generator.
HOW IT WORKS
How to create an llms.txt file in five steps
- 1
Check what you have
Run the AI Readiness Check or the llms.txt Generator's built-in check to see whether an llms.txt exists and whether AI bots are allowed in.
- 2
Pick the pages worth listing
Choose the pages an agent needs to answer questions about you: what you do, pricing, key products or services, policies, contact, documentation. Usually 5 to 20 links.
- 3
Write the file in the spec's order
H1 name, a blockquote summary, optional notes, then H2 sections with Markdown links and a short description after each colon. Or let the llms.txt Generator write llms.txt and llms-full.txt from your URL.
- 4
Upload it to your site root
Place it at /llms.txt (or under a path such as /docs/llms.txt to cover only that section). Serve it as plain text.
- 5
Test and keep it current
Ask an AI agent questions about your business using only the file as a starting point, as the proposal suggests. Update it when you add or retire key pages.
QUESTIONS PEOPLE ASK
Frequently asked questions
Still unsure? Email support@digimetrics.ai and a person will reply.
Is llms.txt an official standard?
No. It is an open proposal published at llmstxt.org by Jeremy Howard, open for community input on GitHub. It is widely adopted for documentation, but no standards body has ratified it.
Where do I put llms.txt?
At the root of your site, /llms.txt. The proposal also allows a file at a subpath, such as /docs/llms.txt, which covers only the pages under that path; agents should use the most specific file.
Will llms.txt improve my Google rankings?
No. Google says Google Search ignores llms.txt and that it neither helps nor harms your visibility or rankings in Search, including AI Overviews.
Is there a free llms.txt generator?
Digimetrics' llms.txt Generator makes both llms.txt and llms-full.txt in about a minute, and it is included in the 7-day free trial. WordPress plugins such as Yoast SEO and AIOSEO, and site builders such as Wix, can also generate one.
What is the difference between llms.txt and AGENTS.md?
llms.txt guides AI agents through a website's content. AGENTS.md is a Markdown file in a code repository that gives AI coding agents build steps, tests and code conventions. Different audiences, different jobs.
Does llms.txt stop AI from training on my content?
No. llms.txt does not control access. To opt out of AI training crawlers such as GPTBot or ClaudeBot, use robots.txt rules for those user agents.
Give AI agents a clear map of your site.
Generate llms.txt and llms-full.txt in about a minute. Start free for 7 days, no card needed.