What Is llms.txt? Does It Work, and How to Create One (2026 Guide)

What is llms.txt?

llms.txt is a plain-text Markdown file, published at yoursite.com/llms.txt, that gives large language models and AI agents a concise, curated overview of a website: what it is, the key facts, and links to the pages that matter most.

It was proposed by Jeremy Howard of Answer.AI in September 2024, and the llms.txt specification was updated to version 2 in August 2026 after two years of real-world adoption. The idea is borrowed from robots.txt and sitemap.xml: a predictable filename at a predictable location that machines know to look for.

The problem it solves is practical. Web pages are built for people, wrapped in navigation, scripts, cookie banners, and layout. Converting that back into clean text is slow and error-prone for an AI agent, and every wasted token costs time and money. An llms.txt file hands the agent the essentials directly.

Does llms.txt actually work?

It depends on what you expect it to do. Here's what the evidence supports:

• It's widely adopted: thousands of sites publish one, documentation platforms generate them automatically, and OpenAI, Anthropic, and Google publish llms.txt files for their own developer documentation.
• It's used on demand by agents: the spec's author notes that llms.txt is mainly used at inference time, when an agent (such as a coding assistant reading documentation) needs information while helping a user.
• Google audits it: Chrome's Lighthouse checks llms.txt as part of its experimental "agentic browsing" audits, describing it as a way to stop agents spending more time crawling to understand a site.
• It's not a search ranking factor: Google's guidance on AI Overviews and AI Mode says you don't need an llms.txt file to appear in them.

The honest verdict: llms.txt won't move your Google rankings, but it's a cheap, low-risk way to make your business easier for AI agents to understand, and it's increasingly checked by standard tooling. Treat it as one small part of generative engine optimisation, not a shortcut.

The llms.txt format (v2)

An llms.txt file uses Markdown in a fixed order, so it's readable by people, language models, and simple parsers alike:

1. An H1 with the name of the site or project. This is the only required section.
2. A blockquote with a short summary containing the key information needed to understand the rest of the file.
3. Optional detail: paragraphs or lists (but no headings) explaining the business, facts, or how to interpret the links.
4. Zero or more H2 sections, each a list of links in the form [name](url), optionally followed by a colon and a short note.
5. An optional H2 section called "Optional" for secondary links an agent can skip when it needs to save space.

# Example Studio

> Example Studio builds web apps and mobile apps for startups in Manchester. Projects from £2,000.

Key facts:

- Founded 2024, based in Manchester, UK
- Contact: hello@example.com

## Services

- [Web app development](https://example.com/web-apps): Custom portals and SaaS platforms
- [Mobile apps](https://example.com/mobile): iOS and Android with React Native

## Guides

- [Pricing guide](https://example.com/blog/pricing): What a web app costs

## Optional

- [Sitemap](https://example.com/sitemap.xml): Every page on the site

Version 2 also allows llms.txt files at subpaths (for example /docs/llms.txt covering just your documentation) and recommends offering clean Markdown versions of important pages at the same URL with .md added. Some platforms also publish an llms-full.txt containing full page content in one file; that's a common convention rather than part of the core spec. Markdown versions matter most for sites whose content is hard to extract from HTML, such as JavaScript-heavy or canvas-rendered builds like Flutter on the web.

What Lighthouse checks (and why our own file failed)

Lighthouse's llms.txt audit is deliberately simple. It fetches /llms.txt from your domain root and then:

When we tested our own llms.txt against these rules, it failed. The file had a proper H1, but listed our pages in a Markdown table with bare URLs, and Lighthouse only counts links written as [name](url). We rewrote it in the v2 format with linked page lists, and it now passes. You can see the result at unlimiteddev.io/llms.txt.

Two other common causes of failure: a server that redirects /llms.txt or returns your app's HTML page for unknown paths, and a file generated by a function that occasionally errors. Serving llms.txt as a static file avoids both.

How to create an llms.txt file

1. Write a one-sentence summary of your business that includes what you do, who for, where, and a price point if you have one. This becomes the blockquote.
2. List the key facts an AI should get right: location, contact details, services, what you don't offer, and current prices.
3. Group your most important pages into H2 sections (for example Services, Guides, Company), each link written as [Page name](full URL): one-line description.
4. Save it as llms.txt and serve it from your site root as plain text, with a 200 response.
5. Test it: check it loads directly in a browser, run Lighthouse (npx lighthouse https://yoursite.com --only-audits=llms-txt), and ask an AI assistant questions about your business using only the file.
6. Keep it current: update prices, services, and links whenever your site changes.

If you're on WordPress, Yoast SEO and AIOSEO can generate the file for you, and Wix creates one automatically. On Next.js, place llms.txt in the public folder so it's served as a static file. If a developer is building your site, add llms.txt and AI crawler rules to the brief (our guide on how to hire a full-stack developer covers what else to ask); it adds almost nothing to the cost of a build. If you'd rather have it built in properly from the start, it's part of how Unlimited Dev builds every site.

llms.txt vs robots.txt vs sitemap.xml

The most common misconception is that llms.txt lets you allow or block AI. It doesn't. Access is controlled by robots.txt.

Controlling AI crawlers: GPTBot, ClaudeBot, PerplexityBot

The major AI companies now run separate bots for separate jobs, so you can allow AI search (which can send you traffic and citations) while opting out of model training.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

• ClaudeBot is Anthropic's training crawler. Blocking it signals your future content shouldn't be used to train Claude, and doesn't affect Claude's search, which uses Claude-SearchBot (Anthropic's crawler documentation).
• GPTBot is OpenAI's training crawler. Blocking it doesn't remove you from ChatGPT search, which uses OAI-SearchBot (OpenAI's crawler documentation). OpenAI says robots.txt changes take around 24 hours to reach its search systems.
• User-initiated fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, because a person asked for the page. Anthropic says Claude-User does respect it.
• Blocking Google-Extended doesn't remove you from AI Overviews or AI Mode, which run on regular Googlebot.

If you want AI search visibility, the key rule is simple: never block the search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot).

llms.txt checklist

• File served at /llms.txt as plain text with a 200 response
• Starts with an H1 containing your business name
• Blockquote summary with what you do, for whom, where, and a price point
• Key facts and current prices in plain lists
• Pages listed as [name](full URL): description, under H2 sections
• Secondary links under an "Optional" section
• Passes Lighthouse's llms-txt audit
• robots.txt allows the AI search agents you want to be found by
• Reviewed whenever prices, services, or key pages change

llms.txt works best alongside the fundamentals: fast, server-rendered pages, structured data, and content that answers questions directly. Whether you're launching an MVP, building a mobile app, planning a new business website, rebuilding a web app, or an agency needing white-label development (how it works), Unlimited Dev builds those foundations in from day one.

Frequently asked questions