Skip to content
agentmarket

Blog

llms.txt explained: what it is, what it isn't, and how to add one

· 3 min read

llms.txt is a plain Markdown file at the root of your site, at /llms.txt, that gives large language models a short, clean briefing: what the site is, what matters on it, and where to find the details. Think of it as a README for AI.

It was proposed in September 2024 by Jeremy Howard of Answer.AI. The reasoning is simple. Web pages are full of navigation, scripts and layout that waste a model's limited context. A curated Markdown summary is cheaper to read and harder to misunderstand.

What it is not

Be clear-eyed about this before you spend time on it:

  • It is not a standard. It is a proposal with growing but uneven adoption.
  • It does not control access. That is robots.txt's job. llms.txt describes. It does not allow or block anything.
  • It is not a ranking factor. No major search engine has said it uses llms.txt for ranking, and no AI company has committed to always reading it.

Why bother, then? It takes ten minutes, it cannot hurt, and when an agent or a user's AI tool fetches your site to answer a question, a clean summary gives it the right facts instead of guesses. Developer tools, documentation sites and coding agents already use it.

The format

The proposal defines a simple structure:

  1. An # H1 with the name of the site or project. This is the only required part.
  2. A > blockquote with a one or two sentence summary.
  3. Optional paragraphs or lists with more context.
  4. ## H2 sections containing Markdown link lists, each link optionally followed by a colon and a note.
  5. An optional section titled ## Optional for links that can be skipped when context is short.

Here is a minimal example:

# Acme Analytics

> Privacy-friendly web analytics for small sites. No cookies. Free up to 10k views a month.

## Product

- [Pricing](https://acme.example/pricing): Free, Pro $9/mo, Team $29/mo
- [Docs](https://acme.example/docs): install snippet and API reference

## Optional

- [Changelog](https://acme.example/changelog)

llms-full.txt

Many sites also publish /llms-full.txt: the full text of the important pages concatenated into one Markdown file, so a model can read everything in a single request. It is useful for docs and blogs. Keep llms.txt short and put the bulk in llms-full.txt.

Writing a good one

  • Lead with facts. What the product does, who it is for, what it costs, what is live and what is not.
  • Use the same names everywhere. If your homepage says "Job Search Agent", do not call it "JobBot" in llms.txt.
  • Say what you are not. "We are not a directory of third-party tools" prevents a common misreading.
  • Use absolute URLs. Agents may read the file out of context.
  • Keep it current. A stale llms.txt is worse than none. Generate it from the same data as your site if you can.
  • Skip marketing language. Models do not need "revolutionary".

Generating it in Next.js

If you use Next.js with static export, you can build llms.txt from code with a route handler, so new pages and posts show up automatically:

// app/llms.txt/route.ts
export const dynamic = "force-static";

export function GET() {
  const body = ["# Your Site", "", "> One-line summary."].join("\n");
  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

At build time this becomes a plain llms.txt file in the output folder. No server is needed.

Make sure crawlers can reach it

An llms.txt that AI crawlers cannot fetch does nothing. Check two things:

  1. robots.txt does not disallow the crawlers you care about, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot.
  2. Your CDN or firewall does not block them. Some providers block AI crawlers with a dashboard toggle that overrides robots.txt. Test with curl -A "ClaudeBot" -I https://yoursite.com/llms.txt.

You can link to the file from your HTML head with <link rel="alternate" type="text/plain" href="/llms.txt"> and mention it in a robots.txt comment so it is easy to discover.

See ours

AgentMarket publishes both: llms.txt and llms-full.txt, generated from the same data as this site. To check whether your own site is readable by AI answer engines, Web Scan includes GEO checks and gives fixes you can paste into a coding agent.