llms.txt: what the file is, how to build it, and whether your site needs it

What llms.txt is and who proposed it

The llms.txt format was proposed by Jeremy Howard, co-founder of Answer.AI. The proposal was published on llmstxt.org on September 3, 2024.

The site itself describes the format as “a proposal to standardize the use of an /llms.txt file.” The word “proposal” matters here: this isn’t a standard adopted by a search engine or a standards body, but one author’s and one project’s initiative. For comparison, the protocol robots.txt is built on is a document that went through the full approval process at the Internet Engineering Task Force (IETF), with Standards Track status (RFC 9309). llms.txt never went through that process.

According to llmstxt.org itself, the format is already in use: thousands of sites publish an llms.txt, some documentation tools generate the file automatically, and in 2026 Chrome Lighthouse added a check for its presence. These are the format’s own author’s claims about its adoption — not confirmation that search engines factor the file into ranking or answers. The next section covers that distinction.

How the file is structured

Under llmstxt.org’s specification, the file’s structure is simple: an H1 heading with the site or project name is the only required element. After that comes an optional blockquote with a short description, and any number of H2 sections containing lists of page links, each with a brief description.

# Company name

> A short, one-sentence description of the company or product.

Documentation

• Page name: what the page is about, in one phrase

Additional

• About the company: a page with history and contact details

This illustrates the syntax, not a ready-made template for any site: the list of links and the wording depend on a specific resource's structure.

How llms.txt differs from robots.txt and sitemap.xml

The three files live at a site’s root, but they solve different problems and have different statuses.

File	Purpose	Who reads it	Required?	Who officially supports it
llms.txt	A short Markdown description of the site for AI tools	Per llmstxt.org’s own claims — documentation tools and certain agentic-browser features; no confirmed use by search engines (see the section below)	Optional: it’s a proposal, not a requirement from any system	The independent site llmstxt.org — not a standards body
robots.txt	Instructions for crawlers on which addresses they’re allowed to crawl	Crawlers that follow the protocol voluntarily	Optional: without the file, or on a 4xx error, Google treats it as if there are no crawling restrictions	A Standards Track protocol from the Internet Engineering Task Force, RFC 9309, 2022; implemented by Google and Yandex
sitemap.xml	A list of a site’s pages that helps crawlers find them	Google’s and Yandex’s crawlers — as a hint, not an instruction	Optional; Google says it isn’t necessary for every site, and Yandex doesn’t guarantee that listed pages will make it into search	The Sitemaps.org protocol; supported by Google and Yandex

The detailed directive syntax is a separate topic: robots.txt holds rules for bots, while llms.txt, over here, is a separate, self-contained file with a different purpose and format. If you need to understand the User-agent, Disallow and Allow directives, see the article on how the robots.txt file works. What to include in a sitemap and how to submit it to search engines is covered in the article on the sitemap.xml file.

What’s known about search engines’ and AI services’ use of llms.txt

The main question for a site owner is whether search engines and AI services read this file and whether it affects visibility. Let’s go company by company, based on their official documentation.

Google

Google has a page about llms.txt, but it’s not in the search-ranking documentation — it’s in the Chrome DevTools section on agentic browsing. Lighthouse checks for the file’s presence as part of its audits for agentic browsers; the page calls the format an “emerging convention” and notes that providing the file is still optional: if it’s missing, the audit is simply marked not applicable.

For Search itself, Google takes a different position. Its guide on optimizing a site for generative AI features states outright: showing up in Search and in generative AI answers doesn’t require creating machine-readable files, other special files, special markup, or Markdown. The same guide adds that you can maintain llms.txt files for other services that use them, but this won’t affect your site’s visibility or ranking in Google Search.

Yandex

As of the check date, Yandex has no official documentation about llms.txt as a format for site owners: no such section was found in Yandex Webmaster’s help for webmasters. At the same time, Yandex’s own developer portal (dev.webmaster) has its own /llms.txt file — an index of the Webmaster API documentation’s sections in Markdown. That’s the same approach llmstxt.org itself uses for its documentation: a short index for AI tools, not a statement of ranking or answer-generation policy.

OpenAI and other AI services

OpenAI splits its crawlers by task: GPTBot collects content that may be used to train models, while a separate crawler, OAI-SearchBot, handles showing sites in ChatGPT’s search results. The documentation on these crawlers doesn’t claim that either one reads third-party sites’ llms.txt files or changes search or training results because of it.

That said, the developers.openai.com documentation page about these crawlers itself links to its own /llms.txt file as an index of its own documentation. According to llmstxt.org, other companies use the file the same way for their own technical documentation — and that, too, has nothing to do with how AI services evaluate other sites.

Does your site need llms.txt

As of the check date, no major search engine has confirmed that it factors llms.txt into ranking or into generating AI answers. Google states outright that the file isn’t required for its AI features and doesn’t affect visibility. That matches the conclusion we already covered in the article on GEO and optimization for generative search: special «for AI» files and markup aren’t part of search engines’ confirmed requirements.

That doesn’t mean the file is harmful. You can create one: the format is simple, and the structure — site name, short description, a list of links to key pages — doesn’t contradict common sense. But llms.txt isn’t a priority when preparing a site for AI search. That work starts with making pages accessible to crawlers, clear entities, and answer-ready content.

If you decide to add one anyway, treat it as an optional extra, not a replacement for technical work on the site. Crawling and the sitemap are governed by robots.txt and sitemap.xml — and those are the files search engines officially support.

Frequently asked questions

Is it a standard, or just someone’s initiative?

One author’s initiative, framed as a proposal on a standalone site. llmstxt.org itself calls it “a proposal to standardize” the file’s use, not a standard adopted by a body like the IETF or the W3C.

Have Google or Yandex confirmed that they read llms.txt?

No. Google states outright that such a file isn’t needed for showing up in Search and in generative AI answers, and that it doesn’t affect a site’s visibility. As of the check date, Yandex’s help for webmasters has no such section.

How is it different from robots.txt?

robots.txt controls which addresses crawlers are allowed to crawl, and it’s a protocol that went through standardization at the IETF. llms.txt is a free-form description of a site’s content, without a comparable process behind it. The detailed syntax is covered in the separate article on robots.txt.

Do ChatGPT or other AI services use this file?

OpenAI’s official documentation on its crawlers has no confirmation of this: GPTBot collects content for training, OAI-SearchBot is a separate crawler for ChatGPT search, but no document says either one reads other sites’ llms.txt files.

Should you remove llms.txt if your site already has one?

There’s no reason to: the file doesn’t interfere with crawling and doesn’t violate other rules. It’s your call, but don’t treat it as a priority task in technical SEO.