Does llms.txt Actually Help With AI Visibility?

The short decision: should your site have llms.txt?

The rest of this guide explains why the table looks this way, what to do instead, and how to check whether any change you make affects your visibility.

What llms.txt is, and what it isn't

llms.txt is a proposal from Jeremy Howard of Answer.AI, first published on September 3, 2024. It describes a Markdown file at /llms.txt that gives language models a curated index of a site, with optional clean .md versions of individual pages (Answer.AI, 2024). The specification describes itself as "open for community input" and is hosted as an informal overview on GitHub (llmstxt.org).

It is a convention, not a ratified standard like robots.txt. It only matters when a specific system decides to read it.

A revised v2 of the spec (August 10, 2026) adds per-section files such as /docs/llms.txt, rel="alternate" type="text/markdown" and rel="describedby" link relations, and an optional byte-order mark (llmstxt.org). How-to articles written before then may describe an outdated format.

llms.txt also cannot block crawling or opt you out of training. Those controls live in robots.txt and user-agent rules, and the spec itself says the two files serve different purposes.

Why llms.txt can't make an AI engine choose you

llms.txt helps an agent move around a site it is already using. It gives an answer engine no reason to pick that site in the first place. That single distinction explains most of the evidence.

The spec's author is explicit about the intended use. The file is "used on demand, when an agent needs information about a topic while assisting a user," and was expected to be "mainly useful for inference rather than training." Its heaviest use is in software documentation, "where coding agents follow them to find API references and tutorials" (llmstxt.org).

Think of it as a store directory. A directory helps shoppers who are already inside the building find the right aisle. It does nothing to bring people through the door. Citation decisions happen before anyone reaches the door, when an answer engine decides which sources are relevant and trustworthy.

Google's John Mueller made a sharper version of this point in June 2026. Because the file is self-reported, an LLM "basically, by design, can't trust what is here as a way of differentiating between different websites" (Search Engine Journal, June 2026). He compared it to the keywords meta tag. Every site describes itself favorably, so the description can't separate you from competitors.

There is also a redundancy problem, which is our own reasoning rather than a documented finding. If a tool can generate your llms.txt from your HTML, an answer engine can extract the same information from that HTML directly. Clean, crawlable pages do the real work, and the index file just repeats them.

What the evidence says about llms.txt and AI citations

Google: not used, and no plans to use it

In July 2025, Gary Illyes said Google doesn't support llms.txt and isn't planning to. He added that getting into AI Overviews takes "normal SEO practices. You don't need GEO, LLMO or anything else" (Search Engine Land, July 2025). This was reported secondhand from a Search Central event, but later statements have been consistent with it.

In January 2026, Mueller said that llms.txt files hosted on Google properties were not an endorsement, and Google's Search team removed the file from its own docs (Search Engine Roundtable, January 2026). By June 2026 he called the file "purely speculative": "the file has existed for years, yet none of the AI systems use it" (Search Engine Journal, June 2026).

Mueller speaks for Google, not for OpenAI, Anthropic, or Perplexity. None of those companies has publicly committed to using llms.txt to select sources. That could change.

Outcome data: no correlation with citations

SE Ranking analyzed 300,000 domains and "found no link between llms.txt adoption and citation frequency in LLM answers," as reported by Search Engine Journal. This is observational data and can't rule out a small effect on one platform. Still, it is the most direct answer available to the "measurably improve" question, and it shows nothing.

Bot fetches: real, but not proof

Some site owners report AI bots requesting the file. One practitioner posted logs in July 2025 showing OpenAI "pinging our servers every 15 minutes" for llms.txt (Search Engine Land, July 2025). That was a single screenshot, and the bot's identity wasn't verified.

Even a verified fetch shows only that a file was retrieved. It doesn't show that the file was used, and it doesn't show that anything was cited.

Why llms.txt looks more important than it is

The file feels like an emerging requirement because it shows up in many places:

• Thousands of sites publish one, and documentation platforms generate it automatically (llmstxt.org).
• Chrome's Lighthouse now audits for it and warns that without it, agents "may spend more time crawling the site" (Search Engine Journal, June 2026).
• AI labs publish llms.txt files for their own developer docs.

None of these signals means answer engines read your file to decide whether to recommend you. Publishing a file and consuming other sites' files are different activities. When an AI lab offers llms.txt for its API docs, it helps developers' coding agents. It says nothing about how that lab's assistant picks sources. Lighthouse's audit is about agent usability, and it uses hedged language. Google Search has said separately that the file plays no role in visibility.

Where llms.txt is genuinely useful

llms.txt earns its place in a few narrow situations:

• Developer documentation. Coding agents and IDE tools follow the file to find API references and tutorials. This is its main real-world use.
• Agents already on your site. Mueller left room for llms.txt to help once an agent is on a site, while noting that agent standards may take another 6–12 months or more to settle (Search Engine Journal, June 2026).
• Users who paste it into a tool. A clean Markdown summary of your docs is convenient context for someone working in an AI assistant.

Mueller's practical test works well as a rule: "When an AI platform that brings you clients complains that it needs the file for your site, then I'd recommend taking the time to create one" (Search Engine Journal, June 2026).

If you publish one, keep it simple

No source shows that llms.txt causes harm beyond wasted effort. If you add one:

1. Follow the v2 format. Use an H1 with the site or project name, a short blockquote summary, and H2 sections that list key pages with one-line descriptions. Put secondary links in an "Optional" section (llmstxt.org).
2. Link to clean Markdown versions of your most-used pages if your platform can produce them.
3. Consider noindexing it. Google has suggested you "probably should noindex it if you do use it," according to Search Engine Roundtable's paraphrase of earlier comments (Search Engine Roundtable, January 2026). Treat this as guidance, not documented policy.
4. Don't pay for promised visibility. Generator tools that claim llms.txt will get you into AI answers have no evidence behind that claim.

Keep expanded files such as llms-full.txt, or the original proposal's llms-ctx.txt and llms-ctx-full.txt, separate in your planning. Those inline the full linked content and serve the same inference-time purpose.

What to do instead to improve AI visibility

These levers address the discovery problem that llms.txt cannot touch. They are ordered by dependency, since each one assumes the previous one is in place.

1. Don't block AI crawlers and agents. Check robots.txt, firewall rules, and bot-management settings for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot. Mueller called not blocking agents "the most basic agentic optimization" and "the biggest" hurdle for most sites (Search Engine Journal, June 2026).
2. Serve crawlable, indexable HTML. Google's position is that normal SEO gets content into AI Overviews. Important content that only appears after client-side rendering, or that sits behind interaction, is harder for any system to use.
3. Publish answer-first pages for the questions buyers actually ask. If buyers ask an assistant "best invoicing tool for freelancers," you need a page that answers that question directly and specifically. A generic features page won't do the job. Our step-by-step playbook for getting mentioned by ChatGPT walks through this process.
4. Get included in the sources AI already cites. Answers often draw on reviews, comparisons, and roundups. Being in those sources can matter more than anything on your own domain. For the tradeoffs, see whether third-party listicles are good GEO or just AI-era link building.
5. Keep your brand description consistent across your site, profiles, and listings, so that assistants don't pick up conflicting or outdated claims.

For agent tasks such as checkout or bookings, Google has pointed to WebMCP and commerce integrations as the clearer path. WebMCP is also still a proposal (Search Engine Journal, June 2026).

How to check whether llms.txt, or anything else, changed your visibility

Treat any change as a test and watch three separate signals. They measure different things, and mixing them up is the most common mistake in this area.

1. Bot requests. Is anything fetching /llms.txt or your key pages? This can only be seen server-side, because browser analytics don't reliably capture crawler requests. A matching user agent is an observation. It is not authenticated identity.
2. Mentions and citations. For a fixed set of buyer prompts, record whether each engine mentions or cites you before and after the change. Answers vary from run to run, so compare several runs over weeks rather than one before-and-after screenshot.
3. AI referral traffic and conversions. Check whether visits from ChatGPT, Perplexity, and similar sources rise, and whether those visitors convert.

A bot fetching your llms.txt is signal one only. If signals two and three don't move, the file didn't change your visibility.

Tracking all three is where most teams get stuck, because the data sits in different tools. Tideflow AI logs AI bot requests server-side through a Cloudflare Worker or Express middleware (Tideflow AI). It marks as verified only the requests that carry Cloudflare verified-bot metadata. It also monitors mentions and citations across ChatGPT, Gemini, Perplexity, Google AI Mode, and Google AI Overview, and it keeps crawler visits, citations, and referral visits as distinct signals (Tideflow AI vs Profound). That lets you see whether bots requested your llms.txt and whether your presence in the answers changed, without treating a fetch as a result. Tideflow can't guarantee recommendations, and no tool can.

Timeline of llms.txt positions

Frequently Asked Questions

Does llms.txt stop AI models from training on my content?

No. llms.txt is a guide for agents, not an access control. To limit crawling or training, use robots.txt rules for specific AI user agents.

Does Lighthouse checking for llms.txt mean Google Search rewards it?

No. Chrome's Lighthouse audit concerns how efficiently agents can use a site. Google Search has said separately that it doesn't use llms.txt for Search or AI features.

Will llms.txt matter more as AI agents mature?

It might, for agents that are already working on your site. Mueller expects agent standards to take at least 6–12 more months to settle, and Google currently favors WebMCP for agent tasks. Recheck vendor statements before investing more than a basic file.

Your next step

This week, check your robots.txt and bot-management settings to confirm that AI crawlers aren't blocked. That one check matters more than llms.txt. Then choose ten buyer prompts, record where you appear across the major assistants, and use the gaps you find to decide which pages to build. Add llms.txt if you have developer docs, and judge it by citations and referrals, not by bot fetches.

Sources

• Jeremy Howard, /llms.txt—a proposal to provide information to help LLMs use websites (September 2024); The /llms.txt file, v2 (September 2024)
• Matt G. Southern, Google's Mueller Says llms.txt Can't Help LLMs Differentiate Sites (June 2026)
• Barry Schwartz, Google says normal SEO works for ranking in AI Overviews and LLMS.txt won't be used (July 2025); Google Does Not Endorse LLMs.txt Files (January 2026)
• Roger Montti, Google Confirms LLMs.txt Has No Current Implementation (June 2026)
• Tideflow AI, AI Visibility & Content Platform; Detect AI Bots; Tideflow AI vs Profound: Which AI Visibility Platform Should You Choose?