Does llms.txt Actually Help With AI Visibility?

Who this is for

This is for anyone who has seen llms.txt recommended as an "AI SEO" tactic, whether by a plugin, an agency, or a blog post, and wants to know if it's worth the time. If you run a documentation site or build tools for coding agents, there's a legitimate, separate use case for llms.txt covered below. If your goal is getting your brand mentioned by ChatGPT or Perplexity when someone asks a buying question, this guide will tell you where to actually put your effort instead.

What llms.txt actually is, in one paragraph

llms.txt is a plain Markdown file you place at yoursite.com/llms.txt. It was proposed by Jeremy Howard of Answer.AI in September 2024 as a curated index of a site's most useful pages, meant to help AI agents work around limited context windows by pointing them to clean, linkable content instead of making them parse full HTML pages. It is not a government or web-standards-body specification. There's no IETF, W3C, or WHATWG process behind it, no numbered spec revisions, and no adoption commitment from any major AI lab to actually read it in production llmstxt.org. It behaves more like a hopeful convention than a protocol.

That distinction matters because most of the marketing content pushing llms.txt implies it works the way robots.txt or sitemap.xml work: files that crawlers reliably check as a matter of course. The evidence says otherwise.

The evidence: two independent studies agree AI crawlers mostly ignore it

The clearest way to settle "does this work" is to look at server logs and see whether AI crawlers actually request the file. Two separate studies did exactly that, using different sites and different methodologies, and landed on the same conclusion.

EZY.ai tracked 83 sites over 12 weeks and compared how often major AI crawlers fetched llms.txt versus robots.txt on the same sites in the same window:

Every crawler tied to a major consumer AI answer engine fetched robots.txt hundreds or thousands of times more often than llms.txt. PerplexityBot never fetched it once. Meta's crawler is the real exception here, fetching llms.txt about as often as robots.txt, but Meta's AI answer products aren't the ones most brands are chasing when they say "AI visibility." As the study's authors put it: "llms.txt is cheap, standards-track insurance… but anyone selling it as the thing that gets you cited is selling ahead of the evidence" EZY.ai.

A separate audit of 1,000 Adobe Experience Manager domains over 30 days of CDN logs found zero requests from GPTBot, ClaudeBot, or PerplexityBot to llms.txt at all. Ninety-five percent of the traffic that file did receive came from Google's ordinary desktop crawler, not an AI-specific bot Longato. Two different panels, two different methodologies, the same result: the crawlers powering the answer engines people actually mean when they say "AI visibility" are not reading this file.

Google has confirmed this directly. John Mueller, a Google Search Advocate, wrote on Bluesky in June 2025: "FWIW no AI system currently uses llms.txt" Longato. Google briefly added llms.txt to its own developer documentation in late 2025, then quietly removed it, which suggests even Google hasn't decided how, or whether, the file fits into its pipeline.

Both log studies are practitioner research rather than peer-reviewed academic work, and each has a narrower scope than the whole web (EZY.ai skews toward small-business sites; the Adobe audit covers one CMS platform). But their directional agreement, combined with a named Google engineer saying the same thing on the record, is about as strong a "not currently" verdict as this kind of question gets.

Why the confusion exists: file adoption isn't crawler consumption

If llms.txt doesn't do anything for AI visibility, why does it feel so widely recommended? Because two very different facts are getting collapsed into one.

Fact one: the file itself is spreading fast. Originality.ai reportedly counted an 8.8x year-over-year increase, to over 36,000 llms.txt files across the 3 million sites it scans Longato. Anthropic, Stripe, Cloudflare, and Vercel publish one. Documentation platforms like Mintlify generate llms.txt by default for every customer.

Fact two: none of that tells you whether AI crawlers are reading those files. Publishing a file and a crawler fetching that file are separate events, and the server logs above measure the second one directly. A CMS plugin that auto-generates llms.txt for thousands of sites doesn't create demand on the crawler side; it just adds more unread files to the web.

There's a related mix-up worth naming: OpenAI, Anthropic, and Gemini each publish their own llms.txt file for their developer docs. That's evidence they see some value in the format for their own documentation. It's not evidence that their crawlers consume other companies' llms.txt files when generating answers to a user's question. Publisher and consumer are different roles, and the research found no evidence linking the former to the latter.

The use case where llms.txt genuinely helps

None of this means the file is pointless everywhere. Howard's own framing was narrower than "AI visibility" from the start: he expected llms.txt to be useful for inference-time retrieval by agents, not for training and not for consumer answer-engine citation. A coding assistant like Cursor pulling in a library's documentation while a developer works is a real, working use case, and it explains why documentation platforms and dev-tool companies adopted the format first llmstxt.org.

If you run a public API, an SDK, or a developer-facing product, an llms.txt file that indexes your docs is a low-cost way to make that content easier for coding agents to fetch cleanly. That's a genuinely different goal than getting your brand mentioned when someone asks ChatGPT "what's the best project management tool for a small team," and it's worth keeping those two goals mentally separate so you don't optimize for one while thinking you're achieving the other.

What to do instead if you want to show up in AI answers

If llms.txt isn't the lever, what is? The evidence points to three things that actually correlate with being crawled, indexed, and cited by AI answer engines.

Make sure robots.txt isn't blocking the crawlers you want. This is the single highest-leverage check, and it takes five minutes. Open yoursite.com/robots.txt and confirm you aren't disallowing GPTBot, ClaudeBot, or PerplexityBot. The log studies above show these crawlers check robots.txt constantly, thousands of times per site over a 12-week window, so this file is the one that actually governs whether they can reach your content at all.

Write content that directly answers the question, in plain, parseable HTML. AI answer engines favor pages that state a clear answer early, in ordinary text, rather than pages that bury the point under marketing copy or require heavy JavaScript rendering to reveal the content.

Earn third-party mentions. Reviews, comparison sites, forum threads, and press coverage are what actually build the kind of topical authority that gets a brand cited by name in an AI-generated answer. No single file substitutes for other sources talking about you.

A framework for judging any AI-visibility claim: three separate signals

Before you add any tactic to your AI-visibility checklist, whether it's llms.txt or the next thing someone tells you matters, it helps to separate three signals that get talked about as if they were one:

1. A crawler visit. A bot requested a page on your site. This tells you the page was discovered, nothing more.
2. A citation. Your brand or content was actually referenced inside a generated AI answer.
3. A referral or conversion. A person clicked through from an AI answer and reached your site.

A crawler visit is a discovery signal, not proof of indexing or inclusion in an AI answer. These three signals are easy to conflate, and most "does X help AI visibility" claims, including the llms.txt hype, only ever touch the first one, if that. Any tactic that only claims to increase crawler visits, without evidence connecting that visit to an actual citation, hasn't shown you anything about visibility yet. Keeping these three separate is the fastest way to spot an overstated claim before you spend time implementing it.

If you want to check where your own site actually stands on these three signals rather than guessing, Tideflow's crawler detection shows which AI bots are visiting your pages, and analytics tracking connects those visits to referral traffic and conversions, so you can see the full chain instead of stopping at "a bot showed up."

Should you still add an llms.txt file?

Yes, if it takes you fifteen minutes and you have documentation or developer-facing content. There's no evidence it causes harm, no duplicate-content or indexing penalty risk has been documented, and it costs nothing to have in place if a major AI lab decides to start consuming it later. Meta's crawler already reads it about as often as robots.txt, which is a real, if narrow, data point suggesting adoption could shift. Just don't let adding the file replace the work that the evidence actually supports: fixing robots.txt, writing answer-first content, and earning mentions elsewhere on the web.

What to check this week

Pull your server logs (or your CDN/hosting provider's access logs) and search for GPTBot, ClaudeBot, and PerplexityBot requests over the last 30 days. Compare how often each one hits /robots.txt versus /llms.txt if you have one. If you see the same pattern the studies found, heavy robots.txt traffic and near-zero llms.txt traffic, that confirms your site behaves like the rest of the web, and you should redirect the time you'd spend polishing llms.txt toward the three levers above instead.

Frequently Asked Questions

Will llms.txt adoption by AI crawlers change in the future?

It's possible. The evidence here is time-bound to studies run between mid-2025 and mid-2026, and crawler behavior could shift if a major AI lab decides to start consuming the file in production. Meta's crawler already reads llms.txt at rates close to robots.txt, which is the one signal worth watching. Treat the current verdict as "not currently a lever," not a permanent one, and recheck your own logs periodically rather than assuming today's pattern is fixed.

Does llms.txt replace robots.txt or sitemap.xml?

No. The three files serve different purposes. Robots.txt governs which crawlers are allowed to access which parts of your site. Sitemap.xml enumerates all your indexable URLs for search engines. llms.txt, per its own creator's design, is meant as a curated overview an agent can consult on demand, not a crawl-permission or indexing mechanism llmstxt.org. Keeping robots.txt and your sitemap correct matters far more than adding llms.txt.

What's the difference between traditional SEO and optimizing for AI answer engines, and where does llms.txt fit?

Traditional SEO optimizes for ranking in a list of links a search engine returns. Optimizing for AI answer engines means earning a direct mention or citation inside a generated answer, which depends more on clear, direct-answer content and third-party validation than on any single technical file. llms.txt fits into neither category with current evidence behind it; it's a documentation-retrieval aid for AI agents, not a ranking or citation mechanism for either search engines or answer engines.

The next step

Check your robots.txt file today and confirm GPTBot, ClaudeBot, and PerplexityBot aren't blocked. That single fix has more documented connection to AI crawler access than anything else covered here. Once crawlability is confirmed, shift your attention to whether your content actually answers the questions your buyers are asking AI assistants, and whether independent sites mention your brand at all. If you want to see which crawlers are visiting your site right now and whether those visits turn into real traffic, Tideflow's crawler and analytics tools show that chain directly instead of leaving you to guess from a single metadata file.

Sources

• Jeremy Howard / Answer.AI, The /llms.txt file, v2 (September 2024)
• EZY.AI Team, We Put llms.txt on 83 Websites. OpenAI Read It 7 Times. (July 2026)
• Flavio Longato, LLMs.txt: Why AI Crawlers Ignore It (2025 Audit) (August 2025)