LLMs.txt Fails — 97% Get Zero AI Requests in 2026
Ahrefs data shows 97% of llms.txt files received zero requests in May 2026, AI retrieval bots accounted for 1% of traffic, and Google now says it ignores them entirely for search ranking.
Key Takeaways
- On June 16, 2026, Ahrefs published analysis of server logs from 137,000 domains showing that the llms.txt standard — promoted…
- The largest segment consuming llms.txt files is not AI search but SEO audit tools. According to the Ahrefs data, SEO audit…
- On a recent Search Off the Record podcast, Google’s Martin Splitt pushed back against the idea that stripped-down markdown is…
🎥 Short video summary — Watch the 2-minute breakdown of this data:
The short version
LLMs.txt — the markdown-based file standard widely adopted throughout 2025 and early 2026 as a way to help AI search engines discover and cite website content — has failed to attract meaningful AI retrieval traffic. According to Ahrefs data published on June 16, 2026, 97% of llms.txt files across 137,000 domains received zero requests in May 2026. AI retrieval bots from ChatGPT and Perplexity accounted for just 1.1% of all llms.txt requests, while Google confirmed on June 15 that it ignores llms.txt files entirely for search ranking and AI Overviews.
Key facts
- 97% of llms.txt files received zero requests in May 2026, per Ahrefs analysis of 137,000 domains
- AI retrieval bots like ChatGPT and Perplexity accounted for only 1.1% of llms.txt traffic
- Google officially stated on June 15, 2026 that it ignores llms.txt files for search ranking
- SEO audit tools and coding agents — not AI search — are the real consumers of llms.txt files
What happened
On June 16, 2026, Ahrefs published analysis of server logs from 137,000 domains showing that the llms.txt standard — promoted by the SEO industry as a GEO tactic for AI search visibility — has essentially no audience among the AI retrieval bots that matter for citation and discovery. Of roughly 38,000 domains with valid llms.txt files, only about 1,100 received any requests at all.
The data lands one day after Google updated its official “optimizing for AI Search” guide to state, in unambiguous terms: “It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.”
This dual development — empirical data showing the files go unread, and Google’s formal dismissal — effectively ends the llms.txt experiment as an AI search visibility tactic.
Who actually requests llms.txt files, and why does it matter?
The largest segment consuming llms.txt files is not AI search but SEO audit tools. According to the Ahrefs data, SEO audit tools accounted for 21% of requests, followed by unidentified bots at 14%, general web crawlers like Googlebot at 13%, and technology profiling tools like BuiltWith at 11%. AI bots across four categories made up 19% of requests — but the composition tells a revealing story. Coding agents sent 10% of requests, training crawlers 5%, and AI assistants 2%. AI retrieval bots — the ones that would cite content in ChatGPT, Perplexity, or AI Overviews — accounted for only 1.1%.
A particularly striking data point: Slackbot, the automation tool for workplace messaging, fetched llms.txt files more often than PerplexityBot did.
Even more telling, 12% of requests came from tools built specifically to audit, scan, or study llms.txt files. As the Ahrefs report noted, “An ecosystem has developed around scoring and cataloging a file format before a significant audience appears.”
What Google’s engineers say about markdown for AI SEO
On a recent Search Off the Record podcast, Google’s Martin Splitt pushed back against the idea that stripped-down markdown is the right format for AI search optimization. Splitt acknowledged the appeal — reducing HTML “cruft” to save tokens — but argued it comes at a real cost to discoverability.
Splitt explained: “Markdown is focused on just one part of the content: the content itself.” He said that removing HTML structure “makes it harder for search engines to see a web page in the context of how it connects to the rest of a website’s content through links, which aid discovery.”
Google’s John Mueller has been consistent on this point for over a year. Pressed by SEO consultant Lily Ray on the gap between Google Search’s dismissal of llms.txt and Chrome Lighthouse adding an llms.txt audit in May 2026, Mueller described llms.txt as a “temporary crutch, perhaps to save some tokens” for AI coding tools — not something built for search.
SE Ranking’s earlier analysis of 300,000 domains reinforces the point: there was no connection between having an llms.txt file and how often a site was cited in AI-generated answers.
What this means (our take)
The llms.txt story is a case study in how the GEO industry can build infrastructure ahead of demand — and ahead of the platforms themselves. A file format that was pitched as essential for AI search visibility turned out to be used primarily by the SEO tools built to validate it, while the AI retrieval bots that actually drive citations never adopted it in any meaningful volume.
There are two practical consequences for practitioners. First, any time invested in building and maintaining llms.txt files has generated zero measurable return for AI search visibility. That resource is better spent on well-structured HTML, internal linking architecture, and schema.org markup — all of which Google’s engineers have explicitly said remain important signals for discovery and ranking.
Second, Google’s alternative — the Open Knowledge Format (OKF v0.1), published June 13, 2026 — signals where the platform is actually heading. OKF is a vendor-neutral markdown specification designed for AI agents, not search crawlers. The distinction matters. Where llms.txt aimed to help AI find content, OKF aims to give AI agents shared, portable knowledge. The difference between “findability” and “usability” is significant, and OKF is worth monitoring — but not yet worth building infrastructure around.
What to do now
- Stop spending time on llms.txt file creation and maintenance. The data shows no AI retrieval impact, and Google has confirmed it ignores them.
- Audit your HTML structure for semantic correctness. Use proper heading hierarchy, descriptive anchor text, and internal links that communicate context between pages — the signals Splitt and Mueller say actually matter.
- Invest in schema.org structured data. Unlike llms.txt, schema markup is consumed by Google Search, AI Overviews, and a broad range of AI retrieval systems — and there is documented evidence linking structured data to AI citation frequency.
- Monitor Google’s OKF specification for developments, but do not build production infrastructure around it until adoption data is available.
- Remove llms.txt-related scoring from your GEO dashboards. If a tool reports on your llms.txt compliance, that metric is measuring something the retrieval bots themselves do not use.
FAQ
Was llms.txt ever useful?
A: For coding agents and training crawlers, yes — 10% of requests came from coding agents like Claude-Code and GPTBot. For AI search visibility and citation in tools like ChatGPT or Perplexity, the data shows it was never adopted at meaningful scale.
Does Google use llms.txt for anything?
A: No. Google’s official documentation, updated June 15, 2026, states that Google Search ignores llms.txt files and they neither help nor harm search visibility.
Should I delete my existing llms.txt file?
A: There is no penalty for keeping it, and a small fraction of coding agents and training crawlers do request it. But maintaining it on an ongoing basis is unlikely to be a productive use of time given the data.
What is Google’s OKF and should I adopt it?
A: Google’s Open Knowledge Format (OKF v0.1), published June 13, 2026, is a vendor-neutral markdown specification for giving AI agents shared, portable knowledge. It is designed for agent usability, not search crawler discovery. It is too early to recommend adoption — wait for usage data and clear adoption signals before investing time.
What actually works for AI search visibility if llms.txt doesn’t?
A: Well-structured HTML with semantic markup, internal linking that communicates content relationships, schema.org structured data, and content written to answer complete user journeys (including follow-up questions) are the tactics that Google’s own engineers and third-party data support as effective for AI search visibility.
Sources
- https://www.searchenginejournal.com/97-of-llms-txt-files-got-no-requests-ahrefs-data-shows/579478/
- https://searchengineland.com/google-says-llms-txt-files-wont-harm-or-help-your-search-rankings-480264
- https://www.searchenginejournal.com/google-says-markdown-for-ai-seo-strips-away-the-parts-that-matter/579496/
- https://ppc.land/googles-okf-wants-to-be-the-lingua-franca-for-ai-agent-knowledge/
- https://www.searchenginejournal.com/googles-mueller-says-llms-txt-cant-help-llms-differentiate-sites/579304/
Frequently Asked Questions
How does AI change search marketing in 2026?
AI Overviews absorb 39.8% of informational clicks and zero-click searches hit 68%. But commercial-intent queries still convert. The playbook: target transactional keywords, build entity-rich content, and track AI citation alongside traditional rankings.
What tools help with AI search visibility?
Tools like Semrush AI Visibility, ChatGPT brand monitors, and Perplexity citation trackers emerged in 2026. Also critical: llms.txt files for AI agent discovery, structured data for entity recognition, and question-answer formatted content for LLM extraction.
