Google’s llms.txt Advice Varies by Product

Google’s latest guidance around llms.txt has created some confusion in the SEO and development community, as different Google products appear to treat the file in different ways. While Google Search clearly states that it is not required for AI-driven search features, Google’s Lighthouse tool has begun testing it within an experimental framework focused on agent-based browsing.

On the Google Search side, the most recent optimisation documentation is fairly direct. It confirms that llms.txt is not needed for visibility in generative AI features such as AI Overviews or AI Mode. In fact, it is grouped alongside other approaches like content chunking, AI-specific rewriting, and specialised schema that Google says are not necessary for its systems. The overall message from Search is that these tactics do not influence how content is surfaced in AI-generated search results.

This position is not new. Google’s Search representatives have repeatedly stated over time that llms.txt is not an official Google initiative and is not used within its ranking or AI systems. Well-known figures from the Search team have previously compared it to outdated SEO concepts, suggesting it does not play a role in how Google processes or understands web content. The consistent message from Search has been that site owners do not need to implement or maintain such a file for SEO or AI visibility purposes.

However, a different interpretation appears within Google’s Lighthouse tool. In version 13.3, Lighthouse introduced a new “Agentic Browsing” audit category, which includes a check for llms.txt. Unlike Search, this is not positioned as a ranking or visibility factor, but rather as part of an experimental evaluation of how well websites support AI agents that interact with pages in a more autonomous way.

Within this audit, Lighthouse checks whether a site provides an llms.txt file and flags technical issues such as server errors when attempting to access it. The documentation describes llms.txt as a machine-readable summary of a website’s content designed specifically for LLMs and AI agents. It suggests that without such a file, agents may need more time to crawl and interpret a site’s structure, content hierarchy, and key information.

Importantly, this does not mean llms.txt is being used for search rankings. Instead, it is framed as part of a broader concept of “agent readiness”, where future AI systems might interact with websites more directly, potentially retrieving and processing information in real time rather than relying solely on traditional crawling methods.

This is where the distinction between Google’s products becomes important. Google Search is focused on indexing, ranking, and generating AI-enhanced search results. Lighthouse, on the other hand, is testing website readiness for a different type of interaction entirely, where AI agents may browse, interpret, and act on web content more dynamically.

The difference in intent explains why the guidance appears inconsistent. From a Search perspective, llms.txt is unnecessary and does not contribute to AI visibility or performance. From a Lighthouse perspective, it is treated as an emerging convention that could support future agent-based systems, even if it is not currently required.

There is also some historical context that adds to the uncertainty. In the past, llms.txt files have appeared on certain Google-owned properties, including developer documentation sites, before being removed or left in place without explanation. This led to speculation about whether the file had any internal significance, although it now appears more likely to have been the result of automated systems rather than a coordinated Search initiative.

Taken together, the situation highlights how different teams within Google can produce guidance that serves different purposes. Search guidance is aimed at helping websites perform well in current ranking and AI systems, while Lighthouse experiments with forward-looking concepts around how AI agents might interact with the web in future.

For website owners, this creates a simple but slightly fragmented takeaway. If the goal is visibility in Google Search and AI-generated search features, llms.txt is not required. If the interest is in preparing for experimental agent-based browsing systems, Lighthouse suggests it may have optional value, but it remains part of an evolving and non-standard approach.

Ultimately, Google has not issued any unified statement that reconciles these two perspectives. As a result, llms.txt currently sits in a grey area—unnecessary for SEO today, but still being explored in experimental tooling that looks ahead to how AI agents might navigate the web in the future.