Google: Noindex Header May Work with llms.txt

John Mueller from Google recently responded to a query about llms.txt and whether it might be considered duplicate content. He explained that it’s unlikely for llms.txt to be treated as duplicate content, but he also mentioned that there could be valid reasons to add a noindex header to avoid it being indexed.

LLMs.txt

The llms.txt proposal introduces a new content format designed to help large language models access the core content of a webpage, without being distracted by non-essential elements like ads, menus, or other surrounding data. It allows website owners to offer a simplified, Markdown-formatted version of their key content.

This file is placed at the root of a website (for example: example.com/llms.txt) and serves as a structured way to highlight the most important information.

Despite some misconceptions, llms.txt does not serve the same function as robots.txt. While robots.txt is used to control how web crawlers interact with a site, llms.txt is specifically intended to deliver selected content directly to large language models.

Will Google View Llms.txt As Duplicate Content?

A user on Bluesky raised an interesting question about whether Google might treat llms.txt files as duplicate content. They pointed out that if others link directly to a site’s llms.txt file, it’s possible Google could index that version alongside or instead of the standard HTML content.

The user asked:

“Will Google view llms.txt files as duplicate content? It feels unreasonable to do so, especially since Google understands the file’s true purpose. Should I add a ‘noindex’ header for llms.txt to prevent Googlebot from indexing it?”

In response, Google’s John Mueller explained that such a file would only be considered duplicate content if it exactly matched an existing HTML page—which, if the file is genuinely useful, wouldn’t typically be the case.

However, he added that using a noindex header could still be a sensible option. Since other websites might link to the llms.txt file, there’s a chance it could end up indexed, which might create confusion for users.

Noindex For Llms.txt

Adding a noindex header to the llms.txt file is a sensible approach, as it stops the content from being included in Google’s search index. There’s no need to block Google using robots.txt, as that would simply stop the file from being crawled—meaning Google wouldn’t be able to detect the noindex directive in the first place.