NotebookLM Ignores Robots.txt

Google has discreetly revised its documentation for NotebookLM, confirming that the platform does not follow robots.txt directives. The update, added to Google’s list of user-triggered fetchers, suggests that NotebookLM operates independently of the usual web crawling restrictions — though there are still ways to block it if needed.

Google NotebookLM

NotebookLM is an AI-powered research and writing assistant that allows users to input a webpage link, analyse its content, and then ask questions or generate summaries from the information provided. The tool can also automatically produce an interactive mind map, helping to visually organise key themes and insights from the webpage.

User-Triggered Fetchers Ignore Robots.txt

Google’s User-Triggered Fetchers are web agents activated by user actions, and by default, they do not follow the robots.txt protocol.

As stated in Google’s documentation on User-Triggered Fetchers:

“Since these fetches are initiated by users, the fetchers typically disregard robots.txt rules.”

Google-NotebookLM Ignores Robots.txt

The main role of robots.txt is to allow website owners to manage how bots access and index their web pages. However, agents such as the Google-NotebookLM fetcher operate differently — instead of indexing content, they act on behalf of users engaging with website material through Google’s NotebookLM tool.

How To Block NotebookLM

Google uses the Google-NotebookLM user agent when extracting website content. So, it’s possible for publishers wishing to block users from accessing their content could create rules that automatically block that user agent. For example, a simple solution for WordPress publishers is to use Wordfence to create a custom rule to block all website visitors that are using the Google-NotebookLM user agent.

Another way to do it is with .htaccess using the following rule:

<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_USER