Google Updates Crawler Rules

Google has updated its documentation for the Mediapartners-Google crawler, clarifying that its role is no longer described as being limited to Google AdSense.

The change is relevant to website owners, publishers and SEO professionals who use the Mediapartners-Google user agent in their robots.txt files or other crawling settings.

What Has Google Changed?

Mediapartners-Google has traditionally been associated with AdSense, with Google’s previous guidance specifically referring to the crawler’s role in helping AdSense display relevant adverts.

The updated documentation takes a broader approach. It now explains that the crawler visits participating websites to help provide relevant advertising and that crawling preferences for the Mediapartners-Google user agent can apply to several Google advertising products.

Google gives Google AdSense and Google Ad Manager as examples of services covered by the crawler.

This represents a change from the previous wording, which referred specifically to AdSense.

Why Does This Matter?

The updated wording suggests that the Mediapartners-Google crawler has a wider role across Google’s advertising services than previously stated in its documentation.

For publishers, this is important because crawling instructions can determine whether Google’s systems are able to access and analyse website content.

If a website restricts the Mediapartners-Google user agent through robots.txt, those restrictions may now affect more than just AdSense-related activity.

Google Still Leaves Some Questions Open

Although the documentation now mentions a wider range of advertising products, Google has not provided a complete list of every service affected by Mediapartners-Google crawling preferences.

AdSense and Google Ad Manager are specifically named, but the updated guidance does not clearly explain whether other advertising platforms and products, such as Google Marketing Platform, Local Services Ads or Google AdMob, are also covered.

This makes it difficult for publishers to understand the full impact of their crawling settings.

Check Your Robots.txt Settings

Website owners who have previously configured rules for the Mediapartners-Google user agent may want to review those settings.

A restriction that was originally introduced to control AdSense crawling could potentially have a wider effect if the same crawler is now used across additional advertising products.

The update does not appear to change how the crawler itself operates. Instead, Google has changed how it describes the crawler and the products affected by its crawling preferences.

For now, publishers should be aware of the broader wording and monitor Google’s documentation for any further clarification about which advertising services use Mediapartners-Google.