Google Updates Googlebot-News Crawler Guidelines

Google has recently revised its documentation concerning the Googlebot-News crawler. The update was made to address and correct a misleading statement that had previously suggested the crawler had an impact on how content appeared in the News tab of Google Search.

The original documentation wrongly implied that publishers’ settings for Googlebot-News could affect whether their content showed up in the News tab. However, this was not accurate. The clarification confirms that crawler preferences directed at Googlebot-News do not influence placement or visibility in the News tab itself.

This correction ensures that publishers are not misled about how their content is indexed and surfaced by Google News. It highlights the importance of clear and accurate documentation, especially when dealing with search visibility and content indexing.

By updating the documentation, Google aims to provide more precise guidance to web publishers, helping them understand the correct role of the Googlebot-News crawler and avoid confusion when setting up their technical preferences.

Google News Tab

The Google News tab is a specific section found near the top of the search results pages (SERPs), designed to highlight news-related content. This feature allows users to filter their search results so they only see articles and updates from recognised news sources.

Generally, the content that appears under the News tab comes from websites that are considered eligible for inclusion in Google News. These publishers must adhere to Google’s strict news content policies, which are in place to ensure credibility, trustworthiness, and relevance.

Eligibility is based on several factors, including editorial standards, transparency, and the overall quality of the content. Meeting these requirements ensures that only reliable and well-established sources are featured within this specialised search filter.

By using the News tab, users can easily access the latest stories and updates from across the web, helping them stay informed on current events without having to sift through non-news-related results.

What Changed:

Google recently issued a correction in its changelog, clarifying a mistake in the user agent description related to Googlebot-News. The error involved a misleading claim about how publishers’ crawling preferences might impact visibility within the Google News tab on Search.

According to the update, the previous description wrongly suggested that settings directed at Googlebot-News would influence how content appears in the News tab on Google Search. Google has now made it clear that this is not the case.

In the previous version of the documentation, the sentence read: “Crawling preferences addressed to the Googlebot-News user agent affect all surfaces of Google News (for example, the News tab in Google Search and the Google News app).” This wording led to confusion among publishers.

The updated and corrected version now states: “Crawling preferences addressed to the Googlebot-News user agent affect the Google News product, including news.google.com and the Google News app.” This revision confirms that such preferences only apply to the standalone Google News platform and not the News tab in standard search results.

This clarification is important for publishers managing their visibility on Google’s platforms. It helps them better understand which tools and directives control where and how their news content is displayed.

By removing the reference to the News tab in Google Search, Google aims to reduce misunderstandings and ensure publishers have accurate information when setting crawling instructions.

This change aligns the documentation with the actual behaviour of the Googlebot-News crawler and reflects Google’s commitment to providing clearer guidance for web publishers.