Google has issued fresh guidance on handling temporary server outages, robots.txt restrictions and URL formats. Other developments include a Cloudflare and OpenAI pilot to help ChatGPT Search discover more web pages, Google’s current limitations around JPEG XL images, and predictions about how AI agents will access websites.

Google Recommends 503 Responses During PHP Maintenance

On 21 October, Google Search Central reminded website owners to return a 503 HTTP status code when temporarily taking servers offline for PHP security patches. The advice followed the release of emergency fixes for two critical PHP vulnerabilities.

A 503 response tells Google that the outage is temporary, helping preserve already indexed URLs while maintenance takes place. Google advises against returning generic 404 pages or pages with a 200 status code when the server is unavailable.

Google’s John Mueller also clarified that the 503 response does not need to be generated by PHP itself. It can instead be handled by another service that remains operational during the outage, such as a content delivery network (CDN) or temporary server.

However, Google recommends keeping 503 responses to a minimum, ideally no longer than one or two days. If a URL continues returning this status for several days, Google may remove it from its index.

Website owners should also avoid returning a 503 response to requests for robots.txt, as this can prevent Google from crawling the site.

Robots.txt Controls Crawling, Not How Content Is Used Afterwards

Google’s Gary Illyes explained in a Search Central video published on 21 October that robots.txt rules govern crawling rather than what happens to content after it has been accessed.

Under the robots.txt standard, the directives tell compliant crawlers which parts of a website they may visit. They do not independently determine how a company can use information it has already collected.

Illyes described robots.txt as a way for website owners to express their access preferences. To block one AI crawler while allowing others, site owners can target the relevant crawler’s user-agent token.

Google’s Google-Extended token provides a separate control for certain uses of content collected by Google. It allows website owners to manage whether that content can be used to train future Gemini models and provide grounding for Gemini Apps and Vertex AI. It does not affect whether a page appears in Google Search, and Google-Extended is not a separate crawler.

A proposed extension to the robots.txt standard, co-written by Illyes, would introduce a Content-Usage rule for expressing preferences about how content is used. However, the proposal remains a working-group draft and has not been approved.

For now, website owners can use Disallow rules to restrict compliant crawlers from specified paths and Google-Extended to manage the relevant uses of their content within Google’s AI services.

Cloudflare and OpenAI Pilot Helps ChatGPT Discover More Pages

Cloudflare and OpenAI have reported that a research pilot helped OpenAI discover and index hundreds of millions of additional web pages, potentially expanding the information available to ChatGPT.

The pilot involved around 1% of Cloudflare’s network. Signals from participating websites helped OpenAI’s crawler identify pages, avoid requests for pages that no longer exist and recognise when content has been updated.

In a blog post published on 15 October, the companies acknowledged that discovering more pages can initially lead to increased crawling. However, they did not disclose how often the newly discovered pages are cited by ChatGPT or how much website traffic they generate.

The companies also stressed that a discovery signal does not grant permission to access a page.

OpenAI maintains separate controls for search and training crawlers. Website owners can, for example, allow OAI-SearchBot to access content for search while blocking GPTBot from using it for training.

Discoverability does not guarantee visibility in ChatGPT responses. OpenAI determines which pages to rank, so a page identified through the pilot is not automatically guaranteed a citation.

Google Treats Unencoded and Encoded Bracket URLs Differently

Google’s John Mueller confirmed on 19 October that URLs containing literal square brackets are different from those using their percent-encoded equivalents, %5B and %5D.

Mueller acknowledged that he had previously assumed the two formats were equivalent and may have given incorrect advice on the subject.

If both URL versions return the same content, Google may crawl them separately, identify the duplicate pages and select one version to retain. Mueller’s recommendation was to choose a single format and apply it consistently across the website.

This includes internal links, canonical tags, sitemap entries and structured data. Google’s existing URL documentation also states that reserved characters must be percent-encoded.

Mixing raw and encoded brackets across a website can leave Google to determine which version should be treated as the preferred URL. Although Mueller expects existing sites using literal brackets to continue working, he would not recommend adopting that approach for a new website.

JPEG XL Is Not Supported in Google Images or Discover Previews

Google’s John Mueller confirmed on 15 October that JPEG XL images are not currently supported in Google Images or as preview images in Google Discover.

The clarification came after JPEG XL support was introduced in Chrome 155, with a wider rollout beginning on 6 October, and Firefox 158 on 13 October.

As of 22 October, Google’s image SEO documentation listed BMP, GIF, JPEG, PNG, WebP, SVG and AVIF as supported formats for images provided through an HTML image element’s src attribute. JPEG XL was not included.

For websites that want to use JPEG XL, a practical option is to provide it as an alternative source within a picture element while retaining a supported format, such as JPEG, WebP or AVIF, in the nested img element. This gives search engines a compatible image source.

Website owners should also check the images used for Discover previews. Google’s guidance points to og:image and structured data for preview images, so these should reference supported formats as well.

Pichai Expects AI Agents to Browse Websites and Use Dedicated Protocols

Google CEO Sundar Pichai expects AI agents to access websites in two ways: by browsing pages in a manner similar to human visitors and through protocols designed specifically for agent interactions.

Speaking on the Sources podcast with Alex Heath, Pichai said both approaches would continue to develop. He highlighted the growing role of agent and payment protocols, including Google’s-backed Universal Commerce Protocol, describing protocols as a more scalable long-term approach while acknowledging that they may take longer to mature.

Pichai also suggested that AI agents could eventually provide a more unified experience across Google Search and Gemini.

His comments indicate that conventional web browsing will remain relevant as agent-specific protocols develop. Google’s documentation also identifies Google-Agent as a user-triggered fetcher. These fetchers generally do not follow robots.txt rules in the same way as standard crawlers.

The Bigger Picture: Understanding What Each Website Signal Controls

The week’s developments highlight the importance of understanding what different technical signals actually do.

A 503 response tells Google that a server outage is temporary, but keeping it in place for too long risks pages being removed from the index. Robots.txt can restrict compliant crawlers from accessing particular content, but it does not independently control how that content is used after crawling. Separate tools, such as Google-Extended and OpenAI’s crawler directives, govern certain uses of collected information.

Likewise, a discovery signal from Cloudflare does not grant OpenAI permission to access a page, while inconsistent URL formatting can cause Google to process two versions of what appears to be the same address. JPEG XL also remains unsuitable as the sole image format for Google Images and Discover previews under the guidance available at the time.

As AI agents increasingly interact with websites, understanding which crawlers are making requests and which rules apply to them will become more important. Server logs can help identify crawler activity, while each platform’s documentation explains the controls that govern access and subsequent use of content.

 

 

More Digital Marketing BLOGS here: 

Local SEO 2024 – How To Get More Local Business Calls

3 Strategies To Grow Your Business

Is Google Effective for Lead Generation?

What is SEO and How It Works?

How To Get More Customers On Facebook Without Spending Money

How Do I Get Clients Fast On Facebook?

How Do I Retarget Customers?

How Do You Use Retargeting In Marketing?

How To Get Clients From Facebook Groups

What Is The Best Way To Generate Leads On Facebook?

How Do I Get Leads From A Facebook Group?

>