Google’s Mueller Slams ‘Markdown-for-Bots’ Idea

Google Search Advocate John Mueller has criticised the idea of serving Markdown files specifically for large language model (LLM) crawlers, questioning whether AI bots can even interpret such files correctly. Speaking on Bluesky, he called the concept “a stupid idea,” highlighting concerns that stripping a page of its standard HTML structure could reduce its meaning and make internal links and navigation unusable for crawlers.

The debate arises from experiments by some developers aiming to optimise their websites for AI crawling. One approach involved using Next.js middleware to detect AI user agents, such as GPTBot or ClaudeBot. When these bots accessed a page, the middleware would intercept the request and serve a raw Markdown file instead of the usual HTML or React content. Early benchmarks reportedly showed a 95% reduction in token usage per page, which developers argued could improve efficiency for retrieval-augmented generation (RAG) bots and reduce computational costs.

However, Mueller raised several technical objections. He questioned whether LLMs could recognise Markdown as more than just plain text and whether they could parse links, headers, footers, sidebars, or navigation elements correctly. Flattening a page into Markdown might strip out essential context, he warned, potentially harming the page’s usability for AI and reducing its overall value. On Bluesky, he emphasised his point with a sarcastic remark, saying that since LLMs can read images, “why not turn your whole site into an image?”

Technical SEO experts have echoed these concerns. Jono Alderson noted that while converting pages into Markdown might reduce token usage, it removes the semantic structure and context that give meaning to content. Others in the community questioned whether such efforts could inadvertently limit crawling rather than enhance it. There is also no evidence that LLMs are trained to prioritise simplified formats like Markdown over standard HTML pages.

This discussion reflects a broader pattern of caution within Google regarding bot-specific content formats. Mueller has repeatedly advised against creating alternate versions of pages solely for AI crawlers, emphasising that clean HTML and structured data are far more reliable for ensuring proper indexing and interpretation. Previous analyses support this approach. For instance, SE Ranking’s study of 300,000 domains found no correlation between having an llms.txt file or other bot-only formats and improved citations in AI-generated outputs. Mueller has likened such formats to outdated meta keyword tags—widely ignored by search engines and AI systems alike.

The problem stems from how LLMs interact with web content. Each URL that bots crawl must be meaningful and properly structured to ensure it can be interpreted accurately. Serving Markdown or other simplified formats might reduce token consumption, but it risks removing critical context, breaking internal links, and omitting page components that contribute to understanding content relationships. This could ultimately hinder AI systems from accurately processing and referencing the site’s information.

Mueller also stressed the importance of maintaining standard HTML pages with accessible navigation, proper headings, and structured data. These elements provide context that bots rely on when indexing and summarising content. Until AI platforms explicitly request Markdown or other bot-specific formats, creating separate pages for LLMs is unlikely to offer tangible benefits.

Looking forward, website owners should focus on optimisation practices that are supported by documented platform guidelines. This includes using structured data schemas, minimising JavaScript that blocks content rendering, and keeping HTML clean and well-organised. By adhering to these standards, sites ensure their content remains accessible not only to humans but also to AI crawlers.

The ongoing discussion highlights a key principle in SEO and AI content management: simplicity does not always mean better. While reducing token usage may appear advantageous, maintaining content integrity, context, and usability should remain the priority. The experiment with Markdown files serves as a cautionary tale, illustrating the potential pitfalls of creating bot-specific versions of pages without clear guidance from AI platforms.

In summary, John Mueller’s response reinforces a consistent message: webmasters should prioritise accessible, structured, and standard HTML content over experimental formats aimed solely at AI bots. Until LLM providers define standards for alternative formats, attempts to simplify pages into Markdown or other specialised files are more likely to create issues than to offer improvements. Clean HTML, proper semantic structure, and adherence to platform documentation remain the best strategies for ensuring websites are effectively crawled and interpreted by AI systems.