Google: Entity Order Shapes AI

New Google research suggests that large language models can struggle to recall facts when a question presents the subject and object entities in the reverse order from how they were originally learned.

The findings could have implications for how information is structured and presented online, particularly as search engines increasingly rely on AI systems to understand and answer queries.

AI Models Know More Than They Can Recall

The research looked at what happens when advanced AI models are asked to retrieve factual information that was already included in their training data.

The researchers found that the problem is often not a lack of knowledge. Instead, the difficulty appears to be accessing information that the model has already encoded.

According to the study, leading models had encoded around 95% to 98% of the facts tested. However, they still failed to recall between 26% and 34% of those facts when answering directly.

This suggests that factual recall, rather than simply learning more information, can be an important limitation for advanced AI models.

The researchers found that even stronger models could still struggle to retrieve information that was already present in their training data.

Why Subject and Object Order Matters

One of the more interesting findings concerns the relationship between two entities in a fact.

For example, consider the statement:

“Oasis played their first gig at the Boardwalk club.”

In this sentence, Oasis is the subject and the Boardwalk club is the object.

If an AI system repeatedly encounters information presented in this order, it may become easier for the model to associate Oasis with the Boardwalk club.

However, asking the same information in reverse can make retrieval more difficult.

For example, instead of asking which venue Oasis played at, a question might ask which band played its first gig at the Boardwalk club.

The underlying fact has not changed. The entities have simply switched positions.

Google’s research suggests that this change in ordering can make it harder for an AI model to retrieve the correct answer, despite the information already being stored within the model.

The Information May Still Be Recognised

The researchers found another interesting difference when testing models with multiple-choice questions.

Although an AI model might struggle to produce the correct answer when asked directly, it can sometimes identify the correct fact when the answer is presented alongside several alternatives.

This indicates that the information may still be available within the model, even when it cannot be easily retrieved without additional prompts or context.

In other words, there appears to be a difference between knowing something and being able to retrieve it at the right moment.

That distinction could become increasingly important as AI systems are used to answer more detailed and specific search queries.

Rewording Questions Made Less Difference

The researchers also looked at whether simply changing the wording of a question affected factual recall.

Interestingly, changing the phrasing did not have a major impact on whether the model could retrieve the information.

The more significant issue was the relationship between the subject and object entities.

This suggests that the problem is not necessarily about using the “wrong” words in a question. Instead, the way entities are connected and ordered may have a greater influence on whether a model can retrieve a fact.

Rare Facts Can Be Harder to Retrieve

The study also examined what happens with less common or “long-tail” facts.

AI models generally performed better with information that appeared frequently in their training data. Less common facts were more difficult to recall.

However, the research suggests that rare information was often still encoded within the model.

The issue was once again the retrieval process rather than the complete absence of the information.

This is particularly interesting for businesses and websites that publish information about niche subjects. A fact may be available to an AI system but still prove difficult for it to retrieve when a user asks about it in a particular way.

More Thinking Can Improve Recall

The researchers tested whether allowing AI models more time to reason could help them retrieve facts they failed to recall initially.

The results were encouraging. Additional thinking allowed models to recover a significant proportion of facts they had previously failed to produce directly.

However, this approach comes with a cost.

More reasoning requires additional computing resources, making it more expensive and potentially slower. There is also the challenge of determining when an AI system should spend extra time trying to retrieve information.

Simply making every query use more reasoning would not necessarily be an efficient solution.

Bigger AI Models May Not Solve the Problem

Another important point from the research is that simply increasing the scale of AI models may not remove these recall difficulties.

If a model already contains the information but struggles to retrieve it, adding more training data or increasing the model’s size may not address the underlying problem.

This shifts attention towards how AI systems organise, access and retrieve the information they have learned.

What Could This Mean for SEO?

The research does not establish that websites will rank higher or receive more AI citations simply by changing the order of their entities.

However, it does raise an interesting consideration for SEO professionals.

If AI systems find certain relationships easier to recall when they are presented in a familiar order, it may be sensible to make important relationships between entities clear and consistent throughout a website.

For example, a property website might consistently describe a particular development alongside its location, developer and key features rather than presenting these relationships in confusing or inconsistent ways.

Similarly, businesses should make important facts about their brand, services, locations and people clear within their content.

This is not a proven ranking technique, but it fits with the wider move towards making content easier for search engines and AI systems to understand.

Clear Information Still Matters

Google’s research provides an important insight into how AI search systems may handle information.

An AI model can have a fact stored within its knowledge while still struggling to retrieve it when the question is presented differently from the way the information was originally encountered.

For website owners, the practical takeaway is fairly simple: make important facts and relationships clear, consistent and easy to understand.

As AI becomes a larger part of search, SEO is increasingly about more than keywords and traditional rankings. How clearly a website communicates the relationships between people, businesses, places, products and topics may become just as important when AI systems decide what information to retrieve and present to users.