Unlock Your LLM's Brain: New Study Reveals Most 'Missing' Facts Are Just Hard to Find

We've all been there: you ask a powerful large language model (LLM) a question, and it comes back with a confident, yet completely wrong, answer – a phenomenon known as hallucination. Typically, developers assume this means the model simply doesn't have the information. The go-to solutions involve beefing up the model with more data, increasing its size, or building complex retrieval systems.

But what if the knowledge isn't missing at all? A groundbreaking new study from Google Research and Technion suggests that for advanced models like GPT-5 and Gemini-3, the facts are often encoded within their parameters, but they're just struggling to be recalled – like having information on the tip of your tongue.

The 'Lost Keys' Problem: When Knowledge is Encoded But Inaccessible

Researchers have found that cutting-edge LLMs encode an astonishing 95-98% of facts they are tested on. The real bottleneck, it turns out, is recall, not encoding. This is a game-changer, implying we can potentially build more reliable AI applications by improving how models access their existing knowledge, rather than just trying to stuff more into them.

The study introduces a concept called knowledge profiling. Instead of just looking at whether an LLM gets a question right or wrong, this method examines a single fact under various conditions to see if it's stored, how easily it can be accessed, and what computational effort is needed to retrieve it.

This allows for a more nuanced understanding of model limitations, categorizing knowledge issues into five profiles:

• Direct Recall: The model knows and can easily access the fact.
• Encoding Failure (Empty Shelves): The fact isn't stored at all, requiring more training data or a larger model.
• Recall Failure (Lost Keys): The fact is encoded but the model can't find it, even with extra time to 'think'. This is the core of the new findings.
• Recall with Thinking: The fact is encoded but requires intermediate reasoning steps (like Chain-of-Thought) to be surfaced.
• Inference Without Encoding: The model doesn't have the specific fact but can deduce it by piecing together other known information.

The implications are significant. A fact might be encoded, but if the question is phrased slightly differently or asks for information in reverse, the model might fail to retrieve it, presenting as a knowledge gap when it's actually a recall issue.

Scaling Isn't Always the Answer: The ROI of 'Thinking'

The study's evaluation of over 4 million responses across 13 LLMs revealed that while scaling up model size (like from Gemma3's 1B to 27B parameters) can reduce 'empty shelves' (encoding failures), it can paradoxically increase 'lost keys' (recall failures). This suggests that as models grow, more information can become trapped in an inaccessible state.

Interestingly, the researchers found that "thinking" – the use of inference-time computation – can recover a substantial 40-65% of facts that models initially fail to recall directly. This is akin to the human "tip-of-the-tongue" phenomenon, where deliberate mental effort can bring a forgotten memory to the forefront.

However, activating this "thinking" capability for every query can be computationally expensive. The key lies in dynamically routing queries to know when deeper reasoning is truly necessary, a challenge that requires models to develop a form of self-awareness or "faithful uncertainty" about their own knowledge.

Actionable Insights for Developers: Beyond RAG

This research offers crucial guidance for developers building AI applications:

• Rethink Retrieval-Augmented Generation (RAG): While RAG is excellent for incorporating fresh or proprietary data, relying on it as a universal fix for hallucinations can add unnecessary latency and cost for facts the model already knows.
• Strategic Use of Inference-Time Reasoning: Implement "thinking" processes selectively, as only a fraction of queries truly require it.
• Implement Generate-and-Verify Pipelines: Models are often better at recognizing correct information than generating it from scratch. Having models review and verify their own outputs can catch errors.
• Focus on Semantic Access: Evaluate models not just on benchmark accuracy, but on their ability to access facts through varied phrasings and contexts.
• Leverage Query Reformulation: Experiment with different ways of asking questions, as prompt framing significantly impacts recall.

While the WikiProfile benchmark used in the study focuses on encyclopedic knowledge, its principles can be applied to enterprise domains. However, researchers caution that for highly specialized, non-public data, genuine encoding failures might still be more prevalent than recall issues.

Ultimately, this study shifts the focus from merely scaling up models to optimizing how they utilize the vast knowledge they already possess. For companies that don't build their own LLMs, this is empowering news, as it highlights accessible post-training and inference-time strategies for improving AI reliability.