source: Google Research: Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
level: research
Google Research introduced knowledge profiling, a framework that separates fact encoding from recall in large language models. The team built WikiProfile, a benchmark of 2,150 Wikipedia facts with ten probing tasks each. They evaluated 13 LLMs, including Gemini and GPT-5, with and without thinking. Results show frontier models encode 95–98% of facts but fail to directly recall 26–34% of them. Even with thinking, 11–12% of facts remain inaccessible, indicating recall is the main bottleneck.
The study found recall failures are tied to how facts were learned. Rare facts are encoded at rates close to popular ones, but the recall gap is much larger. Reverse questions, such as asking for the subject when given the object, are harder in generation but not in multiple-choice verification. Thinking recovers 40–65% of encoded-but-not-recalled facts, especially for rare facts and reverse questions. However, thinking helps only 5–15% of non-encoded facts, showing it mainly aids access to stored knowledge.
These findings shift the focus from scaling model size or data to improving knowledge utilization. Since encoding is near saturation, further factuality gains may come from better recall mechanisms, such as inference-time thinking or post-training methods. The WikiProfile benchmark and profiling methodology provide a diagnostic tool for researchers to measure encoding and recall separately. This could guide development of models that more reliably access what they already know, reducing factual errors without massive new training.
why it matters: For AI practitioners, this work suggests that improving factuality may require better recall strategies rather than larger models, potentially reducing compute costs and enabling more reliable LLM applications.
source: Google Research: Empty shelves or lost keys? Recall is the bottleneck for parametric factuality