Why RAG won't solve generative AI's hallucination problem

Hallucinations — the lies generative AI models tell, basically — are a big problem for businesses looking to integrate the technology into their operations.

Because models have no real intelligence and are simply predicting words, images, speech, music and other data according to a private schema, they sometimes get it wrong. Very wrong. In a recent piece in The Wall Street Journal, a source recounts an instance where Microsoft’s generative AI invented meeting attendees and implied that conference calls were about subjects that weren’t actually discussed on the call.

As I wrote a while ago, hallucinations may be an unsolvable problem with today’s transformer-based model architectures. But a number of generative AI vendors suggest that they can be done away with, more or less, through a technical approach called retrieval augmented generation, or RAG.

Here’s how one vendor, Squirro, pitches it:

At the core of the offering is the concept of Retrieval Augmented LLMs or Retrieval Augmented Generation (RAG) embedded in the solution … [our generative AI] is unique in its promise of zero hallucinations. Every piece of information it generates is traceable to a source, ensuring credibility.

Here’s a similar pitch from SiftHub:

Using RAG technology and fine-tuned large language models with industry-specific knowledge training, SiftHub allows companies to generate personalized responses with zero hallucinations. This guarantees increased transparency and reduced risk and inspires absolute trust to use AI for all their needs.

RAG was pioneered by data scientist Patrick Lewis, researcher at Meta and University College London, and lead author of the 2020 paper that coined the term. Applied to a model, RAG retrieves documents possibly relevant to a question — for example, a Wikipedia page about the Super Bowl — using what’s essentially a keyword search and then asks the model to generate answers given this additional context.

“When you’re interacting with a generative AI model like ChatGPT or Llama and you ask a question, the default is for the model to answer from its ‘parametric memory’ — i.e., from the knowledge that’s stored in its parameters as a result of training on massive data from the web,” David Wadden, a research scientist at AI2, the AI-focused research division of the nonprofit Allen Institute, explained. “But, just like you’re likely to give more accurate answers if you have a reference [like a book or a file] in front of you, the same is true in some cases for models.”

RAG is undeniably useful — it allows one to attribute things a model generates to retrieved documents to verify their factuality (and, as an added benefit, avoid potentially copyright-infringing regurgitation). RAG also lets enterprises that don’t want their documents used to train a model — say, companies in highly regulated industries like healthcare and law — to allow models to draw on those documents in a more secure and temporary way.

But RAG certainly can’t stop a model from hallucinating. And it has limitations that many vendors gloss over.

Wadden says that RAG is most effective in “knowledge-intensive” scenarios where a user wants to use a model to address an “information need” — for example, to find out who won the Super Bowl last year. In these scenarios, the document that answers the question is likely to contain many of the same keywords as the question (e.g., “Super Bowl,” “last year”), making it relatively easy to find via keyword search.

Things get trickier with “reasoning-intensive” tasks such as coding and math, where it’s harder to specify in a keyword-based search query the concepts needed to answer a request — much less identify which documents might be relevant.

Even with basic questions, models can get “distracted” by irrelevant content in documents, particularly in long documents where the answer isn’t obvious. Or they can — for reasons as yet unknown — simply ignore the contents of retrieved documents, opting instead to rely on their parametric memory.

RAG is also expensive in terms of the hardware needed to apply it at scale.

That’s because retrieved documents, whether from the web, an internal database or somewhere else, have to be stored in memory — at least temporarily — so that the model can refer back to them. Another expenditure is compute for the increased context a model has to process before generating its response. For a technology already notorious for the amount of compute and electricity it requires even for basic operations, this amounts to a serious consideration.

That’s not to suggest RAG can’t be improved. Wadden noted many ongoing efforts to train models to make better use of RAG-retrieved documents.

Some of these efforts involve models that can “decide” when to make use of the documents, or models that can choose not to perform retrieval in the first place if they deem it unnecessary. Others focus on ways to more efficiently index massive datasets of documents, and on improving search through better representations of documents — representations that go beyond keywords.

“We’re pretty good at retrieving documents based on keywords, but not so good at retrieving documents based on more abstract concepts, like a proof technique needed to solve a math problem,” Wadden said. “Research is needed to build document representations and search techniques that can identify relevant documents for more abstract generation tasks. I think this is mostly an open question at this point.”

So RAG can help reduce a model’s hallucinations — but it’s not the answer to all of AI’s hallucinatory problems. Beware of any vendor that tries to claim otherwise.