RAG’s Next Act: Retrieval Tool to Enterprise Storytelling Engine

EY’s Dipanjan Sengupta offers this commentary on RAG’s next act, from retrieval tool to enterprise storytelling engine. This article originally appeared in Insight Jam, an enterprise IT community that enables human conversation on AI.

Retrieval-augmented generation (RAG) rose to prominence in 2024 before being quickly sidelined by other artificial intelligence (AI) frameworks. But as agentic AI advances and interest in AI inference grows, RAG’s role in AI agent orchestration and cutting-edge autonomous AI agents has put it back in the spotlight.

So how did RAG resurge? It evolved — from fact finding to storytelling — to become the memory, verification, and context layer that makes agents useful in enterprise environments.

A New Way to Retrieve and Store Data

Before RAG, search was managed via keywords, which made unwieldy sums of information and data searchable without any sense of conceptual analysis.

Regularly, irrelevant data matching a keyword would be returned, and the user was often left parsing through a myriad of false positives to try and locate the correct match they were seeking. Similarly, relevant results that matched conceptually but without the same exact vocabulary were missed entirely.

This process didn’t — and still doesn’t — work. RAG was designed to address these challenges head-on by providing a greater method for capturing meaning: using numbers instead of words.

Its vector-based infrastructure utilized mathematical representations to categorize and organize data into sets of numbers. When a user queries their system to find a specific data point, the query is converted into a vector and then compared to the contents in the vector database. This environment introduced newfound abilities for search and retrieval.

RAG came charging out of the gate due to its ability to improve the output and performance of large language models (LLMs), as RAG helped overcome the limitations of static LLM training.

By locating and incorporating data absent or unavailable in an AI model but located elsewhere in an organization’s data infrastructure (or knowledge base), RAG also enhanced the output of LLMs without interrupting operation and continuity in the system. This made workflows more sustainable by eliminating the need to constantly bring the AI model out of its deployment for ingestion each time new data was required.

Early RAG’s success was defined by the vector system’s ability to gauge the relevance of data conceptually rather than literally, a capability known as semantic similarity. This marked a shift from matching words to matching intent.

RAG’s Downfall and Resurgence

Yet those early RAG systems ultimately disappointed.

While the retrieval of the vector-based system returned better search results than keywords, the use of vectors did not sufficiently illuminate the interrelationships and context that exist between data needed for multi-hop reasoning. “Which of our franchises ordered the most food this year?” would be erroneously split into two separate retrievals, one focused on franchises and another on orders. These were then combined into an answer that cannot be provably verified for accuracy, thus failing the “hop” needed to execute the retrieval correctly.

Further, the process of converting content into vectors was unavoidably arbitrary in nature and did not sufficiently preserve the overarching meaning and intent of text. Similarly, RAG struggled to translate the full meaning of the user query, hindering retrieval from the very start. The enthusiasm for RAG subdued tremendously with some abandoning RAG altogether because of these snags.

However, some RAG researchers continued to work to advance its development, maintaining that its future was viable even as attention turned to other AI paradigms. RAG still had broad applicability across business use cases at an enterprise scale, and today, while additional fine-tuning is needed, RAG has begun to deliver on its promise.

Charting a New Path: GraphRAG

The original method of matching underlying meanings in a vector-based system did not take into account contextual information that escaped the metric of semantic similarity. To target the loss of meaning gained by vector translation, GraphRAG was developed. This new method deploys knowledge graphs to map the relationships between data to improve accuracy.

For example, during the COVID 19 pandemic, one may ask why certain regions within the same jurisdiction experienced higher infection rates despite comparable social restrictions. A traditional RAG system would typically surface relevant studies and reports, but its response is often descriptive and fragmented, lacking a clear synthesis of the underlying drivers.

In contrast, a GraphRAG approach leverages a knowledge graph that takes into account relationships across multiple dimensions such as demographics, healthcare system capacity, and circulating virus variants. By navigating these interconnected data points, it can infer multifactor causal patterns, for instance, the combined impact of the population size, access to specialized critical care and the prevalence of more virulent strains — providing faster added value to those who needed it most, such as doctors and policy makers.

GraphRAG typically produces a more coherent and actionable explanation, illustrating how interdependent factors collectively contribute to observed outcomes rather than treating them in isolation. This highlights its advantage in scenarios that demand reasoning over complex, heterogeneous, and highly interconnected data.

By storing and preserving the relationships among data using knowledge graphs, this new method keeps the detail that was previously lost when data was reduced to embeddings, enabling reasoning capabilities that can analyze the accuracy of the results.

A Necessity, Not Redundancy, in the Agentic Era

Now, some may say that the surge of AI agents could make this rigid structure less necessary and redundant; however, there are clear advantages to having explainability behind why a query was returned. This is why RAG will increasingly work alongside agentic workflows and long-context models rather than be replaced by them.

As AI agents gain stronger reasoning and action capabilities, RAG shifts from simply delivering reference material to functioning as a dynamic information supply system that adapts to an agent’s evolving goals, verification needs, and decision steps.

To put this into everyday terms, early RAG operated like a restaurant with a set menu: The users asked for something, and the system would retrieve the closest match. Modern RAG is becoming more like a skilled maître d’: It understands intent, asks what matters, checks what is available, and guides the user toward the best response.

The enhanced capabilities for preserving context-oriented long-term memory rather than simply retrieving relevant content turns RAG into an assistant over a static participant. By providing agents with the appropriate data along with feedback on accuracy, RAG enables agents to decide and act with greater safety and situational awareness, which is essential for successful enterprise AI adoption.

From Fact Finding to Storytelling

In this context, storytelling does not mean embellishment. It means assembling facts, relationships, source context, and confidence signals into a coherent explanation that a human or AI agent can act on.

There are fair criticisms of current RAG and GraphRAG approaches. Many remain inflexible, overly text-centric, and weak at extracting meaning from tables, diagrams, workflows, and other multimodal enterprise content, leading to challenges, such as missing workloads, extraction, and text. A new solution — one that combines knowledge graphs with state-of-the-art search techniques and supports multimodality, end-user configurability, and transitive relationship derivation — is how RAG will transform from rote retrieval to a dynamic storytelling engine.

This proposed solution, known as adaptive knowledge retrieval, replaces the clumsy assembly of chunks of information with the smooth generation of connected narratives.

This architecture addresses the previously discussed limitations, supporting more adaptive, faster, and context-rich augmentation of LLM responses. By supporting custom retrieval pathways, inference-aware reranking, and multimodal reasoning, this new era provides a scalable foundation, especially for enterprises seeking better compliance, knowledge management, and of course, decision-making.

As RAG garners attention again because of its utility in AI inference workflows, reengagement with it is by no means limited to agentic purposes.

Storytelling may sound like a tall order for retrieval infrastructure. But that is exactly where RAG is heading: from finding information to explaining why it matters.