Query RAG – A New Way to Create Powerful AI Data Agents

Denodo’s Felix Liao offers insights on query RAG and the new way to create powerful AI data agents. This article originally appeared on Solutions Review’s Insight Jam, an enterprise IT community enabling the human conversation on AI.

Today, it has become widely accepted that large language models (LLMs), on their own, are insufficient to enable intelligent AI agents that can deliver real value within an enterprise context. Rather, it is only by integrating and augmenting LLMs with enterprise data that we can deliver truly robust, accurate, and valuable AI agents and applications.

Retrieval augmented generation (RAG) currently offers the most practical and scalable approach for grounding LLMs with enterprise data. By leveraging vector database and embedding technologies, RAG enables LLMs to process requests and deliver responses based on information they were not trained on and have never seen before.

This capability is particularly crucial when applying AI in dynamic enterprise environments, in which it’s essential to access real-time, accurate data. With that said, it has proven to be challenging to implement RAG and deploy generative AI (GenAI) applications within an enterprise environment. One recent study by Gartner predicts that 30 percent of GenAI projects will be abandonedby the end of 2025 due to poor data quality and other factors.

This article delves into how Query RAG, a variation of the RAG pattern with a different set of steps and requirements, can be applied to enterprise data sources to deploy powerful AI agents – specifically, AI data agents.

Query RAG and Enterprise Data

Most RAG contents and examples today focus on augmenting LLMs with unstructured data. As a result, the techniques and eco-systems around implementing RAG using unstructured data such as websites and PDFs have matured rapidly and are proving to be robust and scalable. Unfortunately, the same level of technological maturation has not extended to the world of structured or tabular data, which accounts for a large percentage of enterprise data. It turns out that augmenting LLMs with structured data presents a different set of challenges that are poorly understood today.

Many companies that have begun to implement the RAG pattern using structured data quickly encounter a number of unique challenges that do not apply in the world of unstructured data:

• Structured or tabular data cannot simply be retrieved; it must be accessed and queried via correct SQL or API calls
• Access mechanisms and optimal data retrieval techniques can differ from vendor to vendor and platform to platform (These include file parsing, SQL, GraphQL, and proprietary APIs, to name a few)
• Vector embedding strategies must cover both metadata and actual data, and bridge the gap between natural-language-based queries and often highly technical metadata.
• Structured data (and its associated metadata) changes all the time. So, accessing current, real-time data presents unique changes.
• Existing data security and permission controls need to be respected and integrated

These challenges highlight the need for a distinct approach to handling structured data, which I am calling “Query RAG,” a variation of the RAG pattern with a different set of steps and requirements, as a solution. At a high level, Query RAG should provide:

• A robust, scalable way to create vector embeddings across all metadata and data sources and platforms
• A unified SQL access engine that abstracts away the complexities of the underlying data sources and systems
• A powerful query optimizer that enables the SQL generated by LLMs to be highly performant across multiple data architectures and use cases
• A flexible, robust semantic layer that enables you to develop new data views that encapsulate the relevant business context and language
• A simple process for enforcing user permissions and data security at every step

One of the more unique aspects of Query RAG is that the goal of the embedding search is not to find the text to augment the prompt but to find the right data view to query. Once the right data view and correct SQL have been identified, the tabular data is then retrieved from multiple data sources and is used to construct the relevant response back to the user.

Query RAG provides the ability to utilize enterprise data repositories in a robust, scalable way. By leveraging embedding and exposing real-time data to AI applications, this approach can help organizations to extend their GenAI capabilities beyond unstructured data, delivering real-time insights and improving decision-making in ways that we have never seen before.

The Logical Data Management Factor

Query RAG might sound promising in theory, but it requires a data platform that can integrate data silos in near-real or even real time, to provide a unified, secure access point to the Query-RAG-enabled LLM. Cloud data warehouses and data lakehouses are capable of providing such a platform, but often, some enterprise data may remain outside of the central platform, due to data-export restrictions, multicloud configurations, and other considerations.

Logical data management platforms offer a solution, in that they are implemented as a layer above an organization’s data sources, including cloud data warehouses, data lakehouses, and other modern data platforms, enabling the required data access point: unified, secured, governed, and immediate.

By leveraging logical data management platforms to unify enterprise data sources alongside orchestration frameworks such as LangChain and vector database technologies, developers have what they need to build powerful new AI chatbots and applications using Query RAG.

They can begin to construct conversational experiences atop any data repositories without being constrained by traditional, centralized data access methods. This means supporting analytical-driven questions as well as operational requests that leverage live production data in real time. All the while, the logical data management platform enforces the necessary data security and governance policies, aligning data access and usage with organizational control, without additional manual intervention.

Agentic Teamwork

While an AI agent can be extremely powerful for an organization, imagine multiple AI agents working together, each with their own unique skills and capabilities. This is the future, and data agents will play a key role in this agentic world, enabled by query RAG.

Data agents will be specialists, configured and optimized to retrieve data and interact with all structured data repositories in a scalable, optimized, and secure way. Already, data agents can support open-ended questions rather than users having to rely on hundreds of pre-built PowerBI dashboards. Logical data management platforms should support today’s REST and OpenAPI specifications, as well as all of the expected interaction methods and modes.

From Data Silos to Intelligent Data Agents

Gartner recently talked about the gap between AI-ready data requirements and the capabilities of traditional data management. The issues of data delivery and access associated with data silos in enterprise environments have long been significant challenges. These challenges have become even more pronounced with the advent of GenAI, acting as major inhibitors to the development and deployment of next-generation AI agents.

However, these traditional barriers should now be viewed as opportunities for innovation. With logical data management and new approaches like Query RAG, businesses can unlock transformational capabilities, enabling AI-driven insights that are both immediate and impactful.