{"id":2334,"date":"2025-11-26T16:33:15","date_gmt":"2025-11-26T16:33:15","guid":{"rendered":"https:\/\/solutionsreview.com\/thought-leaders\/?p=2334"},"modified":"2025-12-01T21:03:36","modified_gmt":"2025-12-01T21:03:36","slug":"federate-query-and-optimize-assembling-diverse-inputs-for-ai-innovation","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/thought-leaders\/federate-query-and-optimize-assembling-diverse-inputs-for-ai-innovation\/","title":{"rendered":"Federate, Query, and Optimize: Assembling Diverse Inputs for AI Innovation"},"content":{"rendered":"

Data teams struggle to feed artificial intelligence (AI) models the diverse, often far-flung inputs that they need. On the one hand, they cannot simply consolidate everything, given the complexity of migration. On the other hand, federated querying of data in place can drive up compute costs. These dual challenges deprive AI initiatives of critical inputs.<\/span>\u00a0<\/span><\/p>\n

Compute-optimized queries can ease such challenges. With this approach, data teams assign query jobs to different compute engines based on their processing requirements. By streamlining workloads in this fashion, compute-optimized queries can reduce costs and thereby improve the return on investment (ROI) of AI initiatives. This blog assesses the opportunity and use cases for this type of approach.<\/span>\u00a0<\/span><\/p>\n

Distributed and complex<\/span><\/b>\u00a0<\/span><\/p>\n

We start by assessing the state of data and AI environments. A BARC <\/span>survey<\/span><\/a> earlier this year showed that 47 percent of organizations host their AI data sources in more than one place; that is, in hybrid environments, multiple clouds, or multiple regions on-premises. Other aspects of AI projects, including feature engineering, model training\/fine-tuning, and retrieval-augmented generation (RAG), are similarly distributed. These numbers illustrate that many AI projects straddle more than one data platform. This creates the need for some type of federated querying capability.<\/span>\u00a0<\/span><\/p>\n

Where does your organization host the following components of its AI projects? <\/span><\/b>(n=318)<\/span>\u00a0<\/span><\/p>\n

\"\"<\/p>\n

Federated queries<\/span><\/b>\u00a0<\/span><\/p>\n

Data and AI stakeholders query distributed data at many points in an AI project. A data engineer and data scientist might explore Workday records in the cloud, an Oracle database on-premises, and Internet of Things (IoT) sensor logs at the edge, all to define features for machine learning (ML). The data scientist might further refine features as she trains and evaluates ML models. She may also retrieve subsets of this data, along with technical documentation, to feed a RAG workflow for generative AI (GenAI). Federated queries enable these stakeholders to access distinct datasets as needed.<\/span>\u00a0<\/span><\/p>\n

Data engineers and scientists query distributed data at many points in an AI project.<\/span><\/i><\/b>\u00a0<\/span><\/p>\n

Traditional methods<\/span><\/b>\u00a0<\/span><\/p>\n

Now let\u2019s consider what\u2019s happening under the covers. Building on our prior example, the data engineer executes Structured Query Language (SQL) commands to search, find, and retrieve data from each platform. Then he merges them into a single file on Databricks, ready for the data scientist to transform for feature engineering on Dataiku. Apache Spark engines handle the computation for all these operations. The problem is that the engine, in this case Apache Spark, might consume a lot of costly Amazon Web Services (AWS) compute cycles along the way, exceeding budgets and hurting the project\u2019s profitability. Bulk-processing methods like this can also slow performance.<\/span>\u00a0<\/span><\/p>\n

Enter the compute-optimized query<\/span><\/b>\u00a0<\/span><\/p>\n

The alternative is to take a more surgical approach. Compute-optimized queries orchestrate the query workload and compute engine to make the best use of precious compute cycles while minimizing the need for data movement. In this scenario, the data engineer and data scientist might use Zetaris to query Workday with Google Compute Engine, Oracle with a local Hewlett Packard Enterprise (HPE) server, and IoT sensor logs with DuckDB. And they might merge the outputs in memory to give AI models the real-time views they need for training, inference, and RAG.<\/span>\u00a0<\/span><\/p>\n

Compute-optimized queries streamline the processing, enabling organizations to support a wider array of AI use cases while meeting budget and performance requirements. For example:<\/span>\u00a0<\/span><\/p>\n