According to a new Market Pulse survey commissioned by Matillion and IDG, companies are drawing from an average of over 400 different data sources to feed their BI and analytics. The study, entitled Optimizing Business Analytics by Transforming Data in the Cloud, saw IDG poll more than 200 IT, data science, and data engineering professionals at North American organizations with at least 1,000 employees. Though 400 was the average number, more than 20 percent of the organizations reported drawing from a whopping 1,000 or more data sources.
The IDG survey found that, on average, enterprise data volumes are growing at 63 percent per month. A smaller portion of respondents (12 percent) report that their volumes are growing by more than 100 percent per month. Organizations are reacting to this by using cloud storage, with 9-in-10 companies already placing at least a portion of their data in cloud data warehouses. Among them, 37 percent of organizational data was held in cloud data warehouses at the time of the study, with 35 percent stored in on-prem data warehouses, and 25 percent storing data in offsite non-cloud data warehouses.
The explosion in number of available data sources promises to be a boon to data-driven organizations in the future. However, data and analytics projects are being stalled by complex data environments according to 45 percent of those polled. An additional 38 percent reported that manual coding of data pipelines was a major obstacle, and 32 percent said they were having trouble simply connecting to multiple data sources.
IDG and Matillion provide an important best practice for these increasingly common problems: load the data first and then transform it using cloud-optimized tools. Just 28 percent of those polled are using ELT technologies to do this, and instead they are either transforming data with non-cloud ETL tools or manually coding data into the needed format for use within a business intelligence system.
We encourage you to read the complete survey in full.