A recent 295-person study commissioned by Trifacta reveals the degree to which global organizations are struggling with data preparation. The report, which focuses mainly on IT professionals and data analysts that prepare or analyze data, uncovered that 60 percent of respondents spend at least half of their time on data quality assurance, cleanup, and preparation. Most of those polled (59 percent and 82 percent respectively) believe that they are too dependent on IT and could do more on an ad hoc basis.
Trifacta uses Glassdoor salary figures and IDC estimates to conclude that the global spend on data preparation is more than $450 billion dollars. This isn’t the first time we’ve written about how organizations are spending way too much time on data preparation. One of the main culprits is the continued use of Microsoft Excel, with nearly 40 percent of data analysts and 30 percent of IT professionals using it more than any other tool.
Manual back-and-forth between business users and IT exacerbates these concerns, as well as lacking business context when users are working cross-department. By keeping the use case in mind throughout a data preparation project, users are much more likely to benefit from the output. 59 percent of those polled by Trifacta believe that the overall time spent on data QA and data cleansing is hurting and not helping their organization. Moreover, 60 percent of the IT respondents feel as though they are over-overqualified to be spending so much time doing pre-processing for analytics.
Trifacta’s CEO Adam Wilson commented on the findings and said: “Data preparation is the biggest bottleneck in any analytic process, costing companies billions of dollars and slowing down time to insight.” We got it, everyone is squandering precious time and they know it. We expect to see continued investment in the data preparation marketplace in the months ahead and an even more aggression amongst the enterprise to adopt dedicated tools to deal with this data deluge.
Check out the study’s accompanying infographic here.
- Data Pipeline Automated Testing Best Practices for Beginners - March 20, 2023
- What to Expect at Safe Software’s FME:23 Event on April 13 - March 13, 2023
- The Essential Big Data Engineer Requirements to Know - March 9, 2023