You’re Spending Way Too Much Time on Data Preparation

You're Spending Way Too Much Time on Data Preparation

In a recent executive study by TDWI, 37 percent of respondents said they spent anywhere from 61 to 80 percent of their time doing data preparation instead of analysis and interaction. The survey TDWI Best Practices Report: Improving Data Preparation for Business Analytics also revealed that 28 percent of those polled spend 41 to 60 percent of their time doing the same. If we extrapolate those numbers, we can see that analysts are spending way too much time doing data prep and not enough time on the analytics itself.

After spending virtually all their time doing data prep as a pre-process to analytics, it’s no surprise that more than four-fifths of those polled consider hastening this process to be their number-one priority. Additionally, users cite ETL and Data Integration as slow and key barriers to improving data preparation projects. Other data-prep related wish list items for analysts include improving operational efficiency for end-users and making enterprise vital data more widely available.

In a related study last year, Blue Hill Research recorded similar trends. According to their report Quantifying the Case for Enhanced Data Preparationthe typical data analyst spends two hours per day exclusively on data preparation, accounting for nearly 500 hours per year. In conjunction, 28 percent of all a data analyst’s time is spent doing data preparation, accounting for roughly $22,000 worth of a typical data analyst’s yearly salary.

It’s unclear if the TDWI study polled individuals and organizations currently utilizing dedicated data preparation platforms, but Blue Hill’s research showed that those who deployed standalone or embedded data prep tools reported saving considerable time. To this point, Blue Hill recommends that organizations look into data prep tools to help alleviate some of the time that data analysts must commit to preparation.

It wouldn’t be a post at Solutions Review if we didn’t at least point you in the direction of some tools to meet what’s becoming a time-sink for many. Analyst house Forrester Research recently released the latest version of its Data Preparation Tools Wave Report for Q1 2017. Researcher Cinny Little identifies the seven providers whom are most significant in the category. The Wave details their research and examines how each vendor meets (or falls short of) Forrester’s evaluation criteria.

Timothy King
Follow Tim