Big Data is Innocent, Data Variety is the Culprit

Big Data is Innocent Data Variety is the CulpritIt seems as though IT people and business people alike are running around searching for solutions as big data and all of its fears are on the horizon.  And this behavior in my opinion is justified as big data does present numerous challenges especially if handled incorrectly.

But when I think about Big Data, the image that first comes to mind is quantitative, based on size, which makes sense since it is called “Big” Data. However, a survey of scientists reveals it’s not just the amount of data that is presenting challenges, but the different types of data collected that is causing storage nightmares.

In an eWeek article written by Nathan Eddy called “Data Storage Demands Complicated by Data Variety”, Eddy reports that data scientists are seeing that data variety is harder to handle than data volume.

“Nearly three-quarters (71 percent) of data scientists said big data had made their analytics more difficult and data variety, not volume, was to blame, according to a survey of 111 data scientists from computational database management system designer company, Paradigm4.”

The complexity of the diverse types of data seemed to be the driver to why data was left on the table. This makes sense to me because it’s difficult to analyze information if it’s not “apples to apples” so there will be cases where the “peach” is left out of the analysis. This can be a big problem as data scientists may only be seeing part of the picture as if “looking through a keyhole.”

Eddy states, “The report also revealed 39 percent said their job had become more stressful with the growth of big data, while nearly half of data scientists (49 percent) said they’re finding it more difficult to fit their data into relational database tables.”

So it looks like Big Data is innocent when compared to Data Variety. This presents a major challenge for business leaders, data scientists, and data companies to overcome. Especially considering the resources spent to collect all of this data, one would be leaving money on the table if all of it was not utilized.

Is big data variety more complicated than big data volume? Please let me know what you are experiencing.

Click here to read the entire eWeek article written by Nathan Eddy.

Timothy King
Follow Tim