It seems as though IT people and business people alike are running around searching for solutions as big data and all of its fears are on the horizon. And this behavior in my opinion is justified as big data does present numerous challenges and if handled incorrectly.
But when one thinks about Big Data, the image that first comes to mind is quantitative, based on size, which makes sense since it is called “Big” Data. However, a survey of scientists reveals it’s not just the amount of data that should be acknowledged but the different types of data collected that is causing storage nightmares.
In an eWeek article written by Nathan Eddy called “Data Storage Demands Complicated by Data Variety”, Eddy reports that data scientists are seeing that data variety is harder to handle than data volume.
“Nearly three-quarters (71 percent) of data scientists said big data had made their analytics more difficult and data variety, not volume, was to blame, according to a survey of 111 data scientists from computational database management system designer company, Paradigm4.”
The complexity of the diverse types of data seemed to be the driver to why data was left on the table. This makes sense to me because it’s difficult to analyze information if it’s not “apples to apples” so there will be cases where the “peach” is left out of the analysis. This can be a big problem as data scientists may only be seeing part of the picture as if “looking through a keyhole.”
Eddy states, “The report also revealed 39 percent said their job had become more stressful with the growth of big data, while nearly half of data scientists (49 percent) said they’re finding it more difficult to fit their data into relational database tables.”
So it looks like Big Data is innocent when compared to Data Variety. This presents a major challenge for business leaders, data scientists, and data companies to overcome. Especially considering the resources spent to collect all of this data, one would be leaving money on the table if all of it was not utilized.
Click here to read the entire article.
Check out these additional resources:
- What’s Changed: 2015 Gartner Magic Quadrant for Business Intelligence and Analytics Platforms - October 13, 2015
- Hitachi Data Systems Announces Intent to Acquire Pentaho - February 11, 2015
- Public Service Announcement: Best Practice Guide for Big Data in the Public Sector - January 28, 2015