Data Integration and Data Cleansing Tools are at the Core of Big Data

The growth of big data will present many “mores”. More databases, cloud resources, storage centers, and of course, more data integration and data cleansing tools. Specifically, a rise in the number of data integration and data cleansing tools will be an integral part of managing all the steps associated with keeping big data clean and migrating it faster to more people, more places, and with more frequency.

+ Check out this Free 2014 Data Integration Tools and Software Buyers Guide

At the VentureBeat DataBeat conference in San Franscisco, Tom Davenport, author of Big Data@Work: Dispelling the Myths, Uncovering the Opportunities, spoke about his interviews with data scientists and reported that, “People spend a huge amount of time on what they call munging data or extracting, filtering, cleaning data from various kinds of systems.” Davenport became convinced that data scientists shouldn’t spend so much time doing the “dirty work” or “busy work” involved in data integration and data cleansing. Imagine this times 10 or 100 when considering big data.

What to expect? There will be many challenges. One can be certain of this. It’s not going to be as simple as moving some data from one system to an existing system and expecting it to “plug and play”. There will be all types of interfaces and database models to consider. Keeping the data structure and content intact during these changes will present more complexities than most companies comprehend. Davenport said, “I don’t think you can do all this without adopting some new approaches to data integration and curation. It’s just not going to happen without that.”

The purpose of this article is not to scare you. In fact, it’s to aware you that there is much good news around the topic. First, the concept of data integration is not young, and the technologies and approaches are more than mature. There are many data integration tools available on the market that can work with your current systems as add-ons, with zero footprint architectures, either on premise or on the cloud. Easy-to-use data integration tools that have automation features to assist you in handling the big data workload are common as well. Second, data cleansing tools can be part of data integration platforms, and actually clean the data as it migrates from one place to another in real-time.

In conclusion, the market for data integration and data cleansing will grow in response to the growth of big data. Knowledge will be collected, lessons will learned, and greater support will become available moving forward.

Click here to read more about Tom Davenport’s talk at the at VentureBeat DataBeat conference.