Top 6 Barriers to Successful Hadoop Implementation

Hadoop Jail

Enterprise organizations are turning to Hadoop in increasing fashion, both as a Big Data platform that can accommodate growing volumes of data, and as a computational platform for Business Intelligence and Data Analytics that enables data-driven decision-making. Hadoop has begun to enter into a much wider range of use cases, and it must satisfy stringent requirements that IT and business units demand from their platforms.

TDWI recently released a new Best Practices Report, outlining all things enterprise Hadoop, from the benefits to the barriers and everything in between. The study, which sourced 334 respondents, 70 percent of which held positions in corporate IT or Business Intelligence, revealed that Hadoop usage will become mainstream in the years ahead, and will serve entire enterprises, not just the handful of users with niche applications for the platform. Within that report, TDWI discovered the top 6 barriers to successful Hadoop implementation, here’s a summary:

The Skills Gap

42 percent of respondents believe that inadequate skills and difficulty finding appropriately trained staff are the leading barriers to successful Hadoop deployment. Hadoop is still a relatively new platform, and as of this moment, it is not a complete organizational necessity. This means that a large portion of Hadoop users harness their skills without hiring personnel with Hadoop experience.

Weak Business Support

The study found there to be a lack of compelling business cases, a lack of business sponsorship, and a lack of Data Governance for Hadoop initiatives, and when success isn’t a high probability, a technical implementation will sit on the back burner.

Security Concerns

IT leaders and business stakeholders have real concerns with security when it comes to Hadoop, similar to the concerns these same folks have with other newer enterprise platforms, such as the cloud, appliances, and NoSQL databases. According to TDWI, as organizations grow more accustomed to tools like Hadoop, their confidence in security measures will increase. A number of Hadoop distributions currently help companies cope with these fears by including additional data security, masking, and encryption capabilities that go beyond what Hadoop offers in and of itself.

Lack of Data Management

According to TDWI, enterprise data management professionals that are used to mature relational database managing systems are often deterred by Hadoop’s lack of metadata management, lack of support for ANSI-standard SQL, and limited interoperability with existing systems. Hadoop manages metadata at run time, not prior, and as IT professionals work with the technology, it’s methods will be easier to understand. In addition, Hadoop has its own approach to queries and relational data structures, and these methods have advantages with unstructured and schema-free data – the type that Hadoop is typically used to manage.

Deficiency in Tools

Open-source development tools are quickly evolving, but developing solutions with Hadoop still requires a great deal of hand-coding in languages that few data management professional know how to use, such as Java, R, Hive, and Pig. This is a skills issue to be sure, but it also slows down tool development.

Containing Costs

Those surveyed were confident that Hadoop’s low costs are real and attainable, according to TDWI, although concerns remain regarding the cost of staffing, implementation, and Hadoop operation.

The benefits of deploying Hadoop are many, as the platform supports Advanced Analytics based on techniques for data mining and complex SQL. Hadoop is also being used as a data warehouse as a Big Data source for analytics and as an integration platform for transforming data. Hadoop is ultra-scalable and supports unstructured data sources, making it great for schema-free data. Hadoop can also be deployed inside many other business applications, making it quite versatile.

Though the hype surrounding Hadoop is real, business and IT leaders will have to workaround the barriers-to-implement in order for the platform to evolve from a fairly niche tool to one that can be adopted within entire large enterprises, as TDWI predicts.

Read TDWI’s Best Practices Report: ‘Hadoop for the Enterprise.

Timothy King
Follow Tim