Hadoop vs. Spark; Which Big Data Framework is Better?

Hadoop vs. SparkA recent Forbes post, written by internationally recognized Big Data expert Bernard Marr, covers a key question for today’s data-driven organizations: Hadoop or Spark? You may find yourself asking that very question when looking into which Big Data Framework to deploy at your company. The column was very informative, so I am taking the liberty to summarize it and provide my own thoughts as a courtesy to those of you who may be in this type of conundrum.

Spark has overtaken Hadoop as the most active open source project currently, but while they are not directly comparable in terms of what they offer, both have similar uses. Hadoop and Spark are the two most commonly used Big Data Frameworks in the enterprise.

For years, Hadoop was the leading open source Big Data framework. More recently however, Spark has become the more popular of the two Apache tools. The two tools are not mutually exclusive, and can be paired together in certain circumstances. Spark has been reported to work up to 100 times faster than Hadoop, however, it does not provide its own distributed storage system.

Distributed storage is an important factor to many of today’s Big Data projects, as it allows multi-petabyte datasets to be stored across any number of computer hard drives, rather than involving expensive machinery which holds it on one device. These systems are also scalable.

Spark requires a third party system for organizing files in a distributed way, so for this reason many Big Data projects involve installing Spark on top of Hadoop, where Spark’s Advanced Analytics application can make use of data stored using Hadoop’s Distributed File System.

Where Spark really has the edge over Hadoop is speed, Marr writes. Spark handles most of its operations in-memory, which copies them from the distributed storage system. This reduces the amount of time-consuming writing and reading to and from slow, mechanical hard drives that needs to be done under Hadoop’s MapReduce system. MapReduce then writes all of the data back to the physical storage medium after each operation.

Marr explains: “Spark’s functionality for handling advanced data processing tasks such as real-time stream processing and machine learning is way ahead of what is possible with Hadoop alone. This, along with the gain in speed provided by in-memory operations, is the real reason, in my opinion, for its growth in popularity. Real-time processing means that data can be fed into an analytical application the moment it is captured, and insights immediately fed back to the user through a dashboard, to allow action to be taken. This sort of processing is increasingly being used in all sorts of Big Data applications, for example recommendation engines used by retailers, or monitoring the performance of industrial machinery in the manufacturing industry.”

The bottom line is that the two frameworks aren’t really in competition, especially considering they are born of the same Apache womb. There is some crossover functionality between the two, and the companies that monetizing these frameworks typically offer both, allowing the buyer to choose which functions they prefer to use. One thing to note though is that since Spark is in its infancy, it is likely behind Hadoop in security and support infrastructures. Companies that provide enterprise Big Data solutions will allow companies to work with whichever framework they choose, it all depends on what kind of data the organization has stored.

Bernard Marr was featured in our top Big Data Twitter follows earlier in the year, check it out.

This post was inspired by an article originally published in Forbes.

 

Timothy King
Follow Tim

Timothy King

Editor, Data and Analytics at Solutions Review
Timothy leads Solutions Review's Business Intelligence, Data Integration and Data Management areas of focus. He is recognized as one of the top authories in Big Data, and the number-one authority in enterprise middleware. Timothy has also been named one of the world's top-75 most influential business journalists by Richtopia.
Timothy King
Follow Tim

2 thoughts on “Hadoop vs. Spark; Which Big Data Framework is Better?”

  1. Suresh Chandrasekaran says:

    Liked the article especially the last part. Hadoop has brought in last of interest in everyone to look at this field., making it irrelevant will make new comers run away from this area thinking unstable.

  2. Marcelo says:

    “The bottom line is that the two frameworks aren’t really in competition, especially considering they are born of the same Apache womb.”

    It is not sales / financial competition!

    It is technical competition!

    Apache knows that it has to hit big on the SQL market to get into the mainstream of decent size companies… and Hadoop will never be able to do that… despite of vendors and gurus delusions…

    However… Spark via MISO has the potential for that… if another open-source backend / file system can perform in the relational world… because HDFS will NEVER achieve the required optimization and functionality…

Comments are closed.