14 Essential Hadoop Interview Questions and Answers to Know
-
By
Tim King
, Executive Editor at Solutions Review - Best Practices,
Hadoop is an open-source framework that is written in Java by the Apache Software Foundation. This framework is used to write software applications that require processing vast amounts of data. It works in-parallel on large clusters which could have thousands of computers (nodes) on the clusters. It also processes data very reliably and in a fault-tolerant manner. Hadoop as we know it today began as an experiment in distributed computing for Yahoo’s internet search, but has since evolved into the open-source big data framework of choice in some of the world’s largest organizations. With this in mind, we’ve compiled this list of essential Hadoop interview questions and answers to save you time and help you ace your next interview. We compiled this resource by curating the most popular results from community forums like Quora and Reddit. Our editors broke this resource down into the two main types of Hadoop interview questions focusing on the basic background to technical topics. Prospective data management leaders may also want to consult our directory of top-rated Hadoop books as well.
Basic Hadoop Interview Questions and Answers
Q: What is big data? Provide examples.
A: Big data is an assortment of large and complex data that becomes very tedious to capture, store, process, retrieve and analyze, and usually requires the help of on-hand database management tools or traditional data processing techniques. There are many real-life examples of big data: Facebook is generating 500+ terabytes of data per day, the NYSE (New York Stock Exchange) generates about 1 terabyte of new trade data per day, and a jet airline collects 10 terabytes of censor data for every 30 minutes of flying time.
Q: What are the main characteristics of big data?
A: According to IBM, the four characteristics of big data are:
- Volume: Facebook generating 500+ terabytes of data per day
- Velocity: Analyzing 2 million records each day to identify the reason for losses
- Variety: Images, audio, video, sensor data, log files
- Veracity: Biases, noise, and anomalies in data
Q: What is the difference between structured and unstructured data?
A: Structured data is data that is easily identifiable and organized in a structure. The most common form of structured data is a database where specific information is stored in tables like rows and columns. Unstructured data refers to any data that cannot be identified easily. It could be in the form of images, videos, documents, email, logs, and random text. It is not in the form of rows and columns and requires additional formatting.
Q: What is the difference between traditional RDBMS and Hadoop?
A: Traditional RDBMS are used for transactional systems to report and archive data, whereas Hadoop is an approach to store a huge amount of data in the distributed file system and process it. RDBMS will be useful when you want to seek one record from big data whereas Hadoop is useful when you want to uncover many records to use now and in the future.
Q: What are the core components of Hadoop?
A: The core components of Hadoop are HDFS and MapReduce. HDFS is used to store large data sets and MapReduce is used to process them.
Q: What is MapReduce?
A: MapReduce is an algorithm or concept to process large amounts of data in a faster way. Per its name, the process can be divided into “map” and “reduce.” The main MapReduce job usually splits the input data set into independent chunks. MapTask will then process these chunks in a completely parallel manner (One node can process one or more chunks).
Business logic is written in the MappedTask and ReducedTask. Typically both the input and the output of the job are stored in a file system. The framework takes care of scheduling tasks, monitoring them, and re-executes the failed tasks.
Q: What are compute and storage nodes, and what is a Namecode?
A: A compute node is the computer where the actual business logic will be executed, and a storage node is a machine where the file system resides and can be used to do data processing. A Namecode is the master node on which the job tracker runs. It consists of metadata and maintains and manages the blocks where datanodes are present. It’s also the single point of failure for HDFS.
Technical Hadoop Interview Questions and Answers
Q: What does the Mapper do and where do you specify implementation?
A: Maps are the individual tasks that transform input records into intermediate records. The transformed intermediate records do not need to be of the same type as the input records. A given input pair may map to zero or many output pairs. Mapper implementation is specified inside the job.
Q: What is the InputSplit in MapReduce?
A: An InputSplit is a logical representation of a unit (a chunk) of input work for a map task.
Q: What is the InputFormat?
A: The InputFormat is responsible for enumerating the InputSplits and producing a RecordReader which will turn those logical work units into actual physical input records.
Q: How many maps are there in a particular job?
A: The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files. Generally, the range is from 10-100 maps per node. Task setup takes a whole so it best if the maps take at least a minute to execute.
Q: What are the core methods of the Reducer? What is the Reducer used for?
A: The API of Reducer is very similar to that of Mapper, there’s a run() method that receives a Context containing the job’s configuration as well as interfacing methods that return data from the reducer itself back to the framework. The Reducer reduces a set of intermediate values which share a key to a (usually smaller) set of values. The number of reduces for the job is set by the user
Q: What is JobTracker?
A: JobTracker is a daemon service that submits and tracks the MapReduce tasks to the Hadoop cluster. It runs its own Java virtual machine process, usually via a separate machine. Each slave node is then with the JobTracker node location. The JobTracker is a single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted.
Q: What functions does JobTracker perform?
A: JobTracker in Hadoop performs the following actions:
- JobTracker talks to the NameNode to determine the location of the data
- JobTracker locates TaskTracker nodes with available slots at or near the data
- JobTracker submits the work to the chosen TaskTracker nodes
NOW READ: The Best Hadoop Courses and Online Training
Tim King
Executive Editor
Tim is Solutions Review's Executive Editor and leads coverage on data management and analytics. A 2017 and 2018 Most Influential Business Journalist and 2021 "Who's Who" in Data Management, Tim is a recognized industry thought leader and changemaker. Story? Reach him via email at tking@solutionsreview dot com.
- Data Management News for the Week of April 12; Updates from Collibra, Dremio, Informatica & More - April 12, 2024
- Data Management News for the Week of April 5; Updates from Aerospike, CData Software, Coalesce & More - April 4, 2024
- Data Management News for the Week of March 28; Updates from Astronomer, Dremio, Microsoft & More - March 29, 2024
