What is MapReduce in Hadoop? Architecture & Diagram

By : Prafulla Ranadive Prafulla Ranadive
Hours Updated

⚡ Smart Summary

MapReduce is the Hadoop programming model that turns a large dataset into a small result by running a map function over every input split and then a reduce function over the grouped intermediate values.

  • 🔘 Four phases: Every job runs as splitting, mapping, shuffling and reducing, with key-value pairs flowing between each stage.
  • ☑️ Worked example: Three lines of text become seven word counts, showing exactly what each phase contributes.
  • ✅ Split sizing: One map task runs per input split, and split size normally matches the HDFS block size.
  • 🧪 Intermediate data: Map output is written to local disk rather than HDFS, because replicating throwaway data is wasteful.
  • 🛠️ Coordination: A JobTracker schedules work and TaskTrackers report progress through periodic heartbeat signals.
  • ⚠️ Version note: YARN replaced that pair with a ResourceManager, NodeManagers and a per-job ApplicationMaster from Hadoop 2.x.
Read More

MapReduce architecture in Hadoop explained with an example

What is MapReduce in Hadoop?

MapReduce is a software framework and programming model used for processing huge amounts of data. MapReduce programs work in two phases, namely, Map and Reduce. Map tasks deal with splitting and mapping of data while Reduce tasks shuffle and reduce the data.

Hadoop is capable of running MapReduce programs written in various languages: Java, Ruby, Python, and C++. MapReduce programs are parallel in nature, so they are very useful for performing large-scale data analysis using multiple machines in the cluster.

The input to each phase is key-value pairs. In addition, every programmer needs to specify two functions: a map function and a reduce function.

MapReduce Architecture in Big Data explained with Example

The whole process goes through four phases of execution namely, splitting, mapping, shuffling, and reducing.

Now in this MapReduce tutorial, let’s understand it with a MapReduce example.

Consider you have the following input data for your MapReduce in Big Data program:

Welcome to Hadoop Class
Hadoop is good
Hadoop is bad

The diagram below traces those three lines through every phase, from the input splits on the left to the final word counts on the right.

MapReduce architecture diagram tracing three input lines through splitting, mapping, shuffling and reducing

The final output of the MapReduce task is

bad 1
Class 1
good 1
Hadoop 3
is 2
to 1
Welcome 1

The data goes through the following phases of MapReduce in Big Data.

Input Splits

An input to a MapReduce in Big Data job is divided into fixed-size pieces called input splits. An input split is a chunk of the input that is consumed by a single map.

Mapping

This is the very first phase in the execution of a MapReduce program. In this phase, data in each split is passed to a mapping function to produce output values. In our example, the job of the mapping phase is to count the number of occurrences of each word from the input splits (more details about input splits are given below) and prepare a list in the form of <word, frequency>.

Shuffling

This phase consumes the output of the Mapping phase. Its task is to consolidate the relevant records from the Mapping phase output. In our example, the same words are clubbed together along with their respective frequency.

Reducing

In this phase, output values from the Shuffling phase are aggregated. This phase combines values from the Shuffling phase and returns a single output value. In short, this phase summarizes the complete dataset.

In our example, this phase aggregates the values from the Shuffling phase, i.e., it calculates the total occurrences of each word.

MapReduce Architecture explained in detail

The points below explain how splits, map tasks and reduce tasks are actually placed and stored across the cluster.

  • One map task is created for each split, which then executes the map function for each record in the split.
  • It is always beneficial to have multiple splits because the time taken to process a split is small compared to the time taken for processing the whole input. When the splits are smaller, the processing is better load balanced, since the splits are processed in parallel.
  • However, it is also not desirable to have splits that are too small. When splits are too small, the overhead of managing the splits and creating map tasks begins to dominate the total job execution time.
  • For most jobs, it is better to make the split size equal to the size of an HDFS block, which defaults to 128 MB from Hadoop 2.x onwards (it was 64 MB in Hadoop 1.x) and is controlled by the dfs.blocksize property.
  • Execution of map tasks results in writing output to a local disk on the respective node, and not to HDFS.
  • The reason for choosing local disk over HDFS is to avoid the replication that takes place during an HDFS store operation.
  • Map output is intermediate output which is processed by reduce tasks to produce the final output.
  • Once the job is complete, the map output can be thrown away. So, storing it in HDFS with replication becomes overkill.
  • In the event of node failure, before the map output is consumed by the reduce task, Hadoop reruns the map task on another node and re-creates the map output.
  • Reduce tasks do not work on the concept of data locality. An output of every map task is fed to the reduce task. Map output is transferred to the machine where the reduce task is running.
  • On this machine, the output is merged and then passed to the user-defined reduce function.
  • Unlike the map output, reduce output is stored in HDFS (the first replica is stored on the local node and other replicas are stored on off-rack nodes). So, writing the reduce output does consume network bandwidth, but only as much as a normal HDFS write pipeline consumes.

RELATED ARTICLES

How MapReduce Organizes Work?

Now in this MapReduce tutorial, we will learn how MapReduce works.

Hadoop divides the job into tasks. There are two types of tasks:

  1. Map tasks (Splits & Mapping)
  2. Reduce tasks (Shuffling, Reducing)

The complete execution process, that is the execution of both Map and Reduce tasks, is controlled by two types of entities called:

  1. JobTracker: acts like a master and is responsible for the complete execution of a submitted job.
  2. Multiple TaskTrackers: act like slaves, each of them performing part of the job.

For every job submitted for execution in the system, there is one JobTracker that resides on the NameNode, and there are multiple TaskTrackers which reside on the DataNodes.

Note: the JobTracker and TaskTracker pair belongs to MapReduce version 1 (Hadoop 1.x). From Hadoop 2.x onwards, YARN splits those duties between a cluster-wide ResourceManager, a NodeManager on every node, and one ApplicationMaster per job, although the map, shuffle and reduce phases themselves are unchanged.

The diagram below shows how a submitted job is broken into tasks and tracked across the cluster.

Diagram showing a job split into map and reduce tasks tracked by the JobTracker and TaskTrackers

  • A job is divided into multiple tasks which are then run on multiple data nodes in a cluster.
  • It is the responsibility of the job tracker to coordinate the activity by scheduling tasks to run on different data nodes.
  • Execution of an individual task is then looked after by the task tracker, which resides on every data node executing part of the job.
  • The task tracker’s responsibility is to send the progress report to the job tracker.
  • In addition, the task tracker periodically sends a ‘heartbeat’ signal to the JobTracker so as to notify it of the current state of the system.
  • Thus the job tracker keeps track of the overall progress of each job. In the event of task failure, the job tracker can reschedule it on a different task tracker.

FAQs

YARN did, from Hadoop 2.x onwards. A cluster-wide ResourceManager handles scheduling, a NodeManager runs on each node, and one ApplicationMaster per job tracks its tasks. The map and reduce phases are unchanged.

Models trained on past job history predict runtime, recommend split sizes and reducer counts, and detect skew early. They also watch counter values, flagging unusually slow or failing jobs before a run finishes.

Copilot handles the scaffolding well: mapper and reducer signatures, generics, imports and driver configuration calls. Schema decisions, such as which field is the grouping key, still need a developer who knows the data.

A common starting point is slightly under the number of available reduce slots, so every reducer runs in one wave. Too few create long tails; too many produce many tiny output files.

A combiner is an optional mini-reducer that runs on map output before it crosses the network. It cuts shuffle traffic sharply, but it may only be used when the reduce operation is both associative and commutative.

Spark keeps intermediate results in memory and expresses a job as one directed graph of stages, whereas MapReduce writes intermediate output to disk between phases. Spark is therefore far faster for iterative work.

The partitioner decides which reducer receives each intermediate key, by default hashing the key modulo the reducer count. A custom one is written when that hash leaves a single reducer overloaded.

Hadoop creates one map task per input split, and a split is a byte range rather than a whole file. One large file yields many splits; many small files yield tiny, inefficient map tasks.

Summarize this post with: