MapReduce can means two different things:
- Programming Model
- If you can rewrite algorithms into Maps and Reduces, and your problem can be broken up into small pieces solvable in parallel, then MapReduce might be a potential distributed problem solving approach to your large datasets.
- See  for a sample MapReduce application
- Software Framework
- MapReduce 1 (MR1)
- YARN (MR2)
We will start the introduction of Hadoop MapReduce using MR1 and then briefly with MR2.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system (e.g., HDFS on Apache Hadoop). The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
->map -> ->combine -> ->reduce -> (output)
MapReduce job is useful for batch processing on terabytes or petabytes of data stored in Apache Hadoop and has the following characteristics:
- Cannot control the order in which the maps or reductions are run
- For maximum parallelism, you need Maps and Reduces to not depend on data generated in the same MapReduce job (i.e. stateless)
- A database with an index will always be faster than a MapReduce job on unindexed data
- Reduce operations do not take place until all Maps are complete (or have failed then been skipped)
- General assumption that the output of Reduce is smaller than the input to Map一large datasource used to generate smaller final values
Hadoop MapReduce Framework
It's easy to execute MapReduce applications on Apache Hadoop. Other than simplicity, scalability, and performance, Hadoop MapReduce framework also provides additional benefits such as:
- Failure and recovery
- Minimal data motion
The framework takes care of failures. It is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of servers, each of which may be prone to failures.
Minimal data motion
Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System are running on the same set of nodes (see the above diagram). This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in lower network I/O and very high aggregate bandwidth across the cluster.
Job Tracker & Task Tracker
Applications using MapReduce framework are required to
- Specify Job configuration
- Specify input/output locations
- Supply map, combine and reduce functions
The JobTracker will first determine the number of splits from the input path and select some TaskTracker based on their network proximity to the data sources, then the JobTracker send the task requests to those selected TaskTrackers.
The TaskTracker spawns a separate JVM processes to do the actual work; this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these spawned processes, capturing the output and exit codes. When the process finishes, successfully or not, the tracker notifies the JobTracker.
The TaskTrackers also send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated.
To overcome the drawbacks of MR1, YARN (or MR2) was introduced and provided for:
- Better scalability
- As the JobTracker functionality is split between the ResourceManager and ApplicationMaster. Hadoop may be run on much larger clusters on MR2.
- Better cluster utilization
- As the resource capacity configured for each node may be used by both Map and Reduce tasks
- Non-MapReduce clusters
- May be run on the same cluster concurrently
- Higher throughput
- Uses finer-grained resource scheduling
- Volume Rendering using MapReduce
- MapReduce Tutorial (Apache Hadoop)
- Hadoop MapReduce Framework (Hortonworks)
- How Hadoop Map/Reduce works
- Maps input key/value pairs to a set of intermediate key/value pairs
- The MapReduce framework consists of a single master ResourceManager, one slave NodeManager per cluster-node, and MRAppMaster per application (see tutorial).
- The ResourceManager has two main components: Scheduler and ApplicationsManager.
- Two scheduler plug-ins