So now we've seen conceptually how MapReduce works. In the
next lesson, we'll talk about how to write the code
to perform these MapReduce jobs on your cluster. But before
that, let's go back to the cluster. Just as we
was with HDFS, there are a set of daemons, which
is just a piece of code, running all the time,
running on each of these machines. There were the data
nodes and the name node. When you run a MapReduce
job, you submit the job to what's called
the Job Tracker. That splits the work into mappers
and reducers. Those mappers and reducers will run
on the other cluster nodes. Running the actual map
and reduce tasks is handled by a daemon
called the TaskTracker. The TaskTracker software will run on
each of these nodes. Note that since the
TaskTracker runs on the same machine as the data
nodes, the Hadoop framework will be able to have
the map tasks work directly on the pieces of data
that are stored on that machine. This will save
a lot of network traffic. As we saw, each Mapper
processes a portion of the input data. That's known
as the input split. And by default, Hadoop use an
HDFS block as the input split for each Mapper.
It will try to make sure that a Mapper works
on data on the same machine. If this green
black, for example, needs processing, then the TaskTracker on this
machine will likely be the one chosen to process
that block. That won't always be possible, because the TaskTrackers
on these three machines that have the green block
could already be busy. In which case, a different node
will be chosen to process the green block, and
it will be streamed over the network. This actually happens
rather rarely. So the Mappers will
read their input data. They'll produce intermediate
data, which the Hadoop framework will pass to the reducers, remember that's the
shuffle and sort. Then the reducers process that data and write their final
output back to HDFS. So let's have Ian run a job on our cluster.