So to implement MapReduce code with .NET, we use a technique called streaming. And Hadoop
streaming is an ability of Hadoop to pass data between the YARN job coordinator that’s
managing the MapReduce processing and mappers and reducers via streams. So, we’re passing
the data that’s operated on using the MapReduce processing as a stream of data and it can
be read through standard input and output interfaces by applications. So when we’re
building a streaming MapReduce solution using .NET, what we do is we implement console applications,
which simply read from a standard input buffer and write to standard output buffer. So, the
mapper reads the source data that is passed to it to its input buffer, and it writes key
values after it’s done the initial mapping of the data, and passes that back to the job
coordinator. The reducer then reads the key values that are passed through it by the job
coordinator and writes back the results, which are then written to a file by the YARN process
that is managing the MapReduce job.
So in this technique, we can write very simple console applications that do the job of mapping
and reducing the data, and then deploy them to the cluster and use that to implement a
MapReduce solution.