Hi, and welcome to this course. You're probably here because you have some need to process huge chunks of
data in order to extract insights. You are in the right place. As a programmer, there is a core
understanding that you need to develop before you can dive into managing big data.
That is the art of thinking parallel. And what helps you develop this art?
The MapReduce programming model.
In this module, we start off by understanding the need for distributed computing and what role MapReduce
plays in the distributed computing setup. We'll build up to why MapReduce is useful.
We'll then see a quick sampling of the kind of problems that MapReduce solves.
And then understand the typical flow of a MapReduce task. Where is this MapReduce technology used?
And why do I need to learn it? The Facebook social network, Google Search, and LinkedIn Member Analytics.
These are products we work with every day. More than a billion users, in fact, log on to Facebook every
month, and it's just growing. But even then, even though they have to process records for a billion users,
Facebook takes just a few milliseconds to determine how many friends you have in common with any other user.
This is a pretty big deal. Let's go onto Google Search. The internet comprises of more than 40 billion webpages.
It's even hard to fathom how much information is out there. However, Google Search just seems to be getting
faster day by day. You never have to wait very long to get relevant search results for your query.
Such results are often returned in less than a second. Google in fact is so proud of how fast its search
runs that it advertises publicly how long it took for any search to be performed.
LinkedIn Member Analytics, a more obscure product for sure but nonetheless impressive.
LinkedIn has about 450 million users. Half a billion. This number is also growing albeit a little slowly
compared to Facebook and others. However, whenever you go to your LinkedIn page, you often get to see
interesting insights about your profile. These act as a hook drawing you back to LinkedIn in over and over again.
These products are amazingly creative, super useful, and the technology behind them is very interesting as well.
That's why we are here today. These insights are extracted from a data set that is huge, petabytes in size.
All the random interactions that you have with Facebook, all the connections you have on LinkedIn,
all the comments that you've made on various messages, all of these
come together to give you one clear insight, something that's interesting and engaging and will bring you
back to use their products over and over again. What makes this quick lookup possible is the power of indexes.
So if you step back and think about it, there are two things that you need here.
You need to be able to get the relevant information in a form that can be retrieved quickly.
That is exactly what the index structure is for, quick retrieval in a matter of milliseconds.
But if you think about how this index is built, that is the second challenge to be solved.
The raw data available to these sites to generate information looks somewhat like this.
I am a user. I visited a couple of profiles every day, multiplied this by 450 million.
I get billions of rows of raw data, which need to be summarized and sorted to get it into a form that's
relevant and useful. That is the whole process of building an index.
An index will have the member names and a quick lookup of useful information, in this case, the profile views.
And indexes in such websites when we deal with this amount of data are huge.
These are massive indexes. Since it's pretty clear that building these massive indexes are a hugely data
intensive task, no wonder we require distributed computing and parallel processing.
And these indexes, they have to be ground out over and over again.
You want the latest search results. You want information about your latest profile views.
Data cannot be state, which means building an index is repetitive.
Big data systems are optimized for processing data intensive repetitive tasks.
The technology behind such a system has some pretty heavy lifting to do.
The obvious requirements of such an infrastructure is the ability to store and process huge data sets in a
timely manner. There is a third requirement though, which is, in fact, the most important
in my opinion. In order to understand this third requirement, we've got to cast our mind's eye back to
what we considered big data in the early 90s and 2000s. At that time, a few gigabytes of data was considered
a big deal. But now the rate of growth of the number of users of these sites and the amount of data they
generate is huge. Data grows so fast nowadays that it's not enough to build for today.
Our infrastructure has to support tomorrow, which means the infrastructure needs to keep up with the growing
size of data.
The system needs to scale with data.
If you had a system which deals with, say, one million records really well, and tomorrow you have 50 million
records in your database, it would be pretty annoying, and not to mention expensive, if you had to entirely
change your system to deal with the new scale. You want your current system to adapt to the new requirements.
You want it to grow along with the data, which is why the architecture of distributed frameworks like Hadoop
are built in such a way that they can grow as your data grows.