So in the following lectures, we're going to learn how to process large amounts of tabulated data using
Python and more specifically using Python package that is called pandas, which is very efficient in
processing large amounts of data.
And that also provides you with a lot of tools that are very handy if you want to run statistical analysis.
But before we're going to start coding, we want to download our dataset and you can do this by pressing
on our hyperlink here in our Sandbox AI Python notebook.
And then we land on the page of Kaggle and we can just press here on download.
And if you're not signed up, you can simply sign up and download the dataset then.
So the sign up is for free.
So I would encourage you to simply sign up with your email or whatever your preferred way of signing
up is.
So, yeah, once you signed up, you can go ahead and press, download and download the dataset.
And then we have this CSV file here that is called peer reviews.
And I want to take it and put it into this data folder, which is in the same folder as my sandbox in
Python notebook and I want to drag it in there.
OK, so once I'm there, I want to go back to our Python notebook.
What are we going to do is that we are going to process this file and this file basically contains a
lot of reviews about different types of beers.
So if you can open this CSV file in our Python notebook environment and you can see that here, we have
different columns which contain different parameters like the idea of the brewery, the name of the
brewery and at which time the beer was reviewed.
What the overall review of the beer was and which kind of style this beer is, or which kind of style
this beer belongs to.
And what's the name of the beer is and what the idea of the beer is and so on.
And we want to process this very large dataset, which is probably not the way a very typical dataset
that we would process in sciences.
But in this lecture, we just want to focus on the tools that Python provides you that allow you to
process large datasets efficiently and which kind of data you analyze in the end doesn't matter so much
or doesn't change so much in the end and about your code that you're going to implement.
So, yeah, let's have a look.
Let's explore our data a little bit more.
So we go back to our sandbox file and we do our standard imports.
So important I'm pay and not plot lib.
And now I also want to import pandas as speed, which is the main package that we're going to use in
the following videos, which is a package that is for analyzing efficiently large amounts of data and
more particularly large amounts of tabulated data.
OK, so as a first thing, I want to specify my file path to the to my database.
So my father passed this norm path.
And then I want to go to the folder that is called data.
And then to the file that is called peer reviews.
Does you see?
OK.
So an on path I use as it normalizes the way of how you write paths independent of the operating system
in which you are working.
This is sometimes useful if you are not sure if your code is going to be executed on Windows or on Linux
system, for example.
So it's just a way to to make your code a little bit more robust.
So we have to imported from always the path.
And then we can execute it.
I would Typekit here, so no path.
Yeah, and now I want to load my database as a so-called pandas dataframe.
So I want to call this dataframe beause and then I write PD dot read CSB file path.
OK, so pandas, dataframe, if I just printed here, is basically you can think of it as an excel sheet
or as some kind of tabulated data type, which consists out of column names.
So these are my column names here, and it consists also out of row row indices, which I see down here.
And then basically, I have an entry for each of my column, name and index.
So it's very similar to how you would represent your data also in an Excel spreadsheet or in a Google
spreadsheet or whatever your preferred way of looking at these kind of tabulated data is.
And what's also useful is sometimes to just show basically your first or your last three rows so I can
see print something like pierced or tail and equals three to show the last three rows.
Then I can see, OK, my index this year one million five hundred eighty six thousand six hundred thirteen,
which means that I have so many entries and these are my last three entries that I have in my CC file.
OK.
So in order to make things a little bit easier, we don't want to keep all of the columns, but we want
to drop the ones that we don't need.
And the goal of this lectures of the following lectures is going to be to identify the best and the
worst fears.
And for this, we want to keep focus, especially on the overall review.
As for now, this is our parameter of interest.
So the columns that I want to keep are my brewery name, as I might want to know from which brewery.
My my beer is, then I want to keep my review overall and my beer style.
My your name.
And my beer, your ID, be on Oscorp, your I.D. and then I can simply say for years, which is my dataframe
is pierce at these columns, and if I now look at Pierce the tail and equals three, there was an error.
Yeah, so here I have my column must be named.
We will rename.
And then I can show me the dataframe, and I can see that now I have only the columns that prove your
name.
Rubio, overall your style, your name and your I.D. And this makes processing a little bit easier for
us and a little bit better to understand the following steps.
OK, so one other thing that I want to do is I want to rename this idea to simply be your ID as I find
this more intuitive and we can do this by saying verse equals beers rename.
So I want to rename something and more particularly, I want to rename columns and now I can pass addict
and I can say I want to rename the beer beer ID column to simply beer ID.
And then again, I show my last three entries.
OK.
And now I have my column names exactly as I want.
Great.
Then another very useful tool that pandas provide your pandas dataframe parade you is, they call it
so-called described function.
So you can enter something like used to describe and then it describes it summarizes basically the main
statistical parameters that for each of the columns.
So here for the column review overall and here for the column your ID.
Of course, if you're a column type is a string, for example, it doesn't really make sense to to put
a statistical measure to it.
But for the reviewer for it, overall, you can, for example, see, OK, the count of it, you have
1.5 million counts of 1.5 million entries, more or less.
The mean review overall is three point eight.
So this is your mean rating.
Basically, the standard deviation is zero point seven to the lowest rating is zero and the highest
rating is five.
And then you have here your twenty five percent percentile of your 50 percent percentile and just seventy
five percent percentile for the bride.
Also these quantities, but this is basically because I just came out knowing that the idea is not a
value that whereas statistical analysis would make sense.
So in our case, we can simply ignore it.
And from this, we can already see that, for example, our review goes from zero to five stars, and
we have an average rating of three point eight starts with a standard deviation of 0.7 stars, which
is already quite useful.