Welcome back, everyone.
We're going to continue our lecture series on useful methods on Pande, the series and data frames.
And in part three, we're going to talk about just general description methods and sorting methods.
A lot of these methods are actually self-explanatory.
So we're going to go through quite a few in just a little bit of time.
So let's head back to the Jupiter notebook and explore these methods further.
All right.
Here I am in the same chapter, a notebook that we left off in last time.
Well, I'll go ahead and do just kind of reset everything by saying the F is equal to PD.
Read CSFI and I'm going to read in that tip Slutsky as we file again, just so we can have a fresh slate
here.
And let's start talking about just various method calls.
A lot of these are pretty simple.
And some of these you've even seen before, such as that describe, which just shows you the general
description of statistical methods.
And as we previously discussed, sometimes these methods don't make sense for every column.
For example, it doesn't really make sense to take the stand deviation for the credit card number column
because the credit cards don't really have I mean, there should be some kind of unique identifier versus
it does make sense to take the average total bills or standard deviation of total bills and so on.
So we've seen this grow before.
I also mentioned that I personally like to transpose this sometimes to make it more readable, depending
on what I'm actually looking for.
Now the question comes up how to actually sort my data frame.
Well, what you can do here is simply say D.F. sort values and then you provide the column that you
want to sort by.
So if I wanted to sort by the tip column, I say sort values and now it sorts the tip column starting
at the lowest all the way to the highest.
And if you want to sort the other way around, essentially have a spending equal to false, then you
simply clarify that.
So by default, it goes in ascending order from lowest, the highest.
But you can also say assenting false to go in descending order from highest to lowest.
Something to know is that it doesn't actually reorder your data frame in any way.
It just sorts it by whatever particular column you're looking at.
You should also know that you can actually sort by various columns or more than one column.
So what I mean by that is, for example, we can see here that if I just sort by.
The tip column.
So the first couple of tips are all the same, it's one dollar, one dollar, one dollar, one dollar.
So that means I can sort first by the tip column and then sort by another column.
Let's go ahead and sort then by the size of the party.
And we notice we passed these columns in as a list.
So what this does is it starts first by the tip and there happens to be duplicate tip values.
Then it will sort by the next column over in this case size.
So if I run this notice right now, it's one to two, one for the size.
And I would have gone ahead and done is it sought by the tip and then for each of those same tip values,
it's sought by the next column, which in this case would be size since I wanted to sort by size.
So it goes one one to two and so on.
So this allows you to sort not just buy one column, but multiple column values.
Next, I want to show you how to grab the index locations of the men and max values.
So.
I can simply say.
Total Bill and I get this series of total bill values, if I want the max total bill value, then I
call Max.
If I want the index location of the max value, then I call IDEX Max.
So I say IDEX Max.
And this actually returns the index location of where this max value is happening, which means I could
then sadaf the ILFC Passan 170 and I get back that particular series for that max value and it's the
same thing for Minimum's.
I say grab the minimum or IDEX.
Men to get back to that location, and then I could always pass that into six or seven.
So as you can imagine, I could then grab this guy and pass that in here instead instead of actually
providing six or seven manually.
So this will just grab the entire data framework for whatever the total bill is, the minimum, and
it returns back to the same results.
So again, it's Min Max and then IDEX Max and the X-Men.
So grabbing the minimum maximum values and then also grabbing the integer location, we can also do
a quick correlation check.
So there the IDF that CEOs are functionality and this returns back a pairwise correlation of columns.
So essentially it just tells you how correlated column values are of each other.
And this in general only works with numeric columns since you're doing something like a Peerson correlation
coefficient and you can check out the lecture notebook for a Wikipedia link on correlation and what
it's actually doing.
It's just saying how correlated these values are together.
So do they go up together and down together or when one goes up, the other go down or do they not have
any correlation?
So the more around zero, so the max value is one.
So total bill is perfectly correlated with itself and then the minimum value would be negative one.
Essentially these two values move in exact opposite to each other.
So on one side the other's low and the range is negative one to one.
And always you'll see along the diagonal these ones because each column is perfectly correlated with
itself.
And then you also notice that there's essentially duplicates across the diagonal because you're calling
things twice.
TYP versus size should have the same correlation as size versus tip.
So, again, you can grab those correlations with this core method call.
So now that we know correlation checks, I'd xmen an idea XML.
I want to show you the value counts method.
It's a very useful way to quickly get a count per category and this only makes sense to call on a categorical
column.
So if I take a look at my data frame again and I call D.F. Head here notice, for example, I have the
sex column, which is categorical.
It's either a female or male.
It's not a continuous value here, which means let's imagine I wanted to figure out how many males I
had and how many females I had, essentially, how many counts do I have per possible value?
The way we do this is you simply call the column you're interested in that's categorical, such as a
sex column.
And then say value counts, run this and it's going to return back.
You have 157 rows that are male and eighty seven rows that are female and in a similar vein to the value
counts method.
There's also the unique method and the end unique method.
So maybe you have a column such as the day column where we can say DFG Day.
And if you run this, you're not actually sure what days are present in your data.
Well, what you can do is say.
Unique.
And it returns back the unique days that are present here, data set in this case that Sunday, Saturday,
Thursday and Friday, we can also do than say and unique and that just reports back the number of unique
values.
So there was four unique days there.
That's the same thing as just asking for the length.
Of this unique array.
And then if you again, wanted to figure out, well, how many instances are there of each day, you
could say value counts on this and it shows you how many instances of Saturday, Sunday and so on.
So you have unique and unique.
And if you actually want to count these unique values, you can say, go ahead and give me the value
counts.
Now, I want to talk about switching out or replacing values.
I'm going to show you two methods.
One is the replace method and the other one is the map method.
And they're kind of similar.
One uses a dictionary and one can use a list of values.
So let's first show you the replace method.
So if I take a look at.
Head, let's imagine, I wanted to replace female and male with just the letters F and M, so there's
a couple of different ways I can do this all for show you just how to replace a single value.
Sometimes there's a value that's just incorrect or you want to switch out in your data frame.
So that's actually really easy to do.
You just grab whatever column you're interested in.
Say replace and then the two parameters, it says, what do you want to replace and what's the value
you want to replace it with?
So, for example, I can say.
We're where we have female go ahead and replace it with just the letter F, so you run that and you'll
instantly notice we have AIFS instead of females.
This can also take in a list of values.
We can say, OK, it female.
And then male, and they just make sure that the next list is in the same order, so then you'll say,
OK, F is for female and then go ahead.
And say, Ms.
ReMail, you run that and then that will go ahead and replace multiple values, that's the replaced
method, I highly recommend the replace method if you just are going to replace a single value.
So, for instance, maybe you had just some random no values you can easily replace no values with like
a zero or something like that using the replaced method call.
Next, I want to show you the map functionality.
The first step to use map is you actually create a mapping as a dictionary, so instead of passing in
these lists here, you actually say, all right, female should be maps to F comma and male should be
mapped to M.
So you create this mapping dictionary and then you call sexier and you go ahead.
And in your mapping dictionary, so this will then go ahead and map these values to their matching values
in this column, and you get back the same result.
So I recommend a dictionary approach.
If you're going to do this for a lot of items, I think it's easier to read this way and then replace.
I recommend that if you just intend on replacing one or just two particular values and you want to keep
the rest the same.
So obviously you can kind of interchange these, but replace is easier with fewer items you intend to
replace.
Mapping, I think is easier if you intend to replace lots of items.
And we just have a few more methods to show you here.
Sometimes people ask me how you can deal with duplicated rows.
So in this case, if you have duplicated rows, you can say D.F..
Duplicated and what this does is you'll notice each of these are unique, so there's actually are no
duplicated rows in this.
But if you had a duplicated row, it's going to return true for the very first instance of a duplicated
row.
So let me show you by creating a very simple data frame that actually has duplicates.
I'll say that data frame here, let's say one to two, and then we'll say the columns are a actually
the index names excuse me, are ABC.
So here's my simple data frame, I have one to two if I ask for the duplicates on this, so I say go
ahead and tell me where it's duplicated.
Notice it's going to return true on the very first duplication.
So that means if I have another set of tools here, let's go ahead and have the speed and I run this
again.
So now I have the first unique then the first duplication, the second duplication I run.
This is going to start returning true on and after the first application.
So just keep that in mind.
It's going to mark the subsequent ones as duplicates versus the first ones, which is typically the
behavior you want.
And if you wanted to drop the duplicates, all you need to do is say.
Drop.
Duplicates just using time to complete this, and then it's going to drop subsequent duplicates.
And just a few more recalls left and then we finish off this section.
Last one I want to show you that sometimes useful is the between method call that actually will tell
you which values in a particular column are between two values you choose.
So I have a total bill column and let's imagine I wanted to know the value for the total bill that happened
to be between 10 and 20 dollars.
Well, I simply call between.
And then what I'm going to do here.
Is passing my lower limit, let's say ten dollars and my 20 dollar limit, and then if I want it to
be inclusive of these end points.
So if a bill happens to be exactly ten dollars, I do want that included.
We can say inclusive.
Is equal the true we run that and then it will tell me, true or false, is this value of the total
bill between 10 and 20 dollars inclusive of the endpoints.
So now that we have that, what I could do if I wanted it, just use this as a filter itself.
So if I Sadaf and then pass this in as a boolean filter, essentially using conditional filtering that
we previously learned about, now it gets me only the total bills that happen to be between 10 and 20
dollars.
Very useful tool here.
OK, so that's the between method.
And then finally, I'm going to show you how to grab and largest and smallest.
So what I mean by that is we say D.F..
And largest, so this grabs the first and number of rows ordered by columns in descending order.
So what does that actually mean?
Well, let's say I wanted the ten columns or excuse me, 10 rows that had the largest tip.
I run that and now I get back here at my 10 roads that had the largest hit, if I want to, just the
first two large steps we run, that this is essentially just doing a sort value and then calling ahead.
So you can imagine this is the exact same thing as saying D.F..
Sort values.
By tip.
And then just saying, go ahead and just bring me back, let's say zero to two, and in this case,
we'd want ascending to be false.
So this is just a faster way of doing this, and as you can imagine, there's also an smallest.
Which gives them back the two smallest tips, which in this case would be the same thing as if I said
here, ascending is true.
And here are the same pieces of information, so there's and smallest and largest, just give me back
the end number of smallest or largest values for any particular column.
And last one I want to show you is sampling sometimes.
Just want to take a random sample of a data frame.
You can call D.F. Sample.
And what this does, we take a look at here, there's two ways of doing this, which is really useful.
You can either sample back a certain number of rows or sample back a fraction of your data frame.
So what this means is if I just want to sample five random rows of my data frame, I say sample five
and this will return five random rows.
So each time you run this, you'll notice you should get back five different rows each time unless you
get super lucky.
So this is again, sampling five random rows.
Or the other thing I could do is say D.F. sample.
And say, go ahead and sample based off a fraction between zero and one, so this will sample just 10
percent of the rows of my data frame.
This is really nice because then I don't actually need to know how long or my rows in the data frame
are.
I'm just going to say go ahead and return back a sample of 10 percent.
So grab 10 percent of rows in that table randomly.
OK, and that's it.
For all the useful methods, again, definitely use the notebook as a quick and useful reference for
yourself.
Thanks.
And I'll see you at the next lecture.