‫Welcome back, everyone.
‫We're going to continue our lecture series on useful methods on PANDAS series and data frames.
‫And in part three, we're going to talk about just general description methods and sorting methods.
‫A lot of these methods are actually self explanatory.
‫So we're going to go through quite a few in just a little bit of time.
‫So let's head back to the Jupyter Notebook and explore these methods further.
‫All right, here I am in the same Jupyter Notebook that we left off in last time.
‫What I'll go ahead and do is just kind of reset everything by saying DF is equal to PD, read CSV and
‫I'm going to read in that tips that CSV file again just so we can have a fresh slate here.
‫And let's start talking about just various method calls.
‫A lot of these are pretty simple and some of these you've even seen before, such as DF describe, which
‫just shows you the general description of statistical methods.
‫And as we previously discussed, sometimes these methods don't make sense for every column, for example,
‫it doesn't really make sense to take the standard deviation for the credit card number column, because
‫the credit cards don't really have a mean there should be some kind of unique identifier versus it does
‫make sense to take the average total bills or standard deviation of total bills and so on.
‫So we've seen the scribe before.
‫I also mentioned that I personally like to transpose this sometimes to make it more readable depending
‫on what I'm actually looking for.
‫Now the question comes up, how do I actually sort my data frame?
‫Well, what you can do here is simply say DF sort values and then you provide the column that you want
‫to sort by.
‫So if I wanted to sort by the tip column, I say sort values and now it sorts the tip column starting
‫at the lowest all the way to the highest.
‫And if you want to sort the other way around, essentially have a sending equal to false, then you
‫simply clarify that.
‫So by default, it goes in ascending order from lowest to highest.
‫But you can also say ascending false to go in descending order from highest to lowest.
‫Something to note is that it doesn't actually reorder your data frame in any way.
‫It just sorts it by whatever particular column you're looking at.
‫You should also know that you can actually sort by various columns or more than one column.
‫So what I mean by that is, for example, we can see here that if I just sort by.
‫The Tip column.
‫So the first couple of tips are all the same.
‫It's $1, $1, $1, $1.
‫So that means I can sort first by the tip column and then sort by another column.
‫Let's go ahead and sort then by the size of the party and we notice we pass these columns in as a list.
‫So what this does is it sorts first by the tip, and if there happens to be duplicate tip values, then
‫it will sort by the next column over in this case size.
‫So if I run this notice right now, it's one, 2 to 1 for the size.
‫And now it's gone ahead and done is it sorts by the tip and then for each of those same tip values,
‫it sorts by the next column, which in this case would be size, since I want it to sort by size.
‫So it goes one, one, two, two and so on.
‫So this allows you to sort out not just by one column, but multiple column values.
‫Next, I want to show you how to grab the index locations of the min and max values.
‫So.
‫I can simply say.
‫Total bill and I get this series of total bill values.
‫If I want the Mac's total bill value, then I call Max.
‫If I want the index location of the max value, then I call it x max.
‫So I say ID X max.
‫And this actually returns the index location of where this max value is happening, which means I could
‫then say DF dot ILC pass in 170 and I get back that particular series for that max value.
‫And it's the same thing for minimums.
‫I say grab the minimum or ID x.
‫Men took it back that location, and then I could always pass that into 67.
‫So as you can imagine, I could then grab this guy and pass that in here instead instead of actually
‫providing 6 to 7 manually.
‫So this will just grab the entire data frame row for whatever the total bill is the minimum and it returns
‫back the same results.
‫So again, it's Minimax and then IDC's Max and IDC's Min.
‫So grabbing the minimum maximum values and then also grabbing the integer location, we can also do
‫a quick correlation check.
‫So there the DF.
‫That c0rr functionality and this returns back a pairwise correlation of columns.
‫So essentially it just tells you how correlated column values are of each other.
‫And this in general only works with numeric columns.
‫Since you're doing something like a Pearson correlation coefficient and you can check out the lecture
‫notebook for a Wikipedia link on correlation and what it's actually doing.
‫It's just saying how correlated these values are together.
‫So do they go up together and down together, or when one goes up, the other goes down?
‫Or do they not have any correlation?
‫So they're more around zero.
‫So the max value is one.
‫So total bill is perfectly correlated with itself and then the minimum value would be negative one.
‫Essentially, these two values move in exact opposite to each other.
‫So one one's high, the other is low, and the range is -1 to 1.
‫And always you'll see along the diagonal these ones, because each column is perfectly correlated with
‫itself.
‫And then you also notice that there's essentially duplicates across the diagonal because you're calling
‫things twice.
‫Tip versus size should have the same correlation as size versus tip.
‫So again, you can grab those correlations with this core method call.
‫So now that we know correlation checks, IDC's Min and IDC's out, I want to show you the value counts
‫method.
‫It's a very useful way to quickly get a count per category, and this only makes sense to call on a
‫categorical column.
‫So if I take a look at my dataframe again and I call DF head here, notice, for example, I have the
‫sex column, which is categorical it's either female or male.
‫It's not a continuous value here, which means let's imagine I wanted to figure out how many males I
‫had and how many females I had.
‫Essentially, how many counts do I have per possible value?
‫The way we do this is you simply call the column you're interested in.
‫That's categorical, such as the sex column.
‫And then say value counts, run this and it's going to return back that you have 157 rows that are male
‫and 87 rows that are female.
‫And in a similar vein to the Value Counts method, there's also the unique method and the end unique
‫method.
‫So maybe you have a column such as the day column where we can say DF de, and if you run this, you're
‫not actually sure what days are present in your data set.
‫Well, what you can do is say.
‫Unique.
‫And it returns back the unique days that are present in your data set in this case, it's Sunday, Saturday,
‫Thursday and Friday.
‫We can also do is then say end unique and that just reports back the number of unique values.
‫So there was four unique days there.
‫That's the same thing as just asking for the length.
‫Of this unique array.
‫And then if you again wanted to figure out, well, how many instances are there of each day?
‫You could say value counts on this.
‫And it shows you how many incidents of Saturday, Sunday and so on.
‫So you have unique and unique.
‫And then if you actually want to count these unique values, you can say, go ahead and give me the
‫value counts.
‫Now I want to talk about switching out or replacing values.
‫I'm going to show you two methods.
‫One is the replace method and the other one is the map method and they're kind of similar.
‫One uses a dictionary and one can use a list of values.
‫So let's first show you the replace method.
‫So if I take a look at.
‫DF head.
‫Let's imagine I wanted to replace female and male with just the letters F and M so there's a couple
‫of different ways I can do this.
‫I'll first show you just how to replace a single value.
‫Sometimes there's a value that's just incorrect or you want to switch out in your dataframe.
‫So that's actually really easy to do.
‫You just grab whatever column you're interested in.
‫Say replace.
‫And then the two parameters it says, What do you want to replace and what's the value you want to replace
‫it with?
‫So, for example, I can say.
‫Where?
‫Where we have female.
‫Go ahead and replace it with just the letter F.
‫So you run that and you'll instantly notice we have FS instead of females.
‫This can also take in a list of values.
‫We can say okay, wherever is female.
‫And then mail, and then you just make sure that the next list is in the same order.
‫So then you'll say oc f is for female and then go ahead.
‫And same as for mail.
‫You run that and then that will go ahead and replace multiple values.
‫That's the replace method.
‫I highly recommend the replace method if you just are going to replace a single value.
‫So for instance, maybe you had just some random null values you can easily replace null values with
‫like a zero or something like that.
‫Using the replace method call.
‫Next I want to show you the map functionality.
‫The first step to use map is you actually create a mapping as a dictionary.
‫So instead of passing in these lists here, you would actually say, all right, female should be mapped
‫to f comma and male should be mapped to M.
‫So you create this mapping dictionary and then you call sex here and you go ahead.
‫And pass in your mapping dictionary.
‫So this will then go ahead and map these values to their matching values in this column.
‫And you get back the same result.
‫So I recommend a dictionary approach if you're going to do this for a lot of items.
‫I think it's easier to read this way.
‫And then replace.
‫I recommend that if you just intend on replacing one or just two particular values and you want to keep
‫the rest the same.
‫So obviously you can kind of interchange these, but replace is easier with fewer items you intend to
‫replace.
‫Mapping I think is easier if you intend to replace lots of items.
‫And we just have a few more methods to show you here.
‫Sometimes people ask me how you can deal with duplicated rows.
‫So in this case, if you have duplicated rows, you can say df.
‫Duplicated.
‫And what this does is you'll notice each of these are unique.
‫So there's actually are no duplicated rows in this.
‫But if you had a duplicated row, it's going to return true for the very first instance of a duplicated
‫row.
‫So let me show you by creating a very simple dataframe that actually has duplicates.
‫I'll say PD data frame here, let's say 1 to 2 and then we'll say the columns are a or actually the
‫index names.
‫Excuse me are ABC.
‫So here's my simple data frame.
‫I have 1 to 2.
‫If I ask for the duplicates on this, so I say Go ahead and tell me where it's duplicated.
‫Notice it's going to return true on the very first duplication.
‫So that means if I have another set of twos here, let's go ahead and have the speed and I run this
‫again.
‫So now I have the first unique, then the first duplication.
‫The second duplication.
‫I run this.
‫It's going to start returning true on and after the first duplication.
‫So just keep that in mind.
‫It's going to mark the subsequent ones as duplicates versus the first ones, which is typically the
‫behavior you want.
‫And if you want it to drop the duplicates, all you need to do is say.
‫Drop.
‫Duplicates.
‫Just using tab to autocomplete this and then it's going to drop subsequent duplicates.
‫And just a few more method calls left, and then we finish off this section.
‫Last one.
‫I want to show you that sometimes useful is the between method call that actually will tell you which
‫values in a particular column are between two values you choose.
‫So I have a total bill column.
‫And let's imagine I wanted to know the values for the total bill that happened to be between ten and
‫$20.
‫Well, I simply call between.
‫And then what I'm going to do here is pass in my lower limit, let's say $10 and my $20 limit, and
‫then if I want it to be inclusive of these end points.
‫So if a bill happens to be exactly $10, I do want that included.
‫We can say inclusive.
‫Is equal to true.
‫We run that and then it will tell me true or false is this value of the total bill between ten and $20
‫inclusive of the endpoints?
‫So now that we have that, what I could do if I wanted is just use this as a filter itself.
‫So if I say DF and then pass this in as a boolean filter, essentially using conditional filtering that
‫we previously learned about, now it gets me only the total bills that happen to be between ten and
‫$20.
‫Very useful tool here.
‫So that's the between method.
‫And then finally, I'm going to show you how to grab and largest and smallest.
‫So what I mean by that is we say DF.
‫And largest.
‫So this grabs the first number of rows ordered by columns in descending order.
‫So what does that actually mean?
‫Well, let's say I wanted the ten columns or excuse me, ten rows that had the largest tip.
‫I run that and now I get back.
‫Here are my ten rows that had the largest tip.
‫If I want to just the first two large steps we run that this is essentially just doing a sort value
‫and then calling ahead.
‫So you can imagine this is the exact same thing as saying DF.
‫Sort values.
‫By tip.
‫And then just saying, go ahead and just bring me back, let's say 0 to 2.
‫And in this case, we'd want.
‫Ascending to be false.
‫So this is just a faster way of doing this.
‫And as you can imagine, there's also smallest.
‫Which gives me back the two smallest tips, which in this case would be the same thing as if I said
‫here ascending is true.
‫In here the same pieces of information.
‫So there's smallest and largest.
‫Just give me back the number of smallest or largest values for any particular column.
‫And last one I want to show you is sampling.
‫Sometimes you just want to take a random sample of a dataframe.
‫You can call def sample.
‫And what this does, if we take a look out here, there's two ways of doing this, which is really useful.
‫You can either sample back a certain number of rows or sample back a fraction of your data frame.
‫So what this means is if I just want to sample five random rows of my data frame, I say sample five
‫and this will return five random rows.
‫So each time you run this, you'll notice you should get back five different rows each time unless you
‫get super lucky.
‫So this is again sampling five random rows.
‫Or the other thing I could do is say def sample.
‫And say go ahead and sample based off a fraction between zero and one.
‫So this will sample just 10% of the rows in my data frame.
‫This is really nice because then I don't actually need to know how long my rows in the data frame are.
‫I'm just going to say go ahead and return back a sample of 10%.
‫So grab 10% of rows in that table randomly and that's it for all these useful methods.
‫Again, definitely use the notebook as a quick and useful reference for yourself.
‫Thanks, and I'll see you at the next lecture.