Today, we're going to go over multivariate data selection.
We're going to use the enhanced dataset that you've seen before.
So, first we're going to go ahead and import our NumPy and Pandas as np and pd as usual.
I'm going to import the data from our GitHub.
So, first of all, let's just go ahead and take a look at our dataset that we have.
We can see our variables here.
You'll notice that we have these continuing dots, these ellipses.
So, we can actually see all the variables we have.
There's a little trick to do that,
if you want to change that.
So, we're going to go ahead and change that so we can see all of our variables.
So, to do that,
we're going to set the pandas options.
We're going to change the display of the max columns.
We're going to set that to- I'm just going to say 100.
I don't think there's 100, but 100 is definitely good enough. So, we run that.
Then, if you look at the head of the DataFrame again,
you'll see that the ellipses are gone and we can actually
see all of the variables that we have now,
which is really convenient.
Okay. So, I'm going to just pick a few columns to look at.
So, we're trying now to just pick
a few columns that we're looking at and now look at all the data.
I'm going to go ahead and look at all the body measurement columns,
which is all of these ones, all of the ones with BM in front of it.
So, there's a couple ways to do this.
You can go ahead and look at all the column names at once by calling df.columns.
I'm going to go ahead and label
a list named keep because these are the columns I want to keep.
You can copy paste them if you want.
Once you've called these columns,
you can just select these,
then put them in here and that'll work.
We can go ahead and- our first type of selection is this column selection, which is,
if you put a list inside these brackets of the DataFrame,
then it'll go ahead and select only the columns that are listed in our list to keep.
Another way to do this,
which is maybe a little bit more pythonic,
is to only select the names that have BM in them.
So, I'm going to say I want to keep the column for each column that's
listed in all of the possible columns in our- possible columns in our DataFrame.
I'm going to keep it if BM is in the name.
So, you can see when I do this function,
this is called list comprehension, and it's saying,
"Keep the column that has the word BM in it from our total DF columns."
I'm going to go ahead and just call this keep as well.
This is giving me the same thing as we had before,
but instead of copying and pasting it,
I've done it in a very pythonic way.
Okay. So, again, we can go ahead and do this,
where you just use these brackets and put in a list of the columns we want to keep.
There's a couple of other ways that you can do this as well.
So, when using just the brackets and only putting a list in there,
it's only going to consider column names.
So, you can only put a list of column names into
these two brackets to get anything returned.
But another way you can do it is with this location function.
The location function takes in labels.
So, labels are like the actual indices labels here for rows,
or the column names for the columns.
So, for this one, what we can do is- we want to say we want to keep all the rows.
So, we're going to put this colon here,
which means take everything from the beginning to the end of the rows.
Now, we're gonna put keep in after the comma,
which means keep only those columns.
If you do that, we see we get the same DataFrame as we had above.
So, this DF this location saying
keep all the rows is the same as just specifying which columns that you want to keep.
There's a lot of columns or rows in here.
So, I'm just gonna go ahead and call head-on
both these functions to make them a little bit shorter.
Head, when you call it,
always defaults to five,
but if you only wanted to see a couple,
you could put a different integer in there
and it'll give you as many or as few as you wanted.
The other selection we can use is the iloc,
which is index location.
This one, instead of using labels like just location uses,
this is going to use the indices of it. So, let's see.
If we're going to use this, we first need to find
the indices of what columns the BM is in.
So, let's try and find those.
So, I'm going to say, the index that I'm going to use,
which is going to be a Boolean, and then I'll show you how that works in a second.
This is going to equal to- okay.
So, what I'm doing here, is I'm using NumPy and saying,
"Where are the points where all the keep values are in the column names?"
So, for all of the possible column names,
I only want to return True if that column name is also in keep.
So, if we wanted to look at this index,
that's going to return True and False.
So, it's going to return False everywhere that there's not BM inside the column names,
and True wherever it is it.
We're going to go ahead and use this to run our location one,
or index location and selection.
Again, we're going to get the same DataFrame as before.
So, what's happening here is for the index location and for the location.
You can put- both of these will accept a list of Booleans as possible input.
So, this one is saying,
"Only return where there is True in this list," which is,
we picked out specifically,
that it would be all of the column names with BM.
There's much more on what specifically these [inaudible] in the patent document,
which we're going to link to around these videos. So, you can take a look at it.
But just to be precise,
we have that the brackets are really column indexing and they use the column labels.
This is pretty similar to the location,
which also uses the column labels,
but you can also use- it also takes in row arguments.
You could also use Boolean arrays with these.
The iloc is the indexing locations,
and this is integer position based.
So, it's going to be only from 0 to one,
minus the length of the axes that it's used on,
but it can also use a Boolean array.
Okay. So, now we're going to look at some selections by condition.
So, let's go ahead and say that we only want in this DataFrame,
all of the rows that have the waist measurements larger than the median.
So, we want waists that are larger than median.
So, first, we're going to find the median.
We're going use this by calling
that Pandas-Function on the series which is Pd dot series dot median.
If you try and run the numpy.median,
you'll actually get an error, so I suggest using the one specific for pandas.
Where I get the median is 98.3.
So, let's go ahead and make our first condition,
which is that we want only for when the waist measurements are larger than the median.
So, I'm going to go ahead and call the condition one.
I'm going to let this equal to.
So, now we have our condition.
So, whenever I put this condition in,
it's going to ask this question,
is the waist bigger than the waist median?
You can go ahead and do this with just the single brackets without any rows.
So, we can do our dataframe and just ask,
is condition one met?
It's going to return the DataFrame where
all the rows that have condition one met as true,
and we'll just go ahead and look at the head because it's
much too much data to consider all at once.
That's a little bit easier to see.
So, what happens if we have multiple conditions?
So, here we only have one condition.
What if we add another condition?
Let's go ahead and add the condition that the leg measurement has to be less than 32.
Then, you can go ahead and just use the n symbol and say,
okay, is condition one met and is condition two met?
It will turn all the rows such that condition one and condition two are met.
It's really important to use the n symbol here and not type
and because you'll see very quickly that it will return an error.
This has to do with how and,
the word and n, the symbol work.
We're not going to go into too much of that right now,
but just to note that if you do get this error and you're using and,
the word, go ahead and try and switch it to
the ampersand and you'll get a better outcome.
So, how could we do this if we're using the location instead
of just the brackets with the column labeling?
It's going to be almost exactly the same except your condition one and condition two are
going to be put into the row inputs and not the column inputs.
So, we're going to go ahead and do the same thing.
Say, is condition one and condition two met?
Then we're going to say,
go ahead and check that for all the columns.
So, remember, when you're doing location indexing,
which I should probably put location in there,
specify that's what I'm doing.
The rows are always going to be come first and then it's comma and then the columns.
If you want to include all the rows of the columns,
you include this colon with nothing before or after it.
Let's go ahead and look at the small one again.
You can see that we have the same output through both of these.
So, both of these calls are doing the same thing.
One thing to note is that,
if your index is something different than just numbers,
then this location is really useful for calling specific indices.
So, let me give you an example here.
We're going to take a small DataFrames.
Let's go ahead and do, lets see,
we're just going to go ahead and take a slice or take the first five rows
really of our big DataFrame and go ahead and just call that our small DataFrame.
So, our entire small DataFrame is just five rows.
I'm going to change the indices of these now to be,
instead of integers, I'm going to make them letters.
So, let's go ahead and make these just the first five letters of the alphabet.
So, now we're looking at, you can see that the indices have changed.
Let's make it more obvious what the location function or the location selection is
doing because if I call d of small at location and I only want to look at row,
the first row, which is a and all the columns,
to make sure it's a string, so it knows what I'm talking about.
That's going to go ahead and return
just that first row and I can go ahead and look at the first two rows.
For calling a and b.
If for instant to call one and two and say,
I want to just take the first and second rows,
which is actually should be zero and one,
this is the second and third rows,
it's going to go ahead and give me an error.
Because one and two are not indices anymore because we've changed the indices to letters.
But if I were to instead do the index location,
this would not make sense and it's going to pull
the second and third row because that's taking a purely
by the integer value of where it is in the index and not the label.
So, you've seen a couple of the errors already that you can come across,
but we're going to go over a few more just so when you see these errors,
you're not confused because they're going to show up all
over the place when you first just start using these.
So, let's go ahead and look at this one.
What if I were to call,
we'll just do our small DataFrame.
I just want to do slicing.
I'm just going to say, I only want to look at the column that is the BMXBMI.
Well, that seems to make sense, but I got an error.
Because I said, okay, well,
I want all the rows and I only want this column, but remember,
you have to call the location to do this.
If you're used to using
other programming languages for statistics like if you've ever seen r,
then you might want to try and run something like this and it might not work,
which is why you have to remember that when you include the rows,
you either have to use the iloc or just the regular loc,
location or the integer locations.
Something that is useful, when you do return this,
you'll see that it's going to return the index and if you only want the values,
then there's a pretty easy function for that.
You can go ahead and just call values and it will only return
the array of the actual values for that slice.
So, another thing that you'll often come across as you're like,
okay, I totally know what I'm doing.
I'm going to use the rows and columns because I'm using the iloc
and so I'm going to go ahead and say I want all the rows and I want the BMXBMI columns.
So, you run in to an error again.
Let's see what it says.
Location-based indexing can only have integers.
So, we see here that this is not an integer,
like okay, now that you can see this, you remember.
I have to use only integers for this switch.
I want to say that this one is 22,
not actually sure if that's the BMXBMI,
but you'd have to go back and find the integer for
this specific column if you wanted to use the integer locations.
So, another area that's often come across
is when you're trying to change values in a DataFrame.
So, we can take our small Dataframe and say,
I want to change the values to the BMXBMI.
So, let's see what it is originally.
Then let's just say,
I want to make it one, two, three, four, five.
Well, I've already got an error here.
Well, it's not really an error because it did what I wanted to,
but it's saying that I should maybe use something different,
and this is something that's going to come across a lot.
Where certain of these calls that you do,
like this one with only the brackets and without the location or the index location,
if you try and change values using this,
you'll always come across this error.
Was saying is it's saying you're trying to set a value
on a copy of the slides from the DataFrame instead of the actual DataFrame.
You'll have to look a little bit more into the documentation of this
if you want to really understand why this is.
But the point of me showing you this, is that,
if you want to change values in your DataFrame,
you should always be doing it with either the location or the index location.
So, you can go ahead and just put the iloc right there or the location right there,
go ahead and add the rows and then if we do this again,
we're still getting the error, let's see.
So, the reason that it's giving me this error continually is because
my small DataFrame that I pulled from the large one is actually where the problem begins.
We have to make an entirely different DataFrame.
So, I'm going to go ahead and say that I want this to be.
Let's see, what do I want it to be?
Well, we can go ahead and just, we'll go ahead and look at
our large DataFrame and say that we only want to change a portion of it.
So, we're going to go ahead and change the first five rows.
So, we got rid of this error.
So, what's happening was that my original df small was actually,
it wasn't its own DataFrame,
it was just pointing to the original DataFrame.
So, I couldn't change that without also having some issues in our original DataFrame,
which is why it kept giving me these errors.
So, if I were to try and change the values
of the entire DataFrame using only the brackets.
They have to figure out how long this thing is.
I would do it. There we go.
Those are some of the main areas that you'll come
across and just keep in mind that whenever you're slicing the data,
if you're using labels,
you want to use the location and if you're using indexes,
you want to use the indexing location and if you're just picking by columns,
you can go ahead and just use the brackets. That's it for today.