Welcome back.
In the last video we saw an end to end psychic loan workflow.
Now we're gonna break that down and jump into each of those steps individually.
The first one is getting the data ready.
You probably noticed I've kind of list to find this just because I'm a bit of a nerd so that way we
don't have to scroll right back up to top to see what we're covering.
We can just put this here this little cool trick wherever we want in the Jupiter notebook because we've
run this cell and instantiated this as a list variable.
So we're going to tick off getting the data ready in this section.
And another thing we probably should have done right at the top.
We can usually do it in any machine learning notebook that we're working on.
So we'll just type in here standard imports just so we have them ready at our arsenal.
We've already done some pie pandas as PD import map bot lib dot pie plot as TLT.
And then we'll do a little plot lib inline so our plots appear in the notebook.
So this is this three four lines of code that you can run at the top of most notebooks and then you'll
see throughout as we start to use different socket loan functions we could also start to put them up
the top two but we'll leave it at those for now.
So let's go down to where we were.
This section we're getting our data ready.
Now the reason we have to do so is because most the time data doesn't come ready to be used with a psychic
line machine learning model.
Now Grant a little heading.
So we have to get it ready.
And getting data ready to be used with machine learning and then the three main things that we'll have
to do.
Number one is let's communicate a bit better Daniel.
Three main things we have to do is one split the data into features and labels usually x and y.
This is what you'll find them cold.
Generally if you're on the internet somewhere or this like a loan library usually calls features x and
labels y and then number two is feeling also called imputing or disregarding missing values.
So if any of our rows in our data set have missing values if there's any incomplete fields we may have
to fill them up or we may have to get rid of them completely.
Those samples because a machine learning model can't learn when there's nothing there.
You'll see it throws an era then the final one is converting non numerical values to numerical values
also called feature encoding a.k.a. if we have say we have our car sales or we need to turn that into
markdown beautiful.
So if we have our car sales data and we have Toyota we have the color et cetera.
A machine learning model can't understand Toyota.
What have I done there.
Can't understand Honda.
Come and stand read it.
We have to turn these into numbers and we'll see how to how to do that.
In this section.
All right.
Well we need a data set to begin with first and I think we still have our heart disease data set imported.
So let's check that.
Go ahead.
Wonderful.
So where do step one first we've seen this before in our workflow but we'll just get a succinct version
of how to actually do it.
So in this case we want to use the feature columns to predict why.
So what are we doing here.
We're keeping it nice and simple we use pandas X equals heart disease don't drop.
So we want to remove the target column along Axis 1.
And now remember in a panda's data frame.
Access equals one means.
This access here all the columns axis and axis 0 is the Rose axis.
So X is now going to be every single column except for Target wonderful and you know what our Y is going
to be.
Because that's the labels of our machine learning problem.
Our Y is going to be the target column.
Heart disease and we'll select it like that.
We'll look at why dot head was here first five samples.
Wonderful.
We go back up here one on one on one.
You see why is so beautiful.
Now the next thing we have to do is split data into training and test sets.
So in machine learning one of the most fundamental principles is never evaluate or test your models
on data that it is learned from.
Which is why we split it into training and test sets so suck it loan has a convenient function for allowing
us to do that.
So split the data into training and test sets.
Remember right back at the start when going over concepts if you're looking at the test data it's like
looking at the final exam before you've looked at the practice exam not what you want to be doing right.
If the professor accidentally leaked the final exam everyone would be getting perfect marks and no one
will be actually learning anything.
So we want to from S.K. learn model selection SCA line has a fair few different modules but we'll see
some of the most useful ones.
Import train test split.
Beautiful.
Now when we call train test split it's going to return for different values.
One is X train one is X test one is y train and one is y test.
So let's see what happens.
Train test splint.
And then we pass it our X data and see what it says actually split arrays or matrices into random train
and test subsets.
Beautiful.
That's what we want.
We want some data to train on and we want some data to evaluate our machine learning models on its will
pass it out features and we'll pass it our labels and we'll define the test size as been zero point
two.
So let's see what happens.
Wonderful.
That runs smoothly.
Let's check out the shapes of our new matrices because remember our data here is really just a matrix
in a data frame or a nun pie in the array in a data frame.
So on X trained what shape will test x test shape y train shape.
And finally y test what shape.
Let's see this.
Okay so 242 with 13 columns.
So 2042 by 13 and 61 by 13.
Okay.
This makes sense.
So if we look up here as one two three four five six seven eight nine ten eleven twelve thirteen thirteen
different features.
That's why our X train variable is 242 by thirteen.
But where did this 240 to come from.
Well that's because we've decided here that we want our test data set to be 20 per cent of the overall
data.
So let's have a look at this.
If we go X dot shape because X is before we split it right we've got X up here X dot shape is 303.
So we have 303 samples total.
Let's just check that three a three samples total and 80 percent of them are going to be training data
for the machine learning model.
I trained up shape we want to times that by zero point eight to two hundred forty two point four.
We add these together turn 42 plus 61.
All it's done is it just rounded it down and automatically carve the other 20 per cent into the test
set and it's done the same for y train and Y test.
Beautiful.
Now we've split our data into training and test the next step.
We'll go back up here is to fill it or converting it.
Making sure it's all numerical.
So we might see this maybe we do that.
We fill it first.
Yeah that's a good idea.
We'll come back and we'll do that in the next video.
So maybe have a go at playing around with one of our CSB files and splitting it into training and test
data or splitting it into x and y first and then change around this parameter here to see what happens
when we change that two point three what you think will happen if I press shift enter here the numbers
will change.
Maybe you want to play around with a different fraction.
Otherwise I'll see you in the next video and we'll look at how to fill some missing data.