‫Welcome back.
‫In the last video, we saw an end to end psychic loan workflow.
‫Now we're going to break that down and jump into each of those steps individually.
‫The first one is getting the data ready.
‫You probably noticed I've kind of mystified this just because I'm a bit of a nerd.
‫So that way we don't have to scroll right back up to the top to see what we're covering.
‫We can just put this here, this little cool trick wherever we want in the Jupyter Notebook, because
‫we've run this cell and instantiated this as a list variable.
‫So we're going to tick off getting the data ready in this section.
‫And another thing we probably should have done right at the top, we can usually do it in any machine
‫learning notebook that we're working on.
‫So we'll just type in here standard imports, just so we have them ready at our arsenal.
‫We've already done numpy pandas as PD import map plot lib.
‫Dot pie plot as plot.
‫And then we'll do a little that plot lib in line.
‫So our plots appear in the notebook.
‫So this is this three or four lines of code that you can run at the top of most notebooks, and then
‫you'll see throughout as we start to use different psychic loan functions, we could also start to put
‫them up at the top, too, but we'll leave it at those for now.
‫So let's go down to where we were.
‫This section, we're getting our data ready now.
‫The reason we have to do so is because most of the time data doesn't come ready to be used with a psychic
‫loan machine learning model.
‫Now create a little heading, so we have to get it ready.
‫So getting our data ready to be used with machine learning.
‫And then the three main things that we'll have to do.
‫Number one is let's communicate a bit better.
‫Daniel Three main things we have to do is, one, split the data into features and labels, usually
‫X and Y.
‫These is what you'll find them called.
‫Generally, if you're on the Internet somewhere or the psychic loan library usually calls, features
‫X and labels Y, and then number two is filling, also called imputing or disregarding missing values.
‫So if any of our rows in our data set have missing values, if there's any incomplete fields, we may
‫have to fill them up or we may have to get rid of them completely.
‫Those samples, because the machine learning model can't learn when there's nothing there, you'll see
‫it throws an error.
‫Then the final one is converting non numerical values to numerical values, also called feature encoding
‫a.k.a.
‫If we have say we have our car sales, we need to turn that into markdown.
‫Beautiful.
‫So if we have our car sales data and we have Toyota, we have the color, etc., etc., a machine learning
‫model can't understand Toyota.
‫What have I done there?
‫Can't understand Honda, understand red.
‫We have to turn these into numbers.
‫So we'll see how to how to do that in this section.
‫All right.
‫Well, we need a data set to begin with first.
‫And I think we still have our heart disease data set imported.
‫So let's check that head.
‫Wonderful.
‫So we'll do step 1/1.
‫We've seen this before in our workflow, but we'll just get a succinct version of how to actually do
‫it.
‫So in this case, we want to use the feature columns to predict Y.
‫So what are we doing here?
‫We're keeping it nice and simple.
‫We use pandas x equals heart disease dot drop.
‫So we want to remove the target column along axis.
‫One.
‫And now remember in a pandas data frame axis equals one means this axis here or the columns access and
‫access zero is the rows access.
‫So X is now going to be every single column except for target.
‫Wonderful.
‫And you know what our why is going to be?
‫Because that's the labels of our machine learning problem.
‫Our why is going to be the target column heart disease.
‫And we'll select it like that.
‫We'll look at why not.
‫And so we'll see the first five samples.
‫Wonderful.
‫We go back up here.
‫One one, one, one, one.
‫We see.
‫Why is same beautiful now?
‫The next thing we have to do is split data into training and test sets.
‫So in machine learning, one of the most fundamental principles is never evaluate or test your models
‫on data that it is learned from, which is why we split it into training and test sets.
‫So second line has a convenient function for allowing us to do that.
‫So split the data into training and test sets.
‫Remember right back at the start when we're going over concepts, if you're looking at the test data,
‫it's like looking at the final exam before you've looked at the practice exam, not what you want to
‫be doing, right?
‫If the professor accidentally leaked the final exam, everyone would be getting perfect marks and no
‫one would be actually learning anything.
‫So we want to from Learn Model Selection, Ask, Learn has a fair few different modules, but we'll
‫see some of the most useful ones.
‫Import train test split.
‫Beautiful.
‫Now when we train test split, it's going to return four different values.
‫One is x train, one is x test, one is y train and one is y test.
‫So let's see what happens.
‫Train, test, split.
‫And now we pass it our x data and see what it says.
‫Actually split arrays or matrices into random train and test subsets.
‫Beautiful.
‫That's what we want.
‫We want some data to train on and we want some data to evaluate our machine learning models on.
‫So we'll pass it out features and we'll pass it our labels and we'll define the test size as being 0.2.
‫So let's see what happens.
‫Wonderful.
‫That runs smoothly.
‫And let's check out the shapes of our new matrices.
‫Because remember, our data here is really just a matrix in a dataframe or a numpy and de array in a
‫data frame.
‫So on x train shape we'll test, x test shape, y train, dot shape.
‫And finally, why test shape?
‫Let's say this.
‫Okay, so 242 with 13 columns, so 342 by 13, then 61 by 13.
‫Okay, this makes sense.
‫So if we look up here, there's one, two, three, four, five, six, seven, eight, nine, ten,
‫11, 12, 13, 13 different features.
‫That's why our x train variable is 242 by 13.
‫But where did this 242 come from?
‫Well, that's because we've decided here that we want our test data set to be 20% of the overall data.
‫So let's have a look at this.
‫If we go X dot shape because X is before we split it right, we've got X up here, x dot shape.
‫It's 303, so we have 303 samples total.
‫Let's just check that.
‫Trinity samples total, and 80% of them are going to be training data for the machine learning model.
‫So if we go train shape, we want two times that by 0.8.
‫So 242.4.
‫We add these together.
‫Turner, 42 plus 61.
‫All it's done is it just ran in it down and automatically carved the other 20% into the test set.
‫And it's done the same for y train and y test.
‫Beautiful.
‫Now we've split our data into training and test.
‫The next step we go back up here is to fill it or converting it, making sure it's all numerical.
‫So we might see this.
‫Maybe we do that.
‫We fill it first.
‫Yeah, that's a good idea.
‫We'll come back and we'll do that in the next video.
‫So maybe have a go at playing around with one of our CSV files and splitting it into training and test
‫data or splitting it into x and Y first and then change around this parameter here to see what happens.
‫And we change that to 0.3.
‫What do you think will happen if I press shift enter here.
‫The numbers will change.
‫Ah, maybe you want to play around with a different fraction.
‫Otherwise, I'll see you in the next video and we'll look at how to fill some missing data.