All right before we start pre processing our variables we need to make sure we have the proper setup
to estimate our probability of default model in the best way possible.
A substantial issue we might face while estimating any statistical model is over fitting over fitting
means our statistical model has focused on a particular dataset so much that it has missed the point.
The opposite of overfishing is under fitting when the model fails to capture the underlying logic of
the data that is it didn't learn well so it doesn't know what to do and therefore it provides inaccurate
answers.
Let's explain this with graphs.
Here we can see several data points.
A good algorithm will result in a model that looks like this.
It's not perfect but it's very close to the actual relationship.
Okay we can certainly say a linear model would be an under fitting model.
It provides an answer but it does not capture the underlying logic of the data so it doesn't have strong
predictive power under fitted models are clumsy and have low accuracy.
You will quickly realize that either there are no relationships to be found or you need a different
model.
All right now let's check an overfed model.
Here it is.
As you can see over fitting refers to models that are so good at modeling the data that they fit or
at least come very near each observation.
The problem is that the random noise is captured inside and over fitted bottle since the probability
of default model is a logistic regression model.
Let's illustrate this with logistic regression overfishing problem.
Imagine we have this model on a sample of 10 points.
Now let's say somebody gives us 100 observations from a sample for the same population.
We quickly realize that the relationship has nothing to do with our findings in real life.
The case probably won't be as dramatic.
For instance our population data is likely to be distributed like this looks okay.
But all these points over here are misclassified because if the initial model after a quick examination
we realized that the real relationship is quite similar but say a bit to the left under fitting is easy
to spot.
You have no accuracy whatsoever.
Overfishing is much harder though is the accuracy of the model seems outstanding.
The most basic solution to overfishing is to split our initial dataset into two training and test splits
like 90 percent training and 10 percent test or 80 20 are common here's how it works.
We estimate a model on the train data.
Basically we hide a small part of the dataset from the algorithm.
So we train the model based on most of the data but not all of it.
After we have the model we tested on the test data by creating a confusion matrix and assessing the
accuracy.
The whole point is that the model has never seen the test dataset.
Therefore it can not over fit on it the python package SCA learn offers a pretty neat method of splitting
the data into train and test in order to set it.
We must imported though so from SCA learn model selection import train test split the train test flight
method is very powerful and has many arguments we could take advantage of the two most important ones
however are the input and the target data frames lets indicate them here.
Train test split parentheses our inputs are contained in the lone underscore data data frame for the
time being.
Let's assume they are everything in the data frame except the good underscore bad column.
So we drop the good bad column from the inputs and put them here.
Our outcome is the good bad column from the loan data data frame.
That's clear too.
That itself is enough for the method to split the data the output we obtain consists of four arrays
a trained dataset with inputs then a test dataset set with inputs the train data set what targets then
a test dataset with targets.
Obviously this output is of no use per say.
So let's declare for variables that will contain the four outputs.
We'll call them lone data inputs train loan data inputs test loan data targets train and loan data targets
test.
So let loan data inputs train loan data inputs test loan data targets train and loan data targets test
be equal to the train test method with arguments the inputs and the targets.
Okay great.
Let's see the shapes of these variables.
They will be very indicative of what the train test split actually achieved.
As you can see the training inputs are three hundred forty nine thousand seven hundred thirteen by 207.
While the training targets are three hundred forty nine thousand seven hundred thirteen comma.
This tells us that the training inputs contain three hundred and forty nine thousand seven hundred thirteen
observations along 207 variables.
While the targets are a vector of length three hundred forty nine thousand seven hundred thirteen.
What about the test.
It includes one hundred sixteen thousand five hundred seventy two observations 207 input variables and
one target variable.
So this method basically split the inputs and targets into matching forms.
While this would do we can also take advantage of the rest of the arguments that can be specified in
the train test split method.
The split we just got four hundred forty nine thousand seven hundred thirteen training and one hundred
sixteen thousand five hundred seventy two is three to one.
In other words 75 percent of the observations would help us with training and 25 percent would serve
for testing.
That's obviously the default split usually though we opt for splits like 90 10 or 80 20.
We don't like setting aside too much data for testing because this means training the model on less
data.
How can we specify that we can go back to the function and set the parameter test size equal to a certain
value.
Test size can take values between 0 and 1.
If we write zero point one 90 percent of the data will be used for training and 10 percent for testing.
If we write zero point to 80 percent of the data will be used for training and 20 percent for testing.
Usually I go for the 80 20 split.
Thus the test size argument becomes zero point two okay.
Is there anything else we'd like to tweak in this code.
Sure.
The train test split method has a shuffled parameter shuffle is a boolean so it can be either true or
false by default.
Shuffling is set to true.
So S.K. learn is quite small.
It doesn't take care of the split only but the shuffling too.
Unfortunately this may be a small issue for us when we rerun the code.
We get a different shuffle and this means a different split.
This causes the final model to differ every time due to the shuffling.
So we get slightly different accuracy every time and slightly different model coefficients too.
With that said this automatic shuffling can make things difficult for us.
There is a simple solution though all S.K. learn functions that include some randomness like the random
shuffle here contain a random stay parameter.
Random state takes integer values.
What we can do is set the random state to a number say 42.
This will make the shuffle pseudo random.
It will always shuffle observations in the same random way.
All right.
Finally let's check the shapes.
Once again we've got three hundred seventy three thousand twenty eight observations for training and
ninety three thousand two hundred fifty seven for testing.
So our 80 20 split worked as it should.
Good job.
In the next lecture we'll start the special pre processing of our data for the probability of default.
Model.