Hello In this lecture.
We will see how data preparation works as we pretty process data for one discrete variable discrete
variables don't need fine classing or splitting into categories because they are already categorical.
By definition for convenience let's create working data frames for all the pre processing led def underscore
inputs underscore pre processing be equal to loan data inputs train and Def underscore at targets underscore
pre processing be equal to loan data targets train.
Let's start with a rather simple variable grade.
We want to assess its explanatory power with respect to the outcome of interest that is being a good
or a bad borrower.
We also need the corresponding outcomes contained in the good bad column.
Now we can create a new data frame called DFA one where we will store only the independent variable
grade from the DFA inputs pre processing data frame and the dependent variable good bad from the IDF
targets pre processing data frame.
The values of the great variable are letters from A to G.
They represent external grades a showing the highest credit worthiness and G showing the lowest.
In this lecture we will calculate the weight of evidence of this discrete variable.
The formula was equal to the natural logarithm of the ratio of the proportion of good borrowers to the
proportion of bad borrowers.
Therefore to find the weight of evidence of grade we must first find the proportion of good and bad
borrowers by grade.
Let's start with the number of borrowers for each grade.
To do that we can count the rows that contain each of the grades.
We do this with the group by method grouped by splits the data according to certain criteria.
In our case we want to split by grade.
So we type DFA one dot group by and in brackets we refer to the grade column since we want this code
to be reusable.
Let's parameter rides it.
We know that it is the first column in our data frame.
Recall that counting in Python starts from zero.
So the first column has index 0.
In this case were referred to the first column is d if one dot columns dot values and zero in brackets.
If we group like this the grouped values become indexes in the result we get having them as indexes
is great but rather limiting.
So it's better to have them as ordinary values.
This is something we must specify explicitly in the group by method by setting the as index parameter
to false.
We are actually interested in the number of rows or the count the rows in each group here will refer
to the second column area data frame.
That is the column with index 1.
Since we want the total count of rows.
We'll apply the count method.
Let's run the cell.
We've got a result that contains all grades in the first column and the number of rows for each of the
grades in the second column.
Another piece of information we may need is the proportion of good and bad borrowers are within each
group.
This can be summarized either by the proportion of good borrowers or by the proportion of bad borrowers.
It doesn't matter which one.
Since the proportion of good borrowers equals one minus the proportion of bad borrowers let's calculate
the proportion of good borrowers here.
Remember that the good bad variable has a value of 1 when the borrower is good and the value of 0 when
the borrower's bad hence we would get the proportion of good borrowers simply by calculating the average
of good or bad.
We can apply the same statement as above except that at the end we apply the mean method instead of
the count method.
Okay now let's merge these two results into one data frame.
You can see here that there are two columns with labels grade which are exactly the same.
So we could be certain that the results are matching after merging the two outputs.
We don't need one of the columns so let's override the f 1 retaining all rows and only as first second
and fourth column.
The names of our columns are sub optimal.
So let's change them to something more instructive.
We keep the name that the first column will rename the second column containing the number of observations
to n OBS.
The third column containing the proportion of good borrowers to proportion good.
And here is the clean result.
Great.
From now on we'll only use these two columns and a series of calculations of weight of evidence and
information value by the way.
Let's calculate the proportion of observations that falls into each grade.
It is the number of observations in each row divided by the sum of the number of observations in each
row.
First let's calculate the number of good borrowers and the number of bad borrowers by grade.
Group was store at the number of good borrowers and a variable called end good it is equal to the proportion
of good borrowers multiplied by the total number of observations that have the respective grade and
we'll store the number of bad borrowers in a variable called End Bad it equals the proportion of bad
borrowers which is one minus the proportion of good borrowers multiplied by the total number of observations
that have the respective grade.
Second we calculate the proportions of good borrowers and bad borrowers for each grade.
The percentage of good borrowers in the respective grade category equals the number of good borrowers
in the respective category divided by the total number of good borrowers which is the sum of good borrowers
in all categories.
Similarly the percentage of bad borrowers in the respective grade category equals the number of bad
borrowers in the respective category divided by the total number of bad borrowers which is the sum of
bad borrowers in all categories awesome.
We have everything we need to calculate weight of evidence for the categories of grade let's stored
in the w o a variable it is equal to the natural logarithm of the ratio of the proportion of good borrowers
to the proportion of bad borrowers will use the NUM pi function NFP log to calculate the natural logarithm
and here we have it the weight of evidence of the variable grade it is useful to sort the result by
weight of evidence starting with the category with the lowest weight of evidence.
In this way we'll see the categories where borrowers have the highest default rate first.
When I create PD bottles I like to also calculate the differences in the proportion of good loans between
two subsequent categories and the difference of weight of evidence between two subsequent categories.
Usually we don't need those but is good to have them just in case we'll use the function diff to calculate
them here will get negative values because the function subtracts the value of one row from the value
of the row above it is more intuitive to see positive values as they show how much weight of evidence
would increase from one class to the next.
This is why we'll take the absolute value of that difference.
Finally we can calculate information value.
Let's store it in the Ivy column.
The information value is equal to the sum of the proportion of good borrowers for the respective category
minus the proportion of bad loans for the respective category multiplied by weight of evidence for the
respective category.
The information value column is the same on each row.
Because this metric refers to the information value of the variable grade overall good job in the next
lecture we will see how we can automate this procedure.