The soft classification score outputs that we saw in the last lesson are oftentimes more useful than
hard classification score outputs.
One of the major reasons why is that the hard classification outputs are somewhat arbitrarily defined.
Those hard classification outputs are made based off of our soft classification scores.
The default for classification models in scikit learn is.
If the soft probability score is above 0.5, we'll predict the positive class, and if it's below that,
we'll predict the negative class.
But that threshold is somewhat arbitrary.
And so knowing how to leverage those soft probability scores can reveal a lot more about your model
accuracy than using hard classification scores.
One very helpful visualization that leverages these soft probability scores is plotting predicted probabilities
versus observed event rates, which will give us a visual estimation of our soft classification performance.
What we're going to do is bend our probability scores, and then plot the average positive class rate
that occurs within each bin.
And ideally, our predicted probabilities align closely with our event rate really quickly.
Let's take a look at the code we can use to generate this.
And we're going to dive more deeply into this in a demo.
So I'll move quickly here.
The first thing that we're going to do though is combine our actual observed class values from our training
data with our soft probability scores.
So here I'm creating a data frame that has Y train our real classes and our predicted probabilities
for those same rows.
I'm then creating ten bins for probability scores.
Remember that probability is bounded by zero and one.
So I'm creating deciles for probability that range from 0 to 0.1, 0.1 to 0.2, 0.3 to 0.4, and so
on.
And once we have those bins, I'm calculating the average probability score as well as the average actual
event rate, meaning the percentage of times the positive class occurred in each of our ten bins.
And if we take a look at the visual returned in our x axis, we have predicted probability.
And in our y axis we have the actual event rate.
And so the closer that our predicted probabilities are to our actual event rate, the better.
But at the very least we want to see that these are positively correlated.
As our predicted probabilities go up, our event rate should go up as well.
And so taking a look at this chart here, if we take a look at the bottom left corner where our predicted
probability and event rate is close to zero, our average predicted probability in this bucket 0 to
0.1 is about 0.05.
Which makes sense because that's right in the middle of our bucket.
The average event rate is a bit lower than our predicted probability, probably close to 0.03.
But overall our predicted probability is very close to our actual event rate, which is a great sign.
And we can feel very confident about our probability scores within this range.
If we take a look at 0.6, we can see that our actual event rate was 100% versus a predicted probability
of about 65%.
This would be the bend between 0.6 and 0.7.
And so that is a massive gap.
And this is something that looking at our confusion matrix using a default threshold of 0.5 wouldn't
reveal everything to the right of 0.5 would be classified as a one using hard classification scores,
and everything left of 0.5 would be classified as a zero using hard classification scores.
But understanding that there's some variability and there's going to be positive class instances across
our predicted probability range, actually allows us to do a lot of great things with soft classification
scores, which we'll dive really deeply into in the model evaluation and metrics section of this course.
But let's take a closer look at this diagnostic visual in a demo okay.
And so the first step in creating that visualization is determining how many probability bins that you
want.
And so generally speaking I like to use ten.
But you could choose 20 or whatever number floats your boat.
But most people are comfortable with multiples of five and ten.
You can see that we have 11 values that represent increments of 0.1.
So we go from 0 to 0.1, 0.2, 0.3, 0.4.
So we're getting 11 values equally spaced between zero and one.
And you might be thinking why don't we use ten.
And remember that because we're including both endpoints we need an extra value.
And if you wanted these in increments of 0.05, you could change this to 21.
And now we'd have increments of 0.05.
But these bins are what's going to help us bin our numeric probability scores.
And once you have determined the optimal number of bins for your problem again ten is a good starting
point.
We then need to combine our actual observed class values with our predicted probability scores for those
values.
So we're using our Y training data set.
We could use tests as well.
We'll start with train though to get our class observations.
And then we're going to tie those to our predicted probability scores using Predictproba on our X train.
And we're grabbing the positive probabilities for each of those observations.
And so let's go ahead and comment out the rest of this code for now.
And really quickly we'll take a look at the head of this data frame.
So let's run both cells.
And so all this data frame is, is the actual value for our target class, as well as the predicted
value for that same row.
And now what we need to do is create probability bins.
And that's where pandas cut function comes into place.
So let's go ahead and uncomment this code.
I'm going to leave labels commented out so we can see the raw data before I overwrite it with labels.
But what we're doing is creating a new column called Predicted Probability in this same data frame.
And this is going to be equal to using pandas cut function.
This is going to create bins.
So categorical bins based off of our predicted probabilities that fall within the values of the then
specified here.
And so if we run this again.
We now have a column that represents the range of probabilities.
And so our probability score was 0.24.
That's going to fall into the bin that is greater than 0.2 and less than or equal to 0.3.
This soft parentheses here means that it doesn't include 0.2.
This hard bracket here means that it does include 0.3.
Looking down to the next row, we have a predicted probability of 0.33.
That falls into our bucket of 0.3 to 0.4 and so on.
So we have the correct bins.
And you may choose to label this something differently.
But what I've done here is just grab the first value out of the probability bin and rounded that.
So we'll end up with a label that says 0.2 for the bin that starts at point two, 0.3 for the bin that
starts at point three, and so on.
And so taking a look now.
You can see now that we've almost essentially rounded these.
However, the good thing is, is that these are categorical text based variables.
Now let's go ahead and plot this.
And so what we're going to do here is group by our predicted probability buckets.
So we're grouping by this categorical bucket variable here.
And then we're just going to calculate the mean of the actual target.
This will give us the event rate.
And then we're going to calculate the average probability for those same buckets.
And so once again let's actually create an intermediate data frame here just so we can take a quick
look.
I'll wrap this in parentheses.
And now we can see each of the data points.
It's going to be plotted here.
And I'm getting a little warning here.
Let's go ahead and just pass in that to suppress that.
But here we can see that for our first bin our actual probability was 0.664.
While our predicted probability was very, very close, this bin was very close, etc..
But of course just looking at these bins gets a little bit hard to parse.
So this is where it's going to be good to plot this.
And so in order to plot this all we're going to do is dot plot line.
And we'll end up with a nice line chart here.
And so what we can see is that from 0 to 0.3, our probability scores are almost perfect.
But as we get into this 0.4, particularly at about 0.70.8 range, we start to see a gap between our
actual values and predicted values.
And so our model is under predicting based on reality.
But overall our training set looks pretty good.
These are positively correlated.
And in general this means that we can rely on our probability scores.
One of the good things about this line is that even though there is a gap between actual and predicted,
our actuals are perfectly ordered in the sense that our event rates are always going up.
And that means that our probability scores are correctly arranging our data based on how confident they
are.
There is a gap between reality and predictions, but if we were to cut at any point as probability increases,
so does the actual event rate.
Of course, looking at your training data will always paint a rosier picture than looking at test.
So let's go ahead and see what the test data looks like here.
And so this is probably going to be a little bit more common in practice.
And so what we actually see here is that some buckets.
So for example at 0.7 are actual rate is higher than what it was at this bucket 0.8.
Our probabilities will always look smooth because we're ranking according to our predicted probabilities.
But our event rate, the closer it is to this line, and the more monotonic that our event rate or our
actual line is here, the better off our model is going to do in the real world.
Overall, I would say this is still quite good.
However, we certainly aren't doing quite as well at predicting as we did with our training data, which
makes sense.
But overall, this can tell me a few things about my model fit.
One thing is that our scores near zero are not nearly as accurate as they should be.
So for example, we're predicting essentially zero probability, but in reality there's a 15% chance
we have positive target values hiding in here.
And also again, as I mentioned earlier, observations that our model predicts in the 70% range actually
have a higher likelihood at being a positive class than something in the 80% range.
And so these types of insights can either inform us whether we need to go back and retrain our model,
or try to get more accuracy.
We might also just need to live with the fact that our model is less accurate in certain regions of
probability and plan accordingly.
But now that we've taken a look at this visualization, which again, can be a very nice way to display
your model's accuracy to stakeholders, let's go ahead and get some practice with tuning AK9 model and
an assignment.
We'll see you there.