- [Instructor] Okay. So we've seen again and again,
that statistics and machine learning are quite different
and this is true
but they can work together.
If you look at CRISP-DM,
you'll see that the phase after modeling is evaluation.
I have to admit to a bit of a pet peeve.
You'll hear folks say
that the evaluation phase is running tests
like R squared,
area under the curve, overall accuracy.
It's not true.
That's actually modeling assessment.
It's part of the modeling phase.
The evaluation phase could be called business evaluation.
It's not about the names.
It's about this terribly important reminder
that when you get to business evaluation,
you're doing something very different
than R squared or area under the curve.
It's when you use business metrics to see
if the model is performing well
in a real world setting as a deployed model.
This is a perfect opportunity to use statistics
in support of your machine learning project.
In my experience,
you should always do a partial rollout of the model.
In other words,
use the model in just one city
or one factory or just one sales office
before you deploy it everywhere.
This will allow you to do an apples to apples comparison
of the test city to the other cities,
the test factory to the other factories
or the test sales office to the other offices.
It's an AB test.
So statistics is perfectly suited to the task.
Now you don't want to run just one T test at this point,
you should have determined a set
of business metrics and key performance indicators
back during business understanding.
So you aren't just checking one thing
like if you've been detecting machine failures.
You want a thorough assessment
of several metrics like conversion rate of leads,
number of minutes the machines were down.
This again is in perfect alignment
between what you need and what statistics can offer.
And it's a much, much more thorough assessment
of model performance.
then just train test validation
that you were doing back in the modeling phase.
You do that back in the modeling phase because it's scalable
to lots of models,
but when you get to the evaluation phase,
this is the analysis that really counts.