[MUSIC]. So we talked about re-sampling and we
talked about the bootstrap and we talked about cross-validation, where you're
splitting your training set into different sets in order to validate your
results. So one idea that might occur to you is,
well look, you know, we have all these different subsets of our training set
lying around. Why don't we train different classifiers
on each one, and combine their results? Right?
So instead of trying to use it to validate one particular classifier, why
don't we just make a whole bunch of different classifiers?
And so the question that you need to answer is, well look, can we [INAUDIBLE],
does it actually make sense, does it work?
To take a bunch of not very good classifiers and combine them, you know,
will it actually get better? And the answer is not trivial to find
out, but it was found out and the answer is yes.
Okay. And so you can average the results of
different models and you get a couple of benefits.
One is the strength will go up. The [INAUDIBLE], the classification
performance will be better. And you can drive it up by design.
But more importantly, perhaps, there's more resilience to noise.
And so you reduce the chance of overfitting by having a bunch of
different classifiers all work in tandem, okay?
So why, why wouldn't you do this? Well, there is two reasons.
One is, it's time consuming, right? Obviously, training a bunch of different
classifiers is more expensive than training just one.
But also, you know, they become, the behavior of your model becomes difficult
to explain, right. When you have a whole bunch of different
models also voting on the answer. it's difficult to take some intuition for
what logic is actually being applied. And so, when you remember, we said a big,
just like one rule is easier to understand than a big set of rules one
decision tree was easy to understand. A big set of decision trees is perhaps
not. But that's not going to stop us from
doing it. Okay.
So the idea here is that there's this wisdom of the crowd model.
Well it works when you're talking about these digital artifacts as well, these,
these generated models. Alright.
So how do we actually do this? Well one technique is called bagging,
which is a portment of bootstrap and aggregating.
and so the idea here is to draw your N bootstrap samples, retrain the models
separately on each sample just, we described and then average the results.
[INAUDIBLE] regression problem that we haven't gone into detail yet on.
then you would actually just literally average the results.
If it's classification, where you're only picking a discreet class label, then you
do a majority vote. Okay.
And so this works really, really well for overfit models, it resists overfitting.
because it decreases variance without changing the bias, so without moving the
answer around. Right?
It just shrinks everything or, or, to that answer, and if you remember, we put
up that plot where high variance tended to mean overfit.
It means that you are specialized to the training data and you're very sensitive
to the training data. Okay.
But it doesn't help much with models that do have a high bias with, with wrong
models. Remember high bias, low variance models
are more or less insensitive to the training data.
And so all that you're doing with bagging is taking different permutations of the
training data. So if you're not getting a good signal
out of the training data at all anyway, this isn't going to help much.
Okay? So, another technique is boosting.
Exemplified by this algorithm Eta-Boost. Okay.
So, here, instead of selecting data points randomly with the bootstrap, you
want to favor the misclassified points on each step.
Alright. So take a bootstrap sample, retrain a
model. See what the [INAUDIBLE] inspect the
mistakes of the classifier made. Okay.
And recompute the weights for the next round of bootstrap giving extra weight to
the example data points that the meter model got wrong.
Okay? So let's see what this looks like a
little more precisely. Okay?
So, for each step t, we've got a vector of weights, which represent the
probability of selecting example i in the samples, so that's just D.
So D act t, D of i is the probability of selecting i in the sample.
And these are initialized to equal probability, to uniform probability.
Okay. And then x and y, x sub, x sub i and y
sub i are your record and your label. Okay.
The same data point example and record, I think sort of interchangeably, which
isn't all that helpful, but bear with me. So h sub t is the trained classifier at
step t, using a sample drawn according to the weights D.
Okay. Now, compute the error by adding up all
the weights for everything that was misclassified or this epsilon [INAUDIBLE]
error. [INAUDIBLE] Compute this epsilon by
adding up the weights for everything that was misclassified.
And that's what this, incantation is down here, right.
So this says for [INAUDIBLE] for the model h sub t, apply it to x of i, where
it's not equal to, or the classifier tells you something different than the
actual class label than included in the set and add up all the oops, excuse me.
Added up all the D's for those. Okay.
Then compute what's called the odds, which is the odds, given a probability,
is that probability over one minus that probability.
Okay. So, compute the odds of misclassifying
and then weight, those values, those misclassified values, by the odds.
Okay. Oh, excuse me.
You're going to [INAUDIBLE] that's not quite right.
You're going to, this is a, this is a value between 0 and 1, right?
And so this gives lower weight to D of i and you're only going to apply this
function to the ones that were correctly classified.
And all the other weights you just leave alone.
And so this isn't, I didn't put that data for, in the attempt to clarity I didn't
include the, the entire expression. That may or may not have been a good
idea. So this is only applied to the correctly
classified examples. Okay, and then there's actually one other
step that I also removed in an attempt to clarity, which is, you need to normalize
all these down, because what you've done now is take in a set of weights that
normalize two probabilities. You know, they add up to one.
And you've just adjusted some of them down.
Well, now it's not going to add up to one anymore.
But that's okay because you know the relative weight.
So you need to renormalize all this. and so, it's a pretty simple step, but I
find when you sort of throw a divide by z in here it, it gets confusing.
So, I just wanted to explain it this way. Okay.
And then, that gives you your new weights for the next round and then you repeat
this process. Okay?
So a question I want to ask about is though, given our appreciation for big
data, and for scale out solutions is, the difference between bagging and boosting.
Well I guess I should probably promise you these are, boosting has some nice
properties, and it's a very very succesfull algorithm.
and one thing that's especially cool about this is that it, it's it's a meta
algorithm, right? It works on anything.
All it, doesn't say anything about what this classifier needs to be.
Any, it could be a decision tree, it could be any other method you might come
up with. Right.
All it does is affect the data you use to train the classifier to each step.
And so this is very, very general and very, very powerful.
Okay. And, and, you now, because of this
weighting, it actually does sort of start to zero in on the mistakes that are made,
giving more and more weight to the [INAUDIBLE] errors.
And so you can sort of imagine intuitively that the strength of the
classifier's going to get very much better, better and better over time.
So that's well and good, and it works. but it has a disadvantage with respect to
really big data relative to simpler simpler approaches, such as bagging.
And so that is, it's inherently sequential, right?
We have this for each time step, you know, the weighted time t plus 1, step t
plus 1 depend on the weights of step t. With bagging, we can just go wild in
parallel and do all of this at the same time.
Okay. So that's something to keep in mind, that
you don't see quite as often in the discussions of these various methods,
because they tend to focus on the, the mathematics involved in the sort of
performance. But when we keep in, you know, in a data
science context, we need to keep in mind that we're going to be working on very,
very large data sets at times. and so a slightly worse performing but
easy to paralyze method can win. Alright.
[BLANK_AUDIO].