- [Instructor] There is a frequent claim
that's made that not just some
but all of the kaggle.com winners
on the kaggle.com data mining modeling competition
website are ensembles.
So let's take a moment to talk about ensembles.
Now, keep in mind when you take a bunch of models
and combine them, you've just turned your model into
something more complex
and almost certainly into something opaque.
So I don't think you want to assume on real world projects
that you'll always use an ensemble,
but there's a reason why they win these competitions,
they're very powerful.
So in its most basic form,
a so-called heterogeneous ensemble
would just be a combination of three models.
You could have a logistic regression,
a neural network and a support vector machine,
or any other combination of the algorithms
that we've talked about,
the notion of it being heterogeneous
is that we're not making an assumption here
that all three of these models are trees.
They are a mix.
And we can simply calculate an average,
just as simple as that.
Now, an average of what?
It would probably be an average of the propensity score,
'cause after all, we're talking about binary classification.
Alternatively, you could go with the most confident model,
the max propensity score.
There's all kinds of other options that you could imagine.
Now, keep in mind that if one model was producing a null
for whatever reason, for instance,
it is using listwise deletion
and the other technique is not,
most confident, of course, would also mean
some non null value, and I've leveraged
that kind of a trick when building ensembles.
Now we get to some terms
that you probably have encountered.
Bagging, which stands for bootstrap aggregating,
is actually building a model on different partitions
of our data, that's what the bootstrap is referring to.
So probably, the most famous one is random forests,
which is a combination of three trees into a final model.
Three individual trees, final model is thereby a forest.
Now, of course, it won't really be three samples.
If you're doing something like random forests,
it's gonna be 100 trees or 500 trees or something like that.
This is so closely associated with random forests
that a lot of times people think this is primary trees,
but all of these ensemble techniques
that we're talking about right now can be applied
to any of the algorithms.
Now we'll talk about boosting,
which at first might similar,
but is different in an important way.
What we're doing here is we're building a model.
Now, here the models will tend to be of the same kind,
all logistic regression,
all trees, and once more, typically associated with trees.
So we build a first model
and what we're feeding
to the second model now is information from the first model.
Typically, it's the errors
that the first model made so we're trying
to make fewer errors on that second model.
So think about it now.
We're producing something that's more complex,
and we're trying to increase that signal,
but if we're not careful,
we're also gonna amplify the noise
and we could get something that was overfit,
but again, these algorithms are developed to
try to maximize the benefits and minimize the downside.
Each of these models iterates,
culminating in a final model
that has benefited from all the ones previous to it.
Finally, stacking is a super popular one
on that kaggle.com competition site.
These can be three models of any flavor that you like.
They can be quite different from each other.
They often are different from each other,
and what we do is we produce those models,
we generate those propensity scores,
and those propensity scores get sent to another model
which also can be of any kind
which uses those propensity scores as inputs
and then produces its own combined propensity score,
so ensembling, big topic, but a powerful option for you.