Hello and welcome back to the course on Machine Learning.
In today's tutorial,
we're talking about decision trees and the intuition behind them.
All right so you may have heard the term cart which stands for classification
and regression trees.
And this is an umbrella term that encompasses two types of a decision trees.
And as you've correctly guessed,
they are the classification trees and regression trees.
And in this course we're going to talk about both types.
But specifically in this section we're focusing on the regression trees.
And I wanted to mention right away that regression trees are a bit
more complex than classification trees.
And that's why this tutorial is going to be a bit longer
and is going to require some additional attention.
But nevertheless, we're still going to break this kind of somewhat complex
topic into, very simple, bite sized, elements of information.
So it will all make sense.
And towards the end of it, you'll be quite comfortable with regression trees.
So let's get straight into it.
All right.
So here we've got a scatterplot which represents our data set.
So data set that has been given to us.
And the interesting thing about the scatterplot is that we've got two
independent variables x1 and x2.
And what we're predicting is a third variable a dependent variable which is y.
And you cannot actually see why on this chart.
And that is because this is a simply a two dimensional chart.
When you fit the two variables, y is the third dimension.
And if you think about it, it's like sticking out of your screen.
That's where that dimension is.
And this is just a projection of all the points on the x1, x2 plane.
And so if I add a third dimension, it would look something like that.
But once again we can't see y right now.
And the interesting thing is
that we don't actually need to see y because we need to work with this,
scatterplot first for a little bit to build our decision tree.
And then once we've built it will return to Y.
Now a quick point I wanted to make here is that
I've seen decision trees explained with just one independent variable.
So x1 or just x and y.
And then in and in that case, yes, you can you can just put x1 over here
and then Y would go over here and you would have
a bit of a different kind of diagram, and you'd be able to explain it that way.
But at the same time, I think it might not really drive the point home.
And, it can be a bit confusing when it's explained like that,
although sometimes it is done.
nevertheless, I thought would go, the full way, would do the full Monty
and would look at this problem with two independent variables,
because it'll be a more robust explanation.
So it will make it a bit more complex, but it's definitely worth it in the long
run, because that way will understand the decision tree regression a bit.
or actually I would say quite a bit better.
All right.
So let's continue.
We've got the X1 and X2.
These are independent variables.
The dependent variable.
We cannot see it. And it's the third dimension.
And we're actually going to forget about it for a little while.
Right.
So we're going to just forget about it because we need to work with this
scatterplot to see how our decision tree is going to be created.
So once you run the regression tree or decision
tree algorithm in the regression sense of it, what will happen is
your scatterplot will be split up into segments.
And let's have a look at how an algorithm could go about doing that.
So an algorithm would create a split over here for example at somewhere around 20.
so it would basically split your diagram or your scatterplot into two parts.
Everything has less than 20.
Everything that's greater than 20 for the X1 variable.
Then another split would happen here.
So for all of the elements in this side
they would be compared to 170 greater or less.
And then there'd would be another split here
and then maybe another split over here.
Now how and where these splits are conducted
is determined by the algorithm.
And, it is actually involves
looking at something called the information entropy.
And it is a mathematical concept.
It is quite complex.
So it basically means when I perform this split right.
Is this split increasing
the amount of information that we have about our points?
Is it actually adding some value to our way that we want to group our points?
And the algorithm knows when to stop, is when there's
a certain minimum for the information that needs to be added.
And once the, like, it cannot add
any more information to our set up by split.
These leaves are called leaves.
So each one of these splits is called a leaf.
By splitting these leaves, once it kind of adding more information, then
it stops or, or the algorithm could, let's say stop when you have less than 5%.
if you were to conduct a split, then you'd have less than 5%
of your total points in that leaf, and then that leaf wouldn't be created.
So there are, different variations or different options for that to happen.
And but the most important thing is, of course, where the splits are happening.
And if you'd like to learn
more about that, you'd you'd need to study a bit more about information entropy.
We're not going to go into that mathematical depth right now.
For us, it's sufficient to know that the algorithm can handle this,
and that it is finding the optimal splits of our data set into these leaves.
And the final leaves are called terminal leaves.
And then we're going to focus on the practical application
of this algorithm, how and why we're using these,
decision trees and how this regression is going to work.
All right. So hopefully we're on the same page.
Let's continue.
So we're going to rewind all of this a little bit.
And we're going to create these splits one by one.
And alongside we're going to actually start drawing our decision tree.
So there's our diagram brand new and fresh.
And there goes our first split.
So now we're going to start creating our decision tree.
the splitting happened at 20.
So let's start drawing.
There is our first decision.
And we have two options yes and no.
All right.
So let's let's see what happens next.
Next happens split two.
Split two happens at 170.
And only happens for the points that are greater than 20.
So that means you would check this condition x one is less than 20
meaning you check. No you. The answer is no.
And then you check if x two is less than one, 70 x two is less than 170,
then a split three happens
on the other side and it checks if x two is less than 200.
Let's add that here x two less than 200
and then split four happens at 40.
And it checks if x one is greater or less than 40.
And a split four only happens for the points that answered to split one.
They answered and no, it's not less than 20.
And to split they answered no, it's yes, it's actually less than 170.
So no, it's not less than 20.
Yes, it's less than 170.
And then this is where split world four happens.
X1 is less than 40 is no.
All right.
So that's our decision tree.
It's done. It's drawn.
And so what happens next.
How what do we actually populate into those boxes.
Well this is where we need to remember about our dependent variable.
The third dimension.
And what we need to check here is
how are we going to predict the value of y
for a new observation that gets added to our scatterplot or to our dataset.
So let's say we add a observation which is has x1 equals to 30 and x2 equals to 50.
It would fall somewhere over here and 50 is somewhere over here.
It would fall somewhere over here.
So obviously it falls into this, terminal leaf.
And how does that information.
So as you can see, we've by adding
these splits, we've added information into our system.
So how does that information that now we know that it falls into this
terminal leaf.
How does it information help us in terms of predicting
the value of y for that new element that we're going to add?
Well, the way it works is it's actually pretty straightforward.
The way it works is you just take the averages of each of your terminal leaves.
So you take the average of Y for all of these points.
And that will be the value
that will be assigned to any new point that falls in this terminal leaf.
Same for this terminal leaf.
Same for this terminal leaf, same for this one and the same for this one.
So let's have a look.
Let's say the average for y here is 65.7.
The average for y is here is 300 and point five 1023.
Here -64.1 0.7 here.
So for that point that we just
discussed with x1 equals 30 and x2
equals 50, the predicted value of y that the regression tree
algorithm would predict a value of -64.1.
If it were to fall in any other terminal leaf,
then that's what the value there would predict.
So as you can see, it's actually pretty straightforward.
It's it's very simple.
It's just taking averages.
but you do need to remember that we are, working.
The whole point of this exercise is to add
more information into our chart, into our system, to better predict
y, because if you think about it, what was our other option?
What is our default option?
If the default option were for running any machine
learning on this, data set is to just take all of the points
and take the average across all of the points and whatever that is.
Wherever our new point, the new element of data
that is added to our data set, wherever it falls, we just assign.
It's always that's average for all of the points that we had existing
previously.
What do we did now is we've split our diagram up into these terminal leaves.
The machine learning algorithm has added information to our entire system.
And so now we can more accurately predict the value
or assign the value of y to a new coming element.
And as you can see now, it's average, not just across all of them.
The average is taken into in specific parts or segments of our scatterplot.
And therefore it is or it's supposed to be more accurate.
That's the whole point of the regression tree.
And now last thing we have left to do
is to add the values into our, decision tree.
So basically we just add those values in here.
And now whenever we have a new value,
what would happen is the algorithm which is go through this,
these checks and it would check where it falls and assign the value.
And that's pretty much it.
So the scatterplot is more for like visualization, conceptual purposes.
So you can maybe drive some insights from there.
But the core of Decision Tree is actually held here.
That's why the algorithm is called a regression tree.
I hope you enjoyed today's tutorial.
And, hopefully we did break down this quite complex
topic into some simple and actionable steps,
and I'll look forward to seeing you next time.
Until then, enjoy machine learning.