Hello and welcome back to the course on Machine Learning.
Today we're talking about decision trees.
All right, let's get started.
You might have heard the term cart.
It stands for classification and regression trees.
And this term is an umbrella term for.
Two types of trees which are obviously classification trees.
And regression trees.
Now the difference is that classification trees, they help you.
Classify your data.
So they work with categorical variables such as male or female, apple or orange,
or different. Types of colors and. Variables of that sort.
Whereas regression trees, they are.
Designed to help you predict outcomes which can be real numbers.
So for instance, the salary of a person or the temperature
that's going to be outside and things like that.
So those are the two different types. And we're going.
To be talking about classification trees in this section of the course.
So here we've got an example with.
Lots of points on our two dimensional scatterplot.
Now how does a decision tree work.
So what is going to do is cut it up into slices.
In the several iterations. So let's have a look.
So will be split one.
There'll be split two.
So split one.
Split the data at x two equals 60.
Split two splits our data at x1 equals 50.
Split three.
Split our data x1 equals 70.
Then split four.
Split our data at x2.
It's not shown here, but it's about 20.
So that is how our decision tree works.
And the basis for these splits.
So how are these splits selected.
How does the algorithm know where to select the splits.
Well basically if you have a look at it. Now and then.
The split is done in.
Such a way to maximize the number of a category in each of these splits.
So to maximize for instance, we want maximum red categories here.
And here it is still the same.
But then the next split maximizes the number of green here
and then a red here.
That's a very basic way to explain it.
In reality there's some.
Complex mathematics happening in the background.
The split is trying to minimize entropy.
And and so it's informational.
Entropy is a very interesting term.
It would take hours and hours and hours for us to go
through all of that right now.
And so if you want to get into the deeper mathematics
behind this algorithm, then you certainly can research that.
But for us.
It's sufficient that we're just looking for the optimal.
Split, or the.
Algorithm is going to find optimal splits that are going to maximize
a number of different points in each one of these, new pockets.
So they're actually called Leafs.
So you've got the starting scatterplot.
And then at the end you've got these leaves.
And the final leaves are actually called a terminal leafs.
So that's how the splits occur.
Now let's rewind a bit and let's do that whole procedure again.
But while we're performing the splits we're going to.
Start constructing a decision tree.
An actual decision tree. Let's have a look.
So there's a split number one.
And what it's doing is it's splitting our data.
At the 60 level.
So now let's construct a decision tree that's going to ask exactly that question.
So is. X2 greater than. 60 or less than 60.
So if it's greater than 60 it falls into one branch.
If it's less than 60 it'll fall into the next branch.
So there we go. X2 is less than 60 or no.
Yes and no.
Next is.
Split two only splits.
The data that is above 60 in the x2 variable.
So let's have a look at that.
We're only dealing with data that is above x2.
So it's over here.
And now we're checking.
So I'm going back now.
Now we're checking a split two happens at 50 for the X1 variable.
So here we go. X1 is less than 50.
Yes or no. So if and here you can see that right away.
This split already can tell us whether.
Something is green or red.
So if it's, less so if we already above 60
and then below 50, then it's green, which we can see here.
If we are above 50, then it's red which we can see here.
So that's how this classification works.
And now let's deal with the remainder.
So here we've got a split three happening at 70.
If you're below 70 you're obviously going to be red.
Otherwise we're going to need to do another split.
So below 70 then it's red.
No we got to do another split.
Split for if it's above 20 then it's green.
If it's below 20 then it's red.
If it's above 20 then it's green.
So that's a no. If it's below then it's a yes.
So with these decision trees.
A good way to structure them is to always keep yes yes yes yes on one side.
So like if you if you're looking for yeses they're always going to the left
looking for nos, going to the right or vice versa.
I just don't mix them up.
And then the terminal leaves will.
Predict exactly what color or what class is left.
But at the same time, even if you don't get to the terminal leaf,
because this is a very simple tree, trees can be very, very long.
And so sometimes.
You might not even get to the bottom.
So if you want to classify any observation and for example this observation falls
into this section over here, then it would go down this road then here.
And then we go here.
But let's say you don't even get to the end.
You get to somewhere over here.
Then in these boxes that don't that have still they still have a mix.
So here you can see is a mix of green and red.
Then the rule here is that a probabilistic classification occurs.
So here we know, instead of checking this last condition
we'll just check what is the likelihood of it being green and red.
So here we see that there's more green and red.
So if we just going to leave it at this box
then we'll just say that it's, it's a green dot.
Whereas if we leave it at this whole box, if we just if we just do this part,
so we only going to check the first condition
and then we leave it here, then we would automatically say that it's a red dot.
If we don't go down the decision tree and check more condition.
So that's another
way of using the decision tree instead of going down to the very end.
You can stop at any point and then just use the probabilities
to predict your classification.
And another thing is that it doesn't.
Always have to be the two variables.
So for instance, in a decision tree,
just like with any other machine learning algorithm,
you can have a multidimensional data set which has lots of different columns.
So in our case we only have two.
But you might have lots and lots of columns.
And then you could have a mix of questions being asked here.
And that's up to the algorithm to come up with those questions.
And as a final note,
I wanted to mention a little bit about the history of decision trees
decision trees have been around for very, very long time, and in fact,
they are so old that they have recently kind of started to die off.
They were still popular about 23 years ago, but recently
more sophisticated methods have come to replace them.
And decision trees stop being so popular.
And that's continued for a while.
Until recently, they were reborn with new upgrades.
So to speak.
And those upgrades are additional methods that build on top of decision trees.
And such methods are random forest gradient boosting and other methods.
And in this part of the course, we will look at least at one of those
other methods.
The point is that decision trees are, though, a very simple tool.
They aren't very powerful on their own, but they're used in other methods
that leverage their simplicity
and create some very powerful machine learning algorithms.
And such algorithms even are used to perform facial recognition, like.
On your iPhone, you get, you have facial recognition.
And also some games such as Kinect, which is kind of like the Wii.
But, you can play it without actually holding a controller.
So it's like a game for your addition to your Xbox,
and you can play as without having a controller, your hand.
So it kind of recognizes where you're moving your arms and legs.
And that method, Microsoft decided to use random forests for that method.
And random forests invoke decision trees.
So hopefully you enjoyed this today's tutorial.
It is quite a simple method, but at the same time it lies.
In the foundation of some of the more modern
and more powerful methods in machine learning.
I look forward to seeing you next time.
And until then, enjoy machine learning.