OK. So let's go ahead and understand how decision trees
form the tree of rules that actually form the final decision tree models,
the relationship in the,
in terms of decision in in the form of
the branched trees.
OK. So let's take an example. So this is our input data.
So what we have is that we have four
input features or predictor features which are weather,
temperature, humidity and wind.
And uh in in each of these fields, we have this possible value, sunny, cloudy,
rainy, under weather, temperature, hot and mild and cool humidity, high,
high normal
and on the wind, we have weak and strong.
OK?
And based on the combination of these four values,
we have a label assigned to it,
which is our predictor sorry, which is our target feature,
which is no. So if this is the combination of
features predictor features, then
I cannot play.
If this is a condition, a combination of input features,
then I can play. If this is the combination of our input features, then I can play.
So ultimately, it boils down to a set of rules which are determining whether or not
I can play or not. OK.
And that is coming from some kind of pattern within this predictor variables. Now,
in other
other classification models, the rule building from different approaches
in case of decision tree,
it is going to
create a set of rules in the form of the tree.
OK. Now
let's
pick up a couple of points here as to how decision trees are made. So a
general algorithm for a decision tree can be described as follows,
pick up the best attribute and feature.
Now this is the
big challenge or million dollar question for
decision trees is that how does it pick up the best attribute or feature?
Right.
The best attribute is the one which best split source separates the data,
right? And we will have a definition of what is the best attribute. And
we are saying that is the one which best splits the or separates the data.
And we are going to see
what does it mean
by the best split
split or separation of the data?
OK. Ask the relevant question question are the decision. So greater than
humidity, greater than 75 or humidity less than equals to 75.
Those are the questions that each node of the decision tree asks. So each node
have a condition and those are the questions
follow the answer path. So you have a boolean outcomes from each of the nodes,
right? And based on the data in your observations,
you follow a path the relevant path
go to step one until you arrive at the
answer. Answer is you determine
the final level for your observation.
Then some of the features of a decision tree,
it is simple to understand and interpret the risk can be visualized.
So there are libraries like
Graphviz which you can use on a decision tree model.
And a graph is that the Graphviz library provides a function
that will actually draw
a
tree pictorially with all the information,
all the meta information within it which is going to
make the interpretation of the decision tree rules
after your learning process, very very clear, visually clear. So that
any decision coming out of that model can be easily interpreted by looking at the
visual representation, visual tree representation
created by libraries like Graphviz
right uh requests very little data preparation.
Other techniques often require data normalization or standardization.
So all these are not required
decision trees can work very well with categorical
data and dummy variables need not be created for
decision tree algorithms.
OK. So which is a big RRI
which which saves a lot of time and analysis
and preprocessing time
able to also handle multi output problems uses a white box model.
So this is related to what we have just mentioned is that decision
tree can be interpreted from its visual
representation and they can be visually presented
like many other algorithms, especially for instance, deep learning models.
Interpretation of outcomes are often very
very difficult and almost sometimes almost impossible
because of the black box nature of the
algorithms and the way the complexity are etcetera.
So in contrast
outcomes from decision
can be interpreted very easily by the white box nature of it.
OK?
It is possible to validate the model using statistical
text tests that makes it possible to account for
the reliability of the model and predictions are very, very fast because
uh if you if you look at the decision tree is a binary tree.
OK. So each note gives a binary outcome. So for making prediction, you don't have to
really go through all kinds of
pert
combinations of relationships,
you just have to traverse through the decision tree
in one direction in in a given direction that fulfills
fulfilling the
conditions of the decision tree uh notes.
And thereby
there is absolutely no need for any observation
to traverse to the entire decision tree.
But
a fraction of the overall distances that can be traversed in that decision tree.
So thereby
predictions are made very, very fast,
some concepts about the decision tree.
So you have a root note root is the root node is the first condition
that is there in the decision tree. So the topmost layer is called the root node
and any note and the other notes are called the decision node.
And then then there are parent child notes as well. So
obviously this is the parent node of this child node.
And this is the child node of this parent node, etcetera.
Root note is definitely the one at the top leaf notes
are those ones which do not have any further condition.
And leaf notes are terminating notes
which are like at this notes, you have actually reached a decision as to
uh level outcome, a level. OK. So
at the end of any
branch direction,
it has to end with a leaf node. OK. So if you look at any of the directions,
every direction, every pathway leads to a leaf node and leaf nodes are the
decision nodes
or rather they are the final label nodes,
then splitting is the process of separating the output input dataset into
a node and into multiple output sets based on the condition.
So let's say for instance,
this particular node has a condition and it results into one leaf node.
So final outcome and then
it can also result into another condition uh notes
which can go through further splitting and each, each decision node,
there are binary outcomes possible that is true or false.
And then I mean bullion outcomes possible true or false. And based on
the outcome
and, and uh for conditions,
it may be decided whether it's going to be a
leaf node or another condition node or decision note.
OK. So these are some of the concepts in um
in a decision tree. There is another concept called pruning
pruning is essentially uh
there is this issue with decision trees, which is overfitting,
overfitting happens in decision trees because uh
it, it creates a decisions
or it creates a tree of rules. Now
it keeps on creating the tree of rules based on
the data that it receives at each node.
And its analysis of how can I how can I best split that input dataset? So
say for instance,
you receive the entire input dataset at the root
node and it splits based on a condition.
And that is the kind of splits the dataset into and and redirects them into these two
nodes. And both of them are again decision nodes.
Now this has part of the
data that was received by the
root node and the remaining part of the data are
received by this particular node.
Then there are multiple levels in this part of the
dataset that that was received by this node. And there are multiple levels
in the dataset that is received by this node
and similarly keeps on going. And at each node,
you still have
observation belonging to multiple labels.
OK. So if there are, if you are talking about the
credit card fraud detection classifier,
then you have observations here.
I mean it keeps on branching and the data is getting reduced as
as it goes deeper into the tree.
And the data amount of data that is received by this node
still has observations of both classes.
Now it keeps on creating rules and split the data to find out
a situation where it does not have
mix of
observation of
belonging to both labels.
OK.
Till that point, it keeps on splitting it.
Now, if it keeps on splitting it that way,
what happens is that you get for, for large data sets, you probably will
end up having a very deep decision tree
which is going to lead to
overfitting. So that means that
you are keeping on creating rules
that goes very, very deep to the granular
level that is possible in the dataset.
And
that fits the dataset, input dataset completely.
And any unseen data
runs the risk off
leading to an
inaccurate result because
your model is
overfitting highly.
Now, that is a big issue with the decision trees. And
the pruning process is essentially a hyperparameter
tuning process for a process or regularization process
through which we are going to cut down the decision tree
in a way that it does not go into the granular
level. And it it, I mean, I mean, it
stops splitting further than a predefined level so that it does not overfit.
OK.
So that's, that's about pruning.
Now, let's try to understand what is a decision tree process.
So you have your input dataset,
you have your decision tree algorithm
and it has to create,
it has to learn from the input dataset,
a set of
rules
organized in a
tree like fashion
tree like hierarchical fashion.
So that at the end
when we provide it with an observation, unseen observation.
Without unlabeled observation,
it can
go through the tree of conditions
and give me a outcome level.
OK. So that's, that's that is the problem statement that we start with.
Now. What does it have to do? It has to find out the tree, sorry, root node first
right now, that is a million dollar question
that how does it find out
the feature
which will be placed at the root node?
So there are two things that it has to find to start with
that is
find a single feature key
and the threshold
TK which is which is a condition.
So essentially it has to find out the feature and the condition with the feature
which it will start with.
OK. So that's, that's the riddle that
the decision tree has to solve.
And it is not not only valid for, I mean, this this statement is not only valid for
the root node
at every node, it has to find the next best feature key
to split the data on.
OK.
So you get to the best feature at root note,
you find the condition and you split the data,
go to the next node and you again have to find out the
best feature to split the data on at that level at that,
on those uh split notes, right?
And the condition as well along with it.
So that is the riddle of the decision tree that is what it has to
find out at every level, at every node,
the best feature K and the threshold T to split the data,
right.
So if that is clear,
then the question is how does it find the
best feature or the best condition?
Now it finds out the best feature based condition on the basis of
two important matrix which are similar.
Uh But any of them can be used for your, uh they are pretty similar in any of the,
any of them can be used for your uh for your case.
The two matrix are you are called genie impurity index
or entropy. So one is gene impurity index and the other is entropy.
No, they have different equations. We will get to the equation
in a moment
but just for starting, they are called the genie impurity index and entropy.
Now again, going back to a little while before,
what is the aim
that the decision tree algorithm should take while trying to split the dataset,
the outcome
from the split should lead to
the outcome from the split
should lead to
the purest
form of the data,
the purest possible form of the data
in terms of
mix of the labels.
So that means
that if you have a dataset
where you have
50%
of level level one
and 50% observations are from level zero,
and if you can find out
the condition
along with the feature
and the
social value,
which leads to
the splitting of these levels 50 50%
I mean in in actual ratio of their present in
then I'm done if you can find that condition,
OK?
And finding that condition
is the aim of the aim of the decision tree model.
OK. So what I'm saying is that
if at the root node, you can find a condition, a feature in the condition that leads to
most pure
form of outcomes. So that means
what we are saying is that
in my input dataset, there are 50% of level 1 50% observations are of level zero.
And from the condition,
it leads to at one side 50% that 50% with level one and another 50% with level two.
Is there any further need of split? No, because
we already have received the outcomes which are 50 50%. I mean that
that's that's pure. One side
outcome is
pure or level ones and the other side outcome is
pure all level zeros from the condition we have found out in root note.
But is it possible to find that kind of a condition?
Possibly? No, I mean, realistically you will never find that kind of a condition
which gives you a pure split of ones and zeros. OK. If you find that even at
a
lower level, um you are going to be very, very lucky. So that is why we have to find
those conditions,
those features and conditions
which is going to lead to the most
pure form of mixture of the labels in your outcomes is
is going to be used in the used in the nodes. OK.
It is valid for root node, it is valid for any, any sub
nodes as well.
This approach is done through the use of a genie score
or a entropy score. OK.
Essentially, the decision tree model
by going through all the features and conditions within
those features is going to try to find out
the lowest value of G index or the entropy score.
OK.
So at a particular node starting from the root node,
the decision tree model is going through all the
features one by one and try to find out
which feature
and what condition with that feature
is giving me
the lowest gen index or entropy score.
OK.
With that
it is it is going to split with that feature and condition
then
the data gets divided and it comes to the next layer of
notes and then it is going to follow the same process again.
It is going to go through all the features available in
the
dataset
for that particular set of observation that has come to those specific nodes
and then apply
the same process again and try to find out the feature
that gives the
lowest gene index and the lowest entropy score at that node.
OK. Now for, for our information,
the model starts
with a higher gen index value in upper nodes
and slowly gradually
through the deeper nodes or lower nodes,
the gen impurity index or the entropy score keeps on growing
lower and lower.
And
the tree should stop
going any further when there is no more reduction of gene
index or entropy scores any further for any further splits.
All right. So that is the approach. So
going through this flow chart, what we say is that we get our input dataset,
we calculate the gen entropy values for each feature condition
and choose the feature or condition that results in the lowest
gen index entropy split the dataset using that feature or condition
and repeat steps 1 to 3
till it is constrained by a hyperparameter.
And we will see how hyperparameter or regularization techniques
constrain or limit the data um decision
tree from going any further any further splits
or it cannot find a split that is going to reduce the impurity that is
lower um getting
achieving a lower value of G index or entropy value.
So that is the decision tree steps.