OK, now that we have seen the difference between filter methods, wrapper methods and embedded methods,
one might be curious the criteria that these filter methods they really use to select the features.
And the reason is the wrapper methods and embedded methods, the intuition of feature selection.
And those two methods is is is very clear because there is a machine learning model at the end.
There is some machine learning model that actually is guiding the feature selection process.
Which features are relevant, which are not relevant, is guided by the by some machine learning model,
whether that is a supervised framework or unsupervised framework.
That is some machine learning model that is guiding what features are relevant and what are not.
So that makes sense.
But in filter, in filters, there is no machine learning model.
And one might become curious, what are these criteria that without actually exploiting any machine learning
model, these criteria they are powerful enough to.
To basically select relevant features.
So in the next few videos, I'm going to discuss different filter criteria.
Some of these criteria will be individual, individual, feature based, which means they will rank
features individually and will not require any subset generation, but a few of those, they require
a subset to evaluate a particular criteria.
So let me discuss a few criteria, a few famous and popular criteria that fall under these filter methods.
So these statistical based methods, I mean, all these statistical based methods, I should say, most
of these statistical based feature selection methods rely on some predefined statistical measure.
So there is some measure, predefined statistical measure.
And based on that, the feature selection is carried out and and based on that, measure the irrelevant
features.
They are eliminated.
These methods, most of these methods, they are computationally very efficient.
Their competition cost is really low.
So, for example, one criteria is, is low variance.
That's an unsupervised technique.
And it eliminates the features based on the low variance.
So the criteria is whatever feature has low variation among different instances or examples, that feature
is uninformative and can be eliminated.
So take an example.
Let's say we have features x1 x2 x3 x four.
Dot, dot, dot x d.
Let's say we have D features and let's say we have different values of these features.
That's a value.
And that's a first sample.
Let's say we have another sample that sample two sample two value of X one sample two value of x2.
Then we have this, that and that.
Similarly, we can have another sample.
That's another simple, let's say, simple three.
The color indicates nothing here.
Just just I want to show that these are different samples and so on and so on.
So that's our data.
And let's say the data is has no class label or even if it has class label, the class label is not
required for this particular criteria.
Now, the low variance mechanism works in a following way.
For example, these are all the values of X one, all the values of X1.
And if the values are almost the same.
In extreme cases, if the values are exactly the same among different instances, we will say this particular
feature X1 is not informative because it does not.
It does not have any variation.
It does not have any information.
And it cannot discriminate different instances.
That's the extreme case in.
I mean, it is not required always that all the features that must be same to get all the values
of the feature, they must be exactly same to get this feature eliminated.
We can we can compute the variation or the variance of this X1.
The variance can be computed by easily.
For example, we compute the sample mean we compute all the average values, the average value of all
these values of this feature X1.
And we can compute the variance, which is the sum of squared deviations from the mean, so mean, so
variance at the end of the day is a number.
And if that number variance.
Let me call this variance sigma one.
That is the let's say standard deviation or square of in this variance.
If that variance is smaller than a particular threshold threshold is just a number that is a significance
level.
Whether the feature should be selected or not.
And this threshold, this threshold number that can vary from data to data.
And there is no prefixed threshold for all the datasets in the world.
We have to learn the threshold normally from the data.
Although there are certain heuristics to start with, but it is always good to learn the threshold from
the data.
So anyways, given a threshold, the significance level, if the variance is below that threshold,
that means the feature does not have enough variation to become relevant or informative.
So we can eliminate that feature and that process actually is carried out feature by feature.
So it is individual feature based.
So then we see X2 and we calculate the criteria for X2.
Then we calculate the criteria for X three and and so on.
If the feature values are normalized, we can use the same threshold for all features.
Otherwise, it is okay to use different thresholds for different kind of features if their units or
normalization values are different.
So that's low variance criteria, individual criteria by individual criteria means it rank different features
individually, it does not require any subset to compute to evaluate this particular criteria.
And it is unsupervised, which means it does not require a class label or regression target label to
carry out the feature elimination.
The next criteria that we focus on is T-Score.
And it is a supervised classification criteria.
And it was introduced by David and Samson in 1986.
And it's used for binary classification problems.
The idea is really simple.
It actually tries to maximize the variation between different classes.
The two classes.
So pick the feature that maximizes the the variation which maximizes the separation or
the disk that maximize the discrimination between two classes.
So, for example, again, it's an individual feature based method.
So it does not require subset generation.
So it ranks the features individually.
So very, very fast.
So let me explain.
What does that mean?
So, for example, focus on this feature X3.
And let's say we have some more samples available.
Let's say this simple and then we have, let's say some more samples available.
Let's say these samples.
Let's say these samples. Assume, for example, it's a binary classification problem.
Assume that the first sample belonged to Class one.
The second sample belonged to class, to the third sample belong to class three.
The.
Sorry.
That's class one.
The fourth sample belonged to class two.
Then the fifth sample belonged to class one.
Then class one.
Then two.
And then two.
Let's say here I have drawn the eight samples.
So what we will do here is let's say we pick feature X three.
And we pick all the samples that belong to class one.
Let me highlight the sample.
The feature values that belong to Class one.
So this belongs to class one.
So I will pick this value.
This belongs to class one I will pic.
This value.
Then this value then this value.
So all these green circle values, they belong to class one.
So I will pull them out.
Pick them out.
These and these four values for these four values.
I will compute the mean of these values and standard standard deviation of these values.
So let me call the mean as mean one and standard deviation as standard deviation one for for all the
feature values feature X three for all the values that belong to Class One.
Now, I will pick the rest of the values that belong to basically class two the other class.
So, for example, this belongs to Class two.
This belongs to class two.
And these two values, they belong to class to as well.
Pick them up, pick them and all these values they need.
The number of these values need not to be the same here.
I have taken four and four values.
They may be unbalanced.
One for class one, there may be fewer or more samples than this number of samples for class two.
and I computed mean two and standard deviation two for these values.
Now the T score for are the T criteria for this feature.
X three is the following.
If X three is really relevant, then it will try to discriminate the two classes in a maximum way which
means mean one and mean two.
They should be very far apart.
So a feature that is that is relevant are informative.
As long as this classification goes, that feature should maximise the the difference between the average
values of the of the feature values of different classes.
So here, if you see Mu one, that is the average value of class one.
Mu two is the average value of class two for this feature X three.
And if this value is maximised.
Then X3 can be regarded as an important feature, this value sometimes is normalized by the number of
instances belonging to Class one and Class two.
And this normalization helps to make the things balanced.
And the normalization normally is carried out like so.
So this is.
Sigma one square over n one where n one is the total number of sample, total number of values for
class one for this particular feature.
And then we have sigma two square over.
n two, again, the square root of all that.
OK, so so the the overall goal is you go you try to a feature that actually maximizes the difference
between the mean values averages.
And it basically tries to minimizes the the standard deviations are the only variation within each
class that should be selected.
So it has some resemblance to Fisher score as well.
Where we try to we are the fit in the Fisher score.
The criteria to maximize is the between class variation by minimizing the within class variation.
Although it has no direct relationship to that.
But things can be thought of in that way that you try to maximize the difference between two classes
by maximizing the difference between their averages and then try to make each class as compact in its
own as as possible.
So that's a T score.
And the feature that has this value maximum will be selected.
So at the end of the day, we will compute the T score for x1 x2, x3 X four all of these individually
and we will rank them based on their T score and will pick the top, let's say, top K features.
And those features are selected. A closely related score.
Another score is Chi-Square Score, which basically it was introduced in 1995 by you and said, you
know, it utilizes the test of independence to access whether a feature is independent of the class
label or not.
Again, the difference here between from T-score to chi square score is chi square score.
is multiclass.
So it does not require only two classes.
It is, again, a supervised method, and it is basically it can handle multiple classes.
And the idea at the end of the day is, again, select the features that somehow maximizes the separation
between classes.
And in this particular way, we basically test the independence of a feature, independence of a feature
with a class label.
And the features that are dependent to the class label will consider the feature as a random variable.
And this class label has a random variable.
So if the two random variables are highly dependent, then this feature.
So whatever feature, whatever random variable that is highly dependent with with the class label is
kept and those features that are not dependent or less dependent.
There's a there's a dependence measure.
There they are.
They are eliminated in Chi-Square.
I must mention here, a most recent, most recent criteria that is introduced recently or more advanced
criteria that has the same independence kind of notion, which is called HSIC.
This is a Hilbert-Schmidt independence criteria.
And that criteria also utilizes the independence between to a feature and a class label, for example.
So probability of X and Y, the joint probability of X and Y minus probability of X into probability
of Y.
So if this number is maximum, if the two if this and that they are same, then we say X and Y, they
are dependent because if the joint distribution is the product of individual or marginal distributions,
then the two variables are independent.
And if they are not independent, then this particular number will be large and square of it.
Is the HSIC score.
But but some difference is here.
The HSIC is not computed in the original space.
The features are normally transformed to a higher space.
Higher dimensional space in another space are sometimes called the Hilbert space through a kernel.
And there.
This independence is computed by computing correlations.
Of a lot of different kind of moments.
That was that is a consequence of going into higher dimensional space.
Oh, OK.
I have used a lot of technical, maybe complicated terms by ex in ex in the explanation of this HSIC.
If you don't understand these kernel or Hilbert spaces, something like that.
That's perfectly OK.
I just want to mention that Chi-Square also rely on independence of feature and the class label.
And there is a more recent score called HSIC that also has same kind of notion, although the evaluation
of the criteria is is completely different.
So low variance unsupervised T score individual feature based used on binary classification Chi-score
Chi-square score.
There are so many others.
For example, Gini Index or something called Gini Index or CFS.
There are so many other statistical scores as well and most of them are individually based.
One thing that I really want to mention is the the statistical based method.
Most of these methods.
They are individual, individual feature based methods and hence and hence.
Most of them, they can not handle feature redundancy, because if each feature is selected individually
with respect to the blast label or an unsupervised way individually, then it is possible, for example,
that the feature X I is highly relevant individually and feature X J.
That is also highly relevant individually.
But then what happens is the two features are highly correlated with each other, which means having
only one of them is as informative as having both of these, and one can be eliminated by exploiting
the correlation between these two features.
And so if both X, I and X, J individually are highly ranked, but they are correlated together because
we say that X, I and X, J, they are redundant and redundancy is again not the true redundancy is
or is not a great thing because we can eliminate the dimensions, we can reduce the dimensions again
by reducing the redundancy in in features.
So most of these statistical based methods, they cannot handle redundancy in the next video we will
see information theoretical based methods which actually exploit this redundancy as well.
So, yeah.
So we will see that in information, theoretical based filter methods criteria, they actually exploit
not only the relationship of feature with class label, but also the relationship of features among
themselves to to actually reduce the level of redundancy as maximum as possible.
So hope to see you in the next video with information theoretical based filter criteria.