Welcome to Section 5 in this section I will cover several The methods that rely on statistical tests
to select features from a data set.
These methods still evaluate each feature individually.
But this time in the light of that I get we intend them to predict
these methods.
As I said our statistical tests and formulations specifically I will cover selected features by use
in information gain the fishes score selective features using univariate statistical tests and so little
features by determining the rock you see or the roots.
Million square error depending if it is a classification or regression problem.
In general the feature selection methods from this section are performed in two steps in the first step.
The statistical test any of the statistical tests Rank's the features based on certain criteria.
Each feature is run independently of the other features based solely on their interaction or relationship
with the target in the second step.
The features with the highest rankings are chosen to be in the classification or regression mother's.
How many of the highest ranking features to select is somewhat arbitrary and usually uses that's the
limit.
I will demonstrate how to put the of in them next lectures.
The Pantages of these methods are that they run really fast and not competition will expensive that
this advantage is that they do not contemplate feature redundancy.
You have to screen for duplicated uncorrelated features in previous upper tier steps.
Also this election procedures do not contemplate feature interaction.
This is if one feature in isolation is not a good predictor but it does when combined with the second
feature.
These filter methods will not see that and will remove features on an individual assessment basis.
This is a mess as described in the previous sections evaluated solely.
Each feature for example if they were constant or quasi constant or they compare a feature to other
features in the dataset looking for duplications or correlations the the methods described here.
Work a bit differently.
They evaluate their features in the light of the target.
Each feature is assessed against the target individually and therefore it does not contemplate feature
redundancy.
For example in the case of duplicated features.
They were drunk equally high with any of these methods and therefore both will be selected.
So we see the importance of using these methods in combination with the others to evaluate features
all together.
Let's begin by describing mutually information Luchon information is a measure of the major dependence
between the two variables.
In other words it determines how much by knowing one variable do we know the other one.
The concept is similar to the one of correlation.
However mutual information is more general.
I like that it determines how similar the joint distribution of x and y.
In this case one variable and the target the probability of X given wide is the product of the individual
distributions of x and y which here.
I describe us the probability of X times the gravity of Pi.
This is in other words how similar is the occurrence of y even.
X to the probability of books of both x and y occurring separately.
Intuitively Mitchard information measures the information that x and y share it measures how much you
know in one of these variables reduces the uncertainty of all the other.
For example if x and y are independent then no X does not give any information about white and vice
versa.
So their mutual information is zero.
At the other extreme if x is a deterministic function of y y is a deterministic function of x then all
the information conveyed by X is shared with y.
No when X determines the value of y and vice versa.
As a result in this case the mutual information is the same as the uncertainty contained in white or
the uncertainty contained in X alone.
In other words this is the entropy of why the entropy of X for those interested in the formula of the
mutual information by you I share it here in this light.
Another way of evaluating features is by using the fishless score the fisher score is another statistical
test that measures the dependency between two variables it uses the square distribution.
It is suited for categorical variables I'm finite classification only in attention definable by use
should be non-negative.
Typically a Boolean or frequencies or counts.
This is in essence finity finished and not continuous variables.
This is just called to give you an intuition compares the observed frequencies of divided the class
within each of the labels of the variable against the expected frequency of the class should divide
will have only one label
to give you an indication of how the fish's core works.
I created this example table.
Let's imagine that this is the feature gender from that a tiny data set.
And we're trying to predict whether the passengers survived based on gender.
We can create contingency tables like this one here we can see that in this imaginary data set.
There are two males that survived the tragedy of 10 that did not whereas there are nine women that survived
and only three that didn't.
We can calculate the roll totals by some in the 1 9 which is 11 and then three which is the thing I
would do the same for the column starters which I show here
distorters will allow me to get the expected frequencies here for example to divide it 12 is 0.17 and
then divide it twice is 0.38.
This indicates that only 17 percent of males survived the Titanic for women.
We see that nine divided 12 is 0.75 and 3 divided 12 is 0.25.
This indicates that 70 75 percent of women survive the Titanic.
We can see straight away these distributions are very different and therefore gender should be a good
indicator of survival.
What the official score does is it compares these distributions versus the expected distribution of
all the passengers not segregated by gender which we see here upon 46 is 11 right 24 and 0.50 4 is 13
divided 24 but making this comparison the test concludes in this case that gender is significant with
a p value obtained from the T-square distribution.
For those interested in the formula I share the extract of this article which I also quote later in
the additional reading resources for those interested in further reading
tests on specifically.
I know the measure of the dependence of two variables their center for counting is by levels where the
time it is binary and what the ABA does is it compares the distribution of the value of all.
When the target is one versus the distribution of the value of all where the target is zero NRA assumes
a linear relationship between the target undesirable and it also assumes that the predictors are normally
distributed.
So if your features do not follow these assumptions this method might not be a great choice.
Sgolne has an implementation of a no other can be extended to regression problems where the target
is not binary is rather continuous.
In this case they evaluate the correlation coefficient of the predictor variables and the target and
then using the correlation coefficient and a transformation they can obtain a p value which indicates
the importance of the feature in relation to the target.
It says that the sample size and in very big data sets most of the features will end up being significant.
This is with p values below 0.05 these small p values indicate the distribution defect where the time
it is 1 or 0.
However in fact the difference between these distributions might be tiny.
Therefore the important thing when using this method to select features is to compare the p values among
the different features rather than pay attention to the p value itself.
What I'm trying to say is that the message may be a good indication to compare the relative importance
of the different features but not the intrinsic importance of each feature.
Respect the target because these will be inflated by the sample size.
Finally another slightly different dimethyl consists in determining the rock you see or the root.
Medium square error or medium square error.
For every feature.
Respect to the target.
I would say more on these in a minute this method also evaluates the dependency between two variables.
But this time it uses a machine learning algorithm to evaluate this dependency.
It can be used with all types of Bibles anti-X and it makes no assumption on the distribution of two
variables.
The procedure consists in for every single feature built one machine learning algorithm.
Typically a decision tree and then use in the decision tree make predictions and use those predictions
to determine the rock.
You see other rogue million square error.
Then we rank the features according to these values.
This is true these rock a uses or million square arrows.
We select the highest ranking features from all the methods presented here.
This is perhaps the most powerful but it also has the weakness that it does not contemplate feature
redundancy.
And in the extreme example duplicated features we show the same rock or Usie and therefore both will
be kept
what shall we put the threshold to select or remove features when using this method.
In reasonable OK you see a value of zero point 5 mins random death for all features with rockeries Z
equal to 0.5 could be removed.
Whether we also want to remove those features with a low rock and use the values for example 0.55 is
up to the user and that the session will generally be influenced by how the data set is and other feature
selection methods that we can use in addition for the root mean square error or the medium square error.
The cutoff is less intuitive.
You could select those which got a list of all the mean cut off of all the features.
This is in fact how sikat learn implements the default cutoff for feature selection.
I would say more on this in the coming lectures.
For now this is all I'll see you in the next media.