So let's try running a k-Means cluster analysis in Python.
First, we will call in the libraries that we will need.
In addition to the pandas, numpy, and matplotlib libraries we'll need
the train_test_split function from the sklearn.cross_validation library,
and the pre processing function from the sklearn library.
Finally, we will need the k-Means function from the sklearn.cluster library.
After calling in the data set using pd._read_csv
function we will do some data management.
Namely, we will create a new data set called data_clean in which we will delete
observations with missing data on any of the variables using the dropna function,
then we will create a data set called cluster
that includes only our clustering variables.
In cluster analysis variables with large values
contribute more to the distance calculations.
Variables measured on different scales should be standardized
prior to clustering, so
that the solution is not driven by variables measured on larger scales.
We use the following code to standardize the clustering variables to have a mean of
0, and a standard deviation of 1.
To standardize the clustering variables we will first create a copy of the cluster
data frame and name it cluster var, then we use the preprocessing.scale function to
transform the clustering variables to have a mean of 0 and a standard deviation of 1.
First, we list the name of the clustering variable, then an equal sign, and
preprocessing.scale, then in parentheses we type the name of our variable again and
add .astype, and then in another set of parentheses, float64 in quotes.
Astype float64 ensures that my clustering variables have a numeric format,
and we will do this for all the clustering variables,
then we will use the train test split function in the sklearn cross validation
library to randomly split the clustering variable data set into a training data set
consisting of 70% of the total observations, and
a test data set consisting of the other 30% of the observations.
First we'll type the name of the data set which we'll call clus_train followed
by the name of the test data set which we'll call clus_test,
then we type the function name,
train_test_split, and in parenthesis we type the name of the full
standardized cluster variable data set which we called clustevar.
The test_size option tells Python to randomly place 0.3,
that is 30% of the observations in the test data set that we named clus_test.
By default the other 70% of the observations are placed in
the clus_train training dataset.
The random_state option specifies a random number seat
to ensure that the data are randomly split the same way if I run the code again.
Now we are ready to run our cluster analysis, because
we don't know how many clusters actually exist in the population for
a range of values on the number of clusters before we begin we'll import
the cdist function from the scipy.spatial.distance library.
In this example we will use it to calculate the average distance of
the observations from the cluster centroids.
Later, we can plot this average distance measure to help us figure
out how many clusters may be optimal, then we will create an object called clusters
that will include numbers in the range between 1 and 10.
We will use this object when we specify the number of clusters we want to test,
which will give us the cluster solutions for k equals 1 to k equals 9 clusters.
In the next line of code we create an object called meandist
that will be used to store the average distance values that we will calculate for
the 1 to 9 cluster solutions.
The for k in clusters: code tells Python to run the cluster analysis code below for
each value of k in the cluster's object.
That is to run cluster analysis specifying 1 through 9 clusters,
then we will use the k-Means function From the sk learning cluster library
to run the cluster analyses.
In parentheses n_clusters indicates the number of clusters,
which in our example we substitute with k
to tell Python to run the cluster analysis for 1 through 9 clusters,
then we create an object called clusassign that will store for each observation
the cluster number to which it was assigned based on the cluster analysis.
After the equals sign model.predict asks that the results of the cluster analyses
stored in the model object be used to predict the closest cluster that each
observation belongs to, then we are going to calculate an average distance measure
for each cluster solution independent to our currently empty mean dist object.
The code following meandist.append computes the average
of the sum of the distances between each observation in the cluster centroids.
The formula first calculates the distance between each observation and the cluster
centroids where the number of centroids is equal to the number of clusters that were
specified in the cluster analysis in the first set of parentheses.
We use the cdist function from the scipy spacial distance library
to calculate the distance.
In parentheses we type the name of our clustering variable data set clus_train
followed by a comma, and model.cluster_centers_,
which is the name of the model results attribute that contains the cluster
centeroids, then we type another comma and the word Euclidean in single quotes.
This tells Python to use cdist to calculate the distance between each
observation in the clus_train data set in the cluster centroids using
Euclidean distance, then we use np.min function to determine the smallest or
minimum difference for each observation among the cluster centroids.
Axis equals 1 means that the minimum
should be determine by examining the distance between the observation and
each centroid taking the smallest distance as the value of the minimum,
then we use the sum function to sum the minimum distances across all observations.
Finally, the / clus_train.shape with 0 in brackets divides the sum of the distances
by the number of observations in the clus_train data set where the .shape with
0 in brackets code returns the number of observations in the clus_train data set.
Now that we have the average distance calculated for
each of the 1 to 9 cluster solutions we can plot the elbow curve
using the map plot lib plot function that we imported as plt.
In parenthesis clusters is the object that includes the values of 1 through 9 for
the range of clusters we specified and
meandist is the average distance value that we just calculated.
So what this plot shows is the decrease in the average minimum distance of
the observations from the cluster centroids for
each of the cluster solutions.
We can see that the average distance decreases
as the number of clusters increases.
Since the goal of cluster analysis is to minimize the distance between observations
and their assigned clusters we want to chose the fewest numbers of clusters
that provides a low average distance.
What we're looking for in this plot is a bend in the elbow that
kind of shows where the average distance value might be leveling off
such that adding more clusters doesn't decrease the average distance as much.
You can see how subjective this is though.
There appears to be a couple of bends at the line at two clusters and
at three clusters, but it's not very clear.
To help us figure out which of the solutions is best we should further
examine the cluster solutions for at least the two and three cluster solutions to see
whether they do not overlap, whether the patterns of means on the clustering
variables are unique and meaningful, and whether there are significant
differences between the clusters on our external validation variable GPA.