Let's use lotta's approach of transforming variables on our
Facebook data set. Most of our variables, such as friend
count, likes, comments, wall posts and others are variables
that I would call engagement variables, and they all have
very long tails. Some users have 10 times, or
even 100 ,the median value. Another way to say this
is that some people have an order of magnitudes, more
likes, clicks, or comments, than any other users. In statistics,
we say that the data is over dispersed. Often it
helps to transform these values so we can see standard deviations,
or orders of magnitudes, so we are in effect, shortening
the tail. Here was our histogram of friend count from before,
and notice, we still have that long tail. We can
transform this variable by taking the log, either using the natural
log, log base 2, or log base 10. We could use
other functions, such as the square root, and doing so helps
us to see patterns more clearly, without being distracted by
the tails. A lot of common statistical techniques, like linear
regression, are based on the assumption that variables have normal
distributions. So by taking the log of this variable, we can
transform our data to turn it into maybe a normal
distribution or something that more closely resembles a normal distribution, if
we'd be using linear regression or some other modelling technique.
Now, I know we're not doing modelling here, but let's just
see what it looks like to transform the variable. First, I'm
going to just do this in the summary command. So, here's our
regular summary of friend count. Looks like the median friend count
is 82, and the mean is 196. I can take the log
base 10 of this friend count and get a different table.
Now this seems a little bit unusual since I have negative
infinity for the minimum and negative infinity for the mean. Hm.
So what must be going on? Well, some of our users have
a friend count of zero. So, when we take the log
of base 10 of zero, that would be undefined. For those familiar
with Calculus, the limit would be negative infinity, which is why
that appears here. To avoid this, we're going to add one to friend
count, so that way we don't get an undefined answer, or
negative infinity. There, that looks much better. And, just to show you
another function, let's also use the square root on friend count.
This would be another type of transformation. For me, log base 10
is an easier transformation to wrap my head around, since
I'm just comparing friend counts on orders of magnitude of 10.
Basically a tenfold scale like the pH scale. Now that
you've seen transformations within summaries, let's see if you can apply
a similar transformation to the histogram. Check out the instructor
notes to learn how to use scales and how to create
multiple graphs on one page. Once you've read through those
links, and you think you're ready, try this next programming exercise.
In it, you're going to create three different histograms. The first one will
be our original friend count histogram, and then the second one will
have the friend count transformed using log 10. And then the last
histogram will have the friend count transformed using square root. Now, I
haven't given you all the information to do this exercise, but I
want you to be resourceful. Part of being a good programmer is
being able to read documentation and know how to find your answers
to the questions you may have. Now, we've already done the legwork for
you by providing the resource. So, let's see if you can figure it out.