Okay, so here we are.
Now, we are going to illustrate this web application that we introduced at the end of
the last lecture on sampling distributions and sampling variance.
Again, you can click on the link provided to take a look at this web application and
play around a little bit with these notions
of sampling distributions and sampling variance.
For those of you who are interested,
this web application is in our Shiny App,
we'll touch on that a couple times throughout the specialization,
but this was generated using RStudio actually.
So, let's introduce this web application a little bit here.
We're talking about the notion of sampling from multiple populations, okay?
So, are we drawing a representative sample from an entire population
or do we only sample from a particular subgroup of that population?
What can happen when we make these two different choices?
So, here's the scenario that we came up with
when designing this particular web application.
There's two possible choices in terms of our target population.
We could either be talking about the entire University of Michigan,
or we could talk about University of Michigan students faculty and
staff who go to the gym or who are members of a gym, okay?
So, this is like a sub-population within the larger university of Michigan population.
So, the first choice you can make with
this web application is what your target population is.
Are you only interested in gym goers?
Are you interested more generally in the entire University of Michigan population?
A second option is how many samples you'd like to draw.
So, this is a hypothetical number of samples to select.
Again, we talked about the notion of sampling distributions arising from
hypothetical repeated samples following the same probability sampling design.
So, you can actually toggle how many samples you wish to select.
We're going to start with 50, but you can click and
drag and decide how many samples you want to select,
anywhere from five up to 100.
The second choice you can make or actually the third choice,
is the size of each sample.
So, remember, we talked about in probability sampling,
the notion of the central limit theorem,
and the idea that the larger the size of each sample the less sampling
variability that you're going to have and the less
sampling error that you're going to have in any one individual estimate.
Because you're getting closer and closer to the size of the actual population.
So, the third choice you can make is how big you want each sample to be.
You can see here we're starting at 20,
but you could go all the way up to 100 if
you wanted to with the size of your actual samples.
Then, the fourth choice that you can make is
what percentage of the University of Michigan actually goes to the gym?
So, this defines the features of the target population.
How big of a subpopulation are the gym goers?
We start with them being only 20 percent,
say of students faculty and staff,
but we could move this all the way up to 100 percent,
say that everybody in the University of Michigan goes to the gym.
This dictates what the population distribution of
values on a variable of interest is going to look like.
What fraction of the population ultimately belongs to this subpopulation.
So, when you've made these choices in the web application,
what you can then do is click down here where it says Draw samples.
What this is going to do,
it's going to simulate repeated samples,
50 in this particular example,
from the population that you've selected with the features that you've selected,
what percentage goes to the gym,
and then illustrate the features of that population and the sampling distribution, okay?
What does that sampling distribution look like?
Does it tend to be centered on the value that we're interested in?
Or is it shifted away from the value that we're interested in?
Now, when you're setting out to do these kind of studies,
there always has to be a variable of interest.
When you're collecting these data,
there's a variable that the researchers are interested in when selecting the samples.
For this particular web application,
we're going to keep it pretty simple and just talk about body weight, okay?
So, the variable of interest in this case is body weight.
So, we're going to look at the distribution of values on
body weight for everybody in the population when we draw our samples,
and then we're going to look at the distribution of the sample estimates for
all 50 samples that we're ultimately going to be drawing.
The estimate of interests that we're interested in computing from
each sample is the mean body weight.
So, the variable of interest is body weight,
we measure that for everybody in our sample,
and then we compute an estimate of the mean body weight for the entire population.
We want to see if that sampling distribution of estimated mean body weights
is more or less centered on the true population value or if it shifts away.
Okay. So, let's give this a go here.
We're going to start by selecting all U of M as our target population.
We're going to draw 50 samples.
Each sample is going to be of size 20.
We're going to start by assuming that 20 percent
of the University of Michigan ultimately goes to the gym.
So, I'm going come down here and click on Draw Samples.
It's going to take a second to generate all those hypothetical samples.
Then, you'll see some pictures show up here on
the right-hand side of the web application, okay?
So, let's make a couple of things clear first of all.
This first picture shows the distribution of values on the variable of interest,
weight, for the entire population.
So, remember, this is not a sampling distribution.
This is the distribution of all the values on the variable of interest.
We break this down by different groups here.
So, the red line shows the distribution of body weight values for all of U of M, okay?
So, that red line is that red solid line that you see here,
that's what the distribution of body weight looks like for all of the U of M. That's
the distribution of values on the variable of interest in the population.
The dash blue line,
that's the distribution for gym goers.
So, you can see a key feature of the population here is that gym goers,
their distribution of values tends to be much
higher than the distribution for non-gym goers.
You see, non-gym goers are represented by this dash green line over here,
whereas gym goers are represented by this dash blue line.
So, those two subpopulations,
gym goers and non-gym goers they're very
different in terms of distributions on the variable of interest, body weight.
So, there's a strong correlation between whether or
not you go to the gym and what your body weight is.
We set this up artificially for this particular web application
just to illustrate what can happen when drawing different kinds of samples.
So, you see the overall population distribution.
Why is this piece lower here?
Because we said that only 20 percent of U of M goes to the gym.
So, if we look at the overall population distribution,
there's less weight down here among gym goers.
But the dash green and
the dash blue lines show what each of those subpopulations look like.
So, they're artificially designed to be pretty
different from each other just to establish that correlation.
Okay. Now, we wanted to draw 50 samples of size
20 from this particular population and then look at the resulting sampling distribution.
So, let's scroll down a little bit in the web application.
Now, what we see is another red line and in this second visualization here,
that red line shows the sampling distribution of the estimated
means based on those 50 samples that we selected, okay?
So, we selected 50 samples from this population up here.
For each of those 50 samples,
we computed the mean body weight and then we plotted
the distribution of those 50 estimated means.
You can actually see those 50 estimated means
in the blue dots down here in the web application, okay?
The red line then shows the distribution of those 50 sample estimates.
That's our sampling distribution.
Remember, it's hypothetical.
In practice, we wouldn't draw this many samples and measure each of those samples.
This is purely hypothetical to illustrate these ideas, okay?
So, what do we see from this sampling distribution?
The mean of the sampling distribution is in black, okay?
So, we have this up here in the text.
We want to see how that mean of the sampling distribution,
the mean of all those 50 estimated mean body weights that we just computed in black.
We want to see how that compares to the overall population mean,
and that overall population mean is up here this blue line.
So, you can see that blue line,
it's something pretty close to 160 actually.
So, that blue line is the overall population mean.
That's what we're trying to estimate with our survey and with our sampling.
Okay. So, that blue line is repeated down here in
the sampling distribution and you see
a very key property of probability sampling to begin with here,
this random probability sampling.
Look how close the black line which is the mean of those 50 estimated mean body weights,
and the blue line which is the population mean that we're interested in,
look how close those are to each other based on this random sampling.
So, our sampling distribution of estimates is
more or less centered on the true population value.
So, the mean of those 50 estimates is going to be
more or less equal to the population mean that we're interested in.
What this means is that we're computing unbiased estimates.
So, on average, those estimates from the 50 samples,
on average, those estimates will be equal to the true population mean,
that's one very important property here.
What we also see is the notion of sampling variance.
So, our sampling distribution is centered at the true population mean,
but look at how the estimates vary around that true population mean.
We talked about previously in the lectures,
any one sample represented by these blue dots is not going to
be exactly equal to the true population quantity.
We might get lucky, you see that some of these blue dots are very close
but in general there's going to be sampling error when we draw these random samples.
You can see some of these dots they veer pretty far
away from that overall mean which is close to 160.
Some are higher than the overall population mean, some are lower.
These sample estimates, they all vary.
There's going to be sampling error associated with these different estimates and that
those sampling errors define the variability of this sampling distribution.
So, based on only one estimate,
only one estimate from the data that we're collecting,
we can say we estimate the population
mean body weight to be something based on that one sample.
Then, we can estimate the variability of
this sampling distribution based on that one sample,
and talk about how variable we expect
that estimate to be when making population inference.
We can say that depending on how variable this sampling distribution is,
we're fairly confident that the population mean body weight
is within a certain range based on what we know about that sampling variance,
how variable our sample estimates are.
You can see, in general, the sampling estimates,
they are pretty tightly centered around this overall mean that's close to 160.
They would suggest that there is potentially
some variability in our estimate but overall,
we would expect our estimates to be very close to the true population quantity.
Remember, in reality, we're only selecting one samples,
so we might end up with this sample,
we might end up with this sample.
We want our probability samples to have the characteristic that on average,
they're equal to the true population quantity and we're
seeing that simulated here with this application.
So, that's where we sample,
we drew a representative sample from all of U of M in selecting our samples.
Now, let's see what happens if we take a non-representative sample.
Let's say we're trying to estimate the mean body weight,
and we decide to go up to the on-campus fitness center and select a convenience sample.
We stop everybody as they're leaving the fitness center,
maybe after they worked out and we ask if we can take their body weight.
We collect data that way.
So, it might not be as expensive to do this,
but we still try to do this with
a convenient sample of people who are actually at the gym.
Maybe there's some equipment there that allows us to measure body weight.
So, we decided to draw a sample of gym goers.
Let's keep everything the same here in terms of our parameters.
We're drawing 50 samples,
each sample is going to be of size 20,
and 20 percent of the population ultimately goes to the gym.
So, let's go ahead and redraw the samples.
We're going to draw a new set of samples.
So, population looks exactly the same.
We have the same basic properties as the population,
20 percent goes to the gym.
What really changes now in this new simulation is the sampling distribution.
Blue line, our population mean body weight of interests still in the same place.
It's still pretty close to 160.
But look what happened to the sampling distribution.
Because we were drawing a convenience sample,
non-probability sample of gym goers.
Look how far away from the population mean that sampling distribution got shifted.
Now, that black line which again is the mean of all those 50 sample estimates,
now it's somewhere between 180 and 190.
It looks like it's about 185.
So, in drawing a convenience sample of a particular subpopulation,
when that subpopulation is very different from
the larger overall population in terms of the variable of interest,
remember gym goers had a much higher body weight.
So, to very different subpopulation relative to
the rest of the population are sampling distribution,
all 50 of these estimates,
now they're centered at the mean for gym goer's.
The sampling distribution is no longer
centered at the blue line which is the mean for the entire population.
It's now centered at the mean for gym goers,
because we drew a convenience sample of
a subpopulation that was very different from the rest of the target population.
So, you can extend this idea to say a sample on the street or a snowball sample,
if you only contact people with very specific characteristics and collect data from them,
there's a strong chance that your sampling distribution could be shifted
very far away from the true population value that you're interested in.
This is what we referred to as sampling bias.
If we were to compute an estimate of the U of M,
average body weight based on this particular sample,
we would have a very biased estimate.
Given the sampling variability,
we would conclude that the body weight is somewhere between
maybe 182 and 187, or something like that.
We'd be really far off from
the actual true body weight or the true mean body weight for the overall population.
So, when we draw probability samples,
remember we still have a nice normal sampling distribution here,
we can estimate sampling variance but our mean was extremely biased because we drew
a sample from a subpopulation coming from convenience or something like that,
depending on how we got those data from that particular subpopulation,
but we end up with a lot of sampling bias.
Now, let's see what happens again if we take another scenario.
Let's bump the size of each sample all the way up to 100.
So, we came about some more money,
some more resources to do the data collection.
We're still going to draw 50 samples hypothetically but now,
each sample is going to be of size 100.
We'll go back to simulating all U of M when draw our samples.
We draw a representative sample from all U of M,
not just gym goers.
So, let's draw that sample again.
Population looks the same.
What happens to the sampling distribution?
It shrinks.
We increase the size of each individual sample.
From this particular population,
we increase it up to 100.
Now, look what happens, that black line,
the mean of the sampling distribution lays almost on top of the population mean.
So, we have a more precise estimate,
there's less sampling error on average,
and if you look at the variability of those estimates,
they are much more tightly clustered around the true population mean.
So, it's nice that with that additional money,
we were able to draw larger sample sizes that gave us more precise estimates.
There's less sampling variability in these estimates and we're
more tightly clustered around the true population quantity,
that's a great thing,
but it was more expensive.
It's more expensive to draw samples of size 100.
Let's see if drawing samples of size 100
would help us out at all if we had our bias sampling mechanism.
Again, convenience sample, non-probability sample, volunteers,
whatever the case may be,
but we largely get gym goers in the sample.
Let's see if this helps us at all.
So, we redraw the samples focusing on gym goers only.
Scroll down a little bit.
Again, we have a narrower sampling distribution.
There's less sampling variability,
but we still have the same bias problem.
Because our sampling mechanism,
we only got people who are gym goers,
and gym were very different in terms of their mean body weight.
So, very precise estimate but
a very biased estimate compared to the larger overall population.
So, we can play around this a little bit more.
If we go to all U of M,
maybe just hypothetically we increase the number of samples to 75.
Let's take the size of each sample down to 50,
and let's increase the percentage of U of M that goes to the gym.
Let's make it a little bit less rare and see what happens when we do this.
So, we'll draw the sample again.
Now, the population distribution changes a little bit
because we increased the percentage of U of M that goes to the gym,
so we have a little bit more weight out here.
Now, the population mean has shifted a little bit away from 160 because
more of the population comes from
gym goers which we toggled this feature, the population.
Still we have these very nice features of the sampling distribution.
We can estimate the variability of our estimates
based on those hypothetical repeated samples
and all of those estimates are more or less tightly
clustered around the true population quantity.
But look what can happen.
Some samples may have a very unusual estimate.
That's sampling error.
We again want our sample estimates on average to be equal to
the true population quantity and we can achieve this with random probability sampling.
Now, just to illustrate an extreme,
what happens if all of U of M goes to the gym.
There's no longer distinction here.
Everybody at U of M actually goes to the gym.
Let's draw our samples and look at the population.
Now, everybody goes to the gym.
So, you see the red line in terms of distributions on the variable of interest?
You see that red line lies right on top of the distribution for the gym goers,
and the population mean is the mean of the gym goers.
So, that's changing the features of our population.
Now, with our random probability sampling,
all of the sample estimates are tightly clustered around the overall population mean.
We have a sampling distribution again, bell-shaped again,
but it has lower sampling variability because we're
using a larger sample size and we can make
reasonably good inference given variability in the estimates about
the range of values that the population mean might take.
If we turn the size of each sample down to 10,
let's say and let's say 50 percent of U of M,
went to the gym, in simulating this, let's see what happens.
Now, we have an equal distribution,
half of U of M does not go to the gym,
half U of M goes to the gym.
Look what happens with a sample of size 10.
If each of these samples is of size 10,
look what happens to sampling distribution?
It really spreads out.
It's still nice and centered at the true population value,
you see the blue line and the black line,
lie right on top of each other but there's much more variability in the estimates.
You can see with a smaller sample,
there's much more spread in the different estimates.
The estimates on average, there's still unbiased.
They still equal the true population mean on average,
but there's much more spread.
That sampling distribution is much more spread out when drawing lower sample size.
So, that might not cost as much money but
there's just going to be much more variability in our estimates,
and although the estimates on average will be equal to the true population quantity,
we have much more uncertainty.
The population mean might range between 160 and 180.
We have a lot more uncertainty with smaller sample sizes.
So, you can play around a little bit with this Web application,
but this should give you a flavor of some different design decisions
and how some design decisions could give you very biased estimates.
But this illustrates these ideas of sampling distributions and sampling variance that we
wish to illustrate in this section and
to illustrate how important sampling variances to making inference.
We can have precise statements about
the population or we can have uncertain statements about the population.
It's just very important that on average,
our estimates be equal to the true population quantity.
Probability sampling, where everybody has a known probability of
selection allows our sampling distributions to behave like that.
So, if at all possible,
we'd like to try and work with probability samples in practice.