Hi! In the most recent lectures we
learned about the t-test and how type 1
type 2 errors confidence levels,
and power all are related to each other.
In this lecture we're going to introduce
what's called
[any] an effect size, that measures the size of
an experimental effect
and that in turn is going to enable us
to
be able to design studies, at least the
part where we decide things like
the sample size that we're going to need.
But to start off with, let's just
consider the question of how big
a difference between a sample statistic
and a hypothetical value should be, for
that difference to seem
worthwhile or important. Effect size
measures
are what can help us answer that
particular question.
There are at least two ways to evaluate
how
big such a difference is, we're just
going to focus on two of them here.
The first one is relative to the range
of the scale
that's being used for that variable, this
is
oftentimes known as scale based
evaluation.
And the second one is relative to the
dispersion of scores,
in other words how spread out the scores
are.
So it's relative to how the how the
scores are distributed,
and so it's it could be thought of as
distribution based
evaluation. Scale based effect sizes
use the units of measurement in the
scale of the dependent variable. For
example,
consider reaction time ,let's suppose is
being measured in milliseconds,
then a natural thing to do would be to
compare
the means of two different experimental
conditions reaction times
by just taking the difference between
them and therefore measuring that
difference in milliseconds.
Now how big that difference would have
to be for us [to be]
to regard it as important or not,
would depend
a lot on the nature of the task-- the
context of the experiment
and what kind of difference we were
interested in looking for.
So for example, say, a 500 millisecond
difference in mean reaction time between
two experimental conditions
would not seem like much if the
experimental task
is deciding on
which pair of shoes to purchase, and
we're looking at how long it takes a
person to make that decision,
given the length of time that decision
usually takes and what's at stake
involved in doing it,
a difference of 500 milliseconds would
probably seem trivial most of the time.
However if the task is fighting a fire,
or
piloting a jet aircraft, then a
difference
of 500 milliseconds in reaction time
could mean
the difference between life and death.
Distribution-based evaluation on the
other hand
measures the difference between the
means in
units that relate to some characteristic
of the distribution, and the most popular
characteristic is standard deviations.
Now let's recall that the standard error
of the mean is the standard deviation
divided by the square root
of the sample size: S divided by square
root of N.
And we've already seen that we can write
the t statistic
as the difference between the sample
mean and the hypothetical population
mean:
x bar minus mu divided by the standard
error.
The standard error again is S divided
by the square root of N.
Now what I've done here in this slide is
to take
the x bar minus mu divided by the
standard deviation
component, and I've called that d.
And so that means I can rewrite the t
statistic formula in the first equation
there:
as d times the square root of N.
d stands for the difference between the
sample mean and the population mean
divided by the sample
standard deviation, and is known as
Cohen's d after
Jacob Cohen who invented it. And it
measures
the difference between x bar and mu
in standard deviation units, because
we're dividing that difference by
the standard deviation.
Cohen's d of 2.5, for instance,
tells us that the sample mean
is 2.5 standard deviation
units above its [population] hypothetical
population mean.
A d of negative 0.5 means that the
sampling
is half a standard deviation unit below
the population mean.
Now Cohen's d is useful because it
provides
a measure of the difference between a
hypothetical value for the mean and the
sample mean,
which doesn't depend on the scale that
we're using
for our variable. Now why would we want
this?
Well, a lot of psychological variables of
interest
that end up being dependent on variables
and experiments, or
clinical studies, or surveys, don't have
fundamental units. In other words, the
units
are somewhat arbitrary. It's not like
measuring
reaction time in milliseconds, for
instance,
or measuring weight loss in kilograms,
or anything like that if we've got a
scale
measuring, say, something like
the extent to which person's
self-reporting anxiety,
or self-reporting feelings of depression,
or self-reporting
their level of self-esteem--anything
along those lines.
Usually those scales are made up of
scores by summing the person's response
on the items that go into making up that
scale.
A self-esteem scale, for instance, might
have, let's say 10 items.
Each of which asks a person about the
extent to which they
have positive self-regard. And then
the scale score the self-esteem measure
is achieved by adding up
the the scores that people got on each
of those
10 items. The difficulty is that
there might be several self-esteem
scales out there, and in fact there are
there are several published
and widely used scales for measuring
self-esteem.
And the same is true, for example, for
clinically relevant
scales such as those measuring
depression, anxiety,
psychopathy ... various other things along
those lines. There are [usually there are]
almost always more than one scale out
there.
So if we have one study using,
let's say, the Coopersmith Inventory for
Self-esteem,
and another one using the Rosenberg
Self-esteem scale.
We want to compare the results those two
studies, let's say they use the same
experimental design, and we want to
compare
the difference between the means, [and the]
[two] in, say, two experimental conditions.
The conditions are the same but the
researcher happens to
to have used Coopersmith Inventory in
the study they did, whereas the other
researchers
used the Rosenberg scale and study that
they did.
The problem is that the difference
between the means is going to be
dependent on the nature of the scale
being used there. We can't directly
compare
those differences because the scales are
different.
But if we convert both of those
differences to standard deviation units
now we're comparing the same things with
each other, and we can compare across the
two studies.
Because we're now measuring the effect
size
in a way that is independent of the
scale,
so Cohen's d is oftentimes referred to
as a scale
free measure of an effect size.
Sometimes it's also called a
dimensionless measure of an effect size.
But I think scale free probably
[is the] is the more descriptive term
to use here.
Most of the time, getting back to our
question of
how big a difference is big enough to be
worthwhile or important,
researchers just don't have a clear
criterion by which they can judge how
important
a an effect size is. And
so what's happened to some extent
is that people have used some of the
benchmarks
that Jacob Cohen offered that
he developed by surveying the social
psychological
and clinical psychological literature at
the time. This was back in the 1960s when
he was doing this,
and his reason for surveying those
literatures
was because he realized that quite often
it was difficult to to achieve large
effect sizes
[in those in] in those sorts of areas.
Because
usually the phenomena being studied are
phenomena that are complex, they have
multiple causes,
and so if we're focusing on just the
effect size of one particular cause,
say one particular experimental
manipulation,
we're pretty unlikely to get a large
effect size. So here's what he found: he
decided that,
after reviewing this literature, that
a small effect size would be a Cohen's d
of 0.2.
So in other words, just 0.2, one-fifth
of a standard deviation difference
between
two means or between a mean sample mean
and
a hypothetical population mean. The
medium effect he decided was 0.5--
half a standard deviation difference; and
large effect size
he decided was 0.8,
four-fifths in other words of a standard
deviation difference.
Now a lot of studies, a lot of
experimental studies,
especially in traditional psychological
research areas
would have difficulty reliably detecting
small effect sizes--
Cohen's d is of 0.2. Because their
statistical power is too low.
In other words, the confidence intervals
are too wide. And the main reason for
this
is simply small samples. A lot of
psychological research
has involved relatively small samples.
And so
the statistical power to detect small
effect sizes
usually is not large.
So this naturally leads to a question of
how how many people should we get for
designing a study,
if we're going to be testing a
hypothesis?
To answer this question, we have to
decide three things before we
conduct the study and before we figure
out what sample size we're going to get.
First of all, we have to decide how much
confidence we want to insist on.
Are we content with 90%, 95%,
99% ... the more confidence that we insist on
the wider our confidence intervals are
going to be, the less
statistical power we'll have. Therefore
the larger the sample sizes that we're
going to require-- so there's a trade-off
between how much confidence we insist on
and
the sample size that we require for our
study.
Secondly we have to decide how much
power we want.
Now remember power is one minus the
probability of making a type two error.
So one way to think about this
is how small a type two error rate
we're going to insist on. Would we be
content with
10 percent probability of a type 2 error?
5%? 1%? What would we want?
Okay so if we were thinking in
terms of power,
if we were willing to settle for a 10%
probability of making a type 2 error
then the power that we would need would
be 90--
power 0.9. And then finally,
we need to decide on what size of effect
we're interested
in detecting, because no power statement
makes any sense
without specifying the size of the
effect that we're
calculating the power to detect. So
a researcher needs to decide on what the
smallest effect size is
that they would be interested in
detecting.
So for think back to cohen's benchmarks,
would we
want to insist on detecting small effect
sizes
[um if i have] Cohen's d is of 0.2?
Would we be satisfied instead with the
Cohen's d = 0.5?
How low would we insist on going?
Again,
the smaller the effect size we want to
detect, the
larger the sample size we're going to
need to detect it,
for any given level of power.
So let's return to our example of the
psychologist who
has taken a random sample of primary
school kids
and just put them on a smart diet for
six months then tested their IQs
at the end of that. Now suppose
that the psychologist hasn't done study
yet,
and is trying to work out how many
children she's going to need.
Okay her null hypothesis is that the IQs
won't have changed.
So their IQ scores should behave like a
random sample from a population whose
mean IQ is 100,
and standard deviation is 15.
If only the mean is going to change
because of her diet,
and the standard deviation remains the
same, in other words it's still 15;
then she can determine her sample size
by making
the three decisions that [we just] we just
saw before.
Let's suppose that she decides
to settle for 95% confidence,
she decides she wants power to be 0.9,
and she's willing to
detect at least a medium effect size
so
she's interested in the power to
detect a Cohen's d
of at least 0.5. So 0.5 is the smallest
Cohen's d that she's interested in
detecting.
Okay, well I've got my spreadsheets
set up in front of you here, or
rather a copy of it.
And if we play around with the sample
size,
n equals whatever, then it turns out
that when we get to n equals 42
then we have power of 0.9 to detect
Cohen's d of 0.5.
Okay, so according to our calculations
here,
the psychologists would need at least 42
children to have power of at least 0.9
to detect a Cohen's d of 0.5
or larger. Now just as a side note here,
notice that Cohen's d of 0.5 given that
our
standard deviation is 15
half of 15, 7.5. So
0.5 of a standard deviation above the
mean of 100 means we're going 7.5 points
on up. So she's
going to be looking at trying to detect
an increase of 7.5
points in the mean IQ. In other words, [an]
[IQ] the
mean IQ being boosted from 100 up to
107.5. It's a fairly large boost.
What if she wanted to detect a Cohen's
d of 0.2 instead?
Okay well it's a small effect, 0.2 times
fifteen
is three, and so she would be
trying to detect a shift in IQ
from an average of 100 up to an average
103.
Instead of 42 children, it turns out she
would need
263 children
in order to have power 0.9 to detect a
Cohen's d of 0.2.
So you can see that that shifting
Cohen's d around has a pretty dramatic
effect
[on the number of] on the sample size
that's required
to detect it when power is around 0.9.
What about the situation as sometimes
arises where the psychologist doesn't
have control over the sample size
let's suppose she's gone out to the
schools,
with which she has established a
research relationship,
and the school headmasters all tell her:
"Oh, look. You know, here are the kids we can
spare
for your study, we can't give you any
more than these."
And the total sample size she stuck with
is 50 children.
Okay so what you can ask then is: okay,
I'm stuck with a sample of 50 children,
what is the smallest effect size
I can detect given a sample of 50,
with power 0.9? Okay, well in our
spreadsheet here
we can see that if n equals 50 then
when power is 0.9 and our confidence
level is still
95%, Cohen's d
turns out to be 0.46.
Okay that corresponds to an
IQ boost
from a mean of 100 to a mean of 106.88--
pretty close to 107. So you can see that
with a sample size of 50, she's able
to determine
what the smallest effect size is that
she could detect
with if she wants to insist on power
being equal to 0.9.
Okay, so let's have a look at how to
actually calculate all this
how to calculate power and how to
determine sample size.
Now as I've indicated in an earlier
lecture, this
is exactly the kind of thing that
many researchers in areas like [education]
educational psychology, clinical
psychology, health psychology,
a lot of practical areas would be
required to do
by funding agencies before they would be
granted any research funds
to carry out the research. The reason for
this, as I've indicated before,
is that those funding agencies
are oftentimes [they oftentimes] look into
the ethics
of conducting research, and they
realize that oftentimes
tying up people's time requiring them to
make effort in some cases even
having them engage in in tasks that
are
difficult uh cost them effort and time,
but also may even be counter therapeutic
clinical studies, for instance.
Those those are costs to the
participants
and so the funders want researchers
to justify why they would want to
subject, for example, 263 schoolchildren
to a
six-month diet, probably against their
will.
Right? Who wants to be eating fish all
the time?
Okay so this is where, as I've indicated
before,
ethics meets stats power calculations,
are the things that enable a researcher
to be able to
to justify the sample sizes that they
need
in order to detect effect sizes that
they believe they're going to get
or they're anticipating they might get.
So let's start with this last situation.
Namely [where the] let's suppose the real
mean turns out to be 106.88, or the
psychologist has a good reason for
believing it's going to
be somewhere around there, and the null
hypothesis of course is that the mean
hasn't changed,
so it's still a hundred. Now
instead of using the t-distribution and
the t-test,
what I'm going to do instead, is use the
normal approximation for large samples;
because this will simplify things.
So given that she's insisted on a
confidence level of 95 percent
that means for significance testing
alpha
is going to have to be equal to 0.05
alpha remembers a
probability of making a type 1 error. So
instead of using
t alpha on 2, we're going to use
z alpha on 2. The z score from table 8.2 in
the back of the textbook
or any other normal distribution table,
whose area beyond z
is alpha divided by 2 or 0.025.
And this turns out to be 1.96
so z-alpha-on-2 is 1.96. 1.96
is a figure for z alpha-on-2, you're going
to see a lot.
Because most of the psychological
literature
has alpha levels of 0.05, and so
where people are using the normal
approximation, they'll always come up
with this same
z value here. Okay set alpha on 2 equals
1.96, what does that mean?
It means 1.96 standard errors
above the null hypothesis mean of 100.
Now our standard deviation is 15,
our sample size in the example here
is 50,
so our standard deviation, standard error
rather of the mean is going to be 15
divided by the square root of 50, and
we're going to multiply that by
z alpha on 2 which is 1.96. We do all
that,
we end up with a
an increase of 4.2, so we'll reject
the null hypothesis if we get a sample
mean of 104.2
or higher. And that's the value that was
displayed for rejection
in the table of the previous slide, so
when you're reviewing this lecture if
you go back to that
previous slide, you'll see where the
rejection mean
is, and you'll see 104.2 there.
So power then is the area beyond 104.2
under a normal curve whose mean is
106.88,
and whose standard deviation is the
standard error of the mean
15 divided by the square root of 50. To
find out what that is,
we have to convert 104.2
into the equivalent z score for this
normal curve,
so that means we're going to be taking
the difference between 104.2
and this new population mean of 106.88
and then dividing that by the standard
error.
And if we do that we get negative 1.283.
Okay, so 1.283 standard errors
below the mean of 106.88.
In table 8.2, there is a z
entry of 1.28 which is very close to
1.283,
and the area up to z there is 0.8997
which is very close to 0.9.
Okay so now we're all set to calculate
power and determine the effect size.
The formula that we're going to be using
here is
z alpha on two plus z beta and that
equals our Cohen's d times the square
root of n.
So if we know what d is in advance, we
know what
the effect size is that we want [to want]
to be able to detect,
and we've decided what power is and [what
our]
what our confidence level is, so we know
what z alpha on 2 and z beta are.
Then we can determine what n is, because
that's the only unknown
left in that equation. Conversely,
if we're stuck with a sample size, like we're
being stuck with a sample size of 50.
Then d is the unknown quantity that we
want to get, so the next two equations
below,
that first one, tell us how to get d
if that's what we're trying to estimate
or our sample size n
if that's what we're trying to estimate
instead.
So the psychologist knows in advance
what these two z-scores are:
z alpha abd z beta, then she can
divide their sum
by the square root of n to get d,
and likewise if she wants to know what n
is,
has to be rather in order to have power
0.9 to detect
d of 0.2, or something like that, she can
rearrange that formula
to get n by taking the sum of the z
scores
dividing by d and [multiply and and] then
squaring it.
So let's try all of this using
Table A.2.
Now from Table 8.2, previously we
we got z being equal to 1.28.
But let's be conservative and use z
equals 1.29,
because that's going to give us a
slightly smaller
[error] type 1 a type 2 error rate.
Instead of 0.1, beta is going to be
0.0985.
The area under the normal curve beyond
minus z beta is 1 minus beta and
in this example that's 1 minus 0.0985
which is 0.9015. So [our power] this is our
power and it's just a little bit higher
than point nine.
If we're using our confidence level of
ninety-five percent,
so that alpha is 0.05, then z alpha on
2 is 1.96, and if we plug all of that--
1.96 for z alpha on 2; 1.29 for z beta,
and 0.2 for for d, into our formula
then we take the ratios, square that,
we get 264.1 for our n.
That's pretty close to the n equals 263,
that we got by using a spreadsheet.
Okay, so using power calculations,
we can do three things with this: we can
first of all design our own study,
so we can figure out what our sample
size needs to be; or we can figure out
what effect size we can detect with a
given level of power,
if we're stuck with a particular sample
size; secondly
we can use this to find out how much
power a published study
actually had to detect the effects of
various size.
So let's think about that for a couple
of [couple of] moments here.
Suppose you're reading a study and the
study reports that they failed to find
a significant effect that they thought
they were going to find.
Or let's say that they found the effect
[but um]
[but the] but they're not terribly sure
about how reliable the finding is, or
maybe somebody has raised
a question about the power of the study.
There are two ways we could
go about figuring out how much power a
study had
after the fact, in other words [after]
after the data are in.
One possibility is simply to nominate
your own preferred effect size,
and compute the power of detecting that.
So you can say, well, you know well what
power did this study have to detect
cohen's d of 0.2?
With given its sample size and so
I'll go ahead and use my formula to
calculate how much power it had.
A second possibility, and one that i
haven't actually seen
published used in a publication, but to
me it's just sitting there waiting to be
used,
is: if you calculate a confidence
interval
around the study's sample estimate, say
around that study's mean for instance,
then the confidence interval goes
contains
a range of population effect sizes;
these are the plausible effect
sizes in the population based on the
sample evidence that we have in front of
us.
So if we want to ask how much power the
study
might have had, then we can calculate the
power that it had to detect the
smallest effect down at the bottom of
the confidence interval let's say.
And we can then calculate the amount of
power it had to detect the
largest effect one up the one that the
upper limit of the confidence interval.
Those two power estimates will then give
us a 95,
for example, a 95 percent confidence
interval
for the power of the study.
And finally, of course, [the]
being able to to calculate power
and especially to evaluate how much
power a study had
is something that is rather impressive
knowledge to have even these days.
And so you can use it to impress your
lecturers, supervisors,
and of course your friends. Okay we'll
leave things there for now.