So what is a usability test.
These are the components of what makes a usability test different from some of the other consumer research
methods you may have heard about.
First of all it involves real users.
It doesn't just involve someone at the next desk or a warm body.
Secondly participants perform real tasks.
So they're going to be carrying out things that are meaningful to them not mindless step by step instructions.
Thirdly we're going to observe behaviour we're going to focus on what users do rather than what they
say they do.
And we're also going to observe people.
We're going to record data and then we're going to analyze that data afterwards.
Now you'll notice that when I explain what makes a usability test I didn't include things like the one
way mirror.
I didn't include things like the picture and picture of video recording and so on.
I'm not saying that those things aren't useful because in fact they are and I often use them in my tests
but it's not them that make a usability test a usability test.
It's these features instead.
In fact usability tests don't recall a usability lab or observers.
And some of them don't even require a moderator.
They certainly don't all require a screen recording or video recording all of those.
A nice to have features.
And if you focus on those items on this slide rather than the list of items on the previous slide your
risk of running a cargo cult usability test.
You may not be familiar with cargo cult So let me tell you a little bit about them.
They developed after the Second World War in remote islands in the Pacific.
Japanese and U.S. forces arrived with a great deal of supplies and both sides here dropped clothing
medicine canned food tents and weapons to their troops and sometimes those troops sheared the goodies
with the natives who were there when the war ended.
Obviously the military left and they stopped dropping cargo and in response some charismatic individuals
develop cult among remote Pacific populations and these coult promised to make sure that the food would
be delivered again and the arms would come back and the jeeps and so on.
And in attempts to get the cargo to full by parachute or landing planes of land is imitated the same
practices they had seen the soldiers use.
So behaviours would include things like mimicking the day to day activities and dress style of U.S.
soldiers so they would perform parade ground rules like you can see on the top picture here with wooden
or salvaged rifles and the olindo.
Ireland has also created even mock aeroplanes and they carved headphones from wood and wore them while
sitting in fabricated control towers.
They even waved the landing signals while standing on the runways.
They lit signal fires and torches the line to light up the runways in the hope that planes would land.
Obviously the planes didn't land.
And similarly you find that some people have all of the paraphernalia of usability tests like the things
on this slide but they miss the point of what usability testing is about.
They are another thing you may have heard about usability testing is that you can do usability tests
with a small sample size sample size of just 5 people.
It's quite typical for a usability test and you think well how is that.
If we were running a poll to find out who people would vote for of the next election five people would
be way too small.
How comes we can get away with that in a usability test.
Well let me show you that with an example.
Market research is opinion driven.
Peoples opinions vary considerably.
So you need large samples to generalize but usability research is behaviour driven.
People's behaviour doesn't vary that much so you can generalize from small samples.
Here's a good example of that.
If this was a kitchen hob this is a kitchen OP if it was market research and I wanted your opinions
on the job.
I would say things like does he have enough burners for a family meal or do you think the design would
fit in your kitchen.
How much would you expect to pay for it.
Now I need to ask a lot of people those questions in order to get reliable answers but with usability.
I simply say can you use it.
I voted to give you a task.
So let me do that now.
Which of those dials or knobs on the right hand side would you use to light the gas under the back right.
Hobb.
Would you use number one number two number three or number four.
Now when I ask people that question they typically say it would be norb dial number 1 or number two.
And that's because they expect the mapping to start at the back left or the back right at the back left
would be one or the back right would be one and then it would go clockwise around the hob.
In fact the correct answer is for I know that because I owned this Hobb.
If I'd known I was going to show it to people on a video training course maybe I would have given it
better clean than I did but nevertheless I owned this and it drove me bonkers.
In fact I had to move house because I just couldn't get the mag.
Now there is a logic to it.
The the reason it works that way is because the the mapping actually starts from the front right so
the front right burner is number one the front left is no to the back left is number 3 and the back
right is number four so it's clockwise again.
But it starts from the front right.
It's just counterintuitive isn't it because we kind of read top left down woods so it doesn't make sense
for it to be that way.
You don't need to ask many people.
And find that they experience a problem to realize you need to fix this.
That's why you can get away with small samples.
And then as part of an iterative design process you can come up with an alternative design something
like this.
I'm not saying this is a beautiful design or one I would particularly one in my kitchen but you can't
argue with that mapping of the Hobbs to the bird as if I now said to you which of those would light
the gas under the back right Hobb.
It's a stupid question because it's so obvious because the pattern of the dials matches the pattern
the burners themselves.
And usability testing gives you these kinds of insights to improve designs.
In fact it's led Steve Krug who's written a great book on usability to say testing one user early in
the project is better than testing 50 near the end.
And this number 5 that I've mentioned you'll hear that number widely quoted by people and saying you
can get away with just five users in the usability test.
And indeed most of the time you can.
But I should say first of all the assumptions that that's based on first of all it's based on an assertion
that the design process is iterative.
So if your design process is iterative and you run say five or six usability tests with five people
each.
By the time you release the product you will have run 25 to 30 participants through.
It's also based on an assumption that there's just one segment of users for your product.
So imagine you've got a job recruitment site where you have two user segments people who post jobs and
people to apply for jobs or let's say it's eBay or you've got people who buy products and people who
sell products.
Now those users have got such different goals but you need to test five users within each segment not
just five overall.
And also there's a difference between testing an awful interface with many many problems and finding
subtle flaws in a simple interface which works correctly for the vast majority of users.
So that five user rule is simply not good enough for some applications like medical devices or electronic
voting where most people won't struggle with the interface.
But a small number will and there could be awful consequences if that was the case.
But generally we can get away with five.
And also I should say that most user interfaces I come across are truly awful.
And with five users you're going to find a large number of problems as we all see.
So usability testing itself.
There are two main methods of usability testing.
There are different techniques within these which will cover two main methods.
You'll hear about one is called formative testing.
Sometimes you'll hear record diagnostic or thinking a lay out testing and the advantages of this particular
method.
It is great for finding real problems.
Will users have the fact that people are thinking aloud.
That itself is a valuable source of information.
The Word to the language people use.
You can also use it very early in design.
It doesn't need a sophisticated prototype.
In fact you could do it with prototypes.
It literally made out of paper.
It's also fairly easy to prioritize problems because clearly the problem the five users experience in
your test is going to be more important than one that just one user experience.
So you can prioritize those problems.
It's also got some disadvantages as well though.
One is that the very act of thinking aloud may change the way people behave with the interface.
And also if you do want to analyze the the language of the verbal protocols that people are using that
can take a lot of time it often takes twice as long to analyze a video from a usability test as it does
to actually watch the test itself.
The other type of usability test method is called summative or measurement sometimes called benchmark
testing.
The great thing about this technique is it answers the key question how are we usable.
Is this product.
And that's because it provides real performance data.
It enables you to put a number on the usability of the system that you're testing.
That means you can compare different groups of users different systems or you could compare a system
over time to see if it's improving.
And because you tend to use larger samples for summative tests often recommend about 20 for a summative
test because that way you can have statistics which you can have more confidence in.
It's got high reliability and validity.
However it's got disadvantages.
One is it requires a proper usability lab set up or you could also use a remote benchmarked all.
And we'll discuss those as possible techniques as well.
Sometimes the task you give people can be a little artificial in restricted and that's because you need
to know exactly where people have started and finish the task.
So for example you can start timing it and it may be a while for you to do the statistical analysis
as more that may take some time although you can automate a lot of that but if you're running the same
kind of test over time to benchmark a product.
So two main techniques they both have their advantages and disadvantages.
You'll have to decide which one is best for you.
Let's do that by reviewing the types of test available to you.