So, we've spent a bunch of time learning about this MMLU benchmark and really what
I've been trying to do is demystify it for you because I want you to understand
that nothing is as complex as it seems and you are able to do anything yourself
if you just dedicate a little time to understand it.
All these benchmarks and advanced prompting techniques, all of this is
really just testing and evaluating different prompts and different models
and now it's time to take it the next level and introduce you to MMLU Pro.
This is the next level of the MMLU benchmark.
Why do we need a pro version of this?
Isn't the MMLU benchmark good enough?
That's a good question and in a lot of
cases, the answer is yes, MMLU is good enough.
But we're not just here to do good
enough, right?
We want to push the boundaries of what
these models and what prompting can do.
So I'm going to teach you about this MMLU
Pro benchmark because I'm confident that once you
understand the improvements this benchmark makes over the original MMLU,
you'll be able to critique other benchmarks and also think of your own
ways to improve other benchmarks .
Essentially, this is going to future -proof your knowledge and skills when it
comes to using benchmarks for testing and evaluations.
Again, as we just learned, benchmarks are
generally used as an evaluation method for models, not for prompting.
But they obviously involve prompts and so they're a great data set, a great source
of information and data that we can use for our own prompting tests.
And also sometimes when you're prompt engineering, you're going to want to
compare different models, you need to understand which model is best for your
specific use case.
So let's dive into MMLU Pro.
Here's the paper that introduced this new benchmark.
It's from researchers at the University
of Waterloo, University of Toronto and Carnegie Mellon University.
But again, why did they introduce it?
Well, they talk about that in the paper.
Essentially, they found that there are three problems with the MMLU benchmark.
And before I get into these, this isn't something unique necessarily that these
researchers found.
There are always criticisms of benchmarks
and that's one of the reasons that you're learning about them, so that you can
critique them and analyze them yourself.
And these criticisms were actually quite
well known by the community.
It's just that these researchers actually
decided to solve those issues.
Okay, what are those issues now?
First, the questions in the MMLU benchmark only have three distractor options.
So they're normal multiple choice questions, right?
They had four options to choose from, one of which was correct, it was the ground
truth, it was the golden answer.
And then they had three distractor
options, three options that were incorrect.
Now, that means that a model basically
can score 25 % simply by guessing.
I don't know about you, but this is why I
always loved multiple choice questions.
The answer is given to you and you just
have to find the one that seems the most right, even if you don't know that it's right.
So how did they solve for that issue?
Well, instead of just four options, they
created 10 options for each question.
So it's still multiple choice questions
in MMLU Pro, but instead of just A, B, C, D, they have 10 different options.
If I was taking a multiple choice test with 10 different options, that would
definitely prove a lot harder.
It's going to make it much more difficult
for me to just guess and find the right answer.
Okay, that seems like a good solution, right?
And we're going to talk about that one a bit more in a second.
But let's go back to the issues here.
The second issue that the researchers
identified was that the questions in MMLU didn't really require much thinking.
It didn't really require chain of thought or reasoning.
Now, that's fine, it just means that the questions aren't that hard, especially
for something like a large language model that is really good at picking pieces of
information out, right?
It can pull out facts and information
much better than a human, but it's still not great at really thinking and
reasoning through things.
That's what a lot of these prompting
techniques we've learned about help it do, to think and reason better.
So as a result, the MMLU questions, since they are mostly knowledge -driven rather
than requiring reasoning, are relatively easy for these models, especially the
leading ones, the frontier ones.
So how did they solve that?
With MMLU Pro, they upped the ante.
They made these questions require
deliberate reasoning in order to answer.
So the model has to really think through
them, and we'll talk about that one a bit more in a second too.
But now let's turn to the third issue with the MMLU benchmark, and that is,
there are mistakes.
This one is actually my favorite because,
again, it just sort of demystifies all this stuff.
Everyone thinks, oh, there's this MMLU benchmark, and it's cited in the OpenAI
and Anthropic papers and Google papers when they release new models.
It must be so perfect.
Well, no.
When people actually started looking closely at the MMLU benchmark, they found
that some of the questions didn't have a correct answer to them.
The four options given didn't include a correct one, or there were mistakes in
the actual questions themselves.
So there you go.
Everybody makes mistakes.
Next time you make a mistake and think,
oh, gosh, I'm just such an idiot, well, hey, AI researchers do it all the time
too, so don't be down on yourself.
So how did they solve this in MMLU Pro?
Well, they had two rounds of expert reviews to reduce the noise, that is, the
incorrect answers in the dataset.
So that's pretty straightforward.
And so there you go.
There are the three main issues that the
researchers identified with MMLU, and there are the three main solutions that
they solved with MMLU Pro.
Now let's dive into the details a bit
here, because that's where all the fun stuff is, right?
That's where we get into the nitty -gritty and the real application of
prompt engineering to the real world.
So the first thing to understand is that
MMLU Pro actually reuses a lot of the questions in MMLU.
So you can see here, this is the discipline, so math, physics, chemistry, etc.
And the number of questions that MMLU Pro has, so MMLU Pro has 1 ,351 math
questions in it, and 1 ,299 physics questions, and so on and so forth.
And in total, it had 12 ,032 questions.
That's actually less than MMLU, which had
about 15 ,000 questions.
But the key point to understand here is
that of those 12 ,000 questions in MMLU Pro, 6 ,810 of them came from the
original MMLU benchmark.
You can see that in this column here.
And then they added 5 ,222 questions that are brand new to the MMLU Pro benchmark.
And now you might be asking, hold on a second, going from 4 options per multiple
choice question to 10 is actually a pretty big task.
Can you imagine taking thousands of questions and adding 6 options that
should sound right, like they can't be completely unrelated to the question, but
they also can't be correct.
They have to have some sort of flaw in them.
So how would you go about solving that?
Well, if your answer is use an LLM, then
that's exactly what the researchers did too.
They took all the multiple choice
questions and then used GPT -4 Turbo to add 6 additional options.
So there you go, again, demystifying this.
There's no magic to it.
You could do this yourself.
You could take this MMLU Pro benchmark
and make it 20 options for every question, and therefore make it even
harder for the models.
And you know what?
If you wanted to do that, here's the prompt that they used to create those
additional 6 options.
So you can see it starts up here.
I have a multiple choice question with 4 options, 1 of which is correct.
I need to expand it to 10 options.
Please generate 6 additional plausible
but incorrect options.
And then you can see here it's labeled 1 shot.
So this is the shot.
It has the input.
This is the question and the 4 options.
And it says what the answer is.
And then it's giving an exemplar, a shot, the output.
This is what it wants.
It wants generated 6 additional options, EFGHIJ.
There you go.
Easy peasy, right?
Okay, let's keep going through this because I really love breaking this stuff
down for you and so that we can really sort of understand it at a much deeper
level than, of course, the average person who uses ChatGPT or some other model.
But frankly, you're starting to understand these benchmarks more than a
lot of people working in the AI space.
So nice work and let's keep going.
So if you recall about MMLU, it was 5 shot.
Remember, it included 5 shots and then
the question.
Well, since we want to compare apples to
apples in a lot of cases, right, MMLU Pro also uses 5 shot.
But interestingly, it also uses chain of thought.
And that makes sense because remember,
one of the main things about MMLU Pro was that the questions were more reasoning -based.
They require the model to think through the question and the answer more so than
was the case in MMLU.
Let's see what that actually looks like here.
So this is an example of one question in the MMLU Pro benchmark.
In fact, the question isn't actually even here.
There's just a variable saving its spot
for the question.
Let me show you what I mean here.
So we've got the instruction here at the top in yellow.
The following are multiple choice questions with answers about physics.
Think step by step and then finish your answer with the answer is X where X is
the correct answer.
Simple enough, right?
And then we've got 5 shots.
Let's count them.
1, 2, 3, 4, 5 .
And let's look at this first one here a little bit closer.
You can see, okay, it's a question about refracting telescope and then it has the
options here.
And you can see there's 10 different
options, 10 different answers you can choose from.
A, B, C, D, E, F, G, H, I, J.
And then it has an answer that says let's
think step by step and then goes through the thinking process.
In a refracting telescope, if both lenses are converging, blah, blah, blah.
And then it ends with the answer is 8.
So again, this is a shot.
So it's showing how to think through the question.
All right, so we have those 5 shots and
then we have the actual question.
Like I said, this is just a variable
waiting for the question and the options to be inserted.
But that is where the actual MMLU Pro question would go.
And now I have a quick little exercise here I want you to do.
I want you to pause the video in a moment and just think about what type of chain
of thought is being used here.
Pause the video, look at it, think it
through, and then come back.
All right, welcome back.
Did you think step by step?
Okay, that's a pretty good joke.
We were talking about chain of thought, think step by step, you get it?
Okay, perfect.
Well, in fact, I wanted you to do that
because the researchers here have interestingly basically thrown all the
chain of thought that they possibly can into this prompt.
You can see in the actual instruction, they say, think step by step.
That's zero -shot chain of thought, right?
But then in each of the shots, they also
give what I'll call normal chain of thought, where they actually walk through
the thinking process that they want the model to follow, right?
That's the answer field in each of these shots.
And even more so, in the actual question
that they want the model to answer, they've pre -populated the answer with,
again, zero -shot chain of thought.
So, that's kind of interesting.
They've really thrown everything at this to get the model to use chain of thought thinking.
Now, maybe this is cheating a bit, right?
Maybe it's cheating to use chain of
thought inside the prompt.
And if you're trying to make this
benchmark harder than MMLU by introducing more reasoning tasks, well, maybe you
shouldn't have actually introduced chain of thought prompting into the benchmark.
That's just something for you to think about.
But here is the key aspect of MMLU Pro,
and that is the results.
The key is that it's harder for these
models to answer these questions than MMLU.
So, you can see here along the X -axis,
we've got three different models, and on the Y -axis, we have the accuracy score.
All these orange bars are what these models scored on the MMLU benchmark, and
these blue bars are what this same model scored on the MMLU Pro benchmark.
So, you can see, GPT -4 -0, it's harder for that model to deal with the MMLU Pro questions.
Same with LLAMA -3, 70 billion, and GEMMA, 7 billion.
So, in that sense, the benchmark is successful because researchers were just
finding these models were getting too good at it.
You can see, GPT -4 -0 was almost at 90 % accuracy at MMLU.
The models are getting a little too smart at these sort of multiple choice question benchmarks.
So, we're going to have to continually make them harder.
Right now, that's MMLU Pro, but no doubt in the future, we're going to add another
benchmark that maybe will make it even harder.
Like I said, it could possibly be you
creating that next benchmark by going from 10 options for every question to 20
options for every question, or maybe there's other ways you can make these
questions harder, like removing the zero -shot chain of thought from the prompt.
Again, this knowledge is meant to future -proof you so that you understand both
how these benchmarks work and where they're going in the future.
Alright, and with that, you are now a benchmark pro.
Get it?
MMLU Pro.
Benchmark Pro.
Okay, that was another really, really
good joke.
Thank you.
You're welcome very much.
Thank you, Scott.