Okay, so we have this idea of sort of
measuring uncertainty of things.
And we're going to try and figure out
like what does this mean in terms of language models.
So essentially what this means is
we're going to try and think about,
given the start of a decks,
can we actually guess the next word, right?
So that sort of tells us what's the entropy of English?
How many, how many questions would I
need to ask in order to figure
out what the next word of English is.
And according to one particular study that
looked at one particular text in English,
they were saying that the entropy is about 1.3.
Now this is actually going to depend a lot on
what particular text you give someone to guess about.
But it's in general,
humans are pretty good at this.
And the question that we want to ask is how good
our engram language models, right?
So ideally, if I had a really great model,
then it would have as little uncertainty
about the next character or
the next word as a human would.
We're not going to get there.
But the closer we are able to get to that,
the better our model is, right?
So a good model should assign
high probability to real sentences
and low probability to sentences that are
not real sentences or not good sentences.
And so the, the kind of way that we actually
do that in practice is we use
this thing called cross entropy.
Ok? So what cross entropy
is doing is it's basically measuring
how well the model that we
have predicts the data that we're actually seeing.
And assuming that we have
a fairly long sequence of words that we're looking at.
We can approximate this cross entropy
by basically saying, okay,
let me figure out
the probability of the sequence of words under my model.
And then I'm going to take the negative log of that.
So that's this entropy idea.
And then I'm dividing by n here because what I
actually want to do is not
compute the probability of this whole sequence.
Because in general, the longer the sequence is,
the lower its probability will be.
I actually want to figure out,
on average for each word, that is,
where weather and words, on average,
how well am I doing at predicting that word?
Ok, so this, you can sort of whoops.
You can sort of think of this as a,
as an average prediction probability for the next word.
How good are we at, at doing that?
And sometimes we also use
this other term called perplexity.
Where perplexity is
just two to the power of this cross entropy term.
And I don't know why people
sometimes use one and sometimes use the other.
They're basically equivalent except that
obviously the perplexity gives you a bigger number.
I'm going to skip over
that one and try
and give you this example
of kind of how these things work out in practice.
So let's take a look at an example of a trigram model.
And again, this is using
that Euro pearl dataset that I talked about before.
So the first thing that we can do is we can
write down what's the probability,
maximum likelihood estimate of
the probability of each of these trigrams.
And the first thing you should do is just kind of scan
through these and make sure that
they kinda look sensible to you.
This is the first thing you should always do when you
get a bunch of numbers in front of you is sort of say,
OK, do these numbers seem to make some sense?
And hopefully we can convince ourselves that
they don't look completely crazy
because we can do things like, Okay,
what's the probability that
we see the end of sentence symbol given
that the previous symbol was a full stop.
And that has a very high probability.
So that seems sensible.
We can look at other things like what's
the probability of seeing to after would like,
and indeed would like to is a very common phrase.
So that has a high probability, seems reasonable.
Whereas if we get like to,
there's just about any verb
that could come here and the verb
commend in general is maybe not a very common one.
So it's unsurprising that
this probability is pretty well.
So these things kinda look reasonable.
And then this next column here is
just the negative log of these probabilities.
And on average we can say, okay,
well if we average the negative log probabilities,
then we get 2.6 X3 four, right?
So I mean, that's a kind of
very rough approximation of the cross entropy here.
As I said, normally you need to
have a much longer sequence to
get a good approximation, but roughly.
And then once we do that, we can also sort of look
at comparing between unigrams,
bigrams, trigrams, and four grams.
Okay, so what, what I'm showing in this table is the same
the same average negative log probability for the,
let's say for the unigram model,
this basically just means what's
the negative log of this word?
And then what's this word and then what's this word?
If we're looking at like the bigram model,
what I've got in this column is implicitly
conditioned on the previous word in the sentence, Right?
This is a full sentence here.
Alright? And what you'll notice here is that
the value that we get here that sort
of indicates how uncertain we are
or how good our model is relative
to the sort of true sentence is,
is much worse for
the unigram model than it is for the bigram.
And then it's worse for the bigram
than it is for the trigram
and worse for the trigram than it is for the four Graeme.
Okay, so what that's telling us
is it's following our intuition that
unigram models are not as good as
at being able to predict the next word.
You'll also notice that there's a much
bigger drop off here between unigram,
bigram and then it kind of levels off of it, right?
So there's a big difference between having
no context and having it even a little bit of context.
And then as we increase the amount of context,
the gains are not quite as much.
So in sort of find out
a little transition into what I'll talk about next time.
We have this idea of modelling.
And we can kind of build up our N-gram models.
But we haven't really solved the problem
entirely of trying
to compute the probability of a full sentence.
Because if we think about what happens when we
try and compute something like this, right?
So this is the trigram.
Alright, what's the probability of consuming
given shell commands?
Let's assume that we've seen the words
shall commence in the corpus,
but we've never actually seen this full trigram.
So we're still going to
end up with the same problem
that we've had the whole time,
which is that we're going to
estimate this probability is 0,
which means that we still have no way
of differentiating between a sentence
like this one that's reasonable in a sentence like
this one that's not reasonable because both
of them are going to end up with 0 probability.
Okay? And so this is again coming back to
this problem with maximum likelihood estimation.
Alright? It, by definition,
it estimates probabilities that make
the observed data as probable as possible.
And it does that by assuming that anything that
hasn't been seen before can't possibly happen.
0 probability.
Another way of saying that is that it's
basically overfitting the training data.
And so we tried to kind of get around
that by modelling sentences in smaller chunks.
But even those smaller chunks are sometimes going to have
0 probability if we're
using maximum likelihood estimation.
So what I'm going to talk about next time is how we
kind of get around that even more using
what are called smoothing methods to try to reassign
some probability mass from observed to unobserved events.
This is what I've talked about today, right?
Noisy channel framework,
language modelling and gray models and evaluating them.
I don't have any announcements for
today because I already told you about the tutorials.
However, we do have two minutes
and I would love to take questions.
Yes. But there it
be any better if we use multiple N-gram models?
Do you mean somehow
combining the information from multiple N-gram models?
Maybe. Yeah. So actually
I will talk about that in the next lecture because yes,
sometimes we do actually
combine multiple language models.
Yeah. Yeah. Is entropy related to that?
What is entropy related
to the variance of the probability distribution?
Let me think about that. I think
I can actually answer that off the top of my head.
Let me get back to you on that one. I think so.
I mean, it certainly it's related to the yeah,
I think I think that's right.
So because basically the,
the the, the larger the variance.
Well, yes and no.
So what it's related to is basically
the number of different possible outcomes that you have.
If you're talking about a continuous distribution,
then the wider the sort of number of different districts,
the number of different outcomes
obviously is infinite, but,
but that's sort of roughly related to the variance.
But not exactly because of course you could have
a distribution where it has a very high variance,
but actually almost everything is sort
of centered on one thing.
So it's kind of related,
but it's not exactly the same thing.
Yeah. Yeah. Say it again.
Right. Okay. So the question was,
these N-gram models that I've been
talking about always only
consider the kind of previous words in the sentence.
And what about trying to
condition somehow on the
following words in the sentence as well?
Yeah, so there are models that actually,
so things like skip gram models
and other kinds of models that do
actually condition on words
in both directions in the sentence.
The, this sort of very quick answer
to why we're not doing that at
the moment is that dependent?
It sort of because of the kinds of
probabilistic models that we're using mathematically.
It's actually harder to formulate a model that kind.
But it's, it's not,
it's certainly something that's possible
and has been done.
But it's not something that I'll focus on for
this part of the class. You know.
I think we're actually just about out of time.
So I will Oh, yeah.
So you guys all have tutorials?
I'm still trying to get them to remove
the tutorials for next week
because we only have tutorials every other week,
but you do have them this week.
So please go to your tutorial
this week and I'll see you guys on Friday.