everyone can hear me okay great alright we'll talk about Gaussian processes for
the next couple hours so I wanted to start by saying that Gaussian processes
are a mathematical object with with a great history in theory and also has are
something that's been used to good effect in quite a number of applications
we're not going to focus on either of those two pieces specifically but rather
talk about the piece in the middle which is which is from from a usability
context how as machine learners can we use Gaussian processes so here's how
we're going to go about introducing Gaussian processes I want to start we're talking about gaussians in general both
in words and in pictures this is just going to be sort of an easy introduction
into this so we can think about we can think about what a Gaussian process
actually is from an intuitive perspective then we'll go in and build
out some of the equations we'll talk about using Gaussian processes in a
basic regression stud setting that'll be that'll get us about through the first
hour and then we'll take we'll take a quick break and then we'll come back and
we'll think about moving beyond the basics of Gaussian processes so what
what kind of things can we change we'll connect that to some of the different
technologies and machine learning that we've seen and that'll just about do it
all right so what is a Gaussian as far as machine learning is concerned this
should tie into some of the notions of Bayesian inference that you've seen in
the last couple days so a Gaussian a Gaussian distribution is essentially a
handy tool for Bayesian inference on real valued variables so here's a here's
here's a specific example that we're going to talk about throughout the
course this I'm interested in measuring my heart rate so how might I do this
from a modeling modeling perspective and I'm here I'm going to I'm going to measure my heart rate at 7 a.m. and the
fact that I do that index by time is going to be important we're going to see
that in a moment okay so I'm a reasonably healthy guy so you might
think okay aa priori I have some belief about what my heart rate is going to be when I measured it seven eight seven
a.m. maybe it's somewhere between fifty and sixty beats per minute
so I put some gaussian some Gaussian prior on that and there density now I can go in on a particular
morning and I can measure my heart rate I measure it at 61 I can go in on a
couple other days and I can measure that three three more times so now I've got these four observations these four noisy
observations that I measured on four different days is my heart rate and what
the Gaussian allows me to do and what the notion of Bayesian inference allows me to do is that I can then take my
prior this grey distribution and those four draws from that Gaussian and I can
do posterior inference so I can come up with a posterior P of this of my
underlying heart rate given the noisy observation they've seen you see that
that's again a Gaussian I now have more confidence about where that is and I see that in fact it's it's it's centered
around say 60 62 okay so let's take that univariate Gaussian that up to
multivariate gaussians so we talked about measuring my heart rate at 7:00 a.m. I could also want to measure my
heart rate at 8:00 a.m. and let's think about how how those observations have not just a single real valued variable
but rather a pair of numbers at 7:00 and 8:00 a.m. how that would change so to do
that we can't use the same univariate gaussians very gaussian this should be
an object that we're all familiar with now the multivariate Gaussian we're
going to conventionally look at as these on a flat surfaces of lip sides of I so
probability and so what is this distribution telling us this
distribution is telling us now I've got some prior belief not on on a single
heart rate measurement but on a pair of heart rate measurements it shows that
there's some positive correlation which is to say if I measure if I've got a
higher heart rate at 7:00 a.m. I imagine it'll be higher at 8:00 a.m. and so then
we can do the same thing so here's our prior I can go in and take four
measurements on four different days now remember these measurements now are a
pair of numbers 7 a.m. and 8:00 a.m. so I can get that data
I can use Bayes rule in the same way and I can come up with some posterior inference and now I have I have a
refined belief of what of what my heart rate is at 7:00 and 8:00 a.m. so now I'm
going to take those two measurements the measurement that happened at 7:00 and
the measurement happened at 8:00 and I'm going to represent that in a slightly
different way so this is the same data I've got for data for four pairs of
numbers it's the same data I just indexed it by time now so you see
that the one met the one red point we were looking at before which is the pair
of numbers at seven a and eight am is now is now put there on that axis at 7
a.m. four data points still so then the natural thing you would want to do next
is you want to say okay what if I measure it at 9:00 a.m. what if I
measured what if I measure my heart rate at 10:00 a.m. and so on so what this is getting at is if we wanted to measure at
7:00 and 8:00 a.m. we used a bivariate Gaussian I wanna do that again at 9:00
a.m. maybe we would use a three dimensional Gaussian a four dimensional Gaussian a five dimensional Gaussian
when really what we're getting at what we care about or what we might be interested in is inferring that entire
function over time and that that's how we get to a Gaussian process intuitively so rather than having some finite set of
gaussians that we have to measure strictly from we have if you will an
infinite set of gaussians and that's that's what a function is so here's how
we're going to represent that throughout throughout the course of this so each of
these each of these function curves in color is going to be a single draw from
a Gaussian process that's a function the way we're going to represent the way
we're going to represent the prior distribution is with this this mean
which is this gray line here and this envelope that sits around it which is
two standard deviations around it so what this is saying is that we imagine that
our average draw is going to be something like that and we have some
some distribution that Wiggles around that inside that envelope well this
allows us to do and this is this is this is really one of the key features of a
Gaussian process is that remember before we were talking about measuring rigidly
at those hourly times but now if we've got this nice infinite dimensional
object we can measure really at any time we want and we'll get into all the
mathematical reasons for why we're able to do this so I can measure at this particular time and you can see what's
happened is that I've measured a data point at about 10:30 and what that's
done is that's taken my prior in the same way that we did in in that fixed
dimensional case that's taken my prior and it said actually it's refined it
into a posterior and it says actually I believe that my function is not quite
flat anymore but rather is closer to that data point and you can see that
around the data point that I've measured I now have I now have increased
confidence we can then take another measurement more measurement still and
you see what's happening as we go through this process as we get more and
more data we're scribing out this regression function this nice smooth underlying
function and we're getting more and more confident about the envelope around it
ok so that that intuitive summary is this is as follows when we were taking when we were taking measurements single
measurements at 7 a.m. we were getting real value variables so is the
univariate gaussians was a nice distribution over real valued variables
when we move that to pairs or triplets or what have you that the multivariate
Gaussian allowed us to do that and now when we want infinite numbers of real
valued variables in other words functions of real value variables that's
what a Gaussian process allows us to do so so if you take nothing else away from
this lecture take take that away that this is something that allows us that
the Gaussian process is it effectively something that allows us to have a
Gaussian distribution over infinite numbers of variables and what that
drives us to is this notion of regression
so let's look at regression real quick and talk about a few reminders of what
regression does for us so I'll put this up again so here we've got all these
blue data points that we've that we've observed and we believe that there's
some smooth underlying function that is really what the description of what the
data is doing so one of the one of the things that regression is quite good for
is just that which is denoising and smoothing so we want we don't want to
follow every little wiggle of these data points but rather but rather come up
with some good description of what is noise and what is true signal we also
want to do prediction and forecasting so I've collected all this data that's
great but now I want to know what my heart rate might be a couple minutes
after 9:00 in the morning well to do that you can you can then have some some query at that time and you can say okay
I believe that my heart rate should be centered centered at this point with
some with some variance envelope furthermore one of the one of the things
about regression and and and this you've heard you've heard a bit from from from
Peter batten will hear whether Moore's is the dangers of parametric models so
what I've done here is I've taken this data and I fit a quadratic to it so you
can see okay this quadratic has a reasonably good reasonably good fit to
the data but it seems to miss some of the features and that of course is
because it's a fixed parametric model and it can't it can't respond to a lot
of these features furthermore the parametric models get us into some dangerous places like because of this
because of the way this data was fit there seems to be some magical point
around 11 a.m. where my heart rate Peaks for the day and then it sort of falls
off after that and furthermore if you if you really take this too seriously and
you and you extrapolate from this this has my heart stopping around around
dinner time so there's there's there's some dangerous the parametric models furthermore overfitting under fitting is
always going to be a concern with regression and we'll talk about how
Gaussian process deal night nicely with that so heat you can see here we've got
this model that is that is overfitting there was just chasing all the little
Wiggles all the little noise in the data conversely you can have a data that's
you can have a model that's under fit and so you see here we're still we're
still fitting the data so to speak but we seem to have missed a lot of the
interesting structure alright so that is that's that's basically what I wanted to
go through in this first section of gaussians and words and pictures so now
we're going to now we're going to fill that intuition in with some equations I
should say please interrupt me throughout the course of this if you
have questions about this I suppose everyone's been doing that okay the
multivariate Gaussian this this this this should be a review I hope we say
that F which is a n vector is normally distributed if it has the following
distribution and that following distribution is parameterize by some
mean vector M which is which is an arbitrary and vector and some covariance
matrix K and the only constraint on that covariance matrix is that it's positive
semi-definite the shorthand that we'll use throughout this is we say the F is
distributed normal with mean M and covariance K so as we said before the
loose definition of what a Gaussian process is is a multivariate Gaussian of
an uncountably infinite length in other words take that multivariate Gaussian
vector and just make it longer and longer and longer and what does that get
to that gets to a function that that that's a very loose definition indeed
here's a slightly more rigorous definition we say that F is a Gaussian
process if for any subset indexof that a set of index is T if F of T which is
which is which is the function f evaluated at those index point has a
multivariate distribution according to normal M of T and K of T
all right so I'll say that I'm using tea here as real numbers for familiarity
with regression and time but the domain can be can be any dimension any
dimension X in NRD and we'll show an example of that later all right so I
kind of breezed by this this fact in this definition what are those functions
em and what are those functions kay so let's talk about that now because that's
that's an interesting part of what makes a Gaussian process so the mean function by analogy to that mean vector in the
multivariate Gaussian case or we said the mean vector can be just about anything the mean function can be can be
any function that map's index points T onto real values often in Gaussian in
the Gaussian process literature because you can mean subtract your data and because it makes notation easier we'll
just we'll just set set the mean function to zero because really in the
modeling context what what what often makes things most interesting is
modeling that kernel or covariance function so that that covariance
function is is is again a function that map's your input space onto a real value
except it takes a pair of arguments so it's any valid Mercer kernel so this
connects to all the stuff that you've talked about in kernels already and what this is is just any function any
function that has two arguments and that function has to be a positive semi
definite a positive semi definite function in other words it needs to it needs to obey Mercer's theorem now
Mercer's theorem again is a very rich mathematical theorem from functional
analysis but when it's whittled down to two what we care about in this
particular case what it says is if you give any finite there any any any subset
index of T and you evaluate that function into a matrix K T T in other
words take all your time points evaluate it build it into this n by n matrix that
that matrix K will be positive semi-definite
so to summarize that the GP is fully defined by a mean function and a kernel
function and this requirement that every finite subset of the domain has this
multivariate normal distribution this consistent multivariate normal distribution f according to M of Al u at
the mean function evaluated those points T and the kernel function evaluated
those points T so a couple notes one this is something that we can
conceptualize pretty easily and say okay I've got these two functions the mean
and the kernel function and I can evaluate that and that will give me that
will give me a mean and covariance matrix great and I can stipulate that I
want that I want those to always be this to define this Gaussian but the fact
that that should exist as a valid mathematical object is not at all
trivial and furthermore the fact that this is a full specification for it in
other words that you give me you give me one M and one K and that defines
uniquely a Gaussian process is is not at all trivial one of the things that's
also not trivial and is quite nice is that most of the interesting properties
that were used to in in dealing with Gaussian variables and we'll get into those in a moment that those are all
inherited okay so this kernel function is the only is is the only really
interesting thing that doesn't sort of slot in seamlessly into what we were
talking about with Gaussian so so let's let's unpack that a bit more so the
canonical example for a kernel function is probably the squared exponential
kernel so I know that looks like a Gaussian but ignore that for the time
being just consider that just consider that a kernel function of two arguments
so what I want to do just to just to make a very very explicit connection
between a kernel function and a covariance matrix is evaluate evaluate
this kernel function at a handful of points so to do that we're going to
choose some hyper parameters so you'll notice that I've slipped in I've slipped
in a couple new parameters here we call these hybrid parameters because they
live they live in the kernel this will use the clicker so we've got two we've
got two hyper parameters here we've got L the character its characteristic
length scale and we've got Sigma squared F which is the which is the variance or
the power of this kernel and so what we're going to do is we're just going to
we're just going to evaluate this so let's say I take three index points in T
that's 7:00 a.m. 8:00 a.m. and 10:30 these are the three measurement times
that we care about now what are we asking we're asking what is what's
what's the what's the correlation what's the covariance between between random
variables between my heart rate variable at these times so how do I go about
doing that okay I take this function here I take these pairs of points I
evaluated at all pairs and I build this into a matrix so what we can then do is we can then change these kernel hyper
parameters and see how this covariance matrix change so what I've done here
will just flash back between that we had a length scale of 100 now we're going to go to a length scale of 500 and what we
see is that the diagonal we've still got the same values but what's happening is
as we move away from the diagonal the correlation is falling off much less
quickly so what's this saying with a higher length scale value
between 7:00 and 8:00 a.m. these variables are highly highly correlated
as you get further away to 10:30 this variables not quite as correlated but
still it's still quite highly correlated on the other hand if we make this a
smaller number you see that the correlation drops off very quickly so
this is nearly scaled identity matrix in other words in other words my heart rate
at 7:00 a.m. is nearly independent for my heart rate at 8:00 a.m. we can also
change Sigma F and see how that changes things so this remember we were putting up that
that that envelope arounds around the Gaussian mean around the GP mean that
gray envelope so what we're doing here is changing that so we double that the
envelope doubles okay so that was just a tie in something something where a lot
of people when they're learning Gaussian process you get tripped up is connecting
the kernel function to two covariance matrices so so I'm going to I'm going to
repeat that a couple times but I think but I think it's valuable to make that connection
all right so an intuitive summary of GPS so far GP offer distributions over over
functions and for any finite subset vector we've got this normal
distribution and you see here as promised I've dropped I've dropped the
mean function and the covariance the covariance matrix K is calculated by
just plugging the T the index points into this kernel function so to
introduce some new notation before we were saying F is normally distributed
with with mean zero and covariance K you'll often see written F is
distributed as a GP with mean function m and covariance function K all right so I
mentioned that I mentioned that most of the important
Gaussian properties that we care about are inherited by by Gaussian processes
and so what I want to do is walk through a couple of properties of the Gaussian
that are that are going to be interesting for today's purposes that
are very useful to in the GP context so one is is addict tivity in other words
adding two gaussians together gives you a Gaussian again that'll that'll be nice
to us in forming a joint conditioning so conditioning on gaussian random
variables this is this is important for inference the ability to calculate
expectations which is going to be interesting for calculating posterior
predictive moments and finally our ability to marginalize out variables
that we don't care about so there there are many other gal there are many other
nice properties the Gaussian of course but those are the ones that we want to
that we really want to care about today so let's first talk about forming a joint jointly Gaussian distribution so
I've got some prior I've got some prior on f so f I'm going to use throughout as
our prior I've got some Gaussian prior with M F and covariance k FF I've got
some iid noise that I add to that N and then I want to let y equal F plus n so
what's this saying this is saying the underlying function that I care about the underlying regressor that I care
about is f and I measure some noisy data observations of that Y which is y plus
this independent noise n so it's nice about this is that this allows us then
to form our joint distribution P of Y and F and you see that this is again
so a couple objects that we haven't seen before so kff we gave to you mff we gave
to you and so you say okay what's what's kfy and what's kyy okay you can just
evaluate that out kfy is this expectation and in this case that equals
K FF k YY in this case equals k f f+ the notes so the nice thing now that we have
to connect this back to the regression problem is that the latent F which we
care about and the noisy observation Y are jointly Gaussian okay but wait a
second we just did this all with regular
multivariate gaussians so where did where did the GP go because we've we've
just been talking about GP so the point I want to make here is that if F and Y
are indexed by some some input points T in other words if M F is actually just some mean function evaluated at these at
these end points T these n index points and K FF is about being some kernel
evaluation then it could have just as easily written this as a GP prior F and
some added an additive noise GP N and use this same additive 'ti property but
here when I wrote this this specific Gaussian I would have just indexed this Y at T and Index this F at T so this is
this is you're starting to see one of the really nice features of the GP is
that is that all we need to do is bring in a finite set of index points and then
we're working with multivariate gaussians so so as a warning because of
this is that there is some overloaded notation here so f well people people
are generally pretty loose about that notation it can either be infinite in
other words a GP here I write F is GP so now f is this infinite dimensional
object or a finite multivariate Gaussian and that
that's generally pretty clear depending on the context all right the next
property that we care about is conditioning or doing bait doing
Bayesian inference so here we've got our Leighton F and our noisy observation y
and we know that those are jointly Gaussian and we've got F and Y are
distributed according this distribution so then we can do inference and we can
say that the posterior of F given Y is again a normal distribution so this is
this is this is an important fact of Gaussian distributions and this is just
a stock and trade manipulation of a Gaussian distribution this is actually
proving that this is this is the case is something I think that everybody should do once and only once once once you've
done it just just just forget about it because it's rather tedious but but it's
a cool fact to know so a couple things to point out here you see that we've got
this mean function I'm sorry this this mean and this covariance we can unpack
this a little bit we should all be pretty familiar with this one
interesting thing here is that you see that this is just a linear function of
our observations Y so that's that's nice to know and further this term here K F F
is our prior is our prior covariance so that's the uncertainty that we had about
that latent and you see that we've subtracted here this other term and
that's essentially how much our data explains about what we know about about
our prior so if our data tells us nothing about our prior uncertainty then
this will be a very small term and in other words our uncertainty is still
just around K F F if instead our data tells us a whole bunch then this get
this approaches K F F and our uncertainty decreases considerably
so the main point of this and you can you can not worry about parsing this too
much but rather inference of the latent given the data is simple linear algebra
so we've reduced all of all of the complexity of Bayesian inference and all
the problems that are sometimes associated that associated with that - a simple - a simple set of linear
equations alright the next feature so we talked about forming a joint we've
talked about doing inference so now we can talk about calculating expectations
so again this this simple term the simple conditioning term gave us this
fact and what this allows us to see is that I mean this is this is repetitive I
suppose but the expectation the expectation of F given Y is simply this
mean term and so that's what is that that's the map estimate that's the posterior mean there are a number of
other moments that would be interested in one one other thing I want to bring
up so we looked at the posterior moments f + y we've been talking about this this
joint Gaussian t between the late and F and and the noisy observation Y instead
we can look at we can look at Y which is data that we've collected and y star
which is data that we haven't collected in other words when when I said I want
to query on and see what my heart rates going to be at 9:00 a couple minutes
after 9:00 a.m. this is convention in the literature that YY star is a data that you want to predict so those are
also jointly Gaussian and so there's no different we just use the same we just
use the same conditioning property to get to to get to this fact and you can
see that this is our predictive predictive mean
the final property that I want that I want to explore is marginalization so
again we have these join the Gaussian variables we can marginalize out the
weighting because you say maybe I don't care about the maybe I don't care about
the latent function at all I just want to know how well this Gaussian process
model describes my data so to do that you just P of Y you integrate out f and
another another nice property of gaussians is that you can just read that
right off of here that Y is distributed my width covariance kyy so this is nice
because it gives us the the data log likelihood which then can be useful for
model selection model comparison and things like this oh right so one one one
note because we'll come back to this you notice here that when I introduce those
hyper parameters that lives up in the kernel those have been suppressed here
but actually the data is P of Y given those those kernel hyper parameters and
this this will be the basis of model selection because we want to tune our
our data marginal likelihood based on what those hyper parameter settings
settings are okay so at this point how we do okay at this point you might be
complaining because because you might say okay I'm bored all we've done so far is is is messed around with gaussians
and I'm familiar with gaussians and you know I thought it felt we were coming to
talk about infinite dimensional probability distributions and
interesting stuff and so if that's your complaint you're you're correct and I'm
sorry about that but but in fact this is the whole point right the whole point of
this is that we take this this beautiful mathematical theory and when it comes
down to actually dealing with these objects it's simple in your algebra it's
it's it's simple inference on Gaussian distributions and all this and so what I
want what I want to convince you of is that even with that sort of banal set up
we can do some some really quite remarkable things okay so now let's look
at some of the remarkable things I suppose that GP can do we've talked
about gaussians and words and pictures we've talked about some of the equations so let's talk about using using GPS in a
regression context so our example model which we've introduced throughout the
course the equations section and now we'll now we'll see what it can do so I
say the F is our is our latent it's a GP with zero mean and some and some kernel
k FF and the kernel has this form this is the square root x peole when we get
to talking about talking about kernels we'll mess with that but for now just
just just let that be we say that Y given F this is our noise term right in
other words up if I give you if I give you the latent function value F the data that I observe is distributed some with
some independent noise on top of that that's got some kernel that's the court
that's this is though this is the white noise kernel and that all this is saying is that two different is that two
different the noise that I observe the measurement noise that I observe on two
different time points is independent what this allows us to do again because
of this additive 'ti property is we can we can add this and see again that Y is
distributed as a GP with some kernel kyy and you see this kernel functions add so all right so now we've got the
probabilistic model the distribution fully specified so let's just fill that
in with some of these hyper parameters so I'm going to choose Sigma F equals 10
so that's the standard deviation envelope I'm going to choose a characteristic length scale of 50 and a
noise a noise power of 1 so all right so let's let's let's look at again of our
visual representation of that so this is the prior on F what this is saying to
connect this to the equations is that we've got a mean function of 0 so that
mean continues along zero at all and we've got Sigma squared Sigma F of
10 and so this is the two standard deviation envelope so now we can go
ahead and we can take draws from this Gaussian process so this is a single
draw from that prior F the GP and so now you see hopefully connected to this
notion of of how we can draw a function from from a GP so the steps of this
should be clear this is the only this is the only code snippet I will give you
throughout the course of it but it's but it's but it's only it's only one line
and deal on what will unpack that more ok so how do I get how do I actually get
this draw so to do that I take a whole bunch of index points a finite number so
here I took the integer index points between 0 and 500 I evaluate that kernel
function just like we did with those three index points I evaluate the kernel
function and build that into some into some 500 by 500 matrix kff and then I can take a draw from a Gaussian with
zero mean and this covariance so how that's actually done right and this this
would be your your MATLAB code how that's actually done is according to
this so that will give you this this procedure will give you this nice will
give you this nice draw okay so there was one drawing green now we've taken
four draws of that and I'm belaboring this point just so that just so that we
remind ourselves that a draw from a Gaussian process gives you a function so
four draws from that will give you these nice four functions
before when we were evaluating the kernel matrix we we messed around with
the hyper parameters a little bit to see how that changed see how that changed
the covariance matrix let's do that here in pictures so here is Sigma F of 10 and
the length scale of 50 if I change I'm going to leave that length scale of 50
but now I've changed the power of that the envelope of that from 10 down to 4
and you see what's happened is that the the the envelope has shrunk and the
drawers are shrunk not surprising if I change if I change the length scale of
50 to a length scale of 10 right so what this is saying is as two points get
further apart they fall off their correlation falls off more quickly so accordingly you get more wiggly draws
one final point on this is this this should feel a whole lot like when we
were evaluating those those covariance matrices in other words this is a is a
is a is a closer to the identity matrix the covariance matrix is close to the
identity matrix then for example this which has much longer range correlations okay so I mentioned that we've been
looking at regression in time and taking draws that are nice
temporal functions those are easy to look at and familiar to us you can also
have multi-dimensional input so one of the one of the the histories of the
application areas where Gaussian processes have been used a lot is in geo
statistics and they're they're often interested in spatial Gaussian processes
so for example latitude and longitude instead of regression in time you want
to regress on latitude and longitude so to do that we can make we can make each
input instead of instead of a single real real valued number a pair of
numbers so now we've got some Gaussian process here F is the same GP and the
kernel is almost exactly the same except here there's just an extra term
there's there's a there's a square term for each dimension of that Gaussian process so what might it draw from that
look like instead of one single function what you got now is a field over over
latitude and longitude so that your random function is drawn over that field
so I suppose shameless plug if you are staying for AI stats and you
want to see a multi-dimensional GP in action we've got it we've got a paper on that and I'm sure there will be other
papers with GPS as well okay so we've got the same model that we've just been
dealing with and now let's let's gather some data so this is that this is this
is our model the GP model we've got our prior and we can go in and take we can
go in and take a data point so I want so let's say I gather a data point at time
to 204 so what I know is that I can evaluate Y of 204 and I know that this
is according to the model this is Gaussian distributed according to mean
zero and kyy evaluated at 2:04 comma 204 that's a
simple univariate Gaussian this is all just pulling this right out of right out
of the definition of the GP Y and I'm just evaluating the kernel matrix so then we can use conditioning to update
the posterior so here I've still got the prior P of F but now I've got this data
observation so I can use I can use the inference rule that we talked about and
evaluate Y to ZAR take this data point Y at 204 and run this through the equation
and what does this give me well now this is refined my posterior estimate of F so
this is no longer a flat mean function but I think the mean function comes down
here and goes through F and furthermore because of our choice of noise
parameters you can see that what's happened here is right around 204 I'm
awfully sure my might my covariance envelope has collapsed and I'm awfully
sure that that the measurement is there but as soon as I get further away
well I forget because because I don't think that what happens at 400 is
particularly related to what happens at 2:04 so by the time I get over here I'm
basically back to the prior so a small change is that here we were looking at
the posterior this this I think is important it is important to hang on
just for a second because often when you look at GP work it's not clear whether
people are talking about whether they're showing that put the posterior or
they're showing a predictive model so this is on the posterior and this here
is the predictive the predictive distribution so you'll just notice
there's a very small change here but this this variance envelope has just
increased slight ever so slightly because we think that on top of that on
top of that posterior is actually some measurement noise so this is this is our
belief about the predictive distribution okay so then I had I had point to a for let's say I also gather another data
point at 90 well then I can add that in the same way to this distribution so and you see what's happened here is I've
added this data point at 90 and the same thing has happened that the variance
envelope has decreased the mean function has changed I can do this and add more
and more data points as I go and this is what we've got - and this is again the predictive this is again the predictive
distribution it's Gaussian and all I'm doing is adding more and more more I'm
making this vector here what I've observed longer and longer okay so this
gets us to this gets us to a question which is alright so I get more and more
data but when am I getting to my actual regression function because I can see
that we're doing regression here but when am I going to produce the parameters of my model right but of
course this is a this is a nonparametric regression model so this is one of the
virtues of Gaussian processes that we're not going to just spit out a couple
parameters of a quadratic function but rather as we as we gather all our data
the GP regression gets more and more refined so
this is one of the benefits of GP regression which is it lets it lets the data speak for itself
I suppose the downside of that is that all the data all the data must be so you see that as this Y grows and we collect
more and more data we're doing we're doing a larger and larger problem here
with inverting kyy doing this this this nice piece of simple linear algebra so
you'll often hear you'll often hear this this comment which is nonparametric
models have an infinite number of parameters and I'd like to refine this
slightly in our minds the way we think about this and not say that
nonparametric models have an infinite number of parameters but rather
nonparametric models have a finite but unbounded number of parameters and that
and that number of parameters grows with the data so the way you can think about
it here is that as this gets larger and larger and tends towards infinity yes we
have we have an unbounded number of parameters that can describe what our
prediction is going to be but that's still just growing finitely with with
the amount of data that we get okay
so we are almost we're almost through the basics there's one more piece of the
basics and Gaussian processes and using Gaussian processes for regression that I
want to talk about and that is model selection or hyper parameter learning so
all throughout the course of this we've been adding these data points looking at
the predictive distribution looking at posterior inference and we've been doing this with a fixed with a fixed model F
this Gaussian process with K F F according to this function so I want to
talk about these hyper parameters now L and Sigma squared F so here we've got L equals 50 and we've seen how changing
that can change the fit that we get so if I make L quite a bit smaller you can see that now L is sorry that this GP is
over fitting the data in other words it's chasing it's chasing each
individual wiggle of these of these data points it's recurrent it's forgetting
very quickly probably too quickly such that really it says here that we know
effectively nothing about about about our about our inference on this data
point when in fact given this is probably a better description we believe
that the function should be around here again if we have a length scale that's
too high then we're under fitting the data so here we've got some very
confident prediction but it's probably missing some of the interesting
structure that exists in this data so the question that we want to talk about
right now and what we want to address is how can we how can we tune or integrate
over these hyper parameters l and Sigma squared f so that we get so that we take
our data and we get to this model which is just right
so there's two popular ways to do that the first of which is to use the
marginal likelihood so we talked about the marginalization properties of the
Gaussian which is to say that we can marginalize out we can marginalize out
the latent function f and we can just look at the marginal likelihood of the
data Y and again I said that this is actually hiding these extra parameters
which are now the parameters that we care about these hyper parameters we want to do model selection on these hypo
forevers so just looking at this and particularly when you talk to people
that are outside of Bayesian machine learning it's it's quite common that
people say it's not obvious why this this this should this should do model
selection automatically and that this shouldn't over fit or under fit the data
but it's writing the math so let's let's unpack that for a moment so here is the
log marginal likelihood of the data and you can see that this has three terms
okay let's ignore this one this is just a normalizing constant so what are these two what are these two terms do this
term typically is called a data fit term and this term is often called a
complexity penalty so what you can see here is that if we just consider Sigma F
right so that's the envelope that's the envelope of the Gaussian process so as
Sigma F gets larger this will fit the data better in other words more data
gets more data gets inside that that envelope but as that happens as Sigma f
gets larger this term also scales up so you pay a penalty here so these two
terms these two terms your complexity in your data fit are at odds with one
another and that gives you this this this this automatic determination of
overfitting versus under fitting which happens in a Bayesian model
unpacking this for the length scale is is just a little bit trickier but again
is simple linear algebra and comes to the volume of this the volume of this
ellipsoid which gets larger this gets larger as you get closer to to white in
other words as you get a shorter shorter length scale and and that opposes in the
data fit term as well so that that's that's worth spending some time
unpacking on your own this is why you'll hear the term Bayesian Occam's razor or
Occam's razor is implemented via this regularization or this Bayesian model
selection because there is this you've got this data fit term here and you've
got this automatic regularizer here which which discourages over complex
models so the details of this dealing with model selection will be will be
fleshed out some in the practical
another way to do this and this this falls outside of the this falls outside
of the Bayesian context I suppose but one can also use cross-validation
cross-validation approach which is quite popular machine learning so instead of considering the marginal likelihood we
can consider the predictive predictive distribution for some held out data so
here I'll call this the predictive log-likelihood so here I've holed out some I'll hold out some test data and as
we know this is again a Gaussian P of Y test given Y train so it's again a
Gaussian so you can use this you can take derivatives on this and you can
tune my model hyper parameters in the same way that you do in any cross
validation approach and again that that'll be a model selection will be
dealt with in the practical all right so that gets through the
basics of what I want to talk about with GPS and words and pictures GPS and
equations and then using GPS intubation regression context we're just just a few
minutes ahead of time so why don't we take a few questions nice code so I
suppose I suppose choosing choosing a Bayesian model selection versus
cross-validation is something that people can do for a lot of reasons some
of which is is is just the just their opinion on this divide between between
between Bayesian and frequentist statistics I think one reason why a lot
of times people use cross-validation measures is for out of model data so one
thing that this one thing that this marginal likelihood measure is very much
within this is very much within the model that you've chosen
whereas cross-validation you're saying let me let me ignore the model so to speak and I'm really just interested in
in the predictive ability of that on held-out data so that's that's that's
one distinction that that's that's one distinction I think
with the habré's example as possible - is there a nice cow or give you kind of
periodic yeah so we will get to that in kernel in kernel choices we'll talk
about a periodic a periodic function exactly yes so so that that that kernel
that we've chosen the squared exponential kernel is is is just a nice
and canonical choice that gives you smooth functions over time we can we can
and will mess with that as soon as as soon as we go to the next section if there no other questions why don't we
take why don't we take a five-minute five-minute breather and then we'll come
back and we'll we'll get into some some more interesting details