so unless I'm forgetting
and my fam forgetting I think you guys have a
project milestone due today i'm not wrong about that deadline right
somebody nakauwi say I am i right
that master about the talk
so
one one hard copy to Maori per group and then email
case and and myself with the
Pia as well so by 4 p.m. it's not really very
it was very tough the time sister in a two-page right up
and after this point some point probably over the weekend
we're going to what will divide up each group
among the TA's are now said
improperly the group to be divided up among the case you for an intact
but each will be assigned it a mentor
so you'll be able to kind of have more close contact with 1k about your project
now on
of course you can ask the other shares myself as well
but we found in the past that when you wage for private a mentor
you a lot more attention and kind of makes the project
move forward more easily have any questions about the
I'm the this milestone before we start
thank you
okay I'm the last time we we arm we got most the way to proximal great in a
sense we end with the Feb on major completion we stop a bit more cover with
pokémon trading sent
and we've been a bit behind the last few lectures in terms of we haven't quite
finished all the material
and so spill over the next lecture we're going to try to play catch up on this
one because the mature I want to cover with the numerical linear algebra
primer that's gonna come after this is really not gonna take
as long I think so hopefully we'll catch up and will be back on the schedule
after this lecture
so we learn this album called proximal great descent
it was quite useful because it allowed us to solved with the same speed at
least in terms of the number of iterations required
to in talking virgins the same speed as great an ascent
but applied to non differential will conduct loss functions
and we're looking at particular typeof I'm
looks wrong way functions which were
decompose the functions for the class 3 looking at where we had a differential
conduct function
and then a a convex function it may not be differential will try to minimize the
summer those two
and the case to keep in mind um
is when the surface kinda the log-likelihood are a lost her
some kinda thing we would counter anatomical statistics a machine learning
from
and this is a real riser or penalty but if we don't make them Smith
but still conduct and we saw this very simple arlen was motivated by
the same kind of quadratic expansion idea is for greater sense
we could get um we converge
a at the same speed as great a cent if the function were only Smith
and essentially all we do take a great step for the specter G
that this part here and
after that we apply we call the proximal operator
which is something that is to find only in terms of age
the non Smith part as defined appear
the minimizer this problem I'm the
point that's closest to its argument
once we skillfully school the least-squares term by
one over to see and we at an agency so this proxmire peter tries to keep
as close to the gradient update for GV also making a small
so it's kinda a balance between us two terms can I think this problem operator
who sought doesn't depend on G it's only to find from the beach
and its it's also weekend in close form for many examples that we care about
because usually the right the county's we choose even other
nonsmooth are typically not super complicated and so we saw
example with the the l1 penalty essentially any lost in a one-party
weaken
any convex middle of no one party we can apply the spirits of pressuring our
weather
we saw that and the one that we're looking at now is
the matrix completion problem and some misconduct
tell you what that was to remind you on this is a more complicated proximal
operator
this problem we had something plus
traced our party and the something in the context a major completion
is just only scores loss to the observed entries are some HX the only observe
entries and Omega this matrix why
and the proximal operator we saw
up right involved I'm
a matrix of pressuring sitcom a generalized version what you doing I one
case in a vector case two matrices
we actually prove this in class we didn't prove don't part we did improve
was that
I'm we didn't prove that there's a certain subject French a love
the traced on that we could take for granted that you can improve on your
homework
that comes from Homer besides that we stepped to the proof
think it was fairly sophisticated and he didn't understand not been
you know make sure you go over offline
you can come you know to one of us and ask us about it so how do you drive
props operated in general well
I'm we know that for example for any convex problem we can apply some green
optimally
right if I wanna know something like what is the
minimizer love a problem like this
right with Isis some given
non Smith function but convex function Hz and I can always take a sub great
innocent I decided equal to 0
that's often a strategy we take for driving prop proxmire operators
so that's how we drive the this official in operator one of the 01 or we just
about
several lectures ago that's also how we did it here for the trace time
and this part is differential so we always just get the minus be here
over t so it's really just
D minus be over tea plus land the times the subgrade in
a severe enough the wherever the function is to try to minimize
sabbatical 20
an outdoor tell you what the proximal operator should be
that we did here okay an end up with a pretty nice album called soft Comput
on people use this
not at large scale but this is pretty commonly used for
I say moderate skill matrix completion problems K
the the thing that makes this difficult to do skills that are required an SPD
I'm this is an example proxmire part it's very expensive because
proxmire operator this s the lambda their cars taken yesterday with argument
and then I'm saw fresh warning the
the singular values and then putting up back in
and you know I we leave the left and right singular factors along with
official the singular values
and after them the matrix of pressuring operator returns that's pretty
charging to do it scale ice his example proxmire operator thats
I'm pay sophisticated this keep in mind we talk with ira rations that for
for their worst album it still wanna wraps lawn regenerate on the order of 10
reps lawn
to gin up placards solution regeneration is very expensive okay
are there any questions about a that stuff
so sometimes I hear people call proximal represents different names
the two names that you might hear column in proximity cent are complicated to
send
or generalized gradient ascent for two things that people often refer to us
car was it is the name for that it's kind of obvious it's because it
we try to minimize competent functions smooth placed on Smith
generalize refers to something else which is that
I'm through several special case a problem getting said we can get back
kinda
known algorithms right so it's a more general version of
for example any of these algorithms when you take a chick 10 I claim that we just
get back gradient descent
CNN right that didn't write that out
next flight why is that true with with greatness and remember the proximal
operator
Prakasam operator associate with
with H at some you know level T
it's just defined as
the art min overall see
have won over to T X minus the squared plus
agency the point that's closest
the minimize this criterion so what happens when a sequel 0 what what is
operator
sorry
yet write the answers just act so it has a quadratic
that's right but what's the point being that close to act Sestak's
so we're not sure the proximal operator at any level T doesn't matter
it's just a den the mapping so
what are the proximal there in a sense steps member I just take X plus
which is equal to the procs affects my honesty
times the Graduate
a grouchy but this is just the same thing as X
my honesty gradi
right which is just great dissent on G
hurts 1801 all we have this move part running anything definite great ascent
that's comforting um
when H is the indicator absurd a certain set
we get back but I when called projected good and sent
we can talk about 100 greatest thing with constraints proclamation actually
by the way to do that
when she's there we get something called the proclamation of them
case I'm a we haven't learnt on
yet we learned the subgrade method for solving for
minimizing count in a wreck conduct announcement functions
as an alternative to that all these are currently no have
convergence rate have won over epsilon and they converge right under this
appropriate step size choice that's what we know and by virtue of nine the
convergence behavior for proximal gradient
these are all special cases so have we only learn proximal credit nothing else
we already have been able to see everything that we did about greatness
and
project a great sandwiches nice as I think of proclamation of them
so I was gonna go through these very quickly II
we're already behind but some project a good sign is something that is
is quite useful right if you have a constraints that you try to minimize
a smooth function subject to constraints
and mean than we can use I will call project a great sent provided we know
how to project onto the constraints that
saw what's the problem are pretty here right if I if I give you
H being an indicator function of some set see
which means that 0 whenever that's argument lies in seeing infiniti
otherwise
the minimizing GE
affects plus the indicator on
that's being and see if the same thing as minimizing geoVax subject expediency
I we saw a bit earlier just a different way right to constrain problem
we can we can we reckon strain problem
as an unconstrained binary at in the constraint as an indicator function
and the proximal operator for this this particular choices H
you'll see it's just the projection operator on to see for the same reason
right
if I ask you to minimize
I'm this stealth two squared distance between axes E
divided by two T plus the indicator
function of is eating in C that all gonna do with men try to find the point
in seen as close to stacks
right for the same reason this is gonna be infinity outsider
see as can be 0 inside a city so through it is the projection operator on to see
something that we've seen before
okay so in that case
that the proximal great in the Sun all the time right
repeats repeatedly takes a great step with respect to G
and that applies the procs ok where which is just a projection onto see
so we get back an album called project a great incentive which is
something it's pretty natural just take a great in step with respect to the Los
the criterion G and then because we have to be in the constraints that
project yourself back onto the constraints Etsy find the point its
closest to
where that great step took you it's not obvious that this method converges
right with we learn greta is that we may have to think about how do we prove it
for something like this
but since you already know proximal going into said converters it's a nice
kind kind of remark
we know that this converges it also has the right convergence rate
it emerged just as quickly as ASM
great ascent this projection operator
by the way this could again it could be cheap or expensive
so the preparation complexity could be different for a lot of such projections
I'm are possible on cheap but for others they're not
at Sears the pictures spose our projection are constraints that look
like this
this these are the contras Mr criterion we would take a step towards
along a great enough to cut the crater in function
and then we have to protect myself back under the constraint set for next year
projected grant okay I don't know if we actually mention this but projected
subrident method works too
just so that you know yet this is a year where
that
the other one um but is a special case is called the proclamation albums
not we won the people study that much anymore this is to be more popular I
think
Kevin that something more classic Literature optimization
but its it's an hour in for minimizing I
again a convex but not necessarily differential function this is the case
special case will take JYC 10
so all we have is minimized HFX for some
particular I'm you know under French well but convex
function H and we know how does the subgrade percent right
but we can also do in this payment by just
confirm a special case in g0 whatever just
to is actually just repeatedly applying the procs
right because there is no gradient step there's no G so use repeatedly
apply the procs over and over again to minimize a chip to minimize age
um this we know has convergence rate wanna rap sponsored principles actually
a lot faster than some great method
so great about that had converted but wanna rap on square
we have very different convergence behaviors it's actually not study very
much more because it's not very practical
we have to know the props operator norse do this so
really only in special cases to know approx operator we've already talked
about that so far
units give me a generic conduct function it on Sicily smoother
I will be likely able to compute some guidance that's pretty generic
but being able to compute sprocket operators a lot harder
so people don't really use as much anymore on principle we could it be
faster than some great in math
any questions about those special cases
yes
om so how to slide on
I having you calculate how to calculate projections
maybe I'll put it back in the election not seen any can take a look at it will
put some access in the homework to
essentially the the answers that there is a
for across a different set projections you
you can just look up my projections onto
certain types of polyhedra linear subspaces
certain type the norm balls both things are well understood and you can
just kinda lookup I'm other
other projections you can treat them as a convex opposition problem try to drive
the %ah
now technicians they piss upgrading of my like we've done before and then write
down the projection operator now work
but the truck with projections is that you can move from one example to another
they can get a lot harder and you may be stuck at driving the projection operator
so that's why I put bill lecture slide back in open access in Homer
you can get a feel for kind of what is possible and what's not but
just as an example I
if I an arbitrary polyhedron the set of all
act such that AX is less cynical to be
I'm for an arbitrator for just an arbitrary meant matrix day
then this actually very hard to project onto in general
so projecting onto this the quadratic program
make finding that the point
access close to some other point you know another fixed-point se si
is a quadratic program it but not a closed form for that
and actually very hard to do in general but for certain special cases at a
its a we can do this efficiently that's why projections can be deceiving in a
sense
question so this is one thing that you
inevitably face in practice what you doing the proximal what what what will
you look at prominence in our weather to the come to prom you want to solve
you recognize it as smooth Poznan Smith me think I wanna pipe rocks I'm going to
send
and you go and you can evaluate the proxy option it's something that's not
won the cases that you've seen in class
and you try to write down there so when up now and it looks like it's not
analytically tractable be kept on clothes for the people I never we do
as the actual street that has a separate minimization problem right within
within the step response about the procs the run something off to try to evaluate
this
some other optimization I'll meet learn to try to evaluate the proximate to the
south minimisation prop
and they just plugged in right they generate a bunch of times can
approximate solution
so about the pokémon freighter approximately
and then move on with their I went next iteration and again
finished reading that's like I said as another minimization problem in applying
another population of them should thats something that people in practice
it's not uncommon but in general if we do this
all bets are off in time to the convergence rate but we know that if we
vow at the proxy exactly
and we get a convergence rate wanna rap song if the value in approximately then
you can see that we're not really taking the right
update right we may be actually taking up that that's
some sense it's noisy we should have been taking the proximal the
great in some perspective gábor taking something it's kinda off from that
at both pairs accumulated first kinda a big enough
Aaron each time you approximately about it the
you proximally out about the props up later then
the cookout on my not converge at all if they kinda accumulate over the rations
I'm on the progress and Nathan so
I guess and the take-home message that in general
if you're doing approximately minimum minimize the proxy operator
we should be very careful an estimate would converge however in practice
it it tends to work out a few about it
this prop this proxmire operator kinda high-precision
see run whatever opposition on the way
want to high-level back to see and there's emerging theory that saying that
if this is done
appropriately then still converges so here's a very recent paper
the study is this I'm it basically put the condition how much care you can
occur over each iteration
I've when you approximate minimize the this
in termination problems defined by the problem operator
okay
so the last thing I want to talk about what my came to this proximal great unit
was
expect celebration in past years I've even entire lectures about toleration
I'm I i'm if they want to give more than 11 tracks are Asian
but I think it what it comes down to is that it's something that's very useful
that you should see
and then look look back and use when you when you feel like it would be useful
yep
mean if I get an exact gradient
arm you're in a sense
it's not more sensitive but the
the proximal operator could be very flat on its solution
so you could if you run
if the property is very flat around its solution I welcomed
what controls the curvature of the proximal operator it some
it's really the strongly convex part which is this part and this is strongly
convex with primer 1 over t
to be taking large step sizes
then I'm this doesn't have very much curvature
so you could try to approx when minimize the thing you could say that
you know suppose you exit 1 year you think you're close to the
aquaculture in value the actual solution that year
you're trying to find you could be far off from with the proximal crater
so I don't think it's much more sensitive in terms of the amount
air you can't are then going to send just more like that you encounter know
the updates here
integrated and this this paper has conditions that
I'm say something like if the suppose you in car counter aired
Delta K and aeration k when you're I'm
trying to minimize the procs if it but Delta case %uh mobile
if I add up to alter K overall iteration
the arguments finite then I'll get conversion solution
some I have to encounter less and less air as you go along you wanna make the
you know whatever album the running to fine
the approximate solution here
you wanna make it more and more accurate across types of the oven but it's a good
question I mean
I think there's there's emerging theory for all this
right for what happens when the great in its office nor is it a problem operators
ninety
and we have results that offer to do not the jewish in it just never clear in
practice whether
whether your situation where those are met
okay um so I'm just gonna taiwan acceleration is basically without going
into the details
I don't think the proof the Super I'm insightful as to what why acceleration
help so much for them in previous years but I don't think it's
no stinson really gotten much out of the proof
I'm but it's a very interesting idea
and its do essentially the one person to
I'm teary Nestor of this some whatever
very influential figure opposition especially in in Las
15 years or so so a long time ago
master of have this I this paper on acceleration for small functions that's
the
and a first paper we'll see about this 1983 I think that none of this really
got picked up
until people start looking at proximal methods
and then it really took off so there's a sequence of other papers
rite II bring down four papers here by Nestor of and then
a paper by back into boo hoo took this idea and
made it simpler in some sense and apply to proxmire with M's
proximal grace Ann Arbor them's and all throughout
is the idea that if we use more than just the last interest
if we try to use more information on just where we've been the last step
me carry some kind of momentum through cross rations
and we can actually accelerate the convergence rate apps a great in the
center approximately
percent from wanna perhaps Mon 21 over the square root apps on
and what from something we saw earlier
that's actually optimal convergence rate for first-order methods
so termed the rates we can't do better than that this is achieving the optimal
rate for first-time offense
so what's ex-soviet approximately 8 percent
it looks like Procter gonna send except for we change the argument that we pass
to the procs function and we we change the
step which we make the great update with respect to G
to reach you some we keep track of essentially
on the events axe but before we pass
right if we're doing regular populace that we would just pass
on XK -1 here minus
the step size times the grain FG it
-1 to the proper and move on in XO it approximate method we something
different which is that we take
the lasater and we at something to it
miss a week all momentum term unused you can see that if we're moving from XK -2
XK -1
the next term kind of continues to push us in that direction
correct XK -1 and instead of just about in the proxy XK -1
wheel out some other history the direction that we're going to get there
to push us a little bit further and
I'm and then week the finance BV
this choice of constant actually attended to be quite important
for the kind words for the album so this is taking us
as a fixed constant came I'm sure K plus one not something you have to know you
should turn
all their other conversion strategy is this one is provably convergent
on and what we can see is that at when K is one right at the first deposition
approximately an update
after that were carrying momentum from previous iterations
and what are these weights look like across the duration
so they get bigger or smaller as K increases
sankei is
is one I'm
write this thing is the how
make them it's a case to but in any case it doesn't it
it cases to hear I guess we're going to get the usual proxmire an update
and ice cake its larger with these weights get larger or smaller
what a approaches K gets bigger maker
that ratio came and took a pass one
this approach is one right
hey it's Lauren love this thing get approaches one so here's what the
weights look like
momentum ice gets larger and larger as the iteration of the algorithm proceed
so what does that mean about the momentum when we apply more momentum
were plying momentum later on the steps they are with them
I thought the start for letting the in a red
that were visiting mostly dictate where we're going
as a algun proceeds were allowing the history take push us more money
direction that we're going so why is that helpful
on there's there are kinda a few proof that why this gives you a faster
convergence rate
I don't hunt by Paul they are and honestly missus someone appreciate
statement but I want says this
intuition for why this particular scheme works is really locking terms
exploration
some in Afghanistan mathematically but not intuitively here
n intuitive explanation that i think is pretty good
but doesn't tell the whole story as the
if you're a late step in the prominence an album for example
and the gradients gonna be small because you're going to be close to
the optimum right so on
you not going to be really moving very much because you're trying to move in
the direction of the negative gradient
it's pretty small so the progress they are gonna miss gonna be pretty slow post
op malice by virtue of the fact you're close
the optimum at this point acceleration says that if you're in a good
neighborhood are the solution and you've been traveling there
it's gonna continue to push you in that direction
and pushes you more at least up they are in which the tapping on the great in a
small
writes that the match in a picture like this
release to the contras
the function you try to minimize
than as you
get closer and closer top Mountie say with proximal rent
the step sizes get smaller and smaller at even if this
effective that I think it's Mon smaller even if the step by itself was fixed
because we multiply by the grain up the loss function so effectively
because the grains getting closer and closer 0 the steps are being contracted
what acceleration does that says if you're going the right direction right
then instead of just taking the next
better to be you know where you are minus the great loss function
you can add something which is along the direction where you've been going
but you can add some amount if this was XK -2 and I was XK -1
than you're gonna be adding this traction to the next hitter before you
apply the proximal operator
so thick celebrating you in that in that kinda
direction that you've seen over the history in rats and that acceleration
gets larger and larger because these weights because imported ones you get
caught
farther down Sep they are with them so it
if these weights were tuned in properly then you'd be we would somehow
be accelerating our South maybe in the world the wrong I'm
at the wrong time to the optimization procedure might be pushing herself kinda
far the way mcnally
but this particular tuning up wat this one here it kind of pushes us in the
right direction just the right time
so
computation this is really not much more expensive right and surpassing this to
the props operator generically weirdest
changing the argument yeah
stochastic methods arm
no we don't really of the Cassegrain innocent using
that's interesting question I'm
I'm not sure
I'm not sure where much work that combines those two ideas I'm
thing because the great instruction to some noise already
could be yeah but so what people do in stochastic methods which
we did mention but we may come back to his people often do many bats methods
between Santa and third is picking one function to minimize
and taking the great respect that function you take like a
I'm a subset of the for theta functions
and you its gonna many bash you about the green arrows
them end up being much less noisy so many many many batch
minor fix our nation could work well not really
aware people that I mean it I'm sure it's out there and it's not aware
work that's good mine it okay
I'm yeah
up
our I see know if that's not quite true it's a question I get asked every year
could I think it's it seems like it but when h0 let's
like you said congress no proximal part I'm just doing it so many great in the
sand
so on this is a member this prop goes away
I'm really is taking up it to be the minus TK gotta G the
on let me
actually not answer the question see if you guys can
think about it and figured out on a strong a picture and you can think about
it
just because it ends up barely taking a while to explain
and it's something I think you can think about
figure out if not we can revisit either in a later class in office hours
but the all rephrase the question which was that
on let's suppose my updates were just as follows:
XK for equal to
I'm the
minus T K
got Jia me where V is equal to
I XK -1 plus
k -2 over K plus one times
XK
-1 -2 XK -2 so this would be accelerated gradient descent
which is a special case when we take 860
I quit so the question was business just the same as taking a longer step size
effectively right and
it's not the same it's not fair because you're out when the great in a different
spot
and you can't ourself a picture to kind of cancer stuff that's true
markdown XK -2 X 10 months one XK here
you to actually value in a great in a different spot that has taken a longer
step size
so it's a fair question but not quite the same which is good because you know
the greatest and covers that are 8-1 a rap song
and the skimmers at a much faster rate so this better be doing something
different
okay all good questions
so here's an example of how toleration can really help
I'm we solve this problem known three waste so the first thing I did was take
some grading method
I showed you this is a great lecture this was something I colossal penalize
problem
sister I one penalized regression we can apply so great a method to that has a
career in fact wanna rub
on squared you know that and it's a great steps are fairly simple
last summer yet last time I looked at them
album called established as the iterative saw fresh falling out with him
it's just proximal inner Sun applied to the problem
three generations stop pressuring I was the red curve she's a current rate have
won a rap song
so which is a lot faster wanna rap on square
now I'm applying its operation by national acceleration to progress and
people call it fester the fast interests of echelon our them although those names
I think are kinda silly
you only see them written down bunch of places it has covered state one over the
screw it up so on
so %um these are three different regimes have conversions you can see there
then they really make a big difference in practice one thing you might notice
if you look carefully and if we if we stressed out this
this a XQ here so all you saw was this blue curve
I'm say federations 1 202 some like that
you see that actually the blue curve is not monotone decreasing
so we get these things called Nesterov reports she's
I guess an affectionate term because the fact that nest of
immense declaration you see that actually oftentimes may apply for Asian
this is what the crotch area looks like a cross iterations
and this is just
a I'm trying to the fact that it's not a son method
so television actually does not guarantee that we decide on the couch
and every iteration
unlike progress on which we know does so that that
you shouldn't be surprised if you apply exploration into something like this
this is actually I'm not not unexpected
on homework again if Manning acceleration I think for a prompt them
like this and you're gonna try to be
recreating
sure like this okay
I'm convergence analysis is what we're basically what I said
a under the same assumptions it gets a rate of one over this write-up salon
or the difference between the chief criterion and the operator after case
that says something like one over K
squared the prefer this I'm
I referenced a paper and all national papers at the end you can take a look at
the proof
its it's very different from our personal procurator because it the fact
that this
its operation i think is actually kind of an interesting creation
at the purpose as a bedside on that way a
backtracking as possible exploration I'm
the back talking strategy is really quite someone what we did in
proximal gradient so here's the criterion that we use to exit the
backtracking strategy
this the slide your you'll get to know and love one yet when you got the
homework because
raskin to acceleration with backtracking so that you'll pay attention to
I'm one of the differences between I what we have to do here and in other
strategies
is that if we took a certain step size at Step K
the step can only be smaller not accept K plus one
we actually force ourselves to chew small on smart steps across
durations that's not always true with backtracking the backtracking it for a
greater sense a
we just chose some initial level to always start from like one
here to make it work and this comes from
on freaking out where one can apply backtracking and the proof
we have to make sure that the step size decrease across iterations
so maybe less effective and sometimes the backtracking other methods
because I've ever took a small step by the one direction we've kinda
force myself to take subclasses there are at least that's my on official
rations
lafayette we just I want to talk about
I'm ideas excavation always use force it seems like it's %uh
such a good idea that you might want to use it for every problem and there may
be no reason
to use just proximal gradient on its own without acceleration
biaz talk to our implementations have fast our them
that are not accelerated and that's for pretty good reason
its generally a very expert affected speed up for
but it's not always the thing to turn to and here are a few
I'm a few things to think about
when we thing about applying sarish in first law
be gaining a firm acceleration is often diminished when you're solving
many opposition prompted sequence I'm using what's called one starts
so amenities promptly talked about their some county planner
bike with a las others the count primary lender but major's completion as a
county planner
whenever havoc a lot of course regularize is always a penalty parameter
and what we often do in practice
I'm again you actually importance on your homework is
we solve a sequence of these problems at the largest by the reservation primer
assay
right at one end were convergences easy and then when we finished
I'm not particular problem we move on to the next one
I'm initialize optimization algorithm with the solution up the previous
problem
right so this is called a warm starting our optimization algorithm
such as written down here saw for lamb the one for the biggest value
initialize I'm it
arbitrarily and some sense record the solution at Atlanta by AMCON
X had a plan no one just to go back to the habitation of news in statistics a
then or solving for another valuable and missile and the chair asked
initializer at the recorded solution for the previous valuable and a
and when we do this the number of iterations required to converge
later on in the warm starting sequence is often much less than if we just are
that that valuable and arbitrarily
why is that because there are too close to the solution on the starting of
if we believe that the solution is continuous the functional up the
regulation panel and a
which in most cases it
yes than this one starting already putting up close to the solution rather
starting in 0
for offense for already kinda been a good regime
and acceleration in this context often doesn't do much
and just running some like proximal gradient this one starting off in
supersedes the need for acceleration
doesn't hurt in general the situation but it's often not worth the trouble
so people my do what mess of this prompted for starr said
sometimes don't apply acceleration sometimes acceleration
and even backtracking can be harmful actually sometimes they can actually
make
your life harder when you're playing an opposition over them and the majors
completion problem is a good example that
I'm that the details maybe aren't worth going through
I'm super carefully but take a look at this offense that makes completion at
that something that you understand for your project or or just in general
you can take a look at the slime are carefully essentially
one removed from but say onyx re-approximated took other pokémon it
all we're doing is the evaluating the proxy a different argument rights to the
proximal Perino here's taking a speedy
and applies official into the singular values
I'm accelerated we take an SPD if something else Raksha like adding
something to
the opportunity with original have taken the SPDR the real object will take an
SPD on
in the context of major completion the sum up two matrices
it's something that's part performance low rank
and taking the SPD a combination sparse
low-rank matrices we actually have tricks for how to quickly that's
something that
you know people a study in other contexts a on
and so that s pretty can be done in an efficient manner
the first term as was the projection of Y and
if they're set that sparse if the observed that is quite small the second
part is low rank by construction for constructing estimates to have lower
rank
when we apply acceleration we actually change this part B
this is that the internet I wish for a value in the gradient we change it to
something off switches be
if you look back at the former be it's it's no longer lower rank
actually its rank can change because if the sum up things from previous era
asians so kinda ruins this nice structure and means that the SPD that we
act have to take to get the proxy operator can be a lot slower
to people must never apply coloration matrix completion if for you think
something like pokémon grade
in alright so
conclusion is that it's a useful tool but not something that
say should always expect to get better performance alright
I'm that was about it for the Excel for the proximal great acceleration
lecture the Tana references on the stuff I put some at the end
it's a very popular topic for current research
I'm so to get the some have an exciting time in terms of this topic
any questions about pokémon great an acceleration
before I move on to in america where out my stuff
now okay let's take a cup minute break and then we'll do our
numerical linear algebra lecture
short break I'm by wanna give you some good training with numeric where algebra
this is the last lecture we have in terms are are women unit for for for a
method
and the first one method to support in context
I'm but some pretty classic material for greatness and some rest and
even proxmire greatness and that's a pretty classic material to study for a
long time
acceleration with the only very briefly touched upon is
very recent so it's kinda like I'm a sensor
a regeneration a rebirth of all the exciting ideas and first-order methods
and people are like I said I i find that currently
you'll see paper that net for the conference's the try to improve on
exploration honest and a better
even you know each year so we went from
kasich stuff all the way up to more recent stuff before we move on to our
next unit
I want to talk about I'm summer the kind of computational underpinnings
are the summit the stuff you need to understand
or to figure out how computationally expensive some these methods are
this is gonna be especially relevant when we think about first tournament as
compared to second on methods
and past years I just kinda assume that people would become fourth how expensive
a matrix matrix multiplication and beer matrix all the beat Federer
and I got some people who are really come forth and people who are pretty
confused
so this is this year we're gonna try to do a primer on numerical algebra
so we can't get this guest house in the same page in terms of how expensive are
some its fundamental operations
I'm if nothing else you should walk away from this lecture
we come into effect more methods which are coming up um
and next two albums unit with the with the idea that
solving on your system is generic be more expensive than computing
then not appoint a matrix by vector say or by taking in a park two vectors
and that means that a doing something like
a second one method is more expensive in terms of iteration
complexity and terms that the complexity beat generation then
something like a first-order and we'll see the details about in this lecture
and hopefully keep in mind as we
as we kind of talk about linear systems throughout
so years we're gonna
go through some this may be review for people I think
lot of people may have had this and other courses I'm
some this this might be new to some other people so
will just talk about the complexities and basic operations how expensive is it
to solve linear systems
and then what we get from things like matrix factorizations
and so forth when we talk about the computational cost
a song on the system's okay and and case it's not clear
um solving linear systems to the fence at the heart of water these optimization
methods
so we haven't really talked about that for for personal matters because all its
business computing and gradient
are taking like in our products have actors but we talk with secondary
methods
which really the heart of those is that the linear systems of
so what is one way to measure the complexity a basic operations
we use what's called a flop and for a floating point operation
and a phat counts one of the following things addition and subtraction and
multiplication or division up to
floating point numbers this just means that we're not find them as integers
a memory thing about so in the mess floating-point I'm
with boiling point precision so
in truth a division is actually more expensive than the three operations
but we still count as a flop case that such as the basic in a computation
how many foxes now with Muse is a model for how expensive it would be to run
this is a rough measure of how expensive it is
and we don't think about exact pop counts because I was actually are very
informative
for a lot of other things that affect the practical runtime:
album doesn't flop it's not a perfect measure by any means
but worse than rough of counts because the roughly refer reflect
are kinda stand a computer architecture how how expensive it would be
I'm to perform a given sequence of operations
so let's go through some basic operations up hope we all this
um you know you can internalize the foreign internalize
how expensive is it to add together to the actors
the reserve like them how many fobs that cost
so talk about Ly's addition of two vectors
just and fought right because I'm not adding up each component
to and doing an addition but that skier multiplication so skilled times a vector
also costs and flops listing and multiplications
enter park between two vectors takes on the order inflow of
but caution a two-hour right because I'm having to multiply every component
a maverick upon be and modifications
and add them all up was actually and -1
additions but I'm what does count as an
so it's we're gonna say it's too and flops K flops don't tell the whole story
they're not a perfect measure by any means like I set up the computational
complexity of
a some sequence operations for example
if I want to set every almoner a 2-1 of Iran
something awesome code maybe my Co look something like this issue notation
that would be zero flops think of many flops not doing any addition subtraction
multiplication division
k it limiting actually cost me more like and operations
but in terms of our country 0 okay
such as an example part that flopped don't tell the whole story
we're gonna keep track them anyways because in many cases that's what we
care about
okay
I'm moving on to some more sophisticated operations have a matrix-vector
operations
declare matrix-vector product so 5 a.m. by
n matrix Anna I'm
and dimensional vector what is the product I've eight times be cost
the day
into this is be and my and
what we know that a similar product costs about
two times the like better product flops and so
I did take their part have one wrote a would be
that cost to and pop to the matrix multiplication actually need to do em up
those right any tomato
taken apart every road a with P
so too and and flops if he is sparse
then this is a very nice property of sparse matrices
you commit yourself it takes something like to ask law
presses the number of nonzero elements and a right that
you may have that you can remember that while thinking about the general case
where takes to MN flops am times Aniston after the number of entries in a if they
withstand
when is sparse replace that M times and with
ass right animal I that's true it's because
for every nonzero elements and a if we were storing those in some kind of
clever way and we can
iterate over those but actions matter cuz flops don't count anything but
I'm addition subtraction operation the divisions
don't have to multiply non-germans today
by the course by all means to be an add those up sorry on drama to the total is
just ass
said between a mock patient division that's just to operations for SR mints
makes
2s Fox when a is banded
um it takes two times and Timescape flops
so in that case a looks a
here said they were square for simplicity
is banned it means that it only has non-zero month
I'm in some then with around a stag no
and this from their scale month
they are nonzero in this band and these are all zeros
the k-band major day this i cost to tens and tens que flops
are you thinking about for the same reason and the ass part1 us for example
if a if we could factor a
as a summer break one matrices
and there's are them that means that a has ranked are
then multiplying a by the vector be cost two times are plus
employs and Fox you can see this in a couple of ways
first where you can see this is I'm
by looking as follows: right how
how expensive is it to multiply on
the by a dude in a product
the I transpose be to and flops
how expensive is it to do the inner
the I'm ehlmann wise product view I times this killer
that's just something like I'm am
flops recognises like them and have to do our the so
that and they're being total to our times and PAS and flops
so Bears rank are then action may just not patience a lot cheaper
if are small right this
this I'm two times our times
the summit and plus an it's going to last more than two times and times and
if are as a lot smaller an MRI
so low rank matched we can multiply quickly if we if we have factored them
another way of saying that is I'm
the summer you idea transpose
is saying that a is equal to you times
be transported to Macy's UN be
where these have these you as is and my RN R&B is are by an
for permutation majors how expensive is that
permutation matrix actual cost Europe lot because again wrongly counting flops
that in perfect measure how expensive this operation is
permutation matrix only has one one one
in any given row so this is an example of a permutation matrix
something like this and if we multiply this by a vector be
then we're just rearranging with meeting the elements UPI
across Europe flops does not actually doing anything
thank you
okay any questions
just kind of basic operations are standing at this point matrix matrix
product clock cost
I'm something like two M&P flops for a change
be for being and actor tune and my pee matrix again you can
if you member the matrix-vector product you can see this one because
would estimate expect a part for every column of the matrix be now
that's how we get the product eight times be busy matrix
so we're just multiplying the spiky the number of homes and be
and for an S parse a cost two times as times P flops again for the same reason
was not
wearing a sparse management cation for every comment P its losses be is also
sparks
so theme alert sparse matrices we can do things more cheaply
band images we can do things more cheaply I'm
permutation matrices we can do things for free according to this
measure of complexity so these are the things a long look out for
for for um able to structure things accordingly
matrix matrix-vector product this is one that
I'm when you're learning a programming maybe did the wrong way
and you didn't realize that it was marked pensive how expensive is its most
by a by bebe I see
what either not too bad it hurts
pretty expensive depending how you do it right so if you multiply
a by the time see
for sis eight times be time see
not have different computational complexity into the clock counts I one
case you're in a matrix matrix multiplication
matrix-vector comp calculation now they were doing
a matrix-vector calculation in a matrix-vector calculation they've
pretty drastically different pop cans
here's the the meat
above um what we're gonna study solving linear systems
how expensive is it to solve a given a system
say actually cosby where at this point a square
and its invertible as an inverse how expensive is it to compute
actually cause a inverse times be in general
way should members that cost money order a NQ flops
so AM I N and it stands solving our system is
and Q plus go back to these things a square
then matrix multiplication a baby takes on the order
and squared plus very different right NQ resident squared
and inner product between two vectors that are both the link them
cost and flops so vector vector modifications and flops
matrix-vector Cup calculation is and squared flops
I'm and at linear system solvers and cute flops
so that all very different in terms of their complex is specially when N is
large
I can make a huge difference but they're so make sure that we can actually
I'm for which we can solve linear systems and much more efficient manner
and here are some examples of them if they were stacking all
than solving actually cosby only cost and flops
I because not his only have two for the dire la Mensa
a were a 13 and means that
right it was all zeros except for on the diagonal had these numbers
than solving
AX equals be just requires
dividing every component be by their corresponding component the diagonal
actually and flops for lower triangular a we do something called back
substitution
soul or trying limits is look like the following:
on they have only nonzero elements on the diagonal and the low
so these are all nonzero and these are all zero
how do we solve I a lower triangular system
and efficient manner we do something that's called Ford substitution
so we look at the first
supposed to offer go out the first almanacs we realize that I specified by
I'll
a linear system in x1 by the sauce is that a
11 times x1 equals P .001. because every other
element in the first row of have a is I'm
your so tells us that x1 is equal to be one
over a 1-1 and now he moved to the second row
there's only two non-zero elements in the prior to that role with X
and so Tyler system in x1 x2 30 a 2-1 x1
plus a 2-2 next to it cosby to
and because you are no x1 is
begin a substitute in and solve for x two and
while this cost 14 of this is gonna cost something like on the order
to flops and and what and as we go forward
solving for the K throws can take in the order k flops the number of opsin total
is on the order and squared
the sum up want the some other II
I going from one thru n its like on the order and squared case it's gonna cost
but
and squirt flops to solve a triangular system which again very different from
and to pops
for generic system if upper triangular matrix up instead of having on the rows
below the tide of iron on the Rose
on attacking on above then it's now the shot his car back substitution with us
all for
X and first and of the cost and square flops
for sparse
day there's not a generic way to solve a sparse on your system in a
a tape
aqui sound efficient manner it depends on the sparsity pattern
so there's not a rule of a generic rule like there is for
sale or triangular matrices for Ben images
but they often cost a lot less than and coupons
I'm in the worst-case
it can still be and Q even with the within
with a fairly sparse matrix because actually matters
the structure that sparsity that matters a lot of times have
the fuck on resolving our weapons thats
gets be fairly complicated we're not gonna talk about I'm
sparse our items for when your system
leonard albums for sparse linear systems on
there's entire books about that but the take my point is that it can be done
often a lot more
cheaply than for 10 system can offer we don't even let your time
if the sparsity if the amount of sparse the acid linear
on a linear order and but exact worst-case pop concert are difficult
for more structured matrices
like 10 images which are obviously sparse on
we can get worst-case guarantees that are very good
so if K is if they if they ask a man did which is this picture member at some
is banned in with family K then it costs
n casework flops to solve a linear system actually cosby
k is usually pretty small so that's just usually considered linear
serb and lessons we can solve re officially
worth almost as we can also saw quickly because her and her sister transpose
right so mom if you have an iPhone a matrix again you should recognize that
and would before matrix multiplication rather than
a linear system saw much just on the order and square again
for the same reason we have before and
just to tell you you know again permutation matrices apparently are free
according to measure but
we make a mint ice office more like and operations
with a more generic measure of computational complexity
a for permutation matrix it's a special type without a matrix
was only 11 in each row and in versus its transpose
so many you can check damage suffered by going to some examples
so far no in a system again cost your fault
I was gonna continue blazing ahead unless there are questions to make sure
you guys
interrupt me if you're unsure about something not sure that
I'm not sure how many people for this a who seems kinda stuff before this is
review released actually knew
so he having questions may actually stop manner
keep a pretty pretty fast pace
okay matrix factorizations
are at least in the context of what we call direct methods
these are the way that we solve ministers in general I'm
you might have learned in the previous course potential even high school people
learn about ghosts in elimination
so it's a linear systems they can be
much more useful in general to fact rice a method and perform a week of practice
of technique then got a nomination
and there are various reasons for that on
the biggest reason is that by factor rising
we often consult many linear systems and a
for the right hand side changes and a way that's much cheaper than having to
perform separate linear system solves
so by factor as a major a women the right
expressed as a product a simpler matrices
care them to a career in expressing the product a 13 K
and usually these days are simpler
soap their orthogonal triangular docking or permutation matrices those are the
kinda 131 end up with
we are if I why it's because solving linear systems not made it is a lot
cheaper than solving
an estimate dense matrix usually only factor is a product of two or three
matrices
seems like a small it two or three in order to
solve linear system mexico's be we solve
the scaling systems individually by
inverting this matrix have all these factors are invertible
I December teach wanna reverse the order so I first saw
a a one-time tax equals p/ and I solve the system
on a two-time sexy 'cause
whatever the answer was to that
et cetera okay for computing the factorization is often expensive
so just computing this is often the same cost for solving linear system
such NQ plops so not sense for not doing any better
once we have the factorization because these are special matrices
on applying this factorization
in this map in this manner to solve linear system only cost
and squared obviously that's because I guess that each one of these linear
systems
can be done efficiently because these messages are structured
soak in the factorization in the first place is and sweatshop
13 other factors nation we get the answer and and for NQ Fox
popular accusation the first place once we have it we take and scrape up to get
the solution
so that means that if we want to do this suppose you want to solve the problem XE
cosby
and actually cosby prime
and so on and so forth then we would have factor a once
which cost and
on the order and keep flops and then we would use it to solve
on
you know X because be X because be prime
et cetera and each of these steps so that only take and squared
computations so with we had a lot of them
then this would be saving us a lot of computation time
with their question up there that I with them
I'm
so either Britain yes because I'm writing than
has having members but backers Asian methods applied to the rectangular
masses as well
what do that we'll talk about the QR factorization which doesn't need it to
be
square okay
on to clear any question about that another reason that we like to use
factorization methods is because
often if a under was a simple change like if I told you to solve the problem
mexico's be
I asked you to solve a prime X equals p/ prime
for a prime was just a little bit different a then if you applied
the couch in elimination are some generic technique it would be about the
same cost
over the factorization method we can often get the factorization for a prime
from not that we got for day with a very simple update
if if that if that difference between a prime in a is kinda minor
and we can update the factorization officially now as a helper said
solve many where system for the right hand side changes we can solve many what
I'm an assistant for the left hand side also changes
as long as those changes are too drastic like a rank 1 changed a we can handle
if you give me one more row or column day we can handle that
back kinda thing so we're going talk about
to to factorizations only but they're a lot more
there are I don't know five-plus
the people probably use on and
from the stuff we talked but so far
it's all fairly um assist any material
for matrix factorization all the stuff from talking of rest lecture
there's the definitive reference which is a really great reference
as this book called matrix computations by Kalman van loan
have in reference to the end I really like to pick a lot so
iraq a highly recommend you take a look at it if you want more information about
some other stuff
so is this a metric and positive definite matrix oath
square convertible and saucer symmetric then have what we call to less kitty
composition
which means that we can always write a
a as a product have a
lower trying to make just enough for trying to make checks looks like this
K
and a we can compete this factorization
NQ flops so much members actually on order and cuba over three
and this book I said to tell you how to do that
other unique linen know how do that in practice cuz you would just use
you know your favorite MATLAB to art will do that for you
once we have the choice key factors we can solve linear system very quickly
just by performing to try a system solves right
the first solve I'm for why equals ellen first time
be by four substitution that takes and squared flops
and then we do the we saw for I'm
that should say why I'll transfer them first times why
not be here as the type up buy back substitution
to this matrix is upper triangular answered
pops so compelling factor they should take and cubed
solving pics and square once we have it together takes ends
and coupons and scruff of which is what the cost 10 taken us
just to do something like us nomination order some generic
thing for solving the crisis in the first place but
if I told you that you want now to solve X equals p/ prime
their own take you to an square flops if you store the chill escape a
you already have on all transpose you just have to do these two
trying to assess themselves so
one interesting context the real talk about at least squares problems
the very common setting on people are encountering
linear systems as pretty classic
suppose I wanted you to compute the minimizer the falling least squares
criterion
right I want you to find the be that made the all-time between why next
explain to the smallest course we know the from the normal equations despite
the French in this is that an equal to 0
the answers expense was actually much that can spawn why provide access for
comic
K how expensive is that if we want you to tell us keep factorization
what we fact rice first law we went back to rise X transpose axe
which is the PPIP matrix and use that to solve the system
expense was expected a because X transpose why 3 at the Computex
transpose why
back of some sort of P times and flops if the committee celestial expense was
act that cost
on the order both I'm Peace Corps times
and flops so I'm
so I
which left him a chance for that cost piece went over 34 up just like we were
before
computing act complex in the first place 'cause Peace Corps times on Fox
for matrix matrix multiplication I'm
stuff we learned and then solving linear system once we have a choice key factor
addition costs on the order of p-square pops
so in total this costs about and P squared flops
plus PQ pops or just about NP squared flops
if n is much larger than p the most expensive thing here is actually just
computing act transpose X
that would be now dominate the computational cost
it and much much larger than p sewed
that's one way to solve least squares problems use much less keep back
tradition
here's I'd say its main competitor
I think that kill SQR them two things people use the most
some you may disagree with me by so there's a lot I made it back positions
these I think art are the nice to know because they have
kinda complementary strengths the QR factorization
takes a matrix a and it
decompose it into a product up on a farm in a matrix
with and upper triangular matrix
once you have this by the way
what was a transpose a look like if you had orthogonal columns
a transfer they would be
right a transposed a
it was a cute I'm
are the are transposed times to transpose queue times are which is our
transpose time
are lol transpose if I just rename that
so if we have the QR hey I know it's a lucky for a transfer that
okay
um the computational cost is similar
it takes on the order of here now it's am
and squared flops but if a was square
this would be an BenQ flocks because I hear
we can actually computer QR factorization frame rectangular matrix
you wanna put on the same skills such a less cute thing about a weakening square
and a cost something like and Q plots
I'm in to use this to solve linear systems we use this key identity
which is that if I have any
vector axe and ask for to Norm
and that's going to be
I'm I can write that as the
the to normal
cue transpose act plus qto to transpose axe
square where
I take hugh and
I put next to a cute order and these forman
a square orthogonal matrix 0
this I was gonna be on and by
am this is comes from the fact that this is just equal to
they found a matrix I'll
transpose times X squared we always know that right
the normal vector is invariant if I'm off by a farmer matrix
so how do I solve least squares problems with security composition
I use this key identity and I say that if you want to minimize
y Minas expand a I think you are a axe
and I can multiply everything through
bike you transpose right and I get
cue transpose times why this part
birthplace X with QR to desist our times
beta now substitute you transpose times all this
I strike you total campus and all this work you total was justice a set of
complementary columns
the complete the space the North that they're not actually cancel the part
that involve
beta so all that's left is to transpose totems why
some other words this doesn't matter for the minimisation our beta
and finally scores Palmer the chart accomplish in all have to do is worry
about this part
and now all I have is
some back to college Z minus are times that I want to find a solution to that
when your system
I can do that with the single back substitution because
are as upper triangular so in essence what we have to do that the computer QR
decomposition of Q
compute you transpose why and then just saw this linear system are better
equipped to tamp
comes with why and the flop count comes out to be
to terms and terms P squared on rough them in order and is much bigger than
equal to P
pay let's skip this step on sensitively cuz we're sensitive time this is that
most important point when were you the two less keep versus a QR
this is something that you would face in practice I'm
and I just want to give you the rough guideline for this if you look back at
the computational cost
the QR decomposition took about two times
and there's peace Chris top part of the problem
system and the hosp took about 10 times piece crib flops
go to the park comes to check that such a less key is cheaper
you may think you met always want to use it it also uses less memory
because all we're doing for storing a product up to I'm
well actually only we're storing one
I'll lower band at lower triangular matrix right because this is L
and chill us ke is L times of transfers all we have to do a store that
first both cheaper and and you uses less memory what's the caveat
the county as it is actually a lot more sensitive sensitive to numerical
instability
so X transpose X is nearly singular
or even touched poorly conditioned chill S key is a lot
has a lot more potential for kinda numerical air
the QRL soak you are as a lot more stable the trade of their
you wanna spend twice as much computation and twice as much memory to
get a more accurate solution
you want to use a QR if you want to be faster use less memory
you don't care as much about ACA solution and you want to use the chill
ASCII
okay that's essentially that
trade-off to keep in mind and
I'm like I said we're gonna revisit these
fundamental units comp case we talk about second method because we'll see
the aeration
complex the number competitions for asians really different we saw for
first-order methods
but you wanna think about all this stuff on the conduct the factorization is
because they often make
your life a lot easier when solving linear systems okay
um I was it there some advanced topics here in case you're interested
on these not the only method that you can use to solve linear systems
I put on a bunch of advanced topics and I mean put down some
indirect methods which are popular topic now on current research
so you don't have to worry about the stuff I'm necessarily
from to the homework but I think it's it's interesting so go ahead and go
through
both for some the advanced topics if you're interested okay
I will see you on Monday make sure I'm returning your
milestone by therefore the project