trust us
ok that works ok good we'll get started soon so today we'll be talking about the
recurrent neural networks which is one of my favorite topics one of my favorite
models to play with input into neural networks just everywhere there a lot of
fun to play with in terms of administrative high temps recall that
your midterms on Wednesday this Wednesday you can tell that I'm really
excited I know if you guys are excited very excited to me what a cemetery will
be out due this Wednesday it's so he will be out on Wednesday its due two
weeks from now on Monday but I think since we're shifting it I think to
Wednesday we plan to have released it today but we're gonna be shipping it to
roughly Wednesday so we'll probably the first deadline for a few days and
assignment to him of mistaken was due on Friday so if you're using 38 days then
you'd be having it in today hopefully not too many of you are doing that our
people down with a 72 or many people are done okay most of you looking great for
doing well so currently in the class were talking about coming ashore neural
networks las Casas specifically we looked at to visualizing understanding
convolutional neural networks so we look at a whole bunch of pretty pictures and
video so we had a lot of fun trying to interpret exactly what he's accomplished
all networks are doing what they're learning how they're working and so on
and so we debug this through several ways that you may become a call from a
structure actually over the weekend I stumbled by some other visualizations
are new I found these on Twitter and they look really cool and I'm not sure
how to how people made these because there's not too much description to it
but looks like this is turtles tarantula and then this is chained and some kind
of a dog and so the way you do this I think it's something like the tree nuts
again optimization into images but they're using a different regularize on
the image in this case I think they're using a bilateral filter which is this
kind of a fancy filter so if you put that regularization on the image that my
impression is that these are the kinds of visualizations that you achieve
instead so that looks pretty cool but I am not sure exactly what's going on I
guess we'll find out soon
ok so today we're going to be talking about recurrent neural networks what's
nice about recurrent neural networks is that they offer a lot of flexibility in
how to wire up your network architectures
normally when you work with no let's hear the case on the very left here
where you are given a fixed size picture here in red then you process it with
some hidden layers and green and then produce a fix I saw the better in San
image comes in which is a fix I statements and we're producing a fixed
size picture which is the closest course when the recurrent neural networks we
can actually operate over sequences sequence at the input output or both at
the same time so for example in the case of image captioning and we'll see some
of it today you're given a fixed size image and then through a recurrent
neural network we're going to produce a sequence of words that describe the
content of that image so that's going to be a sentence that is the caption for
that and in the case of sentiment classification in the lobby for example
were consuming a number of words and sequins and they will try to class
driver the sentiment of that sentence is positive or negative in a case of
machine translation we can have a recurrent neural network that takes us a
number of words in say English and then asked to produce a number of words in
French translation so we'd feed this end Andrew recurrent neural network in what
we call sequence to sequence kind of setup and so this work or not work with
just performed translation on arbitrary sentences in English into French and in
the last case for example we have video classification where you might want to
imagine classifying every single frame of a video with some number of classes
but crucially don't want to the prediction to be only a function of the
current time step the current frame of the video but also all the things that
have come before it in the video as a recurrent neural networks allow you to
wire up in architecture where the prediction that every single time step
is a function of all the frames that have come in up to that point now even
if you don't have sequences that input or output you can still use recurrent
neural networks even in the case on the very left because you can process your
fix as inputs or outputs sequentially for example one of my favorite examples
of this is from people from deep mine for a while ago we're trying to
transcribe house numbers and instead of just having this big image feet into a
comment and try to classify exactly what house numbers are in there they came up
with a recurrent neural network policy where there's a small come that and it
steered around the image especially with recurrent neural network and so their
current work learned to basically readout house numbers from left to right
sequentially and so we have pics as input but we're processing it
sequentially conversely we can think about this is also a well-known people
draw this is a general model what you're seeing here are samples from the model
where it's coming up with these digits samples but crucially we're not just
predicting these digits at a single time but we have our current network and we
think of the up as a canvas and the kernel goes in and painted over time and
so you're giving yourself more chance to actually do some computation before you
actually produce you're out that it's more powerful kind of form of processing
data was a question over the specifics of exactly what this means for now
eros just indicated indicate functional dependence so things are so things are
harsh enough things before and we're going to exactly what that looks like in
a bit okay so these are generated house numbers so the network looked at a lot
of house numbers and came up with a way of painting them and so these are not in
a training day on these are made up numbers from the model none of these are
actually the training set these are made up
yeah they look quite real but they're actually made up from the local
so a recurrent neural network is basically this thing he remarks and
green and it has a state and it basically receives through time it
receives an actress so every single time that we can feed in an input vector into
the armed men and it has some state internally and then it can modify that
state as a function of what it what it receives every single time step and so
they're all of course be weights and CNN and so when we turn those wastes the
Arnold different behavior in terms of how its stated goals as it received
exempts I usually we can also be interested in producing and all but
based on the R&S state so we can produce these matters on top of the hour now but
so you'll see a show pictures like this but I just like to know that the Arnon
is really just the block in the middle
worked as a state and it can receive pictures over time and then we can be
some prediction on top of its state in some applications so completely the way
this will look like is the Army has some kind of a state which hereunder noting
as Victor H and this can be also a collection of doctors are just two more
general state and we're going to base it as a function of the previous hadn't
state administration time I T minus one and the current input vector 60 and this
is going to be done through a function which I'll call a recurrence function
and that function will have parameters W and so as we change those W us we're
going to see the Arnold different behaviors and then of course we want
some specific behavior are the Arnon we're going to be training those weights
under seal see examples of that song for now I'd like to note that the same
function is used at every single time step with a fixed function of weights w
and we played that single function at every single time stuff and that allows
us to use the external network on sequences of without having to commit to
the size of the sequence because we applied the exact same function at every
single time step no matter how long the input or output sequences are so in a
specific case of recurrent neural network of recurrent neural network the
simplest way you can set this up in the simplest recurrence you can use is what
other 42 as a bit alarming in this case the state of recurrent neural network is
just a single state h and then we have a cross formula that basically tells you
how you should update your hidden state age as a function of the previous head
of state
and the current input eckstein and in particular and the simplest case we're
going to have these weight matrices whaaa
WX age and they're going to basically project both in the hidden state from
the previous times that in the current input and then those are going to add
and then we squish them with at any age and that's how we update the state at
time t so this recurrence is telling you how a total change as a function of its
history and also the current input at this time that and then we can make
predictions we can base predictions on top of H for example using just another
matrix projection on top of the hill state so this is the simplest complete
case in which you can wire up in your life work so just give you example of
how this will work right now I'm just talked about sex age and why an abstract
forms in terms of actors we could actually end of these factors with
semantics and so one of the ways in which we can use a recurrent neural
network as in the case of character level language models and this is one of
my favorite ways of explaining our next because its intuitive and fun to look at
so in this case we have character level language models using our dance and the
way this will work as we will feed a sequence of characters into the
recurring role at work and at every single time step will ask the recurrent
neural network to predict the next character in the sequence will predict
an entire distribution for what it thinks should come next in the sequence
that has seen so far so I suppose that in this very simple example we have the
training sequence hello and so we have a vocabulary for characters that are in
the ATL and we're going to try to get a recurrent neural network to learn to
predict the next character in a sequence on this training data so the way this
will work as will set up will feed in every one of these characters one at a
time into a recurrent neural network you'll see it in a chat the first time
step and hear the x-axis is the time to time so we'll keep an H II L&L and
Hiromi coating characters using what we call it one hot representation where we
just turned on the bitter that response to that characters order and vocabulary
now we're going to use the recurrence formula that I have shown you wear it
every single time step suppose we start off with 80 and then we applied this
request to compute the hidden state electorate every single time step using
this fix recurrence formula so I suppose here we have only three percent in state
we're going to end up with a three-dimensional representation that
basically at any point in time
summarizes all the characters that have come until
and so we have to apply this requires that every single time step and now
we're going to predict that every single time step what should be the next
character in a sequence so for example since we had four characters in this in
this we're going to protect the phone numbers at every single time so for
example in a very first time that we've said in the letter H and the RNN with
its current setting of weights computer these are normalized lock problem he's
here for what it thinks should come next
so things that H is 110 likely to come next things that eat as 2.2 likely well
as negative three likely and OS 4.1 likely right now in terms of unless lott
probabilities of course we know that in this training sequence we know that we
should follow each so in fact this 2.2 which are shown in green is the correct
answer in this case and so we want that to be high and we will do all these
other numbers to be low as on every single time that we have basically a
target for what next character should come in the sequence and so we just want
all these numbers to be high and all the other numbers to be low and so that's of
course including in the included in the green signal loss function and then that
gets back propagated through these connections so another way to think
about it is that every single time step we basically have a soft max classifier
so every one of these a soft max classifier over the next character and
at every single point we know what the next character should be and so we just
get all those losses slowing down from the top and they will all flow through
this graph backwards to all the arrows were going to get gradients on all the
weight matrices and then we'll know how to shift the matrices so that the
correct problems are coming out of the Arnon so we'd be shaping those weights
so that the correct behavior the army have the correct behaviour you feeding
characters as you can imagine how we can turn this over to other questions about
the diagram
yeah Thank you so desperately lying as I mentioned a scene the recurrence the
same functions always so we have a single WX patient every time step we
have a single WHYY at every time step in the same whah applied at every time step
here so we've used WX awh why awhh four times in this diagram and in back
propagation when we get through YouTube account for that because we'll have all
these gradients adding up to the same weight matrix because it has been used
at multiple time steps and this is what allows us to process you know variably
sized inputs because every time that we're doing the same thing so not a
function of the absolute amount of things and your question what are common
things for initializing the first 80 I think US Senate 20 this quite quite
common in the beginning but does the order in which will receive the data
that matter
yes because so are you asking for these characters in a different order so if
you see if this was a longer sequence the order in this case in this case
always doesn't matter because it every single point in time if you think about
it functionally like this it's a factor at this time step as a function of
everything that has come before it right and so disorder just matters for as long
as you're reading it and we're going to go through a sieve through some specific
examples which i think will clarify some of these points to look at a specific
example in fact if you want to try to characterize the language model it's
quite short so I wrote a just then you can find a good home where this is
hundred line application in numpy for accuracy level are and then you can go
through an actual active steps through this with you so you can see concretely
how we could train a recurrent neural network impact this and so I'm going to
step through this so we're going to go through all the blocks in the beginning
as you'll see the only dependence here is we're loading in some text data so
our input here is just a large collection of a large sequence of
characters in this case a text input that txt file and then we get all the
characters in that file and we find all the unique characters in that file
create these mapping dictionaries that map from characteristic in the season
from indices two characters we basically order our characters so seeming bread in
a whole bunch of to file and a whole bunch of data and we have hundred
characters or something like that and ordered them in a in a sequence so we
associate indices to every character men here we're going to diminish license
first are hidden sizes hyper primary as you'll see with recurrent neural
networks so you aren't using it to be a hundred here we have a learning rate
sequence length here is up to twenty-five this is a parameter that
you'll be you'll become aware of what the problem is if our input data is way
too large say like millions of times the UPS there's no way you can put in
dhahran and on top of all of it because we need to maintain all of the stuff and
memory so that you can do back propagation in fact we won't be able to
keep all of it and two men in memory and a back rub through all of it so we'll go
in chunks through our input data in this case we're going through chunks of 25 at
a time so as you'll see in a bit
we have this entire dataset but will be going in chunks of 25 characters at a
time and every time we're just going to backup get through 25 characters on time
because we can't afford to do back propagation for longer because we have
to remember all that stuff and so we're going in chunks here of 25 and then we
have all these W matrices that here I'm analyzing randomly and some boxes so WX
HHH and HY and those are all of our hype all of our parameters that we're going
to train a backrub
I'm going to skip over the loss function here and I'm going to get to the bottom
of the script here we have a main loop and I'm going to go through some of this
may look now so there are some initialization here of various things 20
in the beginning and then we're looking for ever
we're doing here is a sampling a batch of data so here it is where I actually
take a batch of 25 characters out of this dataset so that's in the list
inputs and the list and puts basically just has 25 integers correspond to the
characters the targets as you'll see is just all the same characters but offset
by one because those are the indices that we're trying to predict it every
single time stuff so so important targets are just list of 25 characters
targets as offset by one into the future so that's what we sampled basically back
from data here we this is some sample code so it every single point in time
training this week and of course I try to generate some samples of what it's
currently thanks character should actually what these sequences look like
the way we use character low-level artists and test time is that we're
going to see that with some characters and then this aren't always gives us the
distribution of the next character in a sequence so you can imagine sampling
from it and then you feat in the next character getting a sample from the
distribution and keep doing it in to keep feeding all the samples into the
rnn and you can just generate arbitrary text data that's what this code will do
and it caused the sample function so we're going to that in a bit then here
I'm calling the loss function the loss function receives the inputs the targets
and it receives also this H prep H pressure is short for his state vector
from the previous trunk so we're going in batches of 25 and we are keeping
track of what is the latest a picture at the end of your 25 letters so that we
can when we meet in the next back we can see that in as the initial h at that
time so we're making sure that the hidden states are basically correctly
propagated from batch to batch through that but we're only back propagating
those 25 time steps so we fit into a function of the loss and gradients and
all the weight matrices and all the boxes and you're just printing the loss
and then here's a primer update we're told us older greetings and here we are
actually perform the update which you should recognize as an undergrad update
so I have all these cash to think of all these cashed
variables for the gradient squared which I'm accumulating and then perform the
autocratic date someone to go into the loss function and what that looks like
now the loss function is this block of code it really consists of forward and
backward method so we're comparing the forward pass and then the back of
passing Green so I'll go through those two steps forward pass you should
recognize basically we get those deficit targets we're waiting receive these 25
indices and we're not trading through them from 1 to 25 we create this text
input vector which is just zeros and then we set the one hot encoding so
whatever the index and impetus we turned it on for with one so we're feeding in
the character with that one hot encoding
here in computing the recurrence formula using this equation so hsi T
their ages and all these things to keep track of everything and every single
time stuff so we compute the state vector and the output using the
recurrence formula and these two lines and then over there I'm computing the
suspects function so normalizing this so that if we get probabilities and then
your loss is negative lock probability of the correct answer so that's just a
softness classifier lost over there so that's the purpose and we're going to
back propagate through the graph so in the backward pass we go backwards
through that sequence from 25
all the way back to one and maybe you'll recognize I don't know how much detail I
want to go in here but you'll recognize them back propagating through a soft max
propagating through the activation functions I'm not propagating through
all of it and I'm just adding up all the greetings and all the prime minister and
one thing to note here especially is that these ingredients and make weight
matrices like woahh I'm using a plus equals because it every single time step
all of these weight matrices getting gradient and we need to accumulate
fit into all the weight matrices because we keep using all these weight matrices
at the same at every time step and so we just backdrop into them over time and
that gives us the radiance and then we can use that loss function from the
primary and then here we have finally a sampling function so here is where we
try to actually get the artist to generate new text data based on what he
has seen an attorney and based on the statistics of the characters and how
they follow each other in the training data so we initialize with some rain and
character and then we go for until we get tired and we compute the recurrence
formula that the problem the distribution sample from the
distribution
encoded in one hot Kate 11 hot representation and then we fielded a
next time so we keep doing this until we actually get 200 texts so is there any
question over just like the rough layout of how this works
$25 South max classifiers at every batch and we back all of those at the same
time and all add up in the connections going backwards that's why do we use
regularization here you'll see that I probably do not I guess I skipped it
here but you can in general I think sometimes I tried regularization I don't
think it is common to use it and recurring nuts as outside sometimes it
gave me like worst results so sometimes I skip it it's kind of a fight promoter
yeah that's right yeah that's right so in the sequence of 25 shots here we are
very low level on character level and we don't actually care about words we don't
know that word exists as just character indices so miss arnelle in fact doesn't
know anything about characters so language or anything like that just in
the series and sequences appendices and that's what we're modeling using pieces
can be used space as the letters or something like that instead of just
constant batches of 25 I think he maybe could but then it kind of just you have
to make assumptions about language will see soon why you would want to do that
because you can plug anything into this and we'll see that we can have a lot of
fun with that ok now we can do we can take a whole bunch of texts we don't
care where it came from a sequence of characters and we feed into the Arnon
and we can train the iron and to create text like it and so for example you can
take all of William Shakespeare's works you can catch all of it is just a giant
sequence of characters and you put into the recurrent neural network and try to
predict the next character in a sequence for William Shakespeare proponents and
so when you do those of course in the beginning the recurrent neural network
has random random parameters so just producing a garbled at the very end so
it's just random characters but then when you train the Arnon will start to
understand that ok there are actually things like spaces there's words start
to experiment with quotes it and it basically learn some of the very short
words like here or on and so on and then as you train more and more disease
becomes more and more refined and the recurrent neural network learns that
when you open a quote you should close it later or that those sentences and
with a cup with a dot it learns all the stuff statistically just from the rock
patterns without actually having to head coach anything and in the end you can
sample entire
shakespeare based on this on a character level so just give an idea about what
kind of stuff comes out a lot I think he shall become approached and the gang
will strain would be attained into being never fed and his but the chain and
subject of his death I should not sleep
that's the kind of stuff that you would get out of this regard network you're
mean up a very subtle point which I'd like to get back to you in a bit okay so
we can run this on Shakespeare but we can run the Sun basically anything so
we're playing with this with Justin I think like roughly year ago and so
Justin Tuck he found this book on algebraic geometry and this is just a
large latex source file and we took that latex source file for this geometry and
finance the art and the artist can learn to basically generate mathematics so
this is a sample submitted this morning just spits out late check and then we
can pilot and of course doesn't work right away we had to tune it a tiny bit
but basically the Arnon after we tweaked some of the mistakes that has made you
can compile it and you can get the generate mathematics as you'll see that
it basically creates all these proofs it puts her stupid little squares at the
end of troops it creates let us and so on
sometimes we are going to create diagrams to varying amounts of success
and my best my favorite part about this is that on the top left the proof here
is emitted
the Sarno is just lazy but otherwise this stuff is quite indistinguishable I
would say from from actual geometry so let X 10 scheme of X ok I'm not sure
about that part but otherwise the gestalt of this looks very good
arbitrary things that it so I tried to find the hardest arbitrary thing that I
could throw the character level I decided that source code is actually
very difficult so I took all of Linux source which is just older like C code
you can copy it and you end up with I think some hundred megabytes and just
see code and header files and then just thrown into the Arnon and then it can
learn to generate code and so this is generated code from the Arnon and you
can see that basically creates function declarations it knows about inputs
syntactically it makes very few mistakes it knows about variables sort of how to
use them sometimes it intends to code it creates its own bogus comments
syntactically is very rare to find that it would open a bracket and not close it
and so on this actually is relatively easy for dornin to learn and so some of
the mistakes that makes actually is that for example it
declare some variables that it never ends up using or do the same variables
that it never declared and so some of these high level stuff is still missing
but otherwise it can do just fine
it also no hostile recite the Jeep the new GOP licensed character by character
that has learned from data and that knows that after the GPL license there
some include files there some macros and then there's some code so that's
basically what has learned that in turn into just a show is very small
just a toy thing to show you what's going on then there's a char and then
which is a more kind of implementation and torch which has just been charged
and scaled up and runs and GPU and so you can play with that yourself and so
this in particular was going to this by then the latter it's a three-layer Alice
team and so we'll see what that means it's a more complex kind of phone
network I just give an idea about how this works so there's a paper that we
played with a lot but this was just an last year and we're basically trying to
pretend that we're neuroscientists and we threw a hair salon on some test text
and so the Arden is reading this text in the snippet of code and we're looking at
a specific cell and his state of the art coloring the text based on whether or
not that sells excited or not
ok so you can see that many of the state
neurons are not interpretable to kind of fire on nothin kind of weird ways
because they have to do some of them have to do quite low level character
level stuff like how often does she come after age and stuff like that let's all
the cells are quite interpretable so for example we find ourselves like a quick
detection so that this cell just turns on when it is a quote and then it stays
on until the quote closets and so this quite reliably keeps track of this and
it just comes out from backpropagation the island of this size that the
character level statistics are different inside and outside of course and this is
a useful feature to learn and so it dedicate some of its head of state to
keeping track of whether or not you're inside a quote and this goes back to
your question which I want to point out here that this RNN was trained on I
think sequence length
hundred but if you measure the length of this quote is actually much more than a
hundred i think is like 250 and so we worked on we only back propagated up to
a hundred and so that's the only place where the cell can actually like Lauren
itself because it wouldn't be able to spot the appendices there much longer
than that but I think basically this seems to show that you can train this
character level detection sell as a useful on sequences less than a hundred
and then it generalizes properly to longer sequences so this so this cell
seems to work for more than a hundred steps even if it was only trained even
if it was only able to spot the dependencies on less than a hundred this
is another dataset here this is I think Leo Tolstoy's War and Peace this is in
this dataset there's a new line character at every single at roughly 80
characters in 80 characters roughly there's a new line and there's a there's
a line link tracking so that we found where it starts off at like one and then
it slowly case over time and you might imagine that a cell like this is
actually very useful in predicting that you like character at the end because
this arnie's to count a tee time steps so that it knows when a new line
character is likely to come next
ok so there's like tracking tell us we found cells that actually respond only a
sudden statements we found cells that only respondents cite quotes and strings
we found cells that I get more excited the deeper you nestin expression and so
all kinds of interesting cells that you can actually find inside these are not
completely come out just from the back propagation and so that's quite magical
I suppose but
this Alice team I think they're about 2,100 cell so you just gonna go through
them and some of them look like this but I would say roughly 5 percent of them
you spot something interesting so you just go through it manually
sorry so we are completely running the entire are in an intact but we're only
looking at a single hidden state fire at the firing of US one single cell in
dhahran so running the are normally but we're just kind of a recording from one
cell and the hidden state that makes sense so this sell just the entire are
among those rising one part of the hidden state basically there's many
other hidden still hittin cells that involved in different ways and they're
all believing in different times and they're all doing different things
inside the Arnon hidden state
but you can get similar results with one layer
these cells were always between negative one in 110 each and this is from
analysis team which we haven't covered yet but the firing of the salsa between
a one and one so that's the scale that's us this picture so are as are pretty
cool and you can actually trendy sequence models over time about roughly
one year ago several people have come to realize that you can actually use the
same very neat application in the context of computer vision to perform
image capturing in this context for taking a single imagine we'd like to
describe it with a sequence of warrants and these are nuns are very good at
understanding how sequences develop over time so in this particular model them
going to describe this actually work from roughly year-ago happens to be my
paper I have I have pictures from my paper so I'm going to use those so we
are feeding a commission and omission to accomplish on your network and then
you'll see that this phone models actually just made up of two modules
there's the comment that is doing the processing of the image and their
current debt which will be very which is very good with modeling sequences as so
if you remember my analogy from the very beginning of the course where this is
kinda like playing with Lego blocks we're going to take those two modules
and stick them together that corresponds to the arrow in between and so what
we're doing effectively here is where conditioning this RNN generative model
or not just telling its sample text at random but we're conditioning that
generate process by the upper to come ashore network and I'll show you exactly
how that looks like so suppose I'm going to show you what the forward pass on
your own that is so suppose we have a test image and we're trying to describe
it with a sequence of words so the way this model with process the images US
policy which take that any plugin to accomplish on your left work in this
case is a VG nett so we go through a whole bunch of comics pool and so on
until we arrived at the end
normally at the end we have this automatic classifier which is giving you
a profit distribution over say 1000 categories of images in this case we're
going to actually get rid of that classifier and instead we're going to
redirect representation at the top of the coalition member into the recurrent
neural network so we begin to generation of the Arnon with a special
art vector so the impetus are nonetheless I think 300 emotional and
this is a special three hundred emotional victory that we always plug
into the first iteration tells me that this is the beginning of the sequence
and then we're going to perform the recurrence formula that I shown you
before for recurrent neural network normally we computed this recurrence
which we've solidarity where we compute WSH time sex but whhhy and now we want
to additionally conditioned as recurrent neural network not only on the current
input and current in a state which we must like 20 so that term goes away at
the first time that but we initially condition just by adding wiht times be
and so this is the top of the comment here and we've added interaction and
added weight matrix W which tells us how this image information emerges into the
very first time since the recurring role at work now there are many ways to
actually play with this recurrence in many ways to actually plug in the image
into there are now and this is only one of them and one of the simpler ones
perhaps and at the very first time step here in this wine zero vector is the
distribution over the first word in a sequence so the way this works
you might imagine for example is you can see that these structures in the mass
hat can be recognized by the Coalition network as strong like stuff and then
through this interaction wiht my condition to hit in state to go into a
particular state where the probability of the word straw can be slightly higher
right so you might imagine that the strong like textures can influence the
probability of strong so one of the numbers inside 10 to be higher because
their structures and so the army from now on has to kind of jungle two tasks
it has to predict the next care and next word in the sequence in this case and it
has to remember the image information so we sent from that sock Max and
supposedly the most likely word that we sampled from that distribution was
indeed the word strong we will take strong and we would try to plug it into
the recording all that work on the bottom again and so in this case I think
we're using word level and beddings so the strong strong word is associate with
a three hundred national Dr we're going to learn to represent the three hundred
national representation for every single unique jewellery and we plug in those
three hundred numbers into the Arnon and forward again to get a description of
the second world and sequence inside why one so we get all these properties we
sample from it again
suppose that the word hat is likely now we take hats 400 much older presentation
and get the distribution of it there and then we sample again and we sample until
we sample a special and token which is really the period at the end of the
sentence and that tells us that the arnaz now done generating and at this
point the army would have described this image as a straw hat period ok so the
number of dimensions and his wife picture is a number of words in your
vocabulary +1 for the special and token and we are always feeding industry
sectors that correspond to different words and a special start talkin and
then we always just that propagates through the whole thing and a single
time to nationalize this at random or you can initialize your BG net with free
trade for a minute and then
distributions and then you encode the gradient and then you backed up through
the whole thing as a single model and just trained at all jointly and you get
a caption or image capture lots of questions ok but yes i three hundred
emotional embeddings they're just independent of the image so every word
has 300 numbers associated with it so we're going to bankrupt get into it so
you initialize it random and then you can back up to get into these better sex
right so those embeddings will shift around there just a parameter another
way to think about it is it's to having a one-hop representation for all the
words and then you have a giant W matrix where every single
multiplied W with that one hundred plantation and it w has 300 out but size
then it's effectively plucking out a single broke up w which and something
I'm betting it's kind of a cold front so just think of it if you don't like those
in bed and just think of it as a one-hopper presentation and you can
think of it that way yes the modelers to up at the end token yes in the training
data the correct sequence that we expect from the art is the first words I can
look forward and so every single training example sort of have a special
and token it go ahead
you can wired differently we plugged into every single state it turns out
that actually works worse so it actually works better if you just plug in the
very first time step and then the Arnon has to juggle these both tasks that has
to remember about the image what it needs to remember through the art and it
also has to produce all these outfits and somehow it wants to do that there's
some headway the reasons I can give you after class right that's true
a single instance will correspond to an image and a sequence of words and so we
would plug in those words here and we will talk in that image and we shall I
come so it's a train time you have all those weren't planning on the bottom of
the image London and then you unroll this graph and you have your losses in
the background and then you can do batches of images if you're careful and
so if your images they sometimes have different lengths sequences in the
training data have to be careful with that because you have to say that ok I'm
willing to process batches of up to twenty words maybe and then some of
those sentences will be shorter or longer a need to in your code you know
worry about that because some some some sentences are longer than others
we have way too many questions I have stuff to go
yes thank you so that propagate everything completely jointly and two in
training so you can pre train with the internet and then you put those words
there but then you just want to train everything jointly and that's a big
advantage actually because we can we can figure out what features to look for in
order to better describe the image that the end so when you train this in
practice we tried this on the census data sets one of the more common wants
is called Microsoft Coco so just to give you an idea of what it looks like it's
roughly 800,000 images and five sentence descriptions for each image these were
obtained using Amazon Mechanical Turk so you just ask people please give us a
sentence description for an image and your record and end up your data set and
so when you train this model the kinds of results that you can expect or
roughly what is kinda like this so this is our in describing these images so
this it says that this is a man in black shirt playing guitar or construction
worker in Orange City West working on the road or two young girls are playing
with Lego toy or boy is doing backflip on a wakeboard and of course that's not
a wakeboard but it's close there are also some very funny failure cases which
I also like to show this is a young boy holding a baseball bat
this is a cat sitting on a couch with the remote control that's a woman
holding a teddy bear in front of a mirror
I'm pretty sure that the texture here probably is what what happened made it
think that it's a teddy bear and the last one is a whore standing in the
middle of a street road so there's no horse obviously some not sure what
happened there so this is just a simplest kind of model that came out
last year there were many people who try to work on top of these kinds of models
and make them more complex I just like to give you an idea of 11 level that is
interesting just to get an idea how how people play with this basic architecture
so this is a paper from last year where if you noticed in the current model we
only feed into images single time to time at the beginning and one where you
can play with this is actually a rowdy recurrent neural network to look back to
the image and reference parts of the image Wireless describing to work does
the words such as you're generating every single word you allow the aren't
actually make a look up next to the image and look for different features of
what it might want to describe next and you can actually do this in the fully
trainable way so they are not only create these words but also the sides
where to look next in the image and so the way this works is not only does the
Arnon out but you're probably distribution for the next one sequence
but this coming that gives you does valium so saying this case we forwarded
the comment and got a 14 by 14 by 512 by 512 activation volume and at every
single time that we don't just admit that distribution but you also emit a
five hundred and twelve dimensional picture that is kinda like a look up key
of what you want to look for next to the image and so actually I don't think this
is what they did in in this particular paper but this is one way you can wire
something like this up and saw this picture is emitted from the Arnon just
like it's just predicted using some weights and then this picture can be dot
product and with all these 14 by 14 locations so we do all these dot product
and we achieved our we compute basically 14 by 14 compatibility now and then we
put a soft max on this so basically we normalize all this so that it's all you
get this what we call in the tension over the image so it's a 14 by 14
probably map over what's interesting for the Arnon right now in the image and
then we use this problem asked to do a weighted sum of these guys with this
saliency
and so this morning can basically a myth of what it thinks is currently
interesting for it and it goes back and you end up doing a weighted sum of
different kinds of features that the Ellis team wants to look at this point
in time and so for example the island's generating stuff and it might decide
that ok I'd like to look for something object like now admits a vector file
numbers of objects like stuff it interacts with cum that's when the
comment a commission and maybe some of the object like regions of that coming
out of that activation falling like light up and ceiling see map in this
4514 irate and then you just end up focusing your attention on that part of
the image through this interaction and so you can basically just do lookups
into the image while you're describing the sentence and so this is something we
refer to as a soft detention and will actually going to this in a few lectures
so we're going to cover things like this where the army can actually haven't
selective attention over its imports as processing the input and that's so I
just want to bring it up roughly an hour just to give you a preview of what that
looks like okay now if we want to make our lives more complex one of the ways
we can do that is to stack them up in layers so this gives you you know more
deep stuff usually works better so the way we start this up one of the ways at
least you can stack recurrent neural networks and there's many ways but this
is just one of them that people use in practice as you can
straight up just plug harness into each other so the impetus for one Arnon is
the director of the State picture of the previous on and so this image we have
the time axis going horizontally and then going upwards we have different
ordinance and so in this particular image there are three separate recurrent
neural networks each with their own set of weights and these are colonel that
works I just feed into each other and so this is always train jointly there's no
train first won second term one that's all just a single competition growth of
a backdrop to get through this recurrence formula to top it
ivory britain it's likely to make it more general rule still we're still
doing the exact same thing is we didn't have the same formula we're taking a
lecture from below and below in depth and effective from before in time we're
cutting them and putting them supporting them through this w transformation and a
smashing the 10 each
so if you remember if you are slightly confused about this there's there was a
WRX H times X plus whah times H you can rewrite this is a concatenation of exxon
H multiplied by a single matrix right so it's as if I stick tacks nation to a
single column vector and then I have this w matrix where basically what ends
up happening is that your WX ages the first part of this matrix and WH
nation's second part of your matrix and so this kind of formula can be written
into formula where you stack with your inputs and you have a single W
transformation so the same formula so that's how we can stop these are
announced and then there now indexed by both time and by later at which they
occur now one way we can also make these more complex is not shared by stacking
them but actually using a slightly better recurrence formula so right now
so far we've seen as very simple recurrence formula for the return to
work in practice you will actually rarely ever use formula like this and
basic network is very rarely used instead you'll use what we call it an
LSD and our long short term memory so this is basically used in all the papers
now so this is the formula would be using also your project if you were to
use are currently works but I'd like you to notice at this point is everything is
exactly the same as with an arlen it's just that the recurrence formula has a
slightly more complex function ok we're still taking the picture from the low
and depth like your input from before in time the previous an estate were
contacting them putting them through aww transport but now we have this more
complexity and how we actually achieve the New Haven state at this point in
time so we're just being a slightly more complex and how to combine defector from
below and before to actually perform an update on heading state just a more
complex formula so we'll go into some details of exactly what motivates this
formula and why it might be a better idea to actually use in Austin
and it makes sense trust me we'll go through it just right now so if you
block 4 p.m. some online video or you go to Google Images you'll find diagrams
like this which is really not helping I think anyone the first time I saw him
being really scared like this one really scared he was really sure what's going
on I understand Ellis teams and I still don't know what these two diagrams are
so ok so I'm going to try to break down the list and it's kind of a tricky thing
to put into a diagram you really have to kind of step through it so lecture
format is perfect for no steam ok so here we have the US equations and I'm
going to focus on the first part here on the top where we take these two vectors
from below and from before so X and HHS our previous in a state an accident but
we met them through that transformation W and now if both Jackson href size and
so there's a number send them we're going to end up producing for any
numbers ok through this w matrix which was put forward by 21 so we have these
four and dimensional vectors I F oMG they're short for input for get out but
and G I'm not sure what that's just you and so the ISI no go through signaled
gates and G go straight tenants gate now the way this way this actually works the
best way to think about it is one thing I forgot to mention actually in the
previous slide is normally require no network to says the single HVAC tried
every single time stopped and asked him actually has two vectors that every
single time and thus we call see the cell state vector so that every single
time step we have both agency in peril and and see vector here as shown in
yellow so we basically have two vectors every single point in space here and
what they're doing is they're basically operating over this cell state so
depending on what's before you and below you and that is your context you end up
operating over the cell state with these
and Ong elements and new way to think about it is I'm going to go through a
lot of this way to think about this is I N O as just binary either 0 or 1 we want
them to be we want them to have an interpretation of the gate like to think
of it as heroes are ones we of course make them later signals because we want
this to be differentiable so that we can back propagate through everything but
just think of Ino as just binary things that were computing base in our context
and then what this from always doing here see you can see that based on what
these gates are and what Diaz we're going to end up dating this see value
and in particular this episode to forget gate that will be used to shut a tus
reset some of the cells 20 solar cells are best thought of as shelters and
these counters basically we can either recent than 20 with us this interaction
this is an element of multiplication their laser pointer is running out of
battery so
interaction 0 then you can see that will zero out the cell so we can reset the
counter and then we can also add to a counter so we can add through this
interaction I times G and since I S between 11 and G is between negative one
in 10 basically adding a number between one and 12 every cell so that every
single time step we have these counters in all the cells we can reset these
countries 2012 forget Kate or we can choose to add a number between one and
12 every single cell oK so that's how we performed the cell update and then the
head an update ends up being a squashed cell so 10 HFC squashed cell that is
modulated by this update so only some of the cell state and up leaking into the
hidden state is modulated by this vector oh so we only choose to reveal some of
the cells into the hen state and learnable way there are several things
to to kind of highlight here one maybe most confusing part here is that we're
adding a number between one and one with I times D here but that's kind of
confusing because if we only had a G there instead then jeez already between
8 11 so why do we need I times G what does that actually giving us when all we
want is to implement a sea by
a number between one and one and so that's kind of my castle part about the
last time I think one answer is that if you think about the G it's a function of
its a linear function of your context no one has to laser printer by chance right
ok so G as a function of your Geo 310 age ok so G is a linear function of your
previous contacts squashed by 10 h and so if we were adding just jeans that if
I time she then that would be kind of like a very simple function so by adding
this I and then having a multiplicative interaction you're actually getting more
richer function that you can actually expressed in terms of what we're adding
torso state as a function of the previous tests and another way to think
about this is that would basically decoupling these two concepts of how
much do we want to add to a cell state which is G and then do we want to
address all state which is so I is likely we actually what this operation
to go through and genius what we want to by decoupling these two that also may be
dynamically has some nice properties in terms of how this all steam trains but
we just end up that's like the Austin formulas and I'm going to actually go
through this in more detail as well
ok so think about this as cell C flowing through and now the first interaction
here is the DOTC so efforts in a bit of a sigmoid of that and so economically
gating yourselves with multiplicative interaction so if f is zero you will
shut off the cell and reset the counter the cytology part is basically giving
you a comp is basically adding to the sole state and then the sub-state leaks
into the hill state but only through a 10 h and then that gets gated by so the
only electric and decide which parts with some state to actually reveal into
the hidden didn't sell and then you'll notice that this interstate not only
goes to the next iteration of the STM but it also actually closed up to a
higher layers because this is the head of state doctrine that we actually end
up looking into teams above us or them goes into a prediction
and so when you unroll this basically the way it looks like it's kind of like
this which now I have a confusing diagram of my own thats I guess we ended
up with but you get your input vectors from below you have your own state from
248
determine your gates fije know they're all in dimensional vectors and then the
end of modulating how you operate over the cell state and the cell state can
once you actually we set some countries and once you add numbers between one and
12 your country's the cell state leaks out some of it leaks out in a learnable
way and then it can either go up to the prediction or can go to the next
iteration of the US team going forward and so that's the so this looks ugly so
we're going to so the question is probably in your mind is why did we go
through all of this there something why does this look at this particular way I
should like to know that this point that there are many various to analyst at
this point but by the end of lecture people play a lot with these equations
and we've kind of converged on this as being like a reasonable thing but
there's many little tweaks you can make to this that actually doesn't
deteriorate the performance by a lot you can remove some of those gates like
maybe the implicate and so on
you can turns out that the stench of see that can be a sea and it works just fine
normally but with a tender age of seats were slightly better sometimes and I
don't think we have a very good reasons for why and CSI end up with a bit of a
monster but I think it actually kinda makes sense in terms of Justice counters
that can be reset to zero or you can add small numbers between one and 12 them
and so it's kind of like a nice actually relatively simple now to understand
exactly why this is much better than our own and we have to go to a slightly
different picture to draw the distinction so the recurrent neural
network that has some state vector right and you're operating over it and you're
completely transforming into through this recurrence formula and so you end
up changing your state vector from time to time stuff you'll notice that the US
team instead has the cell States flowing through and what we're doing effectively
as we're looking at the cells and some of it leaks into the head of state to
state for deciding how to operate over the cell and if you forget gains then we
end up basically just tweaking the cell by
active interaction here so so there's some stuff that looked at as a function
of the cell state and then whatever it is we end up changing the soul state
instead of just transforming it right away so it's an additive instead of a
transformative
interaction or something like that now this is actually remind you of something
that we've already covered in the class with this in mind you that's right yeah
so in fact like this is basically the same thing as be solid resonates so
normally with a calm that we're transforming representation resident has
these skip connections here and you'll see that basically residents have this
additive interaction so we have this X here now we do some computation based on
sex and then we have an additive interaction with acts and so that's the
basic block of residents and that's in fact what happens with an awesome as
well we have these interactions we're here and the ex is your cell and we go
off with you some function and then we choose to add to this cell state but the
LSD and unlike residents have also please forget dates that were adding
these forget case-control choose to shut off some parts of the signal as well but
otherwise it looks very much like a president so I think it's kind of
interesting that were converging on very similar kind of looking architecture
that works both income that's end in recurrent neural networks where it seems
like dynamically somehow it's much nicer to actually have these additive
interactions that allow you to actually that propagate much more effectively so
to that point
think about the the back propagation dynamics between our analysis team
especially in the US team is very clear that if I inject some gradients and
sometimes that's here so if I inject radiance and let the end of this diagram
then these plus interactions are just like ingredients superhighway here right
like these videos will just flow through all the tabs addition interactions right
because edition distributed equally so if I plugging gradient any point in time
here just going to blow all the way back and then of course the gradient also
flows through these acts and they end up contributing their ingredients into the
reading to flow but you'll never end up with what we refer to with our intense
problem called vanishing regions where these gradients just died off go to zero
as you back propagate through and I'll show an example
completely off why this happens in a bit sonar now we have this banishing
gradient problem I'll show you why that happens analyst am because of this
superhighway of just editions these gradients of every single time step that
we inject into the US team from above
just flow through the cells and your ratings don't end up finishing at this
point maybe I take some questions are there questions about what's confusing
here but the last time and then after that I'm going to why arnaz have been in
Greensboro
yes 000 vector is that important
turns out that I think that one specifically is not super important so
there's a paper I'm going to show you what else to answer Space Odyssey they
really played with this take stuff out but stuff in there there's also like
these people connections you can you can add so this cell state here that can be
actually put in with the hidden state better as an input so people really play
with this architecture and they tried lots of iterations of exactly these
equations and what you end up with us almost everything works about equal some
of it we're slightly were sometimes so it's very kind of confusing in this in
this way to show your paper where they took they treated the DS update
equations has just been built trees over the update equations and then they did
this like random mutation stuff and they tried all kinds of different grass and
updates you can have and most of them work about some of them break some of
them work about the same but nothing like really does much better than
analyst team and the questions are going to why recurrent neural networks have
terrible backward flow video also
showing the vanishing gradients problem in recurrent neural networks with
respect to all stems so we're showing here as we're looking at a recurrent
neural network over many periods many time steps and then injecting gradient
and say it's a hundred and twenty eighth time step and we're bankrupting
ingredients through the network and we're looking at what is the gradient
for I think the input type hidden matrix one of the weight matrices at every
single time step so remember that to actually get the full update through the
back we actually adding all those gradients here and so what's what's
what's being shown here is that as a backdrop we only inject ingredient at
120th time steps we do backdrop back through time and the strong the slices
of that propagation what you're seeing is that the US team gives you lots of
gradients throughout this backpropagation so there's lots of
information that is flowing through this art and just instantly dies off that
just greedy and we say banishes just just becomes tiny numbers there's no
gradient so in this case I think indication about a time steps are so
like 10 times steps as all the information that we injected did not
flow through the network and you can't learn very long dependencies because all
the correlation structure has been just died down there so we'll see why this
happens dynamically in a bit there some comments your channel so funny he's like
YouTube or something
ok
ok so let's look at very simple example here we have a recurrent neural network
that I'm going to unfold for you in this recurrent neural network I'm not showing
any inputs we're only have his state updates so wait whaaa church and state
hidden to hit an interaction and I'm going to basically forward a recurrent
neural network does it not for some tea time steps here I'm using T-fifty so
what I'm doing is WHAS time the previous tenant and stuff and then on top of that
so this is just a forward pass for ignoring any input vectors coming in is
just WHAS times H threshold WHAS time sage threshold and so on
that's the forward pass and then backward pass here where i'm directing a
random gradient here at the last time step so in the 50th time step by
injecting gradient which is random and then go backwards and I backed up so
when you back up through this right you have to back up through here I'm using a
rather get the backdrop through a wh multiply than 400 W H multiply and so on
and so the thing to note here is so here I am doing developers brownback
propagate through the relevant just holding anything that where the imports
were less than zero and Here I am dropping the WH times each operation
where we actually multiplied by WH matrix before we do the nonlinearity so
there's something very funky going on when you actually look at what happens
to these DHS which is the gradient on the NHS as you go backwards through time
it has a very kind of funny structure that is very worrying as you look at
like how this gets chained up in the loop like what we're doing here with
these two time steps
zeros yes I think and sometimes that's maybe the outputs the rebels were all
dead and seemed you may have killed it but that's not really the issue of the
more worrying issue is well that would be a show of all but I think one wearing
issue that people can easily spot as well as you'll see that we're
multiplying by this whah matrix over and over and over again because in the
forward pass we multiply by awhh at every single iteration
back propagates through all the hidden states we end up back propagating this
formula wh ich konnte chess and a backrub turns out to actually be that
you take your greeting signaling multiplied by whah matrix and so we end
up the gradient gets multiplied by whah hold it then multiplied by WH official
did so we end up multiplying by does matrix WH age fifty times and so the
issue with this is that the green signal basically two things can happen like if
you think about working with scalar value supposedly scale is not matrices
if I take a number that's random and then I have a second number and I keep
multiplying the first number by the second number so again and again and
again what does that sequence go to their cases right to play with the same
number either I die or just goes to sleep yet if your second number exactly
one year so that the only case where you don't actually explode but otherwise
really bad things are happening either die or we explode and here we have major
cities we don't have a single number but in fact it's the same thing happens a
generalization of it happens in the spectral radius of the WHS major axis
which is the largest eigenvalue of that matrix is greater than one then does
radio signal will explode if it's lower than one degree in civil completely died
and so basically since dr Tan has this very weird because of this recurrence
formula we end up at this very just terrible dynamics and it's very unstable
and just dies or explodes and so in practice the way this was handled was
you can control the exploding gradients one simple hockey as if your greetings
exploding you click it
so people actually do this and practices like a very patchy solution but if
you're reading Does above five min Norman Lin clampett 25 element twice or
something like that so you can do that is degrading clipping that's how you
address the exploding grading problem and then you're you're recording don't
explode anymore but the Greens can still vanish in a carnival at work and Ellis
team is very good with the vanishing gradient problem because of these
highways of cells that are only change with additive interactions with the
gradient just blow they never die down if you are if you because you're
multiplying by the same age or something like that that's roughly why these are
just better dynamically so we always teams and we do do gradient clipping
usually so because the gradients in Dallas team can potentially explode
still made they don't usually vanish
recurrent neural networks as well for Ellis teams it's not clear where you
would plunge in its not clear in this equation like exactly how you would plug
into relative and where maybe instead of the May from G much since then attend
huug here but then resells would only grow in a single direction right so
maybe then you can't actually end up making it smaller so that's not a great
idea I suppose you know so there is basically there's no clear way to plug
in a row here so yeah one thing I notice that in terms of these superhighways
gradients this this viewpoint actually breaks down when you have four get gates
because when you have four get Kate's where we can forget some of these acts
with the multiplicative interaction then whenever I forget gates turns on and it
kills the gradient then of course the backward flow will stop so these super
highways are only kind of true if you don't have any forget gates but if you
have a forget gave their then it can kill the gradient and so in practice
when we play with us teams are we use Austin's I suppose sometimes people when
they initially to forget get to the initializer with a positive bias because
that by
forget to to turn on to me always kind of turned off I suppose in the beginning
so in the beginning the green spoke very well and then the US team can learn how
to shut them off at once you later on so people play with that bias on that for
decades sometimes and so the last night here I wanted to mention that cost him
so many people have basically play with this quite a bit so there's a Space
Odyssey paper where they tried various changes to the architecture there's a
paper here that tries to do this search over huge number of potential changes to
the LST equations and they did a large search and they didn't find anything
that works substantially better than just analyst am so yeah and then there's
the GRU which also has a relatively actually popular and I would actually
recommend that you might want to use this drus a change in the Coliseum it
also has decided to interactions with nice about it is that it's a shorter
smaller formula and it only has a single a tractor doesn't have a Tennessee it
only has an H so implementation wise is just nicer to remember just a single had
a setback in your forward past two factors as just a smaller simpler thing
that seems to have most of the benefits of a nasty but so it's called GRU and it
almost always works about the coolest and in my experience and so you might
want to use it or you can use the last time they both kinda knew about the same
and so somebody is that harness are very nice but the rawr and does not actually
work very well
soyuz US teams are used instead what's nice about them is that weird having
these additive interactions that allow Greece to play much better and you don't
get a vanishing breed problem we still have to worry a bit about the exploding
feeding problems so it's common to see people clip these women sometimes and I
would say that better simpler architectures are really trying to
understand how come there's something deeper going on with the connection
between residents and Ellis teams and there's something deeper about these
interactions that I think we're not fully understanding yet exactly why that
works so well and which parts of it were cool and so I think we need to
understand both theoretical and empirical in the space and it's a very
wide open area of research and so so it's
sport 10 but the end of class where they can I i suppose to explode so it's not
as clear why they would but you keep injecting gradient into the cell state
and so maybe degrading can sometimes get larger
it's common to collect em but I think not as may be important maybe as an hour
and then I'm not a hundred percent sure about that point but urological basis I
have no idea what's interesting yeah I think we should end up here but I'm
happy to take your questions here