In this lecture we're going to do a demonstration to show that Ellis teams really work for capturing
longterm dependencies.
I feel like this is something nobody ever really covers.
They always say that Ellis teams are good at learning long term dependencies as if it were just a fact
you're supposed to take for granted.
Instead I think a better approach is to try and set up an example to demonstrate that this is actually
true and that's what we're going to do.
This lecture is going to walk you through a prepared call lab notebook.
Although a very good exercise which I always recommend is once you know how this is done to try and
recreate it for yourself with as few references as possible.
As usual you can look at the title of the notebook to determine what notebook we are currently looking
at.
So let's run the imports and after the imports we're going to create our dataset.
This is the key part of this exercise.
Let me explain how this dataset will work and how we're going to make it into a long distance time series
problem.
The first thing is you want to be familiar with the XOR problem.
This is if you recall one of the examples in the tensor flow playground.
So no this general shape and know that it's a binary classification problem.
Basically it boils down to classifying which quadrant the data point is in where the quadrants alternate
between classes.
So no two quadrants which are beside each other belong to the same class.
Okay.
So I hope you understand this.
Now the time series itself is just going to be noise generated from the standard normal so you can see
right here.
I'm just generating random noise of size big ti so there's no temporal structure to the data per say
one data point is not correlated with the surrounding data points like how a normal time series usually
would be.
All right.
So what does this have to do with the X or this classification problem is going to look at the value
of three consecutive data points and track their side.
So it's not the absolute value that matters.
It's whether it's bigger than zero or less than zero.
Now let's say I just take the first three points x1 x to an X three what are the different signs they
may have.
Well this is just a counting problem.
There is plus plus plus plus plus minus plus minus plus plus minus minus minus plus plus minus plus
minus minus minus plus and minus minus minus.
All right.
So there are eight possible combinations.
Now of course we can organize these three points on a three dimensional axis.
Then we can have a three dimensional version of the XOR.
So there are eight quadrants and we will just make sure that any two adjacent quadrants do not belong
to the same class so just give me one second and I'm going to run a visualization.
So I made this visualization which is in the script x y 3D so that you can see the kind of pattern I'm
talking about.
So hopefully you see that this is really just the actual problem.
But in three dimensions so you can rotate it and see that there are eight different quadrants and no
two adjacent quadrants have the same color
and by the way it should be clear that a linear classifier like logistic regression would not be able
to separate these classes.
All right.
So now remember that these data points are coming from sequential data.
So here's how we are going to set up our test imagine just a series of random numbers what our classifier
is doing then is it's checking the sign of three consecutive random numbers is it minus minus minus
is it plus plus plus is it plus minus plus and so on.
Now here's how it comes into play for testing the different R9 units.
If I put these three consecutive random numbers at the end of the sequence the RNA in it doesn't have
to remember that long on the other hand if I put these three consecutive numbers at the beginning of
the sequence then the RNA and has to remember that sequence of three random numbers all the way to the
end of the sequence of course this problem will get harder and harder as I increase the length of the
sequence because that means the LSD M has to retain that memory for longer.
Okay.
So I hope that at a high level this concept makes sense at a high level.
We are making a time series signal and there is a pattern within this time series signal that leads
to the classification result.
If this pattern is at the end of the signal then it does not require the iron end to have a long term
memory.
If the pattern is at the beginning of the signal then the Arnon must successfully remember what it saw
all the way to the end of the signal.
And the longer the signal is the longer the RNA has to remember all right.
So this is the major thing you have to understand for this lecture this new dataset
from here on out.
All we are going to do is test different configurations of RNA ends and see how they perform
and so by the way here's the function that I used to get the label.
So when I want to do a short distance the three indices that I care about are minus one minus two minus
three which are the three last data points in the sequence.
And if I want to do a long term pattern then I choose the indices 0 1 and 2 which are of course the
first three data points in the sequence and then I have this function at get label which takes in the
sequence index 1 and next to an end index 3 and then returns the corresponding label so we're going
to start out with just a short distance problem and the first thing we're going to do is try to linear
model.
We know this is going to fail because the decision mandatory is non-linear.
But let's run it anyway.
OK so it does fail.
It only achieves about 50 percent accuracy which is what we expect so if we plot loss
it does go down but it doesn't go down to a low value if we plot the accuracy hovers around 50 percent
so next thing we're gonna do is use a simple or an end to solve this problem.
So let's run our simple earn in
now none of this code is new.
So we're just going to scroll down the results.
All right so let's check out these results accuracy looks pretty high in the loss looks pretty low
so if we look at a loss in the accuracy curves they look good as expected.
So as you can see the simple n can solve this problem since the part of the time series that affects
the classification result is at the end of the sequence.
It doesn't require any long term memory.
Thus the vanishing gradient problem is not a problem here.
Next we are going to recreate our dataset but this time we're going to make it a long distance problem.
In other words the pattern that affects the classification result is now found at the beginning of the
sequence rather than the end so you can see the only thing I've changed here is I've called the get
labeled function with the indices zero wanted to now we're going to stick with the sequence length ten
for the time being so let's set up our simple line and again this is of course all the same.
And now if you look at the loss per iteration in the accuracy per iteration it's clear that the simple
on n now has a harder time learning the pattern so this doesn't look very good at all although the accuracy
does still increase that to 85 percent
so the final accuracy is not as high as it was before.
And remember this is due to vanishing gradients the further back in the sequence you are the harder
it is to figure out how that affects the result.
Now note that if you run this several times you may find that the simple RNA fails to learn the pattern
at all.
Next we are going to test an LSD M on this same problem.
Remember that all we have to do is change simple Arnon to LSD M say so we just change the class name
so after we run this
we can see that the accuracy is very high and the loss is very low.
Is the loss per iteration in the accuracy per iteration All right.
So we can see that the LSD M achieves a much better accuracy and lost perturbation.
This makes sense because the LSD M was built to be able to learn long distance patterns all right.
So next we're going to make this problem even harder.
We're going to extend the sequence length to 20 so that's what we have here.
T equals 20.
And we do our simple and it again
let's look at a loss and let's look at the accuracy okay.
So this completely fails our final accuracy is still 50 percent.
Now let's try our LSD model again.
So it's the same code.
LSD M
loss per iteration and accuracy per iteration.
So it still gets up to almost 100 percent.
And as you may have expected this works our Elysium is able to pick up a pattern that appears 20 steps
behind whereas the simple Arnon cannot.
Although one thing that's apparent from these learning curves is that the problem is becoming increasingly
harder to solve with LSD.
Yeah it's not a steep decrease in the light like we saw before.
It takes a good number of iterations for the loss to start really going down and the accuracy to start
really going up.
So here's the loss again here's the accuracy and this tells us that there might be a point where even
the Atlas team fails and now just for the curious.
Let's also try a G are you on this problem.
Remember that this is very easy using the cross API.
We just replace the word LSD m with G R U.
So after we run this here's what we get.
So you'll notice that for all the other examples I ran them for two hundred epochs whereas this one
I ran for 400 bucks
and if we look at the learning curves we see that the U you doesn't really get to a good accuracy or
loss until after all 400 epochs
so the year you has a much harder time learning this pattern compared with the.
Yeah the last per iteration goes up and down a lot.
And he can't quite figure out the true pattern for quite some time
and here's the accuracy displaying the same pattern.
Now the next thing we're going to do in the script is make it even harder by extending the sequence
length to 30 since the simple RNA couldn't handle sequence length of 20.
We're not going to bother trying it again.
Instead let's just go straight to the Elysium.
So the results seem code as before just now.
Big T is equal to 30.
So we run this and what do we see.
Well it appears we've hit the limit of the LSD arms capabilities so the final accuracy was about 60
percent and only 50 percent on validation.
So here's the accuracy
so although the LACMA can learn long distance patterns to a point the relevant pattern can't be arbitrarily
far back.
The last example I want to do in this lecture is to use the same dataset with sequence length equal
to 30 and to continue using an LSD m.
But this time we're going to take a different approach.
Recall that in our Elysium calculation we have to calculate all of the hidden states for each timestamp
H1 H2 all the way up to h a big T.
The Atlas team by default only returns the last head and state each of Big T.
And that's considered the output of the list.
Yeah but as you've seen in the documentation the Atlas team has an option in its constructor called
The Return sequences.
If you set this eagle to true then you'll get all the hit and states for each timestamp.
So just to reason about the shapes the input is a sequence of size t by D.
If we set return sequences he could a false which is the default setting.
We get the final head and state a vector of size m when we pass this through the final dense layer.
We get a vector of size K where K goes 1 in our case.
But now if you set return sequences equal to true then we get all the hidden states.
So instead of just a single M length vector we get t of them.
That gives us a t by M array.
The question now is what do we do with this.
Well we think back to our old friend global Max pooling which we first encountered when we studied CNS
is now for images we have global Max pool 2D.
This takes in an image of shape h by W by C and collapses the first two dimensions so that you end up
with a vector of size C for sequences.
We have the analogous global Max pool one D.
This takes in a 2 dimensional array of size t by M and collapses only the first dimension so you end
up with a vector of size m.
So in other words global Max Tootie collapses two dimensions and global Max pull one B collapses one
dimension.
In other words it's just an element y's Max operation overall of H1 H2 all the way up to H T.
So ultimately we still end up with a vector of size m and we pass this through the final dense layer
to get an output of size K just like the previous LACMA network we discussed so after running this.
Here's what we get
okay.
Now we see something very cool.
We can see that even though our sequence is very long the Elysium is still able to learn this pattern
and in only one hundred epochs by the way which is much shorter than the previous examples.
So all the other examples I was using two hundred and for the G are you I use 400.
So using global Max pulling allowed us to handle longer sequences by looking over all the hit and states.
This is more powerful than the default setting where you have to remember everything you saw previously
up until the final hit and stay
all right.
So we also reach 100 percent accuracy pretty quickly 100 epochs which again is much better than before.
So global Max pooling is pretty great.
As an exercise there are many ways you could expand upon the experiments we performed in this script.
Here are a few ideas.
First consider the pattern we are looking for is always three consecutive values in the time series.
This probably makes it easier for the iron and to pick up the pattern.
But what if the relevant points are very far away from each other.
For example one at the beginning one at the middle and one at the end.
This is very much like how the English language is structured.
Sometimes you can't infer the meaning of a sentence until you can pass out the relationships between
words that appear very far apart.
Second consider what would happen if you added global Max pulling to the other types of iron in layers
like the simple aren't N and the G are you third you might want to try stacking several iron and units
together.
This isn't as popular as the very deep CNN we've seen but it's worth experimenting with to see what
the results would be
so I hope this lecture helps you understand the different capabilities of each of the RNA units.
The simple answer and unit is the first to fail at finding long distance patterns.
The geo U is capable of finding long distance patterns but is not as capable as the LSD.
Finally the LSD M is pretty good at finding long distance patterns but there is still a point where
even this will fail.
Luckily we can mitigate this weakness by using global Max pooling over all of the hidden states.
This makes it so that we aren't forced to carry all the important information into the final head and
stay all the way from the very beginning.
Instead we can just search along every hidden state we encountered and pick the most useful one.