Hey, guys, and welcome back to Deep Learning in Python - Part One.
I made this lecture as an extra lecture a few months after starting this course because I
realized some people were taking this course and
didn't realize how applicable the stuff they learned is.
So, to begin, I know what a lot of you guys have as
your end goal is understanding
the latest and greatest neural network and deep learning architectures.
Convolutional neural networks and recurrent neural networks like LSTMs come to mind.
And you might ask 'well,
all I learned in this class was how to take derivatives'.
So, how does that teach me about convolutional neural networks and LSTMs?
Well, I'm glad you ask.
In the next few slides I'm going to talk about some newer and
very popular deep learning developments
and how the skills you learn in this class will help.
First, let's go all the way back to logistic regression.
Remember that there are two functions for
these supervised types of machine learning problems, prediction and training.
For a logistic regression the output is y equals nonlinearity of W times X.
So, that's how you do prediction.
For training what do we do?
You take the derivative of J,
the cause function with respect to W and move W in that direction.
So, W gets updated as Eta times the gradient of J.
Notice that J is the same for all these problems.
It's the cross entropy function for
classification and it's the squared error for regression.
Now let's look at neural networks which is what you'll learn about in this course.
How do we do prediction?
It's the same thing with just one extra step.
Actually, it just looks like two logistic regressions in a row.
How do we train?
Same thing as before.
Take the derivative and move in that direction.
Keep in mind that we can make our network arbitrarily deep so you could have W1,
W2, W3, and so on.
You'll see how we can incorporate any number of weights in this class.
So, what about convolutional neural networks?
Well, to do that you first need to know about
convolution which isn't really a machine learning thing,
it's more of a signal processing thing.
Nevertheless, I do go through some examples in my convolutional neural networks course.
So, how do you do prediction?
Well, that's just the same as before except instead of W2 times X,
we have W2 * X,
where the star means convolution.
As with our regular neural networks,
you can have any number of convolutional layers and
regular multiplication layers. How do we train this?
Exactly the same thing as before.
You take the derivative and you move in that direction.
You should be seeing a pattern now, hopefully.
What about recurrent neural networks?
Just like how we introduced one new thing with
convolutional neural networks we will introduce one new thing here which is time.
In particular, we'll split up the two calculations so you will
see that the first calculation depends on its last value.
Pretty simple concept.
And if you guessed that the way we train these is by taking
the derivative of the cost function and going in that direction,
you would be correct.
The last thing we'll look at is autoencoders.
'Auto' in this case means self.
So, this is just a neural network where the target
variable is the input variable instead of some other target variable.
This is an even easier concept than CNNs and RNNs
because you can do these using only the knowledge you learn in this course,
just instead of calling train XY,
you call train XX.
But just because these are really simple doesn't mean they aren't awesome.
One big advancement in deep learning was that we
learned that you could stack a bunch of autoencoders together,
put a logistic regression layer at the end and get
better predictions than just a plain deep neural network.
And you can do all that using just the knowledge you learn in this course.
So, what is the moral of this story?
It's that knowing how to take the derivative of
the cost function is absolutely critical to deep learning.
It's that the things you learn in this course
are fundamental to the things you'll learn about later.
Does that mean you'll be ready to write your own CNN or RNN today?
Probably not. But as you've seen all they
require is the addition of just one more simple step.
So by mastering the material in this course you're making
all these other things that might seem out of your reach today just a small step away.