High in this video we are going to talk about mathematical related stuff.
So if you do not interested in optimization techniques and the mathematics behind our networks feel
free to skip it basically in the end we are going to find a way to make these rather complex equations
into a very very simple ones.
But if you are interested in these kinds of optimization techniques Let's get started.
So the problem is that we usually use X as a factor to denote the training in boats.
So we have this X Factor x1 x2 up to X and and these are basically the features.
Or does it mean that for example for the end a logical relation we have to x ribose.
OK.
I'd noted that with X and Y.
But basically these are the input data the x and y.
Ok so it is a vector because it can have as many items as we want.
And we don't know the corresponding desired output by a y vector and it is the function of the X of
course.
So the output depends on the X inputs of course because if we have for example 0 0 then this is going
to be the output for 0 1 it is 0 as well.
But if we have a one on one then the output is going to be won.
So as you can see that debt outputs are not independent from the input varietals.
Of course it is not that staggering.
So for example of the logical operator training data is like this we have the X and we have the Y vectors.
OK.
And now roll that over as we have discussed several times.
We keep changing the edge weights so that the output from the older network approximates this function
because this is the output we know for certain.
And we train or now on a network that in the end we would like to end up with the same outputs as we
know for certain.
Of course this is the main theory behind super wise learning algorithms that we know that what should
be the result and if we keep training the algorithm until we get the same result as we should get.
So we kind of measure how we are achieving the goal of this approximation.
What is this approximation that basically we are going to get an output from the our network.
And we have an output from the training data.
What we know for certain.
And basically we approximate the output we know for certain we do happen of the now on network output.
So we can measure how we are achieving the goal of this approximation with a cost function and this
cost function can be constructed something like this.
Of course it is the function of the educators because if we change the educates Ford in our network
of course the output will change.
So one divided by two times and where the end is the size of the input data.
So how many training data we have for example for the and logical relation we have one two three four
training data and the why Prime is the output from the now run network.
So this is the y prime the F X is the output we know from the training data.
So this is the x x basically DS outputs 0 0 0 1.
This is what we know from certain from the training data.
OK so if we calculate something like this that what we should get and what we get from the note on that
were the difference between the two of them is something like the cost function or the mean square error
or we square it because we dont bother whether this is greater and this is smaller or twice.
Doesn't matter.
Usually we construct a cost function like this.
And you may guess that it has something to do with the end or term because this cost function is going
to be big.
So its going to have a higher value if the now run network is going to output the different values that
we should get if these f x function is approximately equal to the Y price.
What does it mean that the output from the now our network is approximately equal to what we should
get then the difference between them is going to be approximately zero.
If it's zero it means that the cost function has a very very low value.
So we can come to the conclusion that if we construct the cost function like this we just have to minimize
the cost run by changing the educates and where these cost function has a global minimum doughs educates
will be that good educates for an hour on that over.
If we use those educates for an hour on that work then it's going to make predictions.
So what's the intuition.
FC is huge.
It means there are huge errors in the sense that are now and that work doesn't produce the results we
know from the data set.
If these little means that the no run network is making good productions.
So we have defined the minima for the cost function.
And for example if we have to import varietals and hands to edge weights we'll end up with something
like this.
It is a three dimensional plot.
We have a dependent variable double you want and second dependent variable W2.
And basically this x axis is the error term.
This cost function.
And as you can see that we have to find the minimum.
So we have to minimize the cost function.
Of course its going to be the function of the weights in this case.
We have just to educate and we have to basically change the educates so that we end up paying the minimum.
What is it to mean that we are at a global minimum that we have given adequate W won and W to have this
cost function has a minimum value.
What is it mean if it has a minimum value it means that basically this term is approximately zero facts
is the output.
We know why Prime is the output from the now network.
It means that they are approximately the same.
This is what we are looking for.
We want to end up with an out of network that's capable of learning that when the input is 0 0 the output
is 0 0 1 0 1 0 0 1 1 1 and when it does is the training data.
What we know for certain.
These are the outputs that AF X basically and the y prime is going to be the result of the now on network.
And the best solution would be that they are exactly the same.
So this 0 0 0 0 1.
As the training data what we know for certain and the output of going out on that network should be
0 0 0 1 it should be exactly the same.
So this is why we have to construct the cost function.
We define it like this.
It is working fine as we have seen and we have to make sure that the arrow is a minimum.
So that's why we have to minimize the cost function and that's why usually we end up with something
like this.
OK if we have several educates that is going to be a hyperplane in higher dimensions and we are not
able to view straighted by the intubation is the same that we have to find the global minimum and the
educates at the global minimum will be the optimal actuates for the network and that work and how to
find the minima of a cost function.
I'm not sure whether you are familiar with partial derivatives.
Basically this is the gradient descent approach.
We have to know the partial derivative of the mean square error and go to the direction of the gradient
because the gradient is going to point to the direction of the global minimum.
We are standing here that the gradient is going to point into the direction of the actual global minimum.
So basically the gradient is the partial derivative.
It's going to be a vector by the way.
So we have to calculate the derivative of the mean square error or cost function with respect to double
you on danda derivative of the cost function with respect to W2.
And if we have an edge weights in the network of course we have to calculate the partial derivatives
with respect to every single adulate and basically we can construct a Waechter out of these partial
derivatives and thats going to be the Waechter pointing to the direction of the global minimum because
we should end up with the global minimum in order to make sure that all now what our network will produce
good results and the back propagation.
It is a gradient descent implementation thats about.
But we have to make sure that we match two requirements and basically be the happily back propagation.
We can simplify these equations and when we implement back propagation we dont have to calculate these
partial derivatives.
So where the cost function can be defined as the average of all the cost functions.
So we assign Of course function to every single training example.
And of course function can be transformed into a form that it will be a function of the output activations
if it can be done then we are able to use back propagation.
I'm not going to prove it.
Just take it for granted that we met these requirements.
So we are able to use back propagation.
For now we're on network training and it's quite staggering because you may guess that OK then there
must be a very very complicated back propagation algorithm.
How are we going to implement it in Java.
Do we have to use derivatives.
And so no we don't have to use any kinds of partial derivatives.
We have to apply numbers and we have to divide some numbers.
It's going to be very very easy in the end.
Thanks for watching.