Youtube subtitles download by mo.dbxdb.com
neural networks are good for learning
lots of different types of patterns to
give an example of how this would work
imagine you had a for pixel camera so
not for megapixels but just for pixels
and it was only black and white and you
wanted to go around and take pictures of
things and determine automatically then
whether these pictures were of solid all
white or all dark image vertical line or
a diagonal line or a horizontal line
this is tricky because you can't do this
with simple rules about the brightness
of the pixels both of these are
horizontal lines but if you try to make
a rule about which pic pixel was bright
and which was dark you wouldn't be able
to do it so to do this with the neural
network you start by taking all of your
inputs in this case our four pixels and
you break them out into input neurons
and you assign a number to each of these
depending on the brightness or darkness
of the pixel plus one is all the way
white minus one is all the way black and
then gray is zero right in the middle so
these values once you haven't broken out
and listed like this on the input
neurons it's also called the input
vector or array it's just a list of
numbers that represents your inputs
right now it's a useful notion to think
about the receptive field of a neuron
all this means is what set of inputs
makes the value of this neuron as high
as it can possibly be for input neurons
this is pretty easy each one is
associated with just one pixel and when
that pixel is all the way quite the
value of that input neuron is as high as
it can go the black and white checkered
areas show pixels that an input neuron
doesn't care about if they're all the
way white or all the way black it still
doesn't affect the value of that input
on it all now to build a neural network
we create a neuron the first thing this
does is it adds up all of the values of
the input neurons so in this case we add
up all of those values we get a point 5
now to complicate things just a little
bit each of the connections are weighted
meaning there x a number that number can
be 1 or minus 1 or anything in between
so for instance if somebody has a weight
of minus 1 it's multiplied and you get
the negative of it and that's added it
if something has a weight of 0 then it's
effectively ignored so here's what those
weighted connections might look like
you'll notice that after the values of
the input neurons are weighted and avid
the values come that the final value is
completely different graphically is
convenient to represent these weights as
white Lynx being positive weights black
Lynx being negative weights and the
thickness of the line is roughly
proportional to the magnitude of the
weight then after you add the weighted
input neurons they get squashed and I'll
show you what that means you have a
sigmoid squashing function sigmoid just
means s Shh and what this does is you
put a value in let's say point 5 and you
run a vertical line up to your sigmoid
and then a horrible horrible line over
from where it crosses and then where
that hits the y-axis that's the output
of your function so in this case
slightly less than 25 it's pretty close
as your input number gets larger your
output number also gets larger but more
slowly and eventually no matter how big
the number you put in the answer is
always less than 1
similarly when you go negative the
answer is always greater than negative 1
so this ensures that that neurons value
never gets outside of the range of plus
1 to minus 1 which is helpful for
keeping the computations in the neural
network bounded and stable so after you
some the weighted values of neurons and
squash the result you get the output in
this case 0 point 7 46 that is a neuron
so we can call this we can collapse all
that down and this is a neuron that does
a weighted sum and squash the result and
now instead of just one of those assume
you have a whole bunch there are four
shown here but there could be four
hundred or four million now to keep our
picture clear we'll assume for now that
the weights are either plus one white
lines minus one black lines or zero
which case they're missing entirely but
in actuality all of these neurons that
we created our each attached to all of
the input neurons and they all have some
weight between minus one and plus one
when we create this first layer of our
neural network the receptive fields get
more complex for instance here each of
those end up combining two of our input
neurons and so the value the receptive
field the pixel values that make that
first layer neuron as large as it can
possibly be look now like pairs of
pixels either all white or a mixture of
white black depending on the weights so
for instance this neuron here is
attached to this input pixel which is
upper left and this input pixel just
lower left and both of those weights are
positive so it combines the two of those
and that's it's receptive field the
receptive field of this one plucked
receptive field of this one however if
we look at this neuron it combines our
this pixel upper right and this pixel
lower right it has a weight of minus one
for the lower right pixel so that means
it's most active when this pixel is
black so here is its receptive field now
be because we were careful of how we
created that first layer its values look
a lot like input values and we can turn
right around and create another layer on
top of it the exact same way with the
output of one layer being the input to
the next layer and we can repeat this
three times or seven times or seven
hundred times for additional layers each
time the receptive fields get even more
complex so you can see here using the
same logic now they cover all of the
pixels and more more special arrangement
of which are black and which are white
we can create another layer again all of
these neurons in one layer are connected
to all of the neurons in the previous
layer but we're assuming here that most
of those weights are zero and not shown
it's not generally the case so just to
mix things up we'll create a new layer
but if you notice our squashing function
isn't there anymore we have something
new called a rectified linear unit this
is another popular neuron type so you do
your weighted sum of all your inputs and
instead of squashing you do rectified
linear units you rectify it so if it is
negative you make the value zero if it's
positive you keep the value this is
obviously very easy to compute and it
turns out to
you have very nice stability properties
for neural networks as well in practice
so after we do this because some of our
weights are positive and some are
negative connecting to those rectified
linear units we get receptive fields and
they're opposites you look at the
patterns there and then finally when
we've created as many layers with as
many neurons as we want we create an
output layer here we have four outputs
that we're interested in is the image
solid vertical diagonal or horizontal so
to walk through an example here of how
this would work let's say we start with
this input image shown on the left dark
pixels on top white on the bottom as we
propagate that to our input layer this
is what those values would look like the
top pixels the bottom pixels as we move
that to our first layer we can see the
combination of a dark pixel in the light
pixel some together give us zero gray
whereas down here we have the
combination of a dark pixel plus a light
pixel with a negative weight so that
gets us a value of negative one bitter
which makes sense because if we look at
the receptive field here upper left
pixel white lower left hip so black it's
the exact opposite of the input that
we're getting and so we would expect its
value to be as low as possible minus one
as we move to the next layer we see the
same types of thing combining zeros to
get zeros combining a negative and a
negative with a negative weight which
makes a positive to get a 0 and here we
have combining two negatives to get a
negative so again you'll notice the
receptive field of this is exactly the
inverse of our input so it makes sense
its weight would be negative or its
value with the negative and we move to
the next layer all of these of course
zeros propagate forward here this is a
negative has a negative value and it
gets has a positive weight so it just
moves straight forward because we have a
rectified linear unit negative values
become zero so now it is zero again too
but this one gets rectified and becomes
positive negative times a negative is
positive and so when we finally get to
the output we can see they're all zero
except for this horizontal which is
positive and that's the answer our
neural network said this is an image of
a horizontal line now neural networks
usually aren't that good not that clean
so there's a notion of with an input
what is truth in this case the truth is
this has a zero for all of these values
but a one for horizontal it's not solid
it's not vertical it's not diagonal yes
it is horizontal an arbitrary neural
network will give answers that are not
exactly truth it might be off by a
little or a lot and then the error is
the magnitude of the difference between
the truth and the answer given and you
can add all these up to get the total
error for the neural network so the idea
the whole idea with learning and
training is to adjust the weights to
make the error as low as possible so the
way this is done is we put an image in
we calculate the error at the end then
we look for how to adjust those weights
higher or lower to either make that
error go up or down and we of course
adjust the weights in the way then make
the error go down now the problem with
doing this is each time we go back and
calculate the error we have to multiply
all of those weights by all of the
neuron
values at each layer and we have to do
that again and again once for each
weight this takes forever in computing
terms on computing scale and so it's not
a practical way to train a big neural
network you can imagine instead of just
rolling down to the bottom of a simple
valley we have a very high dimensional
Valley and we have to find our way down
and because there are so many dimensions
one for each of these weights that the
computation just becomes prohibitively
expensive luckily there was an insight
that lets us do this in a very
reasonable time and that's that if we're
careful about how we design our neural
network we can calculate the slope
directly the gradient we can figure out
the direction that we need to adjust the
weight without going all the way back
through our neural network and
recalculating so just review the slope
that we're talking about is when we make
a change in weight the error will change
a little bit and that relation of the
change in weight the change in error is
the slope mathematically there are
several ways to write this will favor
the one on the bottom it's technically
most correct we'll call it the ebw for
shorthand every time you see it just
think the change in error when i change
a weight or the change in the thing on
the top when i change the thing on the
bottom this is a does get into a little
bit of calculus we do take derivatives
how we calculate slope if it's new to
you I strongly recommend a good semester
of calculus just because the concepts
are so Universal and a lot of them have
very nice physical interpretations which
I found very appealing but don't worry
otherwise just gloss over this and pay
attention to the rest and you'll get a
general sense for how this works so in
this case if we change the weight x + 1
the error changes by
minus 2 which gives us a slope of minus
2 that tells us the direction that we
should adjust our weight and how much we
should adjust it to bring the error down
now to do this you have to know what
your error function is so assume we had
error function that was the square of
the weight and you can see that our
weight is right at minus 1 so the first
thing we do is we take the derivative
change in error divided by change in
weight the edw the derivative of weight
squared is 2 times the weight and so we
plug in our weight of minus 1 and we get
a slope the edw of minus 2 now the other
trick that lets us do this with deep
neural networks is chaining and to show
you how this works imagine a very simple
trivial neural network with just one
hidden layer one input layer one output
layer and one weight connecting each of
them so it's obvious to see that the
value Y is just the value x times the
weight connecting them w one so if we
change w one a little bit we just take
the derivative of Y with respect to w1
and we get X the slope is X if I change
w 1 by a little bit then why will change
by x times the size of that adjustment
similarly for the next step it can see
that e is just the value y times the
weight w 2 and so when we calculate dep
why it's just w 2 because this network
is so simple we can calculate from one
end to the other x times w 1 times w 2
is the error E and so if we want to
calculate how much will the error change
if I change w1 we just take the
derivative of that with respect to w1
and get x times w 2 so this illustrates
you can see here now that what we just
calculated is actually the product of
our first derivatives that we took
DW 1 times the derivative for the next
step the ED y multiplied together this
is chaining you can calculate the slope
of each tiny step and then multiply all
of those together to get the slope of
the full chain derivative of the full
chain so in a deeper neural network what
this would look like is if I want to
know how much the error will change if I
adjust a weight that's deep in the
network I just calculate the derivative
of each tiny little step all the way
back to the weight that I'm trying to
calculate and then multiply them all
together this computationally is many
many times cheaper than what we had to
do before of recalculating the error for
the whole neural network for every
weight now in the neural network that
we've created there are several types of
backpropagation we have to do there are
several operations we have to do for
each one of those we have to be able to
calculate the slope so for the first one
is just a weighted connection between
two neurons a and B so let's assume we
know the change in error with respect to
be we want to know the change in error
with respect to a to get there we need
to know dbd a so to get that we just
write the relationship between be a take
the derivative of B with respect to a
you get the weight W and now we know how
to make that step we know how to do that
little nugget of backpropagation another
element that we've seen is sums all of
our neurons sum up a lot of inputs to
take this back propagation step we do
the same thing we write our expression
and then we take the derivative of our
end points Z with respect to our step to
work propagating to a DVD a in this case
is just one we
make sense if we have a sum of a whole
bunch of elements we increase one of
those elements by one we expect the
solemn to increase by one that's the
definition of a slope of 1 one-to-one
relation there another element that we
have that we need to be able to back
propagate is the sigmoid function so
this one's a little bit more interesting
mathematically I'll just write it
shorthand like this the Sigma function
it is entirely feasible to go through
and take the derivative of this
analytically and calculate it it just so
happens that this function has a nice
property that to get its derivative you
just multiply it by 1 minus itself so
this is very straightforward to
calculate another element that we've
used is the rectified linear unit again
to figure out how to back propagate this
we just write out the relation b is
equal to a if a is positive otherwise
it's 0 and piecewise for each of those
we take the derivative so DB da is
either one if a is positive or 0 and so
with all of these little back
propagation steps and the ability to
chain them together we can calculate the
effect of adjusting any given weight on
the error for any given input and so to
train then we start with a fully
connected network we don't know what any
of these weights should be um and so we
assign them all random values we create
a completely arbitrary random neural
network we put in an input that we know
the answer to we know whether it's solid
vertical diagonal or horizontal so we
know what truth should be and so we can
calculate the error then we run it
through calculate the error and using
back propagation
go through and adjust all of those
weights a tiny bit in the right
direction and then we do that again with
another input and again with another
input for if we can get away with it
many thousands or even millions of times
and eventually all of those weights will
gravitate they'll roll down that many
dimensional Valley to a nice low spot in
the bottom where it performs really well
and does pretty close to truth on most
of the images if we're really lucky it
will look like what we started with with
intuitively understandable receptive
fields for those neurons and a
relatively sparse representation meaning
that most of the weights are small or
close to zero that doesn't always turn
out that way but what we aren't
guaranteed is it it'll find a pretty
good representation of you know the best
that it can do adjusting those weights
to get as close as possible to the right
answer for all of the inputs so what
we've covered is just a very basic
introduction to the principles behind
neural networks I haven't told you quite
enough to be able to go out and build
one of your own but if you're feeling
motivated to do so I highly encourage it
here are a few resources that you'll
find useful you'll want to go and learn
about biased neurons dropout is a useful
training tool there are several
resources available from Andre karpati
who is an expert in neural networks and
great at teaching about it also there's
a fantastic article called the black
magic of deep learning that just has a
bunch of practical from the trenches
tips on how to get them working well if
you found this useful i highly encourage
you to visit my blog and check out
several other how it works style posts
and des links for these slides you can
get as well to to use however you like
there's also link to them down in the
comments section thanks for listening Subtitles End: mo.dbxdb.com