hello my name is Magnus Peterson this is
my tutorial number 22 on tensorflow
and this is about image captioning I had
not planned on doing this tutorial but
when I did tutorial number 21 on machine
translation I realized that image
captioning was basically just machine
translation and transfer learning for
images so this tutorial combines
tutorial 21 with tutorials number 8 and
10 on transfer learning and you should
see those other tutorials before you see
this one because I won't go into all of
the details again so remember from the
previous tutorial on machine translation
between human languages that we made a
recurrent neural network using what was
called an encoder and a decoder and the
encoder would take the input text and
process it to generate what we call the
thought later which was an array of
floating-point numbers and it would then
input that as the initial state for the
recurrent units in the decoder so that
they could generate the output text in
the destination language the idea in
image captioning is quite similar but we
have replaced the encoder part of the
network with a pre trained image model
and here we will use the vgg 16 model
that we have also used in previous
tutorials so for example if we have this
input image here we process that through
the convolutional neural network and
normally we would take the output from
the last layer and this would be a
vector with 1000 elements and we would
take the index of the highest element
and that would be the output so for
example here it might output the number
of 340 and that would be the class for a
bear for example so what we do in
transfer learning is that we ignore the
last output layer which has been trained
for the image net data set and instead
we output the vector from one of the
previous layers so if we take the next
to last layer here which is a dense
layer on a fully connected layer then it
outputs a vector with 4096 elements and
we want to use that as the initial state
in our decoder so that the decoder knows
what are the contents of the image and
then it can transform that into a text
however in this tutorial we will use an
internal state size for the recurrent
units of Phi
112 elements so we have to map a victim
with 4096 elements down to only having
512 elements and we do that by adding an
intermediate dense or fully connected
layer and it's important that it has the
ten age activation function so that it
limits the output between minus 101
otherwise I couldn't get it to work so
what we have now is a vector with 512
elements that summarizes what the image
contains now the decoder is actually run
in two different ways and I am Not sure
I made that entirely clear in the
previous tutorial on machine translation
so let me try and make it clear now we
run it differently depending on whether
we are doing training or inference
during training we give the decoder
three inputs we give it the initial
state which we have just calculated from
their pre trained image model and this
summarizes the contents of the image
then we give it a sequence of integer
tokens which have been converted using a
tokenizer so that this start Marga
becomes the integer to the word big
becomes the integer 165 the word brown
becomes the number 110 the word bear
becomes 102 and so forth so we give the
decoder this entire sequence here and
then the last input is the same sequence
but shifted one time step so the idea is
that we want the decoder to learn that
the start Naga should map to the word
big and the word big should map to the
word brown and the word brown should map
to the word bear and so on until we get
down to the int Mogga and then we are
finished and the way this works
internally intensive flow is that
tensorflow first inputs the first
integer to the embedding layer the
embedding layer then converts that
integer into a victim with 128
floating-point numbers roughly between
minus 1 and 1 because the neural network
cannot work on integer numbers directly
then we take that vector and input that
to the first recurrent unit and combined
with the initial state which was taken
from the image model it will then update
the internal state of the recurrent unit
and it will output a vector and what we
show here in this diagram is that we
have unrolled the process
through the time steps so the arrow that
you see here really means that the
internal state is sent to the next time
step so when we get to this point here
we have the number 165 we will put that
through the embedding layer and get some
vector of 128 floating point numbers out
we will input that to the gated
recurrent unit then we will take the
updated internal state we will process
that we will output a new vector and
update the internal state again and then
we move to the next time step and we
continue this until we have processed
the entire input sequence of integer
tokens what we get out of this is a
sequence of vectors of 512 elements each
so that is output here and used as the
input to the second recurrent layer and
then we do the same processing using
these inputs but we still use the same
initial state taking from the image
model over here that summarizes the
contents of the image and then we do the
same and we get a sequence of vectors
with 512 elements out and we process
that in the same way in the third
recurrent layer and then we get a
sequence of vectors out again but these
have 512 elements and we somehow need to
transform that into an integer between 1
and 10,000 if we have a vocabulary of
10,000 words so we do that using a dense
on a fully connected layer and then we
continue doing that for each of the time
steps until we have a sequence as output
when we are training this it doesn't
give us the correct output to begin with
at first it just gives us random output
so we want to Train it by giving it the
correct sequence of integer tokens as
the output so that we compare the
predicted output of the decoder with the
true output that we wanted to produce
then we backwards propagate the errors
all the way through the recurrent units
and also this dense layer here but in
this tutorial we won't be updating the
weights of this model over here you can
try that as an exercise and see if you
can improve the results by doing that as
well and then we continue this process
for millions of images until finally the
decoder produces output texts that are
close to what we want so that it has
learned to map from pixels over here to
text over here
when we're doing inference we do this a
little differently once again we start
by processing the image through the
image model and we get the initial state
out but now we don't know what the text
is so we cannot input any words up here
what we can input is the start Mogga so
we use the integer token 2 which
represents the start mugga and we input
that along with the initial state that
summarizes the image contents and then
we hopefully get the word big out then
we can input the word big again up here
and run that through the recurrent
neural network again and hopefully we
will get the word brown out now when
you're running this on a GPU for example
every time you want to run this through
the recurrent neural network you have to
pack up the data and send it to the GPU
and then you get the result back and
there are different ways of doing that
we could only process one integer token
at a time but then we will have to also
get their initial states of the
recurrent units out and then we will
have to feed them back in for each
iteration for each token or each word we
want to process this makes the code a
lot more complicated and I'm not sure
it's faster than what we're doing here
because it's Python so it's an
interpreted language and it has to
transport data back and forth to the GPU
and so forth so the cleanest way of
programming this is that we actually
built this entire sequence of integer
tokens and we sent the entire sequence
through the recurrent Network in each
iteration so we do it for the start
mugga and then we get the number 165 out
then we add that to the sequence and
then we input the same initial state and
this sequence with two numbers now and
we process that whole sequence but we
only use the last number here 110 then
we add that to the sequence we process
that entire sequence again with the
initial state again and we get this
entire sequence out but we only take the
last number and so forth again this is
very wasteful but it's cleaner to
program and it's better for our tutorial
if you're running something that needs
to process thousands of images every
minute then you will program this more
efficiently okay so let's get to the
implementation and as usual we have some
import
and I actually don't know why I say this
every time because of course we're gonna
have imports I think this is my start
Marga so that now you know that the code
is gonna start so now you're ready for
it okay let's go on so we have all of
these Karass imports here and then we
have to note the data set and I will use
the koko dataset and I have made a
little python module because I thought
that the code in the cocoa dataset was
really really bad and you know how I
feel about that
so I almost nearly hurled my PC into the
ocean while I was yelling profanities so
anyway I have made this module and
before you download the data set you can
just change the data directory if you
want and you should note that this data
set is very large the training data is
19 gigabytes and the validation data is
only about 800 megabytes if you cannot
download the whole training data you can
try and modify this code so it only
downloads the validation data and then
use that as the data set for training
but it's probably gonna over fit you may
still be able to learn how this works
even though the data set is small so you
call this function and it downloads and
it extracts the files if you don't have
them already and this data set contains
a lot of different data because you can
do a lot of different things with this
data set we only need file names for the
images and we need the captions for the
images so I made this little function
that retrieves that from the data set
the training set contains almost one
hundred and twenty thousand images so we
have this little helper function here
which can load and resize an image and
then we have a little helper function
here which can load an image from the
training of the validation set and then
plot it and show all the captions so we
have an example of that here and as you
can see it clearly shows at your ass and
if you look closely there's a second
year a found here and the images have at
least five captions each I think there
were a few of them that have more so we
have one here at giraffe eating food
from the top of the tree at giraffe
mother with its baby in the forest and I
can almost guarantee you that it's a
woman who wrote this okay I'm not gonna
go into a Rand again this time alright
but if you say what I just said today
people have become so
sensitive that I might be called a
sexist for this but it's just a fact
that women are far more interested in
like interpersonal relationships and you
can't actually even see if this is a
baby giraffe it could just be further
away so it looks smaller and it's behind
a hill as well but it's like family
relationships you know when I think
about distant family members I have to
go through my mind and sort of go
through mathematical calculations to
figure out exactly how I'm related with
people if you ask a woman she knows
instantly oh that's the relation oh
actually I can tell a little funny joke
because about 7% of my audience is
female and I actually wish it was a
hundred percent how great would that be
you know all the guys fuck off
unsubscribe get the fuck outta here I
only want women on this channel now
funny story when I started at university
as a computer science student many years
ago they had lumped together mathematics
computer science and physics for many of
the common classes and in the first week
we had this introduction week where the
older students would make a lot of jokes
with us and things like that so one of
the jokes they made was that they said
you know what there's only two girls out
of a hundred students on computer
science so this year we have decided
that some of your classes in computer
science will have to be in psychology
instead or something like that because
there they had the inverse where they
mostly only had women and they said this
with a straight face and I remember one
of the women getting really pissed off
and she said you know what I studied
computer science deliberately because I
knew I wouldn't have any competition I
wanted to get some action as well and it
gets even funnier because a year to
later she was running around school I
can't remember if she even had twins but
yeah so she definitely got what she came
for anyway let's continue so we load the
pre-trained image model vgg 16 and using
chaos this is super easy and this will
also automatically download the model if
you don't have it already now you can
download it with all of the layers
including all the classification layers
for the imagenet data set and
it is more than 500 megabytes download
all you can remove all of the fully
connected layers and then it is only 57
megabytes download now I will connect it
with the last fully connected layer
before the classifier so we need to
download the whole model but I don't
know because I haven't tried it I'm
running on solar power at the moment so
I am limited in how much I can actually
run this but you can try and connect it
to the last convolutional layer instead
and it might work it might actually even
work better and what I have done here
and if you try that then please write a
comment under the video so the rest of
us will know so anyway we can print a
summary of the model here all of the
layers in the model and the one that we
will use as the transfer layer is this
one here called fc2 and as you see here
it has a shape nun and 4096 which means
it outputs a batch of arbitrary size and
each of those is a vector with 4096
elements so we get the transfer layer of
the model out here and then we build a
new model in Karass which takes the
normal input for the image model and
then we take the output from this
transfer layer instead and then we can
use this to run data through the
pre-trained model and get the data out
from the transfer layer so we need to
know the size of the images that we are
inputting to the model and we get that
here and we also need to know the size
of the output which we get here now it
takes a long time to process an image in
the video d 16 model because it's so big
and we will have to process all the
images in the training set multiple
times because we have to train for 20
bucks so 20 times we will go over the
entire training set and what we're gonna
do is we're gonna make a cache file so
we process all the images in the pre
train image model we save the output the
transfer values to this cache file so we
can reload it very quickly and here we
have a little helper function for
printing the progress of this because it
takes quite a while and here we have a
helper function for doing this
processing and it just loads a batch of
images sends that batch through the vgg
16 model and then processes in the next
batch until we are done this is a little
wrapper function which saves the output
to a cache file and if that cache file
exists when we call this function then
we load the transfer values from the
cache file instead of
singing it all again and we have the
same function just for the validation
set instead and running these functions
take about 30 minutes on my GTX 970 GPU
so now we have the tokenizer which
converts text words into integer tokens
and you will have seen this in the last
few tutorials on natural language
processing and machine translation but
it's just slightly different here
because all of the captions are stored
in a list of lists so we just have to
process this with a nested for loop and
then for example for the first image in
the training set we have this list of
captions here where we have added the
start marker and the in-market to each
of the texts and you can see what they
look like without those muggers however
the tokenizer from cara's needs a flat
list of texts then we say that we only
want to use the top 10,000 words in the
data set and then we wrap the tokenizer
class from korra's with a little extra
functionality similar to what you saw in
the last two tutorials and then we can
process the entire data set of image
captions so that this text down here
that marker close up of Bin's our food
blah-blah-blah-blah-blah becomes this
sequence of integers here and this text
here becomes this sequence of integers
here and so forth the problem we have
now is that we have a data set that
consists of an umpire array of transfer
values that are output from their image
model and we have a list of lists with
integers so that for each image on each
victim with transfer values we have
actually a list of sequences with
integer tokens Kinross cannot directly
work on this because each of their
images as at least five captions and we
need to have one-to-one so what we could
do is we could throw away poor off the
captions for each image but this is very
valuable information and it has probably
cost quite a bit of money to produce it
and it is most likely gonna make our
model better that we have fly' captions
for each image rather than just one the
other thing we could do is that we could
replicate the transfer values for each
of the images five times one for each of
the caption so that we have numpy arrays
that match these are bad solutions
because either the throw away a lot of
valuable data or the waste a lot of
memory a better solution is to create a
custom data generator for cras so this
is the function that will generate
random batches of data during the
training first we need a helper function
that given a list of indices for images
it will get a random caption and more
specifically the sequence of integer
tokens for that random caption this
function is then used down here which is
a Python generator function that has an
infinite loop
I suppose the pronunciation is in finite
I don't know but anyway it repeats this
for eternity so that it will just create
random batches for as long as we want so
it starts by getting a list of random
indices of images in the training set
then it gets the transfer values for
those random images and then we use the
helper function above that we just saw
to get random captions or integer tokens
for those captions for those images then
we Pat those sequences of integers with
zeros so they all have the same length
as the longest sequence then we create
the input and output data for the Dakota
by time-shifting these sequences of
integer tokens then we wrap all of this
data in dictionaries where we name the
data so we are sure that it is assigned
correctly in our model and then we
return the data so if we go back and
look at the flowchart what we have done
now is that we have created a cache file
with the transfer values down here 4096
elements for all of the images in the
training set then our generator function
creates a random batch of data by taking
some of these transfer values and the
corresponding sequence of integer tokens
and then time shift in this sequence
here so it becomes like this sequence
down here and it creates a whole batch
of this data so that we can input that
to the decoder model and train it and
let's go back down to the code we will
be using a batch size of 1024 elements
because I want to keep my GPU fully
loaded all the time and we can then just
test if the batch generator works and we
create a new batch like this and these
are the transfer values for the first
image in the batch and this is the
sequence of integer tokens for that
image and it has just been
time-shifted one step down here so now
we can create the recurrent neural
network and we will use internal state
sizes of 512 elements and the embedding
layout will output vectors with 128
elements and we need to be able to input
transfer values to our neural network so
we have that input layer here and
remember that the transform values
contained 4096 elements but the internal
state size for the recurrent units are
only 512 elements so we need this dense
are fully connected layer to map those
transfer values to only 512 elements and
I found that I needed this 10h function
otherwise it wouldn't work so that the
output of this layer are values between
minus 1 and 1 then we need an input
layer for the sequences of integer
tokens and this none here means that the
sequences can have arbitrary length and
here we create the embedding layer and
here we create their recurrent layers
and you will note that the all return
sequences finally we have a dense layer
because the output of the last recurrent
layer is a sequence of vectors with 512
elements each and we want to convert
each of those vectors with 512 elements
into a vector with 10,000 elements so
that we can take the Arg max and get the
integer token for the word out now as in
the previous tutorial there is a bug in
Quran so we have to make a custom last
function because normally you would have
a softmax activation here but because of
our custom loss function we only use a
linear activation function here we are
using the functional API of Karass to
build this and this has actually not
connected all of the layers in the
neural network yet so we have this
helper function here that allows you to
connect different inputs for the
transfer values and then it connects all
of the layers in the neural network and
returns the output of the final layer so
we use that helper function to create a
model that we can train as I described
in more detail in the previous tutorial
there's a bug in Kira's so we cannot use
the built-in loss function and instead
we have to make this custom loss
function that uses this function from
tensorflow I described this in much more
detail in the previous tutorial so I'm
just gonna skip that here now we have to
compile the model before we can actually
train it and I can't remember if I
mentioned this in the previous tutorial
but in many of the tutorials I've used
the ADA optimizer but I found in my
experiments here when I was trying to
get all of this working
Edom doesn't work very well for
recurrent neural networks apparently it
would train okay for a while but then it
would start to diverge and just
completely break down I just changed it
with this optimizer here called IMS prop
and then it worked and as I described in
the previous tutorial also because of
this block days with the loss function
in Geass we need to make a separate
placeholder variable for feeding the
target data of the Dakota and then
finally we can compile it so it's ready
to Train well almost ready because we
want a few callback functions as well so
that we save a checkpoint during
training after each epoch and we also
save our attend support lock and if you
either want to continue training because
the computer broke down or whatever you
can reload there waits for the model
here or if you don't want to train it at
all and you just want to use a model
that you trained a week ago to generate
image captions for new images and
finally we can train it and each epoch
took seven minutes on my GTX 970 GPU and
I ran 20 bucks and you can try and run
more and see if the model gets any
better if we had not used pre computed
transfer values from the image model
then each epoch would probably take 40
minutes to run instead of just seven so
we have saved a lot of time by doing it
this way so now that the model has been
trained we can use it to generate
captions for images and we have this
helper function here first it loads and
resizes an image and because the image
model is working on batches of images so
we just expand the dimensionality and
then we can input that to the image
model so we get the transfer values out
remember this is a vector with 4096
elements then we allocate the sequence
of integer tokens that we will use to
input to the decoder model and then we
use the integer token that represents
the start mugga
therefore SS so that the coder knows
that now we are starting to generate a
caption and the caption is generated in
this loop here so as long as we haven't
seen the in token and we haven't
exceeded the max count of tokens that we
want to produce we wrap the data in a
dictionary so that we assign the data to
the correct inputs of the model and then
we
run that through the model here and get
the output and once again as I have
explained before we sent the entire
sequence of integer tokens into the
decoder model on each iteration and this
is very wasteful but the code would be a
lot more complicated if we had to get
the state of the recurrent units out
when we call predict and then in the
next iteration we would have to beat
those states back in so because we
already know what the model has output
in the last few time steps we are only
interested in the last output so what we
get out here is an array with 10,000
elements because it's 100 coded then we
take the odd max we get an integer and
then we use the tokenizer to map that
back to a text word and then we append
that to the output text and then we just
repeat this loop and when it's done we
plot the image and print the text that
we have generated so we can use an
example here and this is one of the
images that I have used in previous
tutorials and it's an image of a parrot
and the predicted caption is a small
bird perched on top of a tree branch
it's not a hundred percent correct but
it's good enough it shows that the
concept probably works and then you can
experiment and see if you can make a
much better model it also depends on
what the cocoa dataset has if the cocoa
dataset doesn't have any images of
parrots so let's try another image we
have used in some of the previous
tutorials this is Elon Musk and if you
run this through the inception model
back in tutorial 7 I think something
like that the model got confused and
said that this was either a sweatshirt
or cowboy boot but our image captioning
model actually gets it a lot better and
says it's a man in a suit and a bow tie
the bow tie is incorrect I can't see one
unless it's very dark but otherwise it's
actually pretty good it sees a man in a
suit so now let's try and use a model on
images from the training all the
validation set from the cocoa dataset so
we just have a helper function here
which can load an image from that data
set and run it through the image
captioning model so here we have the
same picture that we showed above and
the predicted caption is at you are
standing in a field next to trees this
is actually pretty good
and more importantly it is not equal to
any of the captions in the training set
that is pretty
cool so the image captioning model has
learned to map the transfer values from
the image model which is a 4096 element
vector and that vector somehow encodes
that there's a field and there's trees
and a forest and there's a giraffe and
maybe some other data and then the
recurrent Dakota model has learned to
map those features from the transfer
layer into the correct words and it has
learned to string these words together
in a grammatically correct sentence
that's pretty cool so here's the key
question is the image model and man or
woman now here's another example from
the training set so again this is an
image that the model has seen during
training so it has seen five correct
captions for this and what doesn't model
produce at birth is standing in the
grass near trees now these captions in
the data set were produced with Amazon
Mechanical Turk or something like that
so it's actually humans that have
written them and one of these is
actually wrong to see bruh seem to be
embracing in the wild these are not
zebras these like giraffes but anyway it
is easy to find examples in the training
and validation set that the model can
generate correct captions for and it's
easy to find examples that it cannot
generate correct captions for maybe it
just needs more training but maybe you
also have to improve the architecture
and a final example is this one a brown
bear is sitting in a grassy field and
here I experience that you know if you
ran it for one epoch more or one epoch
less it would generate something quite
differently but something interesting
here is that if you look at the human
generated captions the word bear occurs
in very different positions and also
none of the captions are identical to
the one that we have predicted I guess
the closest one is this one and
especially this part here but even so
our model has added the word is and it
doesn't use the word grassy area it uses
the words grassy field instead so the
data that we have given to train is
actually quite varied sometimes and even
so it still learns to extract meaning
from the data so the conclusion is that
this actually works reasonably well and
it's quite impressive but it's important
to understand that this is not a
human-like intelligence
all right it's just a very clever and
very advanced function approximator that
takes image pixels and maps them to
sequences of integers it doesn't know
what a giraffe is or what a bear is or
anything like that so as usual you can
download this tutorial by clicking on
the link below the video