Hello and welcome back.
So the previous video we showed you the sequence to sequence models and how it works.
So we showed you two applications which is machine translation as well as building a chad box.
Just what we'll be doing in this course.
So now I want to go and discuss image capturing image captioning is the same exact thing.
However right now our inputs or our encoder OK it does not encode a sentence.
So our inputs to the encoder is not a sentence.
It is an image.
So what we're doing is we're going to encode the image.
So as you can see here this is our image and then we're going to use a free drink on social network
using the image in its datasets.
And then finally this we have a fully connected layer at the end.
And then this fully connected layer is transformed to another representation.
OK.
Or a feature vector.
And then this representation or I can say that encoded information off the image is passed through the
decoder.
And this decoder is this good decoder consists of multiple illustrations that decodes this representation
into a description.
So the image at the end is the decoded into a sentence or description.
So in this case this image is the giraffe standing next to each other.
OK.
So we pass this encoded image to the CNN and then we pass the CNN the feature vector to the Elysee I'm
not now you can either do this.
OK.
And then this predicts starts or you can either directly input starts as well as the previously the
lyric to predict giraffes.
So it's the same thing.
OK.
So we predict for us and then giraffes goes down here desperate for Nick standing standing goes down
here.
And so on until the end you finish with the end token.
OK.
So at the end you have giraffe standing next to each other I think.
I think it was something like on a grassy field or something like that.
Yeah but you get the idea.
So it describes the image.
OK.
Or a decodes the representation.
That's that was inputs from the encoder parts which receives the encoder information and then it decodes
it into a description or a sentence OK.
So this is how it's done.
This is what one of the applications of sequence the sequence model of the third applications we're
discussing right now which is image caption.
OK.
I got the representation from this link over here.
OK.
So you can visit and visit this link for more details if you want.
And actually this link is a torch implementation.
So if you want to go ahead and try it out then definitely you can.
OK.
So this these are the results of the image captioning paper which is show and tell.
OK.
In your image caption generator it was it was produced by Google in 2014.
OK.
And this is the same architecture.
OK.
Actually they're they're doing the same thing.
So this is just another visualization of this algorithm or this paper here.
So as you can see here these are the results.
These are the first column is descriptions without errors.
The second column is descriptions with minor errors.
The third column is descriptions that are somehow related to the image.
However they're not correct.
And the final column is the descriptions that are completely unrelated to the image.
So as you can see there's still some errors.
And that is one of the fields of computer vision called Image captioning.
And now now in 2010 18 and there's been a lot of improvements in this field.
OK.
So it's actually really interesting if you want to work with it.
So let's look at some of the examples.
This image here is a description without any error.
So a person riding a motorcycle lighter crudes so completely great.
The second image here two dots playing in the grass.
Now this there is a minor error which because it's three dogs actually painted rust.
Not two dogs.
This imagery skateboarder does a trick on a ramp.
OK.
And this is related to the image.
OK.
Because he is doing a trick when it's not a skateboarder.
He's riding a bicycle.
This is unrelated to the image of a dog is jumping to catch a Frisbee.
This of course is completely incorrect.
OK.
So these are results of show and tell the neural image caption generator produced by Google in 2014.
And this is one of the other applications of sequence to sequence once.
OK.
So I'm going to stop this video here.
Now in the next video we'll talk about.
Attention mechanisms that allow you to focus on the inputs when predicting the words.
OK so we'll visualize it and we'll see how it's done.
Thank you very much and I'll see you next.