Hello and welcome back.
So the previous video we showed you, the sequence, the sequence models and how it works, so we showed
you two applications, which is machine translation as well as building a chat.
But what we'll be doing in this course.
So now I want to go and discuss image capturing image captioning is the same exact thing.
However, right now, our inputs or our encoder, OK, does not encode sentence.
So our inputs to the encoder is not a sentence, it is an image.
So what we're doing is we're going to encode the image.
So as you can see here, this is our image and then we're going to use a pretty global network using
the image in its dataset.
And then finally this.
We have a fully connected layer at the end and then this fully connected layer is transformed to another
representation, OK, or a feature vector.
And then this representation or I can say the encoded information off the image is passed through the
decoder.
And this decoder is going to consist of multiple times that decodes this representation into a description.
So the image at the end is decoded into a sentence or description.
So in this case, this image is the giraffe standing next to each other.
OK, so we pass this encoded image to the CNN and then we pass the CNN, the feature vector to the LC.
I'm not now you can either do this, OK, and then this starts or you can either directly input DART
as well as the previous to the layer to predict JAROFF.
So it's the same thing.
OK, so we predict giraffes and then giraffes goes down here.
You're standing, standing down here and so on until the end you finish with the end token.
OK, so at the end you have giraffe standing next to each other.
I think.
I think it was something like on a grassy field or something like that.
But you get the idea.
So it describes the image, OK, or it decodes the representation that's that was inputted from the
encoder parts.
So if the encoder of the information and then it decodes it into a description or a sentence.
OK, so this is how it's done.
This is one of the applications of sequence to sequence, one of the third applications we're discussing
right now, which is actually I got this representation from this link over here.
OK, so you can visit this link for more details if you want.
And actually this link is Bitauto implementation.
So if you want to go ahead and try it out, then definitely you can.
OK, so this these are the results of the image captioning paper, which is show and tell in your captions
in return, it was it was produced by Google in 2014.
And this is the same architecture, actually.
They're doing the same thing.
So this is just another visualization of this algorithm or this paper here.
So as you can see here, these are the results of this.
The first column is descriptions without errors.
The second column is descriptions with minor errors.
The third column is descriptions that are somehow related to the image.
However, they're not correct.
And the final column is the descriptions that are completely unrelated to the image.
So as you can see, there's still some errors, and that is one of the fields of computer vision.
It's called image captioning.
And now, now in two thousand and eighteen.
And there's been a lot of improvements in this field.
So it's actually really interesting if you want to work with it.
So let's look at some of the examples.
This image here is a description without any error.
So a person riding a motorcycle on a dirt road so completely green.
The second image, you're two dots playing in the grass.
Now, there is a minor error because a stray dog is actually playing in the grass, not to dogs.
This image, you're a skateboarder does a trick on a ramp.
OK, and this is related to the image, OK, because he is doing a trick, but it's not a skateboarder.
He's riding a bicycle.
This is unrelated to the image.
So the dog is jumping to catch a Frisbee.
This, of course, is completely incorrect.
OK, so these are results of show until the neural image caption generator produced by Google in 2014.
And this is one of the other applications of sequence to sequence once.
OK, so I'm going to stop this video here.
Now, in the next video, we'll talk about attention mechanisms that allow you to focus on the inputs
when predicting the words that will visualise it.
And we'll see how it's done.
Thank you very much.
And I'll see you next time.