OK.
So welcome to 20 point to where we talk about all the different types of CNN's going from typical original
on CNN all the way to fast.
Our CNN and mask are CNN.
So let's see what this is about.
So what does our our CNN sign for it sends for regions actually not recurrent.
That's a different type of neural net and this are seen CNN's were first introduced relatively recently
by in 2014 by researchers at the University of College of Berkeley in California.
And basically they had dramatically improved performance on D-Pa. VRC challenge.
This is the equivalent of image for object detection detection testing.
OK.
The example of some of the images in there that isn't so our CNN's attempt to solve the exhaustive search
previously performed by sliding windows by proposing bounding boxes and passing these extracted bounding
boxes to the image ossify.
So how do we how do we find these bones in boxes like holiday proposed proposed by using the selective
search algorithm.
And this is like a simple illustration of what happens here.
So we have an image here.
We have to propose on the boxes and then we have we basically use these boxes perceptual CNN and try
to identify what is in that.
So let's talk about the selective search algorithm selective search attempts to segment image into groups
by combining similar areas such as colors textures and propose these regions as interesting bounding
boxes.
So we have this image of some sheep here.
And what's left of it is going to do is going to try to like merge different similar pixels similar
lighting conditions that sort of stuff.
So we can kind of get eventually get something like this and then it's going to draw a box of each segmented
region and proposed that.
Now it is a lot more tweaking and we can do selective siche.
So it all depends.
Basically we can get little boxes or a lot of boxes
so when selective search has identified these regions of boxes it passes this extracted image to our
CNN example one tree and an image in it.
OK that would be a very good sign into use.
We don't use the CNN directly for classification though although we can we use an SVM to first classify
the scene and extracted features.
Now I didn't mention that before.
But what happens is that instead of actually passing to CNN here using CNN to get to classes what it
does it just gives you to CNN features which are basically a feature maps.
All right.
And then we use an SVM to classified a scene and extract features.
This was probably done to speed because if I if I'm not mistaken R-S.C. an end was the original proposed
object action and it was meant for video.
So speed was a concern.
So after this region proposal has been classified we didn't use a simple linear regression to generate
a tighter danglin box that's holding boxes we're actually basically size and fit around objects and
image.
But how do we know what the good boxes though and that's where we can come up with I an IOU metric that
we discussed before.
Remember you basically had measured how good the overlap of the predicted box was over the over the
labels box.
Now here's a problem that happens in our CNN's own basically or object detection algorithms.
You DO WE they don't often just propose one box they propose many boxes over to see Image image sometimes.
So remember in our view point five was considered a good result.
Well what if we have multiple boxes and you all want all given over one with zero point five.
This is a common problem.
OK.
So in the figures below let's go to this example.
We have four boxes in red.
These are them right here.
The green is a true box a true positive and as such we have one true positive and true false positives.
OK.
So the true false positives here would be these boxes here one two and three.
OK.
So the reason it's true false positives is because we have one box here the dotted line right now this
is basically our best box so that will count as a true positive in our predictions.
All right.
So for now it is ignore the green box.
He doesn't count in this coalition.
We just have one box that's really good.
He's a true positive and treat up on good or false positives so mean average position is a very tricky
metric in my opinion mainly because it's not that intuitive to understand how the formulas fit here.
What I'm going to say though is I'm going to try to explain it to you.
Outgo introduce into amount of left I'm not for you in a slide so you can go over on your own and try
to make sense of it.
There's also a couple of blogs that have some very good but pretty lengthy explanations for it.
But let's go back to the sorry.
So what I'm going to say is that actually in this blog here is a very good explanation for it.
But essentially what mean average position tries to do is that it takes it knows all the boxes the architect
proposes and basically what it tries to do is try to come up with a metric that defines basically what
it is like.
See object that to predict this but object to be predicted maybe for false positives and maybe like
some other boxes that were probably irrelevant and stuff.
So and then we remember this is a for one a class.
Remember there are different classes as well.
So I mean average precision is a way to measure how effective all of my object detectors were were actually
well actually.
OK.
So just remember this is a metric of how we baseline how we measure performance of objective
So now let's move on to not the maximum suppression and this is a technique that object is used to remove
overlapping boxes and thus improve them up scores significantly.
So what this does basically it looks at a probabilities associated with each box that's being generated
and the probabilities it looks at all the probabilities of the object being in the same class effectively.
So if we have like two or three boxes that say this is a call but they have high overlap.
What it does now it looks at the highest probability box.
That's this one here in red.
And what it does it checks the eye or you would do with the other boxes.
So in this image here we see that we have tree boxes here that object texture the doctor has basically
highlighted and these are the probabilities of it being a class so what it does it checks you over these
here and what it does it will drop the boxes that it deems basically not as relevant as main box.
So that's effectively how we clean up these boxes.
You do object that is produce.
So now let's move on to fast.
Our CNN's this was this was announced or released in 2015 a year after the original on CNN came out
and basically what happened was that the problem with our CNNs was that it was effective but pretty
slow as each but each proposed bounding box has to be classified by a CNI by CNN and as such doing is
doing it in realtime was often impossible.
So it required tree models.
And it also required tree models to be trained separately.
So we had to have a feature extraction CNN and SVM to predict a class and a linear regression model
to tighten the Bongi boxes.
You remember all these things we discussed earlier.
So this definitely is a bit of a lot of moving parts in our CNN's.
So what did fasta our CNN's do first.
Our CNN's fiercly reduced number for postboxes by removing the overlap generated.
So how did they do this.
We run the CNN across the image just once instead of many times using a technique called that region
of interest pooling our boy pool.
OK.
So our away pool allows us to share the forward pass of the CNN for images for image across that sort
of subregions.
This works because previously regions are simply extracted from the CNN feature map and then puled which
deadfall you only need to render CNN once image and basically use that output of the CNN going forward.
So combining the training of the CNN classifier and the bonding box regressed into a single model that's
what it did.
So instead of so we have the SVM feature classifier and that now became a sort of actually on top of
the CNN and old linear regression that was taking the boxes basically that became a bounding box.
Apulia parallel to our soft Leo.
So basically what it became is this.
So we have a feature extractor.
CNN we have a soft Max layers at the end of CNN being our classifier for the object type or class.
And then we have this in Peridot a bounding box regressive.
So that is how it our CNN's fast our CNN solve a lot of the delays and sluggishness of the original
RCN and so now a year later in 2016.
Our CNN's media release I should say so faster.
Our CNN's made significant speed increases.
However a region proposal still remained relatively slow as it still relied on selective search algorithm.
Fortunately a Microsoft Research Team figured out how to eliminate this bottleneck.
So how did it speed up rigid proposal select to sit your lies and features extracted from the image
What if we just reused as features to do region proposal instead.
Okay so that was the inside that made the faster.
And is extremely efficient.
So you get to take a look at a diagram from the people here.
So basically this line here is what is important what if we use those features to do rigid proposal.
That is exactly why we don't have to keep running this over and over for the equivalent of a bit using
selective search to generate our proposals.
So how do we do rigid proposals with faster.
Our CNN's so fast our CNN's ad a fully convolutional network on top of the features of CNN to create
a region proposal network.
That's what we're seeing here.
So it is now.
Let's go back to it is now a full fully convolutional network on top of the features of CNN.
So let's think about that.
So we have a CNN producers here and is a fully convolutional at work here that does this region proposal
OK.
So the authors of the paper state the region proposal network slides a window over the features of the
CNN at each window location.
The netbook was a school and a bounty box put anchor hence 4000 box coordinates where kids number of
anchors.
That is basically how it works.
I encourage you to read to people if you want to get into more detail but first our CNN's Anyway back
to this after each pass of the sliding window.
It outputs key potential ballot boxes and a confidence of how good this box is expected to be.
That's pretty cool and pretty complicated as well.
So producing the bombing boxes now.
So previously we mentioned we produced key potential bounding boxes these bomb box proposals are proposals
of common expected boxes of suits and ships aspect ratio and sizes this aspect ratio is called Anca
boxes.
So what's happening here is that all boxes the boxes we propose in this method are basically predefined
and there would be 60 of suits and shapes ratios and sizes.
So it's a region that the proposal outputs a bony box put onco and a scale of how likely the image in
the box will be an object so let's move on to mask our CNN's which is pixel level segmentation now Mass.
Our CNN's aim to combine it.
Obs detection action and classification with segmentation as we previously saw segmentation as the labeling
of objects per pixel level as we can see in the above image.
So how do Musse our CNN's work.
They are basically an extension of fast.
Our CNN's web binary mask is created for the objects detected by the box.
So you remember how you actually created a mask that was basically provided segmentation outputs.
That's what's happening here when in fact our CNN's so binary mask basically bit 0 or 1 is created for
the object in the box.
So we know we're looking at a box.
And we are creating a binary mask for each box that's found.
And this is the architecture of the network and the link to the actual research paper where this publication
was released or made.
So messy CNN's you something called R Y O line.
So the mass of put uses the CNN extracted features to create its binary mask.
So how do the authors of this paper achieve this.
They use our oil line instead of a pool as a Pooles feature map was misalign from the regions of the
original image.
That was something I read in the paper I was confused to why.
But they explained it pretty well.
So mapping a region of interest onto a future map.
So imagine this was the original image here.
And we had a feature map that we generated from CNN that is smaller.
OK because it usually is smaller.
And as we use your reporting and what happens here is I know a 100 by 100 image is now mapped onto the
two by to the to future map.
Therefore a window of 20 by 20 that's just good old size window here on your original image is mapped
to a 6.4 pixel here.
This is how a direct linear mapping would be in the future map.
Our pool however wrong's down pixel maps to six by six.
That is why our pools feature but when we use our pool instead of like a line.
You had a misaligned boxes or regions on the image may have been that important to any looking at a
big image with small objects.
But if you think about it if it's a big box and you're going to do a proposal on being misaligned can
actually be bad.
So I wrote a line of these bio linear interpolation to know exactly what would be the pixel at 6.4.
So it's a pretty cool nifty algorithm that gives you exact mapping without having to roundel pixels.
So these are some examples of mask RCN and segmenting and classifying images is pretty cool to see this
in action.
Actually it's very accurate in some in some videos.