Hi, I'm Brok Bucholtz.
I work on the self-driving car team at Udacity.
Hi, I'm Aaron Brown.
I'm a content developer for the self-driving car Nanodegree.
And today, we're walking through the semantic segmentation project.
Yeah. That's right. We are going over the walk through,
and talking a little bit about it.
And I know you actually worked on it.
Very recently, actually.
Just last week, I was going through it.
Nice.
I can give you some insight of where I got stuck,
and where maybe some more detail would have been great.
And also, just the whole process itself.
So we already have it set up.
We've got everything downloaded.
We've got the data with the project.
Let's start actually looking at the repository and going over that.
Sure.
So if we go over to GitHub, we can actually see the repository here.
And first thing I mentioned is actually the necessities,
so the frameworks, and the packages that are required.
So the dependencies of Python 3, TensorFlow, NumPy,
and SciPy, along with the data set,
and actually starting it.
And we've already installed all the dependencies,
and we already downloaded the data set so we are ready to go.
So let's jump into the main file.
Okay.
So first off, all the import statements we need.
And then we actually have TensorFlow,
which requires version 1.0,
and that's because we are actually using
some parts of the library that we weren't using before.
So before, we were allowing other versions but now 1.0 is required.
Okay. So we have to make sure we have that newer version.
Yeah. And then the next part is actually,
we check to make sure we have a GPU.
So we want to make sure that you're connected to your GPU,
make sure that you're not running off CPU because if you were,
this project would take a very long time.
I know because I did run it off of a CPU on my laptop and it did take a very long time.
Do you know how long that took?
Maybe three hours.
Three hours to go through six ethics.
What did you do in those three hours?
I don't remember.
Okay. So the next function is actually the Load_vgg.
And this is actually where we take the VGG model and we load it.
And if we actually go to the classroom or we actually talk about this.
So if we go to Encoder.
This reminds me of the transfer learning that we're going over in term one.
We had this frozen model in all its layers and weights.
And we can just take advantage of that to learn a completely new topic.
Yes, it's pretty similar to that.
We're using the VGG,
which has already learned from Image Net,
and using that information to actually create a fully convolutional network.
Yes.
And so, that actually brings us in the encoder part because the VGG is
actually the encoder part which we are basing off of the paper here.
Yes.
And what we need to do is actually
load parts of the VGG into our neural network and use that.
Should we get start with that?
Okay, yes. What layers are we dealing with when we're doing this VGG16?
When I was going through it three really stood out to me.
Three?
Yes. It looks like there's two pulling layers and
then there was an input that was about to go into a fully connected layer.
So what we're trying to do is actually replicate the paper,
and if we go down here,
we can actually see the architecture.
What want to do is get the encoding part,
which is the first part here,
and then the decoding part which is the one we will be doing next.
The encoding part we want to take pool3,
pool4, and the pool5.
And those are actually part of the layers along with if we go back to
the main.py file, it is the input layer.
So we're also are asking for input layer so we can feed in the image.
And then we get out a representation of that image from VGG.
And we also keep probability there.
So all the weights are frozen but we still have
the ability to dynamically change the key probability.
That's correct.
How much data we want to just throw away.
Yes. Basically, forcing it to learn more than it has to.
And avoid overfeeding.
Correct. Yes. If you want,
where did you have problems here?
Yes. First of all,
I was just trying to figure out how to load up
these layers and have the names of all of them.
And I noticed that it gives the key takeaways as tf.save_model.loader.load.
So immediately, I just went to Google and I did a search on that,
and it took me right away to the TensorFlow official documentation.
OK. Let's do that right now.
Yes.
And like the parameters it was having is that tag which ends up
being that VGG 16 and a path.
But I noticed that there is some other details in there as well.
And this happened when I found a very specific example on using that.
So, I googled around and it ended up I needed to use the get default graph,
and also get tensor by name.
Right immediately after doing the tf.save_model.loader.load with our tag in our path,
we're just calling on this get default graph,
and then we're asking the graph get this tensor by name and it's basically,
we're using those name variables that you give us.
Okay. So, the first thing you did was get the graph, right?
Yes. Graph is equal to tf.get.
Tf dot.
Get default graph.
Default graph. Okay, cool. So, what do now that we have the graph?
Well, now we're actually getting these different layers
and that's what we're returning from this function.
VGG16.load.
Maybe you can call it W1 is equal to graph.get tensor by
name and then it's that same VGG input tensor name that we're seeing right above that.
So, you want graphed.get tensor by name?
Okay. What name should we use?
Just the name that we're seeing right above there which is
that VGG input tensor name is equal to image input zero.
Which layer should we get?
Let's go ahead and get the input layer.
Input layer, okay. So we're going to get this name?
And call by name. And then, say we want the next one.
Say we want keep probability,
how do we get that one?
Exactly the same process there.
Exactly the same. Man. Geez.
Yeah. Just rename the variable and used to keep probability name.
So like this?
Yes.
So we just do that for all the other layers?
Yes. And we just return those.
One thing that we want to do here is,
right now we have it commented out but it's used tf.save_model.loader.
We actually need to use that.
Basically, right after our tag oior VGG tag,
we would have a line that does this tf.saved_model.load and it's going to be the session.
Okay, so we also want to load the graph from the file is what you're saying.
Yes.
Okay. Let's do that.
What's the first thing I type?
Tf.save_model.loader.
Dot load model?
Dot loader.
Loader, okay.
Dot load. Yes, it's a not full.
And then the session.
Okay.
And then these brackets like syntax, VGG tag.
And then the VGG path which is actually given to us and
the parameters for this function instead of
an input that we have there well, along with the session.
I guess we're using two of those inputs.
All right, and this will load it up for us.
And then basically we just go down the lines
and fetch those other layers and then we just return.
Okay. So we've got the graph loaded from the file through VGG.
Yes.
And then we grab that graph in a variable and then for
that graph we grab each layer by its name. Cool.
All right.
Okay. So let's jump to the next one.
Yes, let's check it out.
I guess another way you could have discovered that syntax is by searching on GitHub.
There is a pretty handy search key on GitHub and you can check out other people's code.
And so, TensorFlow especially 1.0,
it's so new that sometimes it's hard to find the syntax for something specific.
So you can actually search examples of other people doing this stuff on GitHub?
Yes.
Cool.
That sounds really helpful.
Yes. Wink wink. Let's go in this one.
What is this function asking for?
The layers function? This is actually where we're creating the architecture.
This is the meat of the project I feel like.
This is where all the fun happens.
Yes.
ll right, go on.
So this is where are you making that fully connected convolutional neural network.
I see the couple inputs are actually the stuff we just did in load VGG.
Yes. A lot of these functions are set up where we run it
and then the output from one function becomes the input for another function.
Okay cool.
It's all streamlined.
What are some of the problems we had when you're trying to do this?
Basically, is just understanding what that architecture looks like exactly.
And what really helped me out is,
I was searching in the classroom and I found theconcept.
Oh yes? Which one?
It was in Scene understanding and it was called decoder.
And it turns out this references exactly the way that you want to
be using the skip layer architectures and upsampling.
Okay, so this one right here that I've got up.
Yes, exactly.
Basically, we just take our input which is
that final frozen layer that we call the VGG layer 7,
and we're upsampling it by two.
Okay.
But before we do that actually there's one by one convolution.
So even before this concept there is an encoder part,
that also goes over the one by one convolutions.
We have to do this to preserve spatial information [inaudible] .
Yes, instead of doing the fully connected layer.
Right. If we just did the fully connected layer we'd lose
all our spatial information and that wouldn't be good.
Is there any other information we're talking about or should we just jump right into it?
Yes, let's jump right into it and see what it looks like.
This concept talks about how to
do the skip layers and everything which is actually just adding it together.
I guess for this example maybe we'll just check out how to do the one by
one convolution of one layer, and upsampling.
We'll basically start with the first two or three layers.
Yes. Let's see what it'd look like if we did that layer 7.
Okay, so we want to create you said a one by one convolution?
Right. So we'll call it something.
Maybe just call it convolution one by one.
Okay, one by one.
And then that's equal to tf.layers.conv2d.
And let's say we're doing this on our VGG layer 7 out.
This is the layer that furthers into this frozen say VGG16.
Yes.
And then we have the number of classes.
Okay, number of classes.
And what does that represent?
In our case we're just doing binary classification.
Is this pixel wrote or is it not wrote?
Okay. Is that the number of features that we'll be getting?
Yes. You'll be creating this depth.
Okay. And then how about the kernel size?
What should we do for the kernel size?
Well, the kernel size is just one because it's a one by one convolution.
Padding is the same.
And then this part I actually found really helpful for doing the training,
and you need a regularizer.
If you do not,
your weights will become too large and it will be really prone
to overfitting and producing garbage.
Oh, actually you provided me with some of those image.
So let's look at those. So which-
Yeah, so that one I used a regularizer,
and that one I didn't,
and they're both trained on six epochs.
So this one we are looking at right now,
this is with the regularizer?
Okay.
Looks nice. You can see that the green is representing the road well.
And on the other one it's just all over the place.
Oh man, that does not look good.
Yeah, so it's overfilling.
So by doing regularization you go from this image to this image.
Right. You really want to penalize it,
the weights get too large.
And so just setting up this regularizer right here
within the layer is a really helpful way of doing it.
So how did you do regularization for this layer?
So we'll just say that
kernel_regularizer=tf.contrib.layers.l2_regularizer.
Is it l2 underscore regularizer?
Yes.
And then some value,
I'm using a pretty small value of 1 to the power of negative 3.
Get the smaller value,
less l penalize it, and there we go.
That's our 1x1 convolution and now we are
preserving that special information and we are ready to up-sample it.
We want to up-sample it to the original image size.
So what's the next layer we're doing?
It's a deconvolution.
It's a transpose convolution.
Okay, so we've got a 1x1 and now we're doing a deconvolution layer.
Um-hum. And so we will maybe just call it output.
Output=tf.layers that convolve2D transpose.
Oh, you want the transpose. Okay.
That's important.
And then we'll use that layer that we just created,
so they convolve 1x1.
And now we have the num classes.
Okay, number classes. So that's the two features?
Yes. And now we have the kernel size which is 4 and we have the stride number which is 2,
and that stride amount is actually going to be what's up sampling it by two.
And this is important,
is the padding, it needs to be the same.
Or if we don't do this,
we do valid, we'll get a different size than we want.
And we have to keep that since we want the same exact size as the output?
Yes.
Okay, cool.
It is really important that we get all the dimensions lined up,
other than that same, all the syntaxes in the classroom.
Now you did talk about a kernel regularizer for the 1x1.
Do we do that for the output layer that we've just created to as well?
Yes. Let's do a regularizer for every layer.
Every layer?
You must really like it.
Yeah.
Okay, regularizer. Is this the same?
Yeah, exactly. You could just copy and paste it from the last one.
And you can do a little bit of experimentation of course to
try to see what values work for this.
I am just using 1 to the power of negative 3 for all of them,
and I got pretty good results.
So you can experiment, see what works best.
Now we're going to actually up-sample again by two.
So basically exactly the same code,
but now our input's going to be the output.
I hope that's not too confusing.
So the next stuff is actually almost doing the exact same pattern as this, right?
Exactly.
But it's doing it for all of the VGG layers.
Yes.
So what we do here for doing those skip layers,
is we'd be adding them in as we're doing in this process of up-sampling.
Because these different layers are different dimensions and different sizes.
Some are bigger than others,
so we scale one up and now it is the same size as this skip layer,
so we can add them together now.
And then we take those
that we added together and up-sample it and now it's
the same size as this skip added together.
And so that is what would be happening in between kind of
doing these up-samplings and that's all in the classroom.
Oh, the concept we saw before?
Yeah, for the decoder.
It's actually showing you what it looks like to up-sample it and then add.
Right here under skip connections.
Yes, exactly.
The only thing I'd want to add here is to be sure to use the right padding,
so be careful with that,
and then also the regularizer. And then you are good to go.
Okay. Awesome. So that is it for layers then?
Yes, that is basically it.
So we up-sample it by 2 then by 2 again and then by 8.
So a lot of those layers were really close together, just separate by pulling,
which generally has that two times difference.
And then we return that final output which is the same size as are our image.
Okay. And where did you get the- when you decided to
upscale by 2 and upscale it by 2 again and then eventually upscale by 8,
where did you get those numbers from?
The classroom.
Classroom?
That's where all the answers are.
Yes, it actually uses that same thing.
So if you look at the strides we'll see that it's a 2 by 2 2, 2 2,
8 8 and also has the kernel size as well for what to use.
Another thing to look at might actually be the paper that this is based off of.
Oh, for sure.
So if we actually look at the paper,
you can see they've got their same architecture which
matches relatively the same as the one you created.
So you've got the skip layers here and
here and then you actually have up-sampling for each one.
Yeah.
Oh, nice. Okay.
So we have that 32 times already in there.
Let's jump to the optimizer.
This is the next function.
Were there any major problems you had?
Well I just checked up the next concept in the classroom and lo and behold,
it was called the Classification & Loss.
Oh, this concept right here.
Yes. So right away it tells you that you need to do this resize of the logits.
And that's because it was 4D and now we are doing it 2D.
So basically it's like a flat image where the height is
the number of classes and then its width is how many pixels.
Okay, I see. You're flattening the images.
Yeah. It was like 4D, it had like height and width,
and now it's just flat.
With the number of classes which is the prediction of?
Okay, cool.
And then that perfectly sets it up for doing Softmax.
And guess what, that's the next concept.
The cross entropy.
So we are doing a Softmax cross entropy with logits.
And so if we look at that,
what are the two inputs for that?
We have logits and we have labels.
So our labels also need to be matching up with the size,
so if that comes into our optimization function,
we're just going to reshape it the same way we're
doing with the logits and sure they're the same shape,
and that will set it up so it is able to do the softmax,
look at the probabilities for those two classes and then
have it so it's like picking the one that's the highest value.
And for that we need to do an Adam optimizer.
So after we kind of declare cross entropy loss,
we also need to do our optimizer.
So let's just go ahead and use,
I'd recommend using an Adam optimizer,
because it's fewer hyper parameters for us and it
has things built in like decaying learning rate,
and we can just feed it that cross entropy loss.
So the first line would actually be this line,
we'd get the logits from the flattened image?
Yeah.
Sp let's jump over to the next function.
Yeah, which is train_nn.
And so here we get a ton of different variables,
epochs, batch size, we have this function get_batches_fn,
which is going to be pretty helpful for us.
So what kind of problems would you have when doing this function?
Well, basically it just all comes down to understanding,
I think what this get_batches_fn is doing.
Because it's doing a lot of the heavy lifting for you.
We just maybe need to look into the helper code and see what it's returning,
but it is actually returning,
so based on batch size,
an image and a label pair,
you can just kind of feed into your session that is running.
So how would I go about and implementing that?
When you're doing this,
I was looking through the number of epochs.
And then would we get the number of the epochs?
I was just getting that later on.
So this is being fed into this function itself.
And then the next part,
I need to actually fetch an image in the label
and I was getting that from the get_batches_fn.
So you could say for parentheses image comma label and get_batches_fn batch size.
This will actually loop through and get all those pairs of images and labels.
Inside of that you are just kind of creating
maybe this feed dictionary and that has this template for the input image,
the correct label, even things like key probability which we are
seeing as an input argument to this function and the learning rate,
which is also an input.
So basically if we loop through epochs and loop through the batches,
we have all the information we need to do the training?
That's right. And then we can actually run it.
We can define loss=session.run and
then do this on our train optimizer and cross entropy loss.
The same functions we just implemented.
Yes.
Awesome.
So if we jump into the next one, this is the run function.
What exactly is the run function doing?
So this is where we're actually going to be defining things like
how many epochs we want to be working with.
It's going to be creating some templates for
doing learning rate and also the correct labels,
so that's something we'll need to create.
Learning rate is like a float value where correct labels is actually a 4D value,
it has batch, height,
width and then number classes.
I kind of instantiate it with like none, none,
none and then number of classes for that correct label, it's a tensor flow placeholder.
And then, I go ahead and I say number of epochs,
maybe I just want to do 6 and then batch size,
we're dealing with a lot of information.
So, maybe keep that small if you can.
So, let's start off with the first thing it's asking for,
it's asking for us to build a NN using load_vgg,
layers, and optimization function.
What should we do for this?
So we're actually just going to be calling the functions that we created above.
Okay. So, we'll call load_vgg,
and then we put in the session,
and then the VGG path.
Yeah.
What do we get out from it, the load VGG?
A couple of things. Well, quite a few.
Input image keep probability.
Input image, the keep probability.
We have layer3_out.
Layer3.
Layer4.
Um-hum, and then, finally layer7.
And then layer7_out.
Yes.
Just like that.
And now that we've all these layers, what do we do with these layers?
You put into the layers function.
Man, that sounds like a plan, let's do that.
So, we use the layers function to implemented,
and then what so we first pass into it?
So we're going to pass in layer3.
So, layer3.
Layer4.
Layer 7.
Layer7.
And then, number of classes.
The number of classes.
Yeah.
Okay. What do we get from these layers?
We get this layer output which is kind of like, this final layer.
So, the final layer of the neural network we just built.
Yeah.
Okay.
It's result.
And then,
we actually call our optimizer.
So, it's kind of like in the order that we created in.
Yeah. So we call the optimizer and then we use
that with the train function we implemented,
and then we use that to inference.
Yeah, that's right.
So after we do that,
so our optimizer gives us our logits or optimizer and cross entropy,
and then, just like you said,
we just go ahead and we do train_nn,
and we call that function,
and that will actually go over all the data,
and to do the number of epochs,
and batches, and then finally,
once it's done processing that we have our model,
and we can test it on our testing data and save it off.
So, I mean, that's just about it.
That's the whole project right there.
That is the whole project, yeah.
Oh, nice. So what do you want to talk about next?
Is there anything you saw during the project that you might have had questions about?
So one question that I kind of had is,
once we have this base model,
it's nice try to do something extra?
Yeah.
So, immediately. We did all of these cool videos in term one.
So, I know that you made a video,
how was the process?
Oh, so you're talking about this optional thing
that we have in the course, in the project.
We're we basically say,
one of the are the optional things to stand out and to show employers is actually
to use us this neural network and apply it against a video.
That's right.
So what I recommend for doing that is actually to take each image,
sorry, the video from each image.
So every video is made out of images,
it's 30 frames per second,
then you get 30 images a second in the video.
So, you take those images out, you export them,
and you apply each image in batches to inference.
And then, you have now an inference on that video.
But to turn it back to a video,
you take those images and put them back in order to create a video.
Sounds like something very similar to what
we're doing in term one with Advanced Lane Finding.
So, we should have all the tools necessary to do that.
Exactly.
Awesome. Yeah. Well, I'm looking forward to creating my own video then.
What else could we do that's kind of optional?
If you have a GPU you can do some exciting things, right?
Yeah, that's right.
You could do more than two classes.
Yeah, you could. You could do with three classes which actually is part
of the images in the dataset.
Oh really?
Yeah.
Maybe we should check that out.
If you actually take a look at the dataset,
we'll go to the training.
Um-hum. Yeah, there's three different colors there.
Yeah, that's right. Because when they labelled it,
they labelled the street in front of the car along with other streets.
Okay.
So, you have this pink color here which is the street in front of it,
and then the red line is what they call the background, so not the street,
and the black line is actually a street that is either going parallel or perpendicular,
but it's not the street that the car is on.
Okay.
But, for this one we kept it simple only doing two labels.
So we basically say,
only label where the street is or where the background is.
Okay.
So, if you wanted to make things more complex you could actually do three labels,
one for the street the car's on,
one for where other streets are,
and the third level with the background.
So, I was actually checking out one of those functions is in that helper.py,
and I was checking out the get batch function,
and the logic for that was basically looking at red pixels and labeling that as false.
And then everything else is true.
Yeah, that's correct. And if you wanted
to implement for three labels you would have to modify that option.
Okay, so, maybe just have an integer for each of those,
say, if it's a red pixel,
let's say it's zero.
Yes.
If it's a black pixel,
let's say it's one. Last ones too.
Sure.
Okay, cool.
That sounds like it'd be a pretty fun project to do.
Yeah. And then the other one is actually to train on the cityscape data set.
Which actually, I recommend this being the hardest one because it takes a lot of time to
train in that city data set and we've kind
of made the problem a little bit easier to do in this data set.
Could you use a simulator as well?
We have that open source self-driving car simulator,
we could get the ground truth,
we can modify it, and then train the same architecture on it.
Yeah, that's actually based on a paper where they took GTA and actually did
segmentation on the simulator because the simulator itself,
they could actually segment all the pixels in the simulator.
You don't have to do the hand annotation.
You don't have to do manual in annotation.
Kind of speeds things up.
It does, yeah.
Yeah, it's nice. So, lots of awesome data that's readily available than for training.
Yeah.
Okay.
Well, do we have any questions on this project, Brock?
Yeah, we actually do. We've got one from Kelly Smith.
Asking, how are training labels effectively created for
generating data sets for semantic segmentation,
and are they created manually?
Yeah, actually, they are created manually.
Most labelled data is created manually,
whether you're marking spam,
or if you decide not to buy a hat,
somewhere that's been logged,
and is labeled data for us to use or anyone else.
Okay.
And then the next question is from Jean-Francois who asks,
how do we get the dimensions of the layers that are loaded from the VGG-16 model?
I keep getting question marks that are returned as dimensions.
I got that problem too.
I just try to print it out and it's question mark, question mark,
question mark, and then maybe 512 or 4096.
Yeah, this question marks indicate that dimensions aren't set yet,
and that's because we want to allow any size images.
So, we've only been using convolutions in this neural network which allow any dimensions.
You change it up, it will still give you a matching dimension to that.
So, if we use fully connected,
we're actually restricted to a certain dimension.
That's why you saw on the past you had the exact dimensions.
But we can actually, what we can do in the classroom is
actually see what those dimensions look
like while we're running the neural network. That's great for debugging.
Okay.
So if we say,
let's take our layer's function.
Yeah.
Go into that and say we want to know
what output in the commercial one by one dimensions are.
What we can do is tf.Print,
make sure you've got that capital P that that's very important,
there's no other print segment.
And then we want our output.
So this is, tensor flow is a tree.
So as you pass stuff in,
it gets passed to the next node.
So if we want print to happen,
we have to have a print node.
So to do that we take,
what we want as our output,
pass it through and now this print statement will be attached
to to the tree and run as the nral network gets run.
So the next thing you want is actually what we want print out.
So we only do array and then actually all the items will then print out.
So, say we want the dimensions output.
We could do output but we also get shape, so,
do shape function on output.
Sorry, to tf.shape on the output and that will actually give us our shape.
You could do output.shape,
but what happens here is,
tf.shape won't be called until the node note is running where
dot get shape will actually get the dimensions before it runs.
That's very important. That's what we want.
Okay.
And what you might actually get is
the first three dimensions with the fourth one cut off.
And then if you wanted more dimensions or just wanted to see one dimension,
you could just call it by index.
Okay.
So if you just wanted the x and y dimension of the image you could say,
starting adds in that index one ending at index three.
Cool. That's very helpful. Yeah.
Well, I guess that wraps up the semantic segmentation project.
Yeah, it does.
Yeah, that's a fun project.
It's pretty awesome.
Having it be a full image and you're classifying it per pixel, feels pretty amazing.
Yeah. It's very important right now in the world of self-driving cars actually.
And if you guys have any questions or feedback for us, let us know.
Yeah.
Thanks for coming.
Yeah, thanks for joining us, guys. Catch you later.