Welcome back.
In the last video we made a function
called createModel which does what it says.
It creates a model and the transforms to
go along with it.
Namely a ResNet -50 model and the
transforms from training it on ImageNet.
And we also hinted at the fact that
PyTorch 2 .0 relative speedups will be most noticeable when as much of the GPU
as possible is being used.
So in the intro we discussed that one of
the main costs of training time is bandwidth costs.
So transferring data from memory to the GPU and you want to reduce that as much
as possible.
And one of the ways to do that is with
operator fusion.
And then another way to do it is graph capture.
Now graph capture isn't specifically for reducing bandwidth costs.
It's to memorize the operations a model graph or a PyTorch model needs to do
ahead of time.
So it doesn't have to look them up while
it's doing it.
But let's jump down into 2 .2 which is
where we're up to.
Speedups are most noticeable when a large
portion of the GPU is being used.
And if you hear some crows chirping in
the background, well that's one of the unmissable sounds of Australia.
So there's a bunch of them.
I think you call a group of crows a
murder in a tree not far outside my window and they're singing their
afternoon song.
So if you hear them enjoy it with me.
So let's write this down.
Speedups are most noticeable when a large
portion of the GPU is being used.
So let's break this down a little bit more.
We've given an example before but let's make this a bit more concrete.
So since modern GPUs are so fast, fat, they're fast at performing operations.
So tensor manipulation, matrix multiplications.
You will often notice the majority of relative.
Now I say relative because that's an important point.
Is that it won't always be like -for -like speedups but they will be relative speedups.
Relative speedups when as much data as possible is on the GPU.
So in practice you generally want to use as much of your GPU memory as possible.
You don't want GPU memory and compute power to go to waste, right?
So what are some ways to do that?
Well the first one is by increasing the
batch size.
So previously throughout the course we've
been using the ever -reliable batch size of 32.
Now that is good but as you start to train larger and larger models on bigger
and better GPUs you'll likely want to increase that batch size.
Namely putting more GPU or more data, sorry.
So we've been using batch size 32 but for
GPUs with a larger memory capacity you'll generally want to use as large as possible.
E .g. 128, 256, 512 etc.
A multiple of 8 is generally good practice.
And then you want to also increase the data size.
So for example instead of using images that are 32 by 32 use 224 by 224 or even
336 by 336.
Now I'm not quite sure what the default
image sizes are across the standardization of different image models
but this is something that you can explore with.
The main point here is it that it's increasing in size and that could be also
you could use an increased embedding size for your data.
And now let's keep going on.
We want to increase the model size.
So for example instead of, we hinted at this one before, using a model with 1 mil parameters.
Use a model with 10 mil parameters.
And now that might actually be considered
a small model in today's world but we're not going for very specific examples here.
We're just saying that you'll notice the trend is just increase, increase, increase.
As much as you can put on the GPU as possible that's a good thing.
And then we want to decrease data transfer.
So this is the one thing we want to decrease.
So since bandwidth costs, so transferring data will slow down a GPU because it
wants to compute on data.
But if it doesn't have the data it can't
compute on it right?
So you want to use as much CPU as power
as you can to put as much data as you can on the GPU and avoid unnecessary transfers.
Which is one of the things that PyTorch does behind the scenes with operator fusion.
Instead of writing and saving data and computing it, it just computes as much as
it can through a series of operations.
So let's go here.
As a result of doing the above your relative speedups should be better.
E .g. the overall training time may take longer but not linearly.
So although we're increasing here you might increase the batch size by 4x.
It doesn't mean to 128 it doesn't mean that the training time will increase by 4x.
The same with data size, the same with model size, the same with data transfer.
It's not a linear increase just because you're doing more.
So now let's go to the keynote.
I have a beautiful little diagram here.
So this is our A100 with some data on it.
If we put this one square of data on it
the training time might take 10 minutes.
But then if we decided to step up our
experiment and create a larger experiment we may have six times as much data across
two GPUs and then we have six times as more data but only twice as much training
because we're taking advantage of the parallelism computing powers of our GPUs.
And again these training times are for illustration purposes only.
I'll leave it to you to do some experimentation with different batch
sizes, different model sizes, different data sizes to see how this actually plays
out in practice.
So let's jump back into here.
I might just put a resource here.
Resource for learning how to improve
PyTorch model speed.
This is something that just came to mind
the other day.
Improve PyTorch model speed Sebastian Rushka.
So Sebastian Rushka is one of the best machine learning researchers that I know.
Publishes some great information about AI and machine learning in general.
So this is a great blog post I'll leave as extra curriculum to read how plain
PyTorch 2 .0, which is what we're just learning about how to use can go from 21
.3 minutes to 1 .8 minutes.
So isn't that a beautiful increase?
Check that blog post out when you can.
But for now we've discussed some options
for improving relative speedups and most of them deal with either increasing the
size of your data, increasing the size of your model or decreasing how much data
you're transferring.
So do as much computation on the GPU as possible.
And so speaking of increasing the batch size, how about in the next video we
write some code to check the memory limits of our GPU.
So we can do that via Nvidia SMI but it would be good to check this
programmatically so that if we have a larger GPU available we could increase
the batch size.
So let's see how to do that in the next video.