Welcome back.
In the last video, we made a function called createmodel, which does what it says.
It creates a model and the transforms to go along with it, namely a Resnet 50 model and the transforms
from training it on image net.
And we also hinted at the fact that PyTorch 2.0 relative speedups will be most noticeable when as much
of the GPU as possible is being used.
So in the intro we discussed that one of the main costs of training time is bandwidth costs.
So transferring data from memory to the GPU and you want to reduce that as much as possible.
And one of the ways to do that is with Operator Fusion.
And then another way to do it is graph capture.
Now graph capture isn't specifically for reducing bandwidth costs.
It's to memorize the operations model graph or a PyTorch model needs to do ahead of time so it doesn't
have to look them up while it's doing it.
But let's jump down into 2.2, which is where we're up to Speedups are most noticeable when a large
portion of the GPU is being used and if you hear some crows chirping in the background, well, that's
one of the unmissable sounds of Australia.
So there's a bunch of them I think you call a group of crows a murder in a tree not far outside my window
and they're singing their afternoon song.
So if you hear them, enjoy it with me.
So let's write this down.
Speedups are most noticeable when a large portion of the GPU is being used.
So let's break this down a little bit more.
We've given an example before, but let's make this a bit more concrete.
So since modern GPUs are so fast that they're fast at performing operations, so tensor manipulation
matrix multiplications, you will often notice the majority of relative.
Now I say relative because that's an important point, is that it won't always be like for like speed
ups, but they will be relative speed ups, relative speed ups when as much data as possible is on the
GPU.
So in practice you generally want to use as much of your GPU memory as possible.
You don't want GPU memory and compute power to go to waste, right?
So what are some ways to do that?
Well, the first one is by increasing the batch size.
So previously throughout the course we've been using the ever reliable batch size of 32.
Now that is good.
But as you start to train larger and larger models on bigger and better GPUs, you'll likely want to
increase that batch size, namely putting more GPU or more data.
Sorry.
So we've been using batch size 32, but for GPUs with a larger.
Memory.
Capacity you'll generally want to use as large as possible e.g. 128 256 512 etcetera.
A multiple of eight is generally good practice and then you want to also increase the data size.
So for example, instead of using images that are 32 by 32 use.
2 to 4 by 2 to 4 or even three.
Three six by three, three six.
Now, I'm not quite sure what the default image sizes are across the standardization of different image
models, but this is something that you can explore with the the main point here is that it's increasing
in size and that could be also you could use an increased embedding size.
For your data.
And now let's keep going on.
We want to increase the model size.
So, for example, instead of we hinted at this one before using a model with one mil parameters.
Use a model with ten millimetres parameters, and now that might actually be considered a small model
in today's world.
But we're not going for a very specific examples here.
We're just saying that you'll notice the trend is just increase, increase, increase as much as you
can put on the GPU as possible.
That's a good thing.
And then we want to decrease data transfer.
So this is the one thing we want to decrease.
So since bandwidth bandwidth costs, so transferring data will slow down our GPU.
Because it wants to compute on data.
But if it doesn't have the data, it can't compute on it.
Right?
So you want to use as much CPU as power as you can to put as much data as you can on the GPU and avoid
unnecessary transfers, which is one of the things that PyTorch does behind the scenes with Operator
Fusion.
Instead of writing and saving data and computing it, it just computes as much as it can through a series
of operations.
So let's go here.
As a result of doing the above, your relative speedups should be better.
Overall training time may take longer, but not linearly.
So although we're increasing here, you might increase the batch size by four x.
It doesn't mean to 128.
It doesn't mean that the training time will increase by four x.
The same with data size, the same with model size, the same with data transfer.
It's not a linear increase just because you're doing more.
So now let's go to the keynote, have a beautiful little diagram here.
So this is our A100 with some data on it.
If we put this one square of data on it, the training time might take ten minutes.
But then if we decided to step up our experiment and create a larger experiment, we may have six times
as much data across two GPUs and then we have six times more data, but only twice as much training
because we're taking advantage of the parallelism computing powers of our GPUs.
And again, these training times are for illustration purposes only.
I'll leave it to you to do some experimentation with different batch sizes, different model sizes,
different data sizes, to see how this actually plays out in practice.
So let's jump back into here.
I might just put a resource here resource for learning how to improve PyTorch model speed.
This is something that just came to mind the other day.
Improve PyTorch model speed.
Sebastian Rashka So Sebastian Rashka is one of the best machine learning researchers that I know publishes
some great.
Information about AI machine learning in general.
So this is a great blog post.
I'll leave as extra curriculum to read How Plain PyTorch 2.0, which is what we're just learning about
how to use, can go from 21.3 minutes to 1.8 minutes.
So isn't that a beautiful increase?
Check that blog post out when you can.
But for now, we've discussed some options for improving relative speed ups, and most of them deal
with either increasing the size of your data, increasing the size of your model, or decreasing how
much data you're transferring.
So do as much computation on the GPU as possible.
And so speaking of increasing the batch size, how about in the next video we write some code to check
the memory limits of our GPU so we can do that via Nvidia SMI, but it would be good to check this programmatically
so that if we have a larger GPU available we could increase the batch size.
So let's see how to do that in the next video.