Let's step back for a bit to remind us why we're focusing on memory the way we are.
Our overarching goal, of course, is just to make code fast.
So we say great, GPUs are fast, let's use those. But why are they fast?
Gpus are fast, first, because they are massively parallel, with hundreds or thousands of
processors on a single chip, working for you to solve your problem, but also because they have
an extremely high bandwidth memory system to feed those massively parallel processors, okay?
So if the memory system can't deliver data to all of these processors
and store results from all those processors, then we're not going to get the full speed out of our GPU.
And that's why, on a memory limited kernel like transpose,
our subgoal is really to utilize all the available memory bandwidth.
Hence our focus on global memory coalescing, DRAM utilization, and so on.
Now I really want to ask a question a little bit more rigorously.
What do we mean by utilizing all the available memory bandwidth?
And this is going to bring us to a very important, very simple principle called Little's Law.
Let's have the talented Kim Dilla illustrate this for us.
Now John Little is a MIT professor who studies Marketing.
He formulated his Eponymous Law, when writing about queuing theory in business processes.
And Little's Law is usually used to reason about things like optimizing the number
of customers in a line at Starbucks, or maybe the size of queues in a factory.
But Little's Law is really very general and can be applied
to many things including memory systems in computers.
In that context, Little's Law states that the number of bytes delivered
equals the average latency of each memory transaction times the bandwidth.
Let's be a little more precise and emphasize
that we care about the useful bytes delivered, and the problem with uncoalesced
global memory accesses is that not all of the bytes in every memory transaction are actually being used.
That's why coalescing global memory accesses helps
ensure that every byte delivered in a memory transaction will be used.
So given this definition, what can we do to improve our bandwidth? Let's check all that apply.
We can increase the number of bytes delivered, we can increase the latency--
meaning the time between memory transactions--we can decrease the
number of bytes delivered, or we can decrease the latency or time between transactions.