Let's step back for a bit to remind us why we're focusing on memory the way we
are. Our overarching goal, of course, is just to make code fast, so, we say
great, GPUs are fast, let's use those. But why are they fast? Gpus are fast,
first, because they are massively parallel, with hundreds or thousands of
processors on a single chip, working for you to solve your problem. But also
because they have an extremely high bandwidth memory system to feed those
massively parallel processors, okay? So, if the memory system can't deliver data
to all of these processors and store results from all those processors, then
we're not going to get the full speed out of our GPU. And that's why, on a
memory limited kernel like transpose, our subgoal is really to utilize all the
available memory bandwidth. Hence, our focus on global memory coalescing, DRAM
utilization, and so on. Now, I really want to ask a question a little bit more
rigorously, what do we mean by utilizing all the avail, available memory
bandwidth? And this is going to bring us to a very important, very simple
principle called Little's Law. Let's have the talented Kim Dilla illustrate this
for us. Now, John Little is a MIT professor who studies Marketing. He formulated
his Eponymous Law, when writing about queuing theory in business processes. And
Little's Law is usually used to reason about things like optimizing the number
of customers in a line at Starbucks, or maybe the size of queues in a factory.
But Little's Law is really very general and can be applied to many things
including memory systems in computers. In that context, Little's Law states that
the number of bytes delivered equals the average latency of each memory, memory
transaction times the bandwidth. Let's be a little more precise and emphasize
that we care about the useful bytes delivered and the problem with uncoalesced
global memory accesses is that not all of the bytes in every memory transaction
are actually being used. That's why coalescing global memory accesses helps
ensure that every byte delivered in a memory transaction will be used. So, given
this definition, what can we do to improve our bandwidth? Let's check all that
apply. We can increase the number of bytes delivered, we can increase the
latency, meaning the time between memory transactions, we can decrease the
number of bytes delivered, or we can decrease the latency or time between
transactions.