thank you for the introduction
my name is Tonya and this is a joint
work with mark silverstein and
publishers from Tekken GPUs have come a
long way from fixed-function
accelerators to fully programmable high
performance processors that support them
and paging yet their integration with
the host OS is still quite limited in
particular GPU memory which can today be
as large as 32 gigabytes has been
traditionally managed entirely by the
GPU driver without any host OS control
now one crucial implication of such
design is the OS cannot provide core
system services to GPU kernels such as
memory mapped files or efficient file
sharing between the CPU and the GPU nor
can if utilized the additional memory
links in disposal for optimizing CPU i/o
or support seamless peer-to-peer between
the GPU and the disk but you might ask
yourself why are you telling us about
this I mean why do we even need these
functionalities let's take a look at
this side an example and decide together
shall we
so this needs no introduction I'm sure
everyone uses Road navigation well at
least I do but let's say think how its
implemented and we'll focus on the
server side of it because this is where
the interesting stuff happens so you
want to get from the airport to our
hotel we can look at a map as a weighted
graph with weights being the travel
times on the roads and we have a
planning service taking this graph as an
input and calculating the shortest path
for us to travel on and in parallel we
have the traffic updater service
periodically updating the travel times
on the roads and as a result of the
update we have to reactivate our
planning service and our optimal path
may change now how would we implement
this so our map is saved in the file and
we will take two CPU processes each of
them and mapping the file so the code is
pretty simple our updated service just a
map the file and uses it as if it was a
regular memory access and same for our
planning service we just a map the file
and pass the pointer to the route
calculation routine but what if we now
wanted to speed up the planning
by offloading it to GPU unfortunately we
can't no longer just MF the file we have
to read the whole file into the CPU
memory and then copy it to GPU and know
that we have to repeat these two steps
before each kernel activation regardless
of the fact that perhaps not even the
whole file is required by the planning
service so the concludes that the
broader example of some cpu application
sharing file with multiple GPUs and each
of them is working on a different
portion of the file but which portion of
the is that is calculation dependent
source application just M Maps the file
as before but for the GPU would have to
copy the whole file into each of the GPU
memories so the conclusion is that
without OS management of GPU memory we
can't have data dependent GPU access to
files nor can we efficiently share files
between the CPU and the GPU and as a
result we have to repeatedly copy the
whole file into each of the GPU memories
which is not just tiring but also pretty
and efficient right so in order to
overcome this limitation and
architectural changes required we need a
tighter integration between the GPU
memory and the host OS and from this
understanding came gaya a distributed
weekly consistent page cache for
regional systems with gaya will enable
memory mapped files for GPU kernels with
the port efficient file sharing between
the CPU and the GPU and we enable CPU
and GPU AI optimizations that were just
not possible in the current system
design it turned out we had some really
interesting challenges ahead of us we
had to decide the consistency model
suitable for the new system and we had
to understand how it will all integrate
both with the GPU driver and with the
host OS but today we'll talk only about
the consistency model we chose and you
are welcome to read the paper for more
details so let's take a look at our
system with the unified page cache what
we actually created is the distributed
shared memory in which the same file
data can now reside in both the CP
you and the GPU memory and not only that
now both the CPU and the GPU can update
the file in parallel so what do we do
about consistency
for example if the CPU update the file
when will the GPU become aware of the
change let's start by focusing on the
GPU side of the system and talk a bit
about unified memory and NVIDIA GPUs so
in the recent GPUs Nvidia enabled the
same pointer to be accessed by both CPU
and the GPU and implemented by utilizing
the page fault mechanism to trigger the
data migration between the two memories
but now if the CPU and the GPU can
access the same pointer Nvidia also had
to address the consistency question
right and the consistency model Nvidia
chose is the one of strict quadrants in
64k granularity which is one GPU page
according to this model the same page
can be mapped by only one processor at
any given moment so for example if the
page now resides in CPU memory and the
GPU requires it the GPU will trigger a
page fault
which will result in the page being
migrated to GPU memory and if the CPU
needs the page next it will also raise
the page fault and the page might may
will be migrated back to CPU so as you
can see if the CPU and the GPU are right
sharing the page it results in a
constant ping-pong covet between the two
memories but what if the CPU and the GPU
are actually false sharing the page for
example the CPU writes to the lower 32 K
and the GPU writes to the upper 32 K in
theory they could have worked in
parallel right if not for the strict
currency limitations false sharing is
actually a known issue with you VM and
the official advice from Nvidia in such
cases is to avoid them by having the two
processors work each on a private buffer
and to merge the output at the end so we
did a small test to evaluate how false
sharing affects performance we had two
GPUs ride-sharing 64 K buffer
and we compare this to the official
advice from Nvidia
in which the true GP used were each
working on a private copy of the buffer
and we merge the result at the end so
what we did is we varied the number of
loop iterations performed by each GPU
and by doing so we actually varied the
number of page migrations and this is
what you can see on the x-axis y-axis is
x the slowdown compared to our baseline
in which the two GPUs are each working
on a private buffer so we can easily see
that the slowdown due to false sharing
increases linearly with the number of
page migrations which is pretty expected
right but the slowdown is not the only
side effect of false sharing it turned
out that the false sharing effects v M
as a whole we did another test where we
took a CPU only benchmark called k-means
comes from the Phoenix test with and we
limited it to run off on four of our
system cores in parallel to it we ran
our full sharing benchmark and we
allocated the remaining two cores for
GPU management which is a common
practice with GPU accelerated workloads
so we ran two tests in the first test we
just ran our k-means on the for course
we allocated for it and we measured its
run time of course is constant right
what we did next is we ran our full
sharing in parallel to K means now what
we expected to see is that came in run
time will remain constant as well right
because it's running on different calls
but to our surprise this was not the
case and performance isolation was
affected beautiful sharing so this
brought us to an conclusion that a
different consistency model is required
and the one we chose is the one of laser
release consistency in the paper we
discussed several alternatives which I
want to go over today so laser release
consistency defines tube to consistency
operations called acquire and release
and the general rule is that
modifications to share object will
become visible only after the writer
releases and the reader acquires so
let's take a look an example of two
writers sharing the same file
they both acquire the first writer write
something to offset an and in parallel
to it the second writer also acquires
and writes something to officer 20 next
they both release and by releasing they
actually notify the other that the file
changed so the next time the first
writer for example tries to access the
file it will issue an acquire which will
result in the data being propagated from
the second writer and nourished with the
local changes now you might have already
noticed a small issue with this protocol
if one of the writers is a CPU because
we are now actually forcing the CPU to
issue some consistency operations before
each access to files right and this is
of course a no-go so we have to somehow
make our synchronization operations
transparent to cpu applications to keep
them unaware of the weekly consistent
page cache and we do this by limiting
the consistency control to the CPU code
that invokes GPU kernels now how do we
do that the idea here lies in the
understanding of what do we want to
happen on the acquire and release
sequence and the purpose of the
acquiring release is basically to make
sure that the next reader knows from
where to get the latest version of the
data from so keeping that in mind we can
play with when the acquire of the CPU
occurs and move it back to when the GPU
releases and the similar trick can be
applied to CPU release by moving it to
when the GPU acquires and this way we
made the synchronization operations
compare until a transfer to CPU
processes now let's put it all together
and look how our code will look with
with road navigation will look with gaya
we can now just M
the file and pass the pointer to the GPU
colonel and that's it
all the synchronization disk access the
copy to the between the memories is done
transparently for us by the operating
system we have several more interesting
features that enable us to improve CPU
and GPU i/o in the paper as well as
implementation details we didn't go over
today and this brings us to evaluation
in the paper you will find detailed
analysis of overheads added by gaya in
as well as benchmark results including
closed source libraries like coolest and
today we will go over two of the three
real life applications we used so the
first one is microscope image stitching
when I take a picture with a microscope
you'll get a mosaic of small images
overlapping with each other and you need
to post process those images in order to
get the original original shot now this
post processing was efficiently
implemented in by the help of GPUs and
we hence this process even further by
having the CPU work in parallel on half
of the output image so we compare three
different implementations in the first
one we have the CPU and the GPU each
working on a private buffer and we merge
the result at the end and the next
become we implement this with uvm in
unified virtual memory in which the CPU
and the GPU right share the output
buffer and as we saw earlier in this
case the experience Falls sharing which
results in 31 perform percent
performance degradation compared to our
previous implementation and last we
implemented with guy in which we just M
mapped the file and the CPU and GPU work
in parallel as you can see with guy we
were able to completely eliminate the
false sharing effect without having to
merge the output file at the end and
we'll of course conclude with our road
navigation service that been following
us throughout this whole talk we
implemented it using gang Rock which is
the open source library for graph
processing and it implements the
shortest paths application we had two
processes one of them calculating the
shortest path on the GPU and the other
was the legacy CPU process periodically
up
the weights on the graph and we have had
the two processes interleaved and we
compare three different implementations
the first one is CUDA in which before
each kernel activation we had to read
the whole file and copy it to GPU memory
the second one is UVM in which before
each kernel activation we had to read
the whole file but the copy to GPU was
done transparently for us and the last
one is gaya
in which we just M mapped the file and
made sure to acquire and release before
kernel activation so we varied the
portion of the file changed by a legacy
CPU process and this is what you see on
the x-axis and Y is the test duration as
expected both you VM and CUDA
implementations are not affected by the
portion of the file change because prior
to each activation we still have to read
the whole file but with gaya we were
able to get up to eight times speed up
because the unchanged pages remained
cached in GPU memory to conclude we
extended the OS page cache into GPU
memory by implementing laser release
consistency model and eliminating false
sharing effect we added support for
memory mapped files for GPUs and all of
our changes remain transparent to legacy
CPU processes and compatible van with
unmodified GPU kernels and last but not
least although unfortunately we had no
time showing that we were able to
improve IO performance for CPU legacy
processes by caching data in GPU memory
so all of our code is open source and
you can find it online thank you for
listening and I will be happy to take
your questions now
[Applause]
let me ask you a question
so imagine the hardware architects
decide to come up with a new set of
primitives and provide coherence between
CPU and GPU and obviously this is coming
if not already there how will it change
your work
well actually the idea of the unified
page cache still remains valid but I
agreed that the laser release
consistency model won't be required
because as your self currency will be
handled by the hardware but this thing
is this work is the we aimed it for
providing a solution for this screed
GPUs in x86 systems and we don't see the
hardware currency becoming available
anytime soon in the systems while Gaia
is thank you thank the speaker again
thank you