foreign
today I'm going to be presenting my
research group's work on zukway failure
is not an option it's an exception this
work was performed in collaboration
between the non-volatile systems
laboratory at the University of
California San Diego and the
architecture research group at the
University of Colorado Boulder
first I'll give you a brief one slide
overview of our contributions we
developed a simple pre programming model
for persistent memory systems which use
flush on fail semantics like eadr or gpf
this model requires little or no effort
from the application developer to use
typically no effort and performs at
least 3x faster on average than prior
work with flushes and fences removed to
make it fair
now that I've hopefully caught your
attention I will step back and provide
some motivation and background for the
work so as we're probably mostly aware
persistent memory is persistent across
power failures it is bite addressable
and it offers comparable latency and
bandwidth to dram
so a naive approach to making a program
persistent might be to Simply uh
program's memory and put it all in
persistent memory however that doesn't
really work because the cash has been
volatile so updates that are in the
cache but not yet flushed will be lost
if there's a power failure and
applications need to explicitly evict
cache lines the backing store for crash
consistency
this has some significant consequences
for performance and for usability
in terms of performance the biggest
issue is that explicit cash flushes
amplify rights to persistent memory
you're going to end up writing to the
backing store a lot more often if you
have to explicitly evict lines after an
update rather than waiting for Hardware
to evict them
second correctly placing cash flushes in
the associated barriers is difficult
some many would say it requires expert
knowledge and it is a quite a burden for
the application developer if they want
to make their program persistent
and third the required memory barriers
and multi-threaded programs also install
incur further overhead from pipeline
stalls and synchronization overhead
between coherence agents
uh so there's been a wide body of work
created to make pmm programming easier
and faster
um these works very significantly in how
they are implemented but their
programming models are mostly similar
they Define a failure Atomic section of
where they divide the programs rights
and affiliate intersections and after a
failure each section's rights are either
entirely present in the persistent store
or entirely absent
so we will divide so we divided these
Works into three groups I will discuss
each of these based on their programming
model and I will discuss each of these
three models now
uh the first and most common is the
transactional model which Imports the
traditional acid transaction from
database systems
so these either have an explicit
transaction begin transaction end
interface or they rely on the
application's locks since transactions
have to match the Locking semantics of
the program or rather transactions
restrict the Locking semantics of the
program and specifically they restrict
them such that lock acquire and release
must Nest perfectly and the persistent
memory can only be accessed when all the
locks are held
uh unfortunately this prohibits common
and useful locking patterns such as hand
over hand locking uh which is where
you're iterating over a pointer chain
and you lock the next item before
releasing the previous one in order to
ensure consistency
uh and this is a common pattern in
multi-threaded graph applications that
use fine-grained synchronization
so phases were created to address this
limitation by relaxing the definition of
the failure Atomic section to the region
between the first lock of acquisition
and the last lock release not
necessarily of the same lock
so this solves the programming model
problem it supports any locking scheme
compatible with Legacy applications but
in order to support this model these run
these generally require runtime tracking
of dependencies between the threads
specifically read after write
dependencies because a thread can read
an already written value and act on it
and possibly end up in an inconsistent
state if a failure occurs
uh unfortunately this runtime tracking
incurs significant overheads in practice
and can also lead to unrecoverable
scenarios in adversarial cases which we
discussed further in the paper
the third model which is uh sort of
different from the others is whole
system persistence
and it's pretty much what it says on the
box everywhere you would normally use
dram in the system you use pmem instead
uh and the other big difference between
the other models is there are no
explicit cash flushes uh the only time
the caches are flushed is if a failure
occurs and they expect the system to
deliver and interrupt in order to do
this
so they used experimental hardware for
this original paper about a decade ago
but in the past decade manufacturers
have indeed brought flush on fail
support to commodity Hardware in the
form of gpf for cxl 2.0 devices and eadr
for nvdim devices like optane
uh these systems generally
require the platforms to provide a
guarantee that the caches will be
flushed by a low-level firmware
interrupt if power failure occurs and
that there's enough power to flush the
caches
so from our perspective we would say
this essentially gives us persistent
caches
however we still believe that the
software model of whole system
persistence is uh insufficient
first it only preserves the contents of
the application memory and provides it
to them at restart there's no
well-defined model for Recovery of the
application
and second we would say that the whole
system does not really need to be
persistent and that introduces
significant engineering challenges and
probably some performance overhead and
you're probably but you're really
probably only interested in making some
specific user space applications uh
persistent
so in order to address these criticisms
we developed a new programming model
called whole process persistence
in this model we transform all memory
allocated by the process into persistent
memory
if a power failure occurs we deliver an
operating system signal to the process
after it is recovered
and execution simply continues from the
point interrupted by failure
so this model has two main benefits over
prior models first it is easy to use
there's no restrictions on locking or i
o and our libsy-based implementation is
binary compatible with existing existing
applications
second it's low overhead there are no
explicit cash flushes and there are no
right amplification from mechanisms such
as logging or replication
so now I'll talk a little more about the
design of our libsy based runtime and
how it works
so at a high level it's a modified
version of lib C which implements the
whole which implements the whole process
persistence model we chose to implement
this in libc because libc provides the C
bindings for Linux system calls and the
threading API and so that's where we
interposed we intercept and transform
the arguments to these calls to
accomplish our goals
and it is entirely transparent to the
application we simply all you have to do
is set an environment variable and run
your program as you usually would
so what happens when you run your
program uh during normal execution uh we
simply we mostly modify in map so we
return persistent memory for all
Anonymous and map allocations and we
also return persistent memory for all
private writable file mappings that will
be more esoteric things like your
initialized globals in your program and
your loader relocations
and the anonymous is your peep and your
stack
and we also uh since the kernel stack is
volatile in our implementation we would
lose the user space context if a power
failure occurs in kernel mode
so we also save that state that context
to a separate pmem region on every
kernel entry and we modified the kernel
to do this
if a failure does occur uh so as I
mentioned before we expect that the the
system will deliver and interrupt
interrupt the application or the
operating system to flush the caches and
we assume that this flushes the caches
correctly however there's still the
model of the volatile architectural
State uh the register file the floating
point and Vector unit context and so on
uh on x86 at least uh so the firmware
interrupt that flushes the caches does
save this state to memory however uh
it's not to persistent memory it is to
some memory chosen by the uh device
manufacturer
uh so we to our understanding uh we
should it should be pretty
straightforward to change this interrupt
to save to persistent memory but it is
diff it is impossible to replace
firmware on Intel systems without the
platform manufacturer's cryptographic
key so we weren't able to do that uh
instead for evaluation we simulated this
process using user space signals which
push similar State on the stack but
would not be suitable for use in a real
failure in a real power failure scenario
so after a failure when we restart the
program we have the virtual memory map
and this volatile architectural state in
hand
and so to restart execution of the
program we first restored the program's
address space
and so that's remapping the persistent
memory files we previously created
uh we restore operating system specific
state so we are also we also track the
state of threads and files and recreate
them at restart
and uh finally we restored the volatile
architectural State and since that
includes the program counter and the
stack pointer for each thread restoring
that is equivalent to restarting
execution of thread at this point uh if
the application provides a failure
Handler we run it before execution
continues
and finally I should note that there are
some requirements for applications to
use zookway first is that threading file
descriptors in virtual memory must be
managed through the libc interfaces that
we interpose on in line says calls
bypass those and so that will not work
with us
and second is that applications must
check error returns from system calls
which interact with components beyond
the process boundary for instance if a
file system was not remounted after a
restart uh zukui cannot do anything
about that
now I will close with uh performance
results
uh so first here are some comparisons of
zooquay on an older version of memcache
D which all comparison systems can run
the green bracket marks are improvement
over a prior work as you can see we
moderate Improvement
and moderate overhead versus the native
implementation
uh because the Red Bar as you as you may
note uh this version of memcache D
doesn't really scale at all uh it
doesn't really gain any throughput from
adding more threads this is a known
problem with with early versions of
memcache D and later versions of
memcache D were improved to fix this
including by adding fine grain locking
incompatible with transactional schemes
however that's not a problem for zuquay
we can easily run that and so we can
gain significant we can run a much newer
version of memcache D that is
significantly faster than any prior
version of persistent memcache D note
that the y-axis on this figure is five
times larger than the previous figure
uh the and again the green bar so the
black bar on these figures is uh
zuquayon old memcache d
and this orange and the orange bracket
is our overhead versus native which is
also much lower than any previous
persistent implementation especially on
the read heavy benchmark
and finally I will say that if the
applications are less memory intensive
the picture is even better for us
uh so these are results of from some
selected benchmarks on the pi
performance Benchmark Suite we ran the C
python interpreter on top of zukway and
in Native versions and as you can see we
introduced minimal or no overheads
in most cases
so uh to summarize our contributions uh
we introduced a new programming model
for flush on fail persistent memory
systems and built and tested ellipse
based prototype implementation
uh we found that zookway outperforms the
state of the art programming PM
programming systems and can run a wider
range of applications than prior work
and additionally it's not on this slide
but we also provide formal arguments in
the paper as to why our prior systems
are insufficient and fail to recover
from some common scenarios
so looking forward to your questions
thanks for your time
foreign