so this is don't work with my advisor
Patrick McDonald and collaborators from
University of Wisconsin and ARL here we
go so just to get you started
consider an adversary who's trying to
crash a driverless car the driverless
car today uses a camera to identify and
locate signs on the side of the road and
then it's going to use a classifier to
process this raw image and output the
type of the sign and in this work we're
considering adversary's who are able to
manipulate the input of the neural
network classifier and who are
interested in making it forcing it to
make mistakes
for instance here you have an image of a
stop sign which is correctly processed
by the model and now if I consider an
adversary capable of introducing this
perturbation to the image the adversary
is able to force the model to produce
the wrong class and previous work has
shown that this is possible with
indistinguishable perturbations and very
reliably so this is not a problem
specific to driverless cars companies in
the industry use neural networks to
detect financial fraud malware and
advanced persistent threats they also
process medical data and genetic data
with neural networks also some companies
provide deep learning as a service for
other developers to include in their
products so before I get started let me
introduce briefly some basics about
neural networks so I can lay out the
rest of the presentation so this is an
example of a neural network which is
solving a classification test that is
that you are trying
to map an input domain into predefined
finite set of classes for instance here
the neural network is trained to
recognize digits in images so the first
layer of the neural network is simply
reading the image and encoding it as a
vector of real values then the second
the first hidden layer sorry is applying
a weighted sum to this input layer and
then nonlinear functions to produce the
first representation of the input the
second hidden layer repeats the same
process with different parameters in the
weighted sum to produce a second
representation of the input and so on
until you reach the last hidden layer
which has as many neurons as there are
classes in the prom and each neuron
produces a score value which indicates
the likeliness of the sample of being in
this class and finally this is processed
by a softmax layer which is simply
taking the class values into
probabilities so normalizing and summing
up to one and so this whole model is
trained by showing it a bunch of
input-output pairs and back propagating
error gradients through the model and it
turns out that neural networks have been
shown to produce state-of-the-art
accuracies in many mission learning
tasks in the past ten years another
example is this architecture used in the
industry to extract meaning from raw
audio and it shows the success of DNN
because it is able to replace a chain of
tools that require hand feature feature
engineering and here the neural network
is able to extract our our key of
increasingly abstract representations
automatically so however even though
neural networks perform very well on the
JIT innate inputs previous work has done
traded that it is extremely easy for
adversary's with knowledge of the neural
network to find indistinguishable
perturbations that will force the model
to it put a wrong class so for instance
here on the left you have our digit data
set on the diagonal all the samples are
correctly classified by the model and
off the diagonal is you will find at
virtual samples that are all miss
classified by the model so you to read
the matrix the rows are input classes so
they're the legitimate class of the
sample that a human being would assign
and the columns are the output of the
model and as you can see we can craft
perturbations that take any sample from
any class to any target class chosen by
the adversary for instance the first row
is a zero classified by the model as a
zero one two three through nine this
works with many data sets on the right
you will see the Sifford data set where
you can classify dogs as birds planes as
trucks and below it is a traffic sign
data set where you can for instance
classify or no truck sign into a yield
sign and how this is done is by
computing the Jacobian of the model with
respect to the current input which gives
you an estimation of the sensitivity of
the model and then you combine these
values into add virtual sailin c maps
which will indicate for each input
feature how likely it is to contribute
to your Mis classification goal and you
do this iteratively perturbing the
highest-selling c features until you are
able to force the model to misclassify
the sample in a class that you chose so
now I'm going to go to the topic of
interest for this talk which is
defending again
these attacks so what are we trying to
achieve so if you consider a simplified
picture of the models output surface X
is a legitimate sample and the adversary
is trying to find the add virtual sample
X star that is closest to the legitimate
sample so basically the one that's on
the other side of the decision boundary
and what we are trying to do as
defenders is push this decision boundary
further away from the legitimate sample
we want to do this with a low impact on
the architecture so that the solution
can be easily deployed in existing
systems we need to maintain accuracy
because otherwise the model becomes
useless related to this we have to
achieve robustness in something that is
close to the legitimate expected
distribution or else you could imagine
this constant classifier where the
decision boundaries to infinity which
would be perfectly robust and finally we
have to maintain the speed of networks
because neural networks are widely used
in the industry because they're able to
make predictions a test and very quickly
so we don't want to degrade that so if
you remember the model that I showed you
earlier the last layer is the softmax
layer and this is what we leverage to
create our defense the softmax layer
works by applying an exponential
function to the class scores and
dividing by the sum of the exponential
values and there is a parameter in that
function which is the temperature T
which divides the class scores before
the exponential function is applied and
what this parameter does is that when
you set a low temperature the
probabilities will be extremely discreet
so you will make confident predictions
however if you increase the temperature
the probability vector will smooth
and converge towards one over the number
of classes so typically people will set
the temperature to one because they want
to make confident predictions what we
saw is we can instead increase the
temperature to increase the robustness
of the model and I will show you how if
you have your training data which is a
set of input points and labels that you
show to the network you can train a
first Network as you would typically do
to the exception that you raise the
temperature to a large value typically
around 40 to 50 and you will produce
this way probability vectors that are
very smooth you are going to then use
these probability vectors to label your
data that is you replace the labels that
simply indicate the correct class with
the probability vectors you train the
same architecture from scratch using
this new labelled data set and you
obtain a new model which we call the
Distilled model and to make predictions
using this new model all you have to do
is simply set the temperature back to
one a test time this way you still have
discrete predictions which are confident
and it turns out that this is very
efficient in making add virtual sample
crafting much harder so why does it work
what's the intuition behind this there
is actually two of them first is that
when models are trained they typically
are trained to minimize a loss function
which is something along the lines of
the cross entropy between the expected
labels and the predictions and if you
replace a label of the correct class by
probabilities instead of constraining
only the correct class to be the output
you allow the network to produce some
class scores for
classes that are not the correct class
and you can see how this is useful if
you look at the two images at the bottom
of the slide this three here is very
similar to an eight there's only a few
pixels that are missing to make it into
an eight and what this solution will do
is that it will allow the model to
output a class score for eight here as
long as three is larger as a larger
value and also it turns out that the
Jacobian amplitudes are inversely
proportional to the temperature so as
you increase temperature reduce the
Jacobian amplitudes so we validated this
defense on two data sets the digits data
set which previous work has shown you
can force two misclassify with a ninety
seven percent success rate by only
modifying four percent of the inputs and
the second data set is a data set of
color images with objects and animals on
both of these data sets we trained
state-of-the-art or close to
state-of-the-art architectures using
convolutional layers rectified linear
units and the softmax so here are the
results you can see the blue curve
corresponds to the digits data set and
the gray curve to the objects data set
and the x-axis holds the temperature
parameter and the y axis the success
rate of adversaries and as you can see
clearly as the temperature increases
during training the adversary's have a
harder time crafting at virtual samples
and eventually for the digits data set
we reach levels less than 0.5 percent
success rate and for the image data set
we go below six percent so you may
wonder ok so he's selling us the defense
doesn't impact the accuracy of the model
does it in fact the performance and it
turns out that the impact on the
accuracy is limited on the left you can
see
increasing temperatures the accuracy
impact is always constrained between a
little less than 1.4 percent and a
little over 1.2 percent increase so that
is sometimes distillation even increases
the accuracy of the model and this
allows you to find sweet spots where for
instance for the digits data set for a
temperature of 40 we found that we could
bring the success rate down to 0.45
percent while degrading the accuracy by
only a little less than 0.5% as well and
the same thing holds for the image with
colors of objects you can set the attack
success rate 6% using a little improve
in the accuracy so this clearly shows
that there's a more general question
here whether you'd like to identify the
relationship between accuracy and
robustness and this is an open problem
that we are still looking into so also
the impact of the of the distillation on
Jacobian amplitudes as if you remember
earlier I said that the Jacobian
amplitudes are inversely proportional
temperature and you can see that we
measured this empirically each histogram
is for a given temperature the one on
the left is no distillation and the one
on the right is the highest temperature
you can see that initially the Jacobian
amplitudes are between 10 to the minus
of 3 and 0 and as you gradually increase
the temperature you decrease the
Jacobian amplitudes up to while actually
down to 10 to the minus 40 so there is a
reduction by a factor of 10 to the power
30 of the Jacobian amplitude which
effectively smooth the models decision
and we estimated the robustness because
it is not possible to compute it exactly
so what we did empirically is we
considered the test set for each data
set and we computed all possible adverse
all targets for each input and we we we
identify the smallest perturbation which
achieves miss classification in any
class that's different from the
legitimate class and it turns out that
this gives us a way to compute the
average minimum perturbation to achieve
miss classification which is the
definition of robustness and you can see
that for the digits in blue and the
objects in yellow we indeed have a
reduction sorry an increase in the
average minimum perturbation so to
conclude distillation significantly
reduces attack success and yields model
smoothness because of the Jacobian being
smaller we have an easy implementation
all you have to do is loop twice through
training and an acceptable impact on
accuracy even sometimes improving it if
you have any questions I'll be happy to
answer them thank you
hi I'm pen Alaska from my research in
Munich and thanks for very interesting
talk on such an unusual subject does
your method somehow consider the
constraints on the attacker that's a is
an attacker constrained in so obtaining
adversarial samples in a certain number
of iterations so are you asking
basically about the assumptions that we
make on the universal knowledge right so
what what what sounds a little
counterintuitive to me is that your
method essentially is trying to
introduce to increase the smoothness of
the Jacobian and intuitively with very
unstable de kalb an attacker can very
quickly attain his goal yes if the
Jacobian becomes smoother than the
relative effect of the attack would
become smaller that an attacker might be
willing to wait for a long time might
still achieve his goals have you
considered this sort of temporal factor
in the behavior of the attack so this
attack here when you craft an adverse
ample you you do not impact the training
set so the impact of an adverse ample is
instant it's instantly misclassified
what we showed here is that it takes
more perturbation to produce a virtual
samples with distillation we didn't say
that we completely solved a virtual
sample the problem of them all that we
have is that the adversary's need to use
much more perturbation to achieve their
goal okay and are they constrained on
the amount of perturbation there are to
have with a well would it be it depends
on the application data for instance
here for the digits data set if you
start increasing too much the
perturbation you introduce then even as
a human
the the you will miss classify them
because there are so much noise in the
image so we that's why the adversary is
interested in minimizing the
perturbation okay oh thanks
thank you do you have any backups lights
that you where you can show us what the
what the perturb minimum perturbed ones
look like with your system what the ones
and what the ones I do not have that do
do humans look at them and say oh that's
a nine instead of it's a one we haven't
studied that with distillation we in
previous work we studied the perception
of humans on virtual samples what we saw
was that they would miss classify them
above a certain threshold of
perturbation and the defense here is
basically bringing the minimum average
perturbation to that threshold so I
assumed that if you were to repeat the
same study on human perception you would
find that humans would find these
samples is not acceptable as well thanks
thank you just a question for the
district model so do you give the neural
network description also to the
adversary oh I'm sorry what is the exact
knowledge of the adversary do you give
him the description of the so this the
attack that we use to evaluate the
defense assumes that the adversary has
access to the model parameters and the
architecture however in more recent work
we show that this knowledge is not
required to conduct the attack and as a
matter of fact adversary's can conduct
blackbox attacks on any machine learning
technique by training substitute models
for their target so you can ask a very
quick question and we'll move on so the
applications that you focused on were
computer vision tasks
and I'm wondering how this would work
with other sorts of classification like
suppose just looking at the sensors on
your SmartWatch or smartphone and doing
activity recognition so there I haven't
made that public yet but I have studied
attacks on malware classifiers and the
attack is still possible if the neural
network is trying to detect malware you
can invade it this way I haven't studied
the impact of the defense yet