ellis cose nephews
good morning this is for signifier
Scrivener I'm Alfonso Moniz and we're
gonna talk to you a little bit about AI
which seems an important subject for
everybody but related to computer
security we're talking about offensive
and defensive security with machine
learning that's the title of offensive
security because we're seeing that many
attacks arriving we'll look and see
whether we can be defended from them
that's a little bit of information about
who we are
I'm Alfonzo and you can see the other
person with me is for Cigna Thea so
there are contact details if you need to
contact us you can contact us by email
or on our Twitter accounts
what's our aim a github published if you
want to replicate anything that we've
done what we want to do is to remove all
the worries the concerns the mist that
surrounds this subject we'll look at the
different issues theoretical and
practical and we're going to spend some
time especially talking about all the
different attacks on artificial
intelligence systems some are practical
they're tools that can be downloaded and
can be plugged in and others that are
still theoretical but just go to show
what can be done so we're going to do
the following firstly we're gonna give
you an overview of the context of
artificial intelligence so
if somebody's new to this world and
there's nothing about artificial
intelligence just a brief context
setting we'll talk about what makes a
right unique will then have Jose giving
you a little more practical approach
we're talking about machine learning or
give some demos we'll see how your own
once we've done this context will then
move on to security we'll talk about the
limitations of context security and
we'll give you a brief overview what's
being done in defensive security and
with those limitations and context we'll
move on to what they what's less spoken
about which is offensive as security
we'll talk about all difference black
box white box attacks a I be safe from a
computer safety viewpoint well then draw
some conclusions so the context as you
probably know artificial intelligence is
a machine learning it is part of
artificial intelligence and what it
allows you to do is via software allow
machines to learn independently in to a
certain extent we're trying to imitate
the reasoning of human behavior but
automatically and in fact this kind of
technology is so important and I don't
know if you see companies in many
sectors are trying to use this
technology to
which is how to play AI to cybersecurity
and I say such ly anecdotic all because
the control we have it is very limited
because there are many things that mean
that in our field it's very difficult to
apply this so if you look at this table
there's a lot of information about tools
environments etc if you know and you've
worked with these technologies this is a
very good summary of how they work what
is AI and why is everybody so excited
about this technology we think this
slide summarizes it quite clearly
there's a whole series of apps that
actually profit from the possibility of
automating certain human behavior with
certain different techniques that are
already known there are many scenarios
where this is successful so we've got
the classics fraud detection image
classification recommendation systems a
whole series in the prying to reach
common style they show many of the
mistakes of a
recent decades cybersecurity how things
could go badly wrong if we don't do
things properly typical written about
machine learning they try to bring
together information about this subject
and another problem that many people are
concerned about is its impact of impact
of AI on the economy and society and in
fact if if you've read some people are
talking about charging robots taxes and
where the robots are going to take our
work away from us but what we're worried
about is not just the economic side of
things but the societal side of things
whether these areas may bring out the
worst in human beings they may end up
becoming us making us more racist
because it's been shown that terrible
things may happen for example BOTS that
identify people by race and that
encourage them to be racist so what
we're saying is that we need to be very
careful with the technology and at the
end of this of the day what it's
summarizing is the following
because a eye is what's called a
singularity so that's the time when this
kind of AI can solve a problem as
efficiently as a human being when is
this going to happen well it depends on
many things the problem that we have is
as human beings that we're very bad at
predicting the future we virtually
always get things wrong in fact nobody
knows when this
is really going to happen the only thing
that we can refer to is some service
carried out by the future of humanity
Institute and here what they've said is
that between now and 40 years hence many
activities that humans do today will be
done by AI at least equally as well as
humans and they've given for example a
20 year time span for more or less at
the time when the majority of tasks that
humans carried out will be able to be
carried out by a AI my robots we're
talking about surgery talk about all
sorts of things so this is the uncertain
scenario that we find ourselves in and
what makes people so concerned about
these kinds of technologies we we don't
know is when these kind of algorithms
are going to be able to replace the
ability of a senior analyst for example
of testing reversing etc so our specific
opinion on this is that if you've got a
good security and analysts normally AI
isn't going to replace that person but
what AI has is it can be replicated for
example I may have a very good person
that's very good at doing reversing but
I don't have five people or ten people
or 100 people that could twenty
four-hours-a-day
repeat that so what's good about a is
its scalability it may not be as good as
human beings but as long as they're
reasonably good too they're gonna be
able to replace many tasks that we do as
I said earlier the main problem is that
people don't really know where this road
is heading in fact the onus of major
organization such as Tesla and Facebook
are continually debating whether this is
gonna have an impact or not and what its
reach will be and there's a great deal
of uncertainty once again let's now we
quickly look
with a couple of overheads of justice or
fairness are you to what extent AI is so
powerful and whether it will affect
security here
we've given you a series of examples
especially from video games which is the
typical one that people always struck by
to see how this progress has developed
and here you can see how in I think the
most recent 20 years AI has made fairly
slow progress it's dealt with certain
specific problems some of them it's
resolved very well but it can't be used
as generally as human intelligence so if
you look at artificial intelligence you
can see actually really dates from the
1950s but it's over the last 20 years
that things have been done but not
perhaps as quickly as many people might
think in fact this is a typical example
that people are always struck by these
are different simulations that's really
made of AI playing video games and you
can prove that for many video games ai
plays better than a human being so you
can have algorithms made that actually
play better than a human so for many
scenario Keith's are persecuted maybe
for security these are interesting we're
not worried about these games because
it's just an anecdote but what people
begin to get worried about is the
ability to play games and develop AI
algorithms that actually reflect human
intelligence and an example of this or
or a disruptive point was with the
different bets that are made to beat for
example alpha go zero
so today things are evolving of former
rapidly we can do very good specific
algorithms we can develop them and we
get the results that we all know about
with them I like this study carried out
by Google especially I don't know if
you've seen it what Google did was an
experiment with urine neuronal networks
in blue and red you've got two neurons
neuronal networks that are competing
they represent apples what happens is
the neuron neuronal networks as there
are lots of apples they collaborate that
is they they decide to collaborate if
there are lots of apples but when there
are few apples they decide they have to
compete so they attack each other these
red dots are represent but there's
another scenario that is if you're faced
with different neuronal networks one
hears greater computation or ability
other so therefore it can train rapidly
than the other one it doesn't matter how
many apples there are they'll always
attack then never collaborate and that
is deduced by the network and that's
what many people are concerned about
what extent a I can be controlled and
from a security viewpoint it's okay i if
I use this kind of method for defensive
security is it going to turn on me
that's what we want to know and here I'm
gonna give you a couple of references to
articles talking about these subjects I
actually recommend you watch this a
video a video published by C seek
there's also an article about how Google
has got things wrong and ask about I
mentioned earlier AI is useful at the
end of the day but we need to remember
in what context in what domain
it should be applied or not and
finishing this section about context and
I'll move on to the more technical side
of things there are some aspects which
some people think about forget which for
us in the security world is very
important which is everything related to
legislation there are scenarios where
even if we had AI algorithms that were
better than human behavior they cannot
be used because legislation is going to
prevent us from doing from let's imagine
a bank that has to give a loan the data
protection law may force that those
decisions have that impact a person's
normal daily life have to be reviewed
especially if they impact for example a
URL and that's because you can have a
black box made you can have inputs but
not outputs and you don't know why
inputs don't lead to outputs because
there's no reasoning so there many
scenarios in which this technology
cannot be used if we want the best
security system it may well be that we
can actually use it because of legal
problems having given you context let's
now move on to the technical side of
things let's talk about machine learning
machine learning are basically data and
algorithms you have seen data and
algorithm is applied to a data and we
obtain a model a model that allows us to
carry out different tasks the data can
be in any kind here they represent as
binary data but they can be text images
audio recordings whatever we want takes
us onto algorithms we can classify the
algorithms depending upon the type of
learning and the task that they saw
let's look at learning first there are
different times there's supervised non
supervised and back up learning
the supervisor and what it tries to do
is to learn a task through tagged data
if we want to identify for example
pictures of dogs and cats in each
picture we add a tag a tag to say
whether it's a dog or a cat so what this
function does is it learns from these
tax unlike non supervised learning where
those tags don't exist the aim of this
kind of learning is to obtain the
internal structure of the data and
finally back up learning which is
different to the other two and in this
case we've got a robot in a certain
environment and this agent of robots
knows the states and carries out
different actions for every action that
it carries out in a state it receives a
reward and the idea of this agent is to
maximize the reward we can also classify
things according to the task that's
solved machine learning can be applied
to different kind of applications but it
can be grouped into four different types
which are called tasks classification
regression clustering and dimensionality
reduction classification and regression
our part of supervised learning and the
other 2 are non supervised learning so
what happens here you have a series of
predefined objects or classes like the
cats and dogs
as we said previously and the idea is to
differentiate between these two classes
between the cats and the dogs regression
in the case this case would have a
finite group of classes but rather an
infinite interval so we try to approach
a function to these data cholesterin
involves grouping together objects
depending upon their properties to give
a typical example we've got market
segmentation for example if we've got a
million customers now we want to offer
these customers a tailor-made tariff for
each one that's impossible
so we segment the market into four or
five user groups and each user group is
offered a certain tariff a certain rate
and then dimensionality reduction
normally we have data with some
variables but there's lots of data some
may be redundant and in the case of
machine learning training this can take
its time because if you reduce all these
variables you can lose information but
this is a way of making training more
efficient each type has several
algorithms so what we've done is search
in kaggle for a page with different
machine learning competitions are
carried out and here they've carried out
a survey where many questions are asked
here they're just two the first question
is what
data science methods are used at work
for those that use Kegel and what tools
are used at work and if we look the ones
that are most use are up there on the
screen you've got logistic regression
and decision trees random forest neural
networks and as for programming
languages the one that's mostly used is
Python followed by R which is used by
mathematicians and statisticians then
we've got others such as Jupiter
notebooks or elearning frames
[Music]
the major the problem of machine
learning isn't so much algorithms but
where they get their data from either
because we need to apply or we want to
make an application for something but we
don't have the data because the data
aren't publish because a very bad dirty
data so there are a series of data
repositories where you can see the kinds
of data each data type has different
categories maybe the the most
interesting one is is the cattle data
sets because they do all kinds of
competitions and all kinds of themes
also the data from the web service from
the US and from Amazon is also
interesting and also sick repo has also
have several security data sets one of
the data kinds most used in security or
machine learning in general are pictures
images and here you can see a series of
repositories on the scheme screen and
normally in these data sets what's
happened is you get a comparison between
different algorithms there's a benchmark
to see whether the algorithms are
working or any good one of the most
important monomers typical which is all
very small which is M NIST which are
handwritten digits you can see it on the
screen here what we try to do is to
classify each of these digits from
naught to 9 for example also see for 10
and see 400 or we don't have any digits
but in this case we've got pictures of
objects and depending on whether it's a
C for 10 or C 400 you've got either 10
types of objects or 100 types of objects
another set is celibate with 200,000
images of famous faces with over 40
attributes and other databases such as
image net
Cocco dataset okay
now we've seen that machine learning is
datasets and algorithms but then how do
we make something out of that can we
apply the algorithm you can have data
and algorithms okay on the screen you
can see that so during the data stage
you have to obtain the data from date
that we generate ourselves through
public repositories of other sources
normally when a database has a set of
datas you have to clean them because
some data are dirty you have to get rid
of data that are atypical some data are
missing once you've cleaned the data
what we have to do is to reduce their
size through the algorithms through
statistical applications or through
expert intervention so that's the data
once you've got the data and then you
move on to algorithms and depending upon
the applications you want to create
whether we want to solve a
classification or regression problem
what we just recreate an algorithm and
normally these algorithms have several
one or several parameters that need to
be set initially how do you do that
either through trial and error or
through initial studying and here we're
ready to train our algorithm then we can
divide our datasets into two or three
parts the majority of data I used it for
training
and the remainders used for test and
validation although of course validation
depends on what we're going to be doing
and those parts that are used to train
the model are called the training
reports and for the other two parts of
our data set that's what our trained
model is based on and we get a series of
metrics that we've pre-established early
and depending upon the application may
be error positive data phases depending
on the metrics that we achieve what if
they're correct if we've got for example
you know if we decide that our model
will be valid if we get 90% precision
and if from the metrics so that we get
from where we've got 95% precision then
we'll deploy the model if they're not
valid then you have to either choose
algorithm or change the parameters just
something to add to what Jorge said and
somebody may be wondering what do I want
to know this for I'm not interested in
hacking into an AE system it's like you
have to study all sorts of things
protocols and architecture and it's
seriously you can understand what's
going to happen after you need to
understand this first because this cycle
is very important because if we
understand the cycle we can see where
the access points are for attacks so
it's important to have this mix of
people in your tape team of those that
are AI experts and those are the
security experts so as I said earlier
majority problem at data with algorithms
ransom any problem because the majority
of algorithms in several libraries the
majority you can see some of them of the
majority are in Python because that's
the language that are used pythons are
used to program it's the majority are in
Python but one of the ones that we like
a lot is ticket learn and hto AI if
you're more big data or it is you've got
this spark em lib and then you've got
other libraries such as WIC I'm at my
house well so here they are up on the
screen for you you've got deep learning
frameworks as well wear classic machine
learn algorithms can be applied although
they're not specifically designed for
that there's not what the one that we
like a lot which is chaos and chaos as
some celeritas of deep learning together
with some frameworks such as tensor fern
theano so for the things that we're
going to be seeing afterwards for deep
learning this is a very interesting free
mug framework got a very simple one
apart from a thing that we've said we
can run this locally as you see later
these algorithms are difficult so we've
got specific services in the specific
cloud services where we can train models
the transparent cloud for us or we can
type in the code upload it into the
cloud with our data and run it all on
the cloud
that happens with cloud platform Amazon
Web service and other specific service
such as flight hub and big ml we'll see
something later with fluid hub which
allows you a free account to trial
things but we'll see that afterwards
then we'll see a demo so this is the
part of the deep learning which is a
subsets of machine learning and then
we'll talk about a gang neural network
that is basically used in security
what is the deep learning it's new
neural networks basically let's see it
demo to understand what this is in this
demo you see the basic concepts of a
neural network in deep learning
especially for offensive attacks you
will see the second part are the tool to
see how we can cheat classification
systems a neural networks is a set of
layers composed of neurons what is a
neuron will define it very quickly it's
a set of entries and in inputs and
outputs and the outputs go to the next
layer these entries and exits are
weights and the goal of the new world
networks is to adjust these waves so the
loss functions are minimized in this
will depend on the algorithm that we
apply we have three types of layers the
injuries basically the data that are in
put into the algorithm yeah outputs and
interestingly user of the hidden layers
and this allows us to learn complex
concepts that more layers we have the
deepest the learning that's why it's
called deep learning
and the more complex concepts we can
learn you can use the URL veces
simulator for neural networks if you
want to understand concepts much better
you can use this simulator you can
design neural networks and see the
effects what the algorithm does is to
minimize a loss function here it doesn't
converge but it what should happen here
is that where you can see here in blue
composes a complex circle separating the
orange and the blue classes here they
should be separated let's change the
architecture of this network and
normally something that is quite
difficult to adjust is the number of
hidden layers because we don't know how
many we need how many neurons we need on
on each layer this is one of the biggest
problems we do it with a test and a
trial and error and we adjust the
different number of neurons on each
layer etc let's remove layers let's see
if this works
to train a neural network what we do is
to apply a pox it's the training test
process in whole that we apply we will
get to 500 a pox and 500 times the
training so there's a clear separation
here from the two classes you see the
colors the the orange and the blue class
and the error is 0 so this is a good
classifying system for defensive
security and offensive security the
problems is a pox this is done very
quickly by the similar simulator but
this is that our very high computational
course
so we can do it very many times and and
test and train the model as an offender
and as a defense defenders we can be a
better offender or a better defender we
can play with various parameters for
instance the the training data ratio we
can include noise so that it's not that
easy to classify here you will see that
this is not separated into two
concentric circles here they are just
near the blue ones it's difficult to
classify it with this algorithm we can
also play with the training rage how the
algorithm learns the activation rate we
can also play with the type of problems
of classification and training and let's
see the convergence here this is just
basic new your neural network there's a
training process going on but we haven't
given you any technical details so this
is not too boring otherwise we should
enter into mathematical details that we
don't want to give you here today with
that simulator we've seen the same the
simple model the multi-layer training
but we have other types of neural
networks called
architectures and there are many of them
one of the most popular ones are
recurrent networks even though they're
not used in that way we use variants
like PS the HTML group the convolute
conversational networks and there's Jam
Network that we use in security the OT
encoders here's the link where you can
find the detail on the full architecture
of these networks the description and
what the purpose is let's see what are
the
ganz j-a ends this is an interesting
concept for attack and defense it's not
a neural networks that you generator and
discriminator how does it work
when the generator gets some noise and
it generates false examples and the
discriminator who gets the data
generated by the generator plus real
examples in such a way that a generator
and discriminator compete the general
rated tries to cheat on the
discrimination and the discriminated
tribes to defend from the generator the
generator tries to generate more and
better examples real examples real
generated samples and the discriminator
tries to distinguish between the real
and the generated this game finishes
when they get to balance the generator
cannot generate better detail data - for
the discriminator cannot distinguish
between the real samples and the
generated samples or the fake samples
this is the original paper - the link
where you can see all the specific
details of the loss function studied
what uses or how is it it is implemented
because we're not going to elaborate on
this this idea of the Gann Network is
very interesting indeed and there are
many variations out of the original idea
the Gann networks were created in 2014
and this is the number of papers
published per month where the word Gann
appears so it is growing exponentially
the director of a research in Facebook
Jana Luca thinks that this type of
networks are the most interesting ideas
in the last 10 years in machine learning
and going back to the deep learning
the neural network concepts are not new
this is a paper where the first neural
network was defined based on the
biological functioning and it goes back
to the 40s and other articles on the
earliest TM it's a recurrent network
created in 97 and the convolutional
networks were created in the 80's 90's
why is it so fashionable now for two
reasons because of data 20 or 30 years
ago when architectures were defined we
didn't have so many data but now we are
in the time of big data so we can use a
big amount of data and there's a second
reason the greater computational
capacity neural networks require many
date data and the more layers are more
hidden layers and more data you will
have the training is quite expensive so
we have GPUs and graphic cards and
because they're less expensive we can
train very complex models in reasonable
times so deep learning is fashionable
now for those two reasons because there
will have more and more data big data
that we can use and because we can have
more computational capacity
thanks to GPUs and the lowest prices
let's see neural networks examples
so we can see the examples it's quite
easy let's see the Stone Paper Scissors
game this is the example with a browser
we can play rock-paper-scissors against
the machine this example is quite
interesting and we have to see it from
the technological point of view some
years ago this was it just incredible to
apply this type of intelligence in a web
browser with libraries that allow you to
do this it's a very short example you
play rock-paper-scissors against a
machine tie what is its weighting is the
rock to start put the game again then
you lose if you don't fit and you don't
have friends you can play against your
computer it's so funny and so you can
spend a whole afternoon doing this it's
just an example this is just to show you
an example of neural networks another
example a very typical one you know
Tesla that is developing the autonomous
car and NVIDIA is another company which
is now developing applications and in
their own autonomous car and this is an
interesting video for two reasons the
first one is because companies don't
tell us how they trade the cars all of
them the errors that are made until it
works and these companies never tell us
about that
and yes this that the courageous person
inside the car when he or she trains the
car before it works and then there's
another approach the traditional machine
learning approaches that we haven't a
complex problem and then we divided into
sub sub tasks
and in for each task that we apply a
specific model here we have e to e deep
learning approach we train the car as a
whole and this gives us gives it a
certain robustness more robustness and
then if we had it done it in an
abandoned away and they apply
computational networks 30 years ago
these networks existed already another
company that is working for the
autonomous goes bye-bye dude it's the
Chinese Google and it's putting a lot of
money to develop the autonomous car but
what's interesting in Baidu project is
that it's a pen sous project so anybody
can download the software you and they
give you the hard work to train you to
Animus car and then you can create your
own autonomous car without investing
money on hardware because they give it
to you the Baidu idea is that in 2020
December 2020 this type of a operative
system can work in highways and open
roads
let's see another demo of the
convolutional network example it's the
the style transfer is it's the last one
and then we'll get to the security
applications
one of these services in fact we haven't
said it we have published it in in
Google we have a github we have a lot of
you have a lot of papers we have the
github you can use it the idea of this
demo
it's the convolutional Network examples
you can have so make examples of images
and frame it's a frame well a certain
style with specific colors the paintings
used specific colors the the lines and
then you can do a transfer from the
style of the of this painting on the
images let's do it let's apply freehub
fluid hub is a cloud service you can use
libraries in a simple way you can't do
don't need to install them locally you
can have a CPU a GPU on demand what's
great in this is that you have to three
hours of GPU for these tests we are now
using the free account that you have to
be you hours for free this is step of
graphic cards and you don't need to use
your credit card to open an account
this is fluid hot what fluid help can
you help you to do is to have your own
codes and the tab set data sets
data data sets that you can use in an
easy way here two data sets for size
pre-rendered styles with frames so here
we have six of them
that's the names and then you can see
that the images or pictures on which we
will apply this specific style this
algorithm so we can create our own
project and within each project we can
execute jobs it's just to upload and
apply a code you can execute the code on
specific data let's transfer the style
what we do is in github how we do it
let's transfer the style for the code
and we use the CLI that is provided to
us by fluid hub so with a very simple
command we can transfer the style or to
execute what we want so it uploads all
the code it's important to separate the
code from the data because if we have a
very big set of data it's going to
upload every time and it can be quite
long according to your power so we
execute this the Freud it's the command
line and we execute it they the tensor
flow environment with version 1 3 and
python 2 and that will apply to datasets
the ones I showed you before security
and M blend
data sets and the style what's
interesting in fluid hub is that you can
see the data set we can create various
versions of our data set and we mounted
on styles in the same way we mount our
photos on in the directory called photos
or pictures we execute
this with Pythian environment where this
which is already installed with this
command evaluated octopi and we say that
it has to continue and to apply this
style wave on the directory called
pictures and then it's going to be put
into a special directory in fluid hub
that allows us to have the outputs or
the exits we have different sizes in our
pictures so we can have different sizes
at the end and for each picture we do it
one by one let's see let's see all those
times that is are generated here all the
jobs
here should have finished but still it's
still running anyway let's show them the
images that we have on our base I don't
want to waste time but what it generates
in fluid job with fluid hobbies these
pictures this is the wave style that we
have applied and this is the result this
is what is it's generated for each of
the images the original a picture while
the style is applied to the the photos
we can do it for each of them we have
predefined styles but but it's quite
costly it can take between four to six
hours according to your network so fine
this is half of our told thank you for
being here there's the most intense part
but it's important we can't speak about
AI in cybersecurity without knowing
these basic concepts with this basis
we're going to eat about cybersecurity
what can we do with this what is being
done with other than the constraints
several months ago I heard a talk by
Samir and he said a sentence and because
of this sentence he said we are here
today giving you this talk Adi Shamir he
is tiptoeing tog refer one of the most
brilliant people that we have on our
planet said this sentence I think
machine intelligence is useful for
defense but not offense in this hour
well when we get to the end I have
reached the conclusion that this is just
the opposite it's more useful in offense
than any defense we see that but if
Shamir sets a said that it's because
there is a contact something that we
can't can't grab something is
moving and that's what tribe we try to
compile all the tools that I exist that
not many and let's try to explain it in
a more human way what is being done in
theory is a cybersecurity what are the
problems in cybersecurity that humans
can't solve and what about AI is able to
complement it this is a great list of
problems to such a solve and we're not
able really to solve this yeah just an
some of them the software exploitation
problems systems are vulnerable because
we make very bad software for many
reasons here and this is violating their
systems although in fact we don't know
how to configure the systems and then
also because there's a failure safety
failure security failure is us we are
the reason why when we speak about
applic applying a is security many
people would say that they know how to
apply but many many problems out there
and people don't say anything about that
and they just they keep quiet it's about
the amount and quality of data if we ask
somebody how many data you need to solve
a specific problem that the answer is
the more data have the best but that
answer is not very scientific normally
they say ok maybe we don't need a lot of
data but we want quality data what does
it mean you apply all the domain domains
but in the case of security there are
scenarios what this is useless we have a
threat that we've never detected before
how can you detect a threat that has
never been detected people who say oh
it's easy you are no lies what it's
normal and what it is strange well
there's an alert there they solve this
alert in a human being and you know
wasting your time do you know if this is
reasonable or not we don't know to do
that better than 20 years ago we're not
making progress there's another problem
it's a
the static training problem this is
useful in certain scenarios you have a
model you can train it you classify and
that's it but that's not really in
security because the attacker is
evolving and is changing you have a
model to detect and specific attacks but
the attacker is now doing it in a
different way your system would be able
to make continuous training continues
training in the presence of an attacker
it's about gamm but there are no
solutions for that it's not me saying
it's my friend DARPA I can't justify
that there's no solution for that then
there are these two points that are
really worrying the application there
are attacks to steal AI models so you
can attack in an offensive way so the
idea of this talk is that offensive
security is real and offensive and
defensive security has to learn from
that and this is something important for
us if you use AI in security technology
or in any with product this element has
to be audited from the IT security side
when we do a reverse in the Tetra it's
something they'll have to add have a lot
of things to do but this is essential
for a security professional the greater
problem has no solution and couple of
months DARPA this network has opened up
new competitions if you have a solution
you'll get millions of dollars from this
agency DARPA to make AI systems that can
be learning in a continuous way because
there's an adversary there's a rival
there this the attacker it's not about a
static model era change from now and
then
no because the attacker
is learning how to cheat on these
systems so here we go back to what Xhosa
was saying before it's the gang networks
in security this scenario is very well
known for many months we know that the
attacker lives with us the attacker is
in our heating system is it at home with
us the attacker is living with us and we
need to adapt to that person being there
it's not a medieval type of defense with
a wall in fact the attacker is with us
in the happiness in our home so you have
to use the attacker to protect your
system so that your AI system is more
robust one of the most complex things to
do in AI today and with no clear
solution is how to apply privacy as
course I said to train the systems we
don't have the calculation capacity we
need to have a Google Amazon or third
party so we are we upload our motto it
could be trained or not and it's there
how do we know that a third party is
stealing this module how do we know that
these people are not modifying a model
with backdoors we don't have a solution
we have other proposals the academic
repose especially with neural networks
will ease ciphered miss eyford neural
networks you can upload everything on
the public cloud and if people steal it
from you it's ciphered they cannot make
consultations and treesa
it's because it's encrypted because they
need and a cryptographic key it's quite
interesting because it's the only place
an effective solution to this type of
stealing model attacks you could put
crypto net and this can help you will
find more information let's see aspects
of defensive security as an introduction
to offensive security
so how do we use AI in defensive
security I've been working on this for
20 years these are the scenarios in
defensive security normally it's used in
access control or sanctification of the
relation identification and profiling
it's always been used in or related to
traffic registrations IPSS ideas is logs
fising detection and malware prevention
and fraud detection and it's been used
in AI I don't know how to say it it's
it's more less professional with static
training processes etc but there are
other scenarios where this is applied
especially in software development risk
development and security in operations
with the data yes we probably don't have
data cos if something is attacked
usually the attacked actually changes so
if you model your attacker on what's
happened in the past that's not going to
help you predict the future and
everything related to automatic training
there increasing number of proposals to
collect information accessible
information related to defensive
security and one of the most interesting
is called ml SEC what it tries to do is
to collect up all sorts of projects and
AI tools just for defensive security so
if you're interested specifically in
that niche have a look at it and you'll
be able to find different ad effort
torques hacking solution talks in this
slide and I'm not sure if you can see
the cyber part of it it's in red here
what I'm trying to show you is that all
large companies are concentrating on
using AI for security programs for
monitoring for traffic analysis and for
malware detect
and there's some other actors you know
small startup smaller companies mainly
from the UK and Israel and the US these
are normally the birthplaces of the
major advances that try to take
different approaches but the majority
that we know of because many of these
progress of course are kept secret of
course they'd want to so they can't be
hacked they want to keep it for their
clients and actually the majority of
them aren't doing anything new that then
what's been done in the past but I'm
going to give you a couple of examples
here of interesting different kinds of
things are being done in defensive
security and then we'll move on to
offensive security and the two examples
that I want to show you are related to
data protection one is related to
cryptography and I'll just tell you
about it and if you read a while ago
that Google had invented a single use
algorithm encrypted algorithm if you
know a bit about cryptography the
problem for the next decades is the
progress that's going to be made in
Quantic computing Quantic computing is
going to create probe problems in
certain cryptographic algorithms so what
Google said is that maybe we could use
new neural networks to generate
character graphic networks and we can
change the keys in each session in each
process and what Google says is that if
you can create a cryptographic algorithm
with a similar level to the one that a
human will make with with just one
single use if we've got a computer
that's powerful we make that may be
sufficient
Soviet to pretty says so what did Google
do it looked at the typical scenario we
have Alice and Bob and then you've got
the spy Eve different neural networks
that trip were trained to try to do
deduce their own algorithm the only
condition here is that the attacker has
to be more and more clumsy when
deciphering or when trying to understand
the cryptic messages so they're trained
and after a series of attempts Alice and
Bob deduce an algorithm that Eve isn't
able to clarify it's quite a
revolutionary idea what's the problem if
you look at it in detail what the neural
network finds is that a primary school
child could do a better algorithm it's
not very sophisticated but there are
some properties of human cryptography
that can deduce this for example the
Avalanche effect that you get in
cryptography so the certain things that
neural networks can detect it's very
promising and we'll ever be able to do
this truly in the future but it's an
interesting idea but it's not what has
been published it's just an experiment
but it's a trend that's and shows how
large companies such groups such as
Google are trying to broach security in
this way here we put an example and this
is a steganographic example what we've
got is an attacker who's not aware of a
hidden communication we've got a gang
type setting we've got images that are
generated and here we can have a
discriminator it's going to say to what
extent the image that is included is
real or not this is perhaps a classic
way of trying to deceive of
amplification systems but basically here
we use something more simple which is to
hide information and also they add a
steganographic analyzer what they're
trying to say
is that this can be detected with
current steganographic algorithms what
this does is with this neural networks
you can generate hidden more robust
algorithms and the ones existed today
and can deal with the most robust
attacks so with the same setup in the
neural networks you can make new
defensive security proposals analyze
traffic static training etc this is a
great context it's making process we're
trying to go beyond the limitations but
what happens if we try to do this in
reality if we want to introduce an
attacker in this kind of scenario which
is that's when problems crop up and the
next slide shows different attacks that
we've collected in offensive security in
AI that is hard to attack systems that
use AI and we've divided them into four
blocks classic attacks for example the
typical hacking attacks that use AI and
therefore work better what we've got
synthetic attacks which are basically
attacks that use image processing that
are used for attacking AI algorithms the
most interesting ones which is AI
against AI and then a mixed bag
wellwe've includes all sorts of things
model robbery backdoors etc etc but
anyway if you try to assess the security
of these systems you always have to
think about this diagram it's just as if
you are pen testing a black box what you
have to do is to think about what an
attacker can do if they only know inputs
or only know outputs
what about if they can affect the
training stage or what about if they can
only do it in a test stage what can I
take I do if they only know the model
and what about if they can manipulate
the algorithm either remotely with
inputs or physically by introducing
electric modifications or power
modifications so let's go point by point
firstly classic attacks yeah these are
just a couple of examples and this one
at the top pass can is how to use neural
networks for a password diction has been
stolen from linked in the neural network
is able to deduce a rule a rule for
creating passwords this rule can
actually do deduce new passwords that
look similar to the others
that's what pass can does but what does
this tool used for it shows that we can
create quicker passwords so in the real
world this tool gives better results
than the classic ones that we get
another rapid example this is from this
summer in which deep learning was used
for detecting vulnerabilities in systems
and in these use detecting certain
parameters such as a table etc etc you
can do that with statistics you could do
it with brute force but what we saw with
this investigations that using neural
networks you can get the same results
with less effort and this can be
interesting to use in deep learning so
there are many examples
of time we've just got these two about
how you can improve the classic attacks
code facing phishing using AI we've got
another specific example which isn't
like the previous one in which a rule is
generated what this one is you see a
list of passwords and what this neural
network does is bring out passwords that
literally look like the other ones in
certain scenario they could it could
also be useful so I'll leave you with
the Python that you need to do that and
more information about how to do it
synthetic attacks this is a very very
famous one that was published a lot in
the press I'm not going to spend a lot
of time on it but it's basically how to
use external systems to modify a data
source so that the machine learning
system classifies them wrongly here
we've got face recognition this isn't it
let's look at how what we want to do is
to make the system recognize this girl's
face as if she was this actor and all
you do is put some glasses on her and
the system can get it wrong why why does
this work because this glasses have some
pixels on them so that the training
system fails there are other interesting
ones you may have seen this one which is
to fake videos in real time you get take
the video of a person if you connect as
different pages it's interesting you can
see it with Donald Trump and others what
you do is you get real vidi you take an
actor and the video in real time adapts
to what you do with your face so if you
open your mouth maybe on shorts nigga
will open his mouth as well
and it's very good possibility of making
3d masks using 2d images that's um still
a little bit of a long way off but
things are happening and here we've got
another one
something that we're also concerned
about and many others are concerned
about which is the possibility with just
very few audio of a person's voice they
can synthesize that person for us it can
steal that voice in fact they've done it
with Obama's voice just 30 seconds of
that person's voice you can synthesize
it and to make and make anything it's
kind of frightening probably we won't be
able to differentiate between a real
audio and a real video this is what's
going to happen and we'll see that
coming in the next five years this
reality is very important well we are
going to take more time is other kinds
of attacks such as machine learning
against machine learning and what we've
called mixed attacks here that we wanted
to something that we wanted to highlight
is the fact that an increasing number of
scientists at warning of the difference
between offensive security and the
difference between the more academic
world and the more military world for
example real offensive security is
something there's a real problem we need
to consider when we talk about AI
attacks against AI tax will be both
basically talking about evasion and
poisoning attacks let us talk about a
evasion attacks
for example the system classifies
something as smart where as not smart
where I but these types of attacks use
evasion tech techniques and poisoning
techniques which is basically to put
poison in the system to produce
different effects of all of this there's
a lot of theory behind many papers have
been written about it a lot of
information but increasingly if
information is being published in
libraries in at a more practical level
so that you can understand it here we've
got a couple of themes some of them are
the most important ones here we'll see a
couple of videos with demos we've got
clever hands and deep wing let's look at
different demos with clever hands and
deep wing and I'll make a couple of
comments about them let's look at care
of hands first these kind of attacks -
attacks are from white boxes the first
to a white box attacks and the final is
a black book attack black box attacks
basically what these algorithms do is an
adversary in a case of the images you've
got a different image which is to the
human eye is almost the same as the
original one but if you apply it to a
machine learning algorithm can be
confused when classifying so let's use
clever hands sorry
Flo how to simulate this attack and then
we'll have a black box what's this
setting it out I'm going to summarize to
explain how these attacks work if you've
got a white box attack basically what
we're gonna be able to do is have
massive attacks and directed attacks
well with a white box attack we need to
know what the training model is and the
setup parameters are so in this scenario
we get inputs and outputs with that
information we can get the system to
classify things badly in general that's
a massive attack I can just introduce a
zero and get the system to pass by
anything that's not a zero so the system
doesn't understand then we've got
directed attacks and we'll see a video
of a demo that we've prepared for that
in which we can get the system to
classify whatever we want I can put a
zero and the system can classify it as a
one or two or nine these are what we
call white box attacks I'll talk about
the black box once afterwards basically
to run this demo
what we did is as before the only
difference here is that we we have a GPU
and we apply the code in this case we
don't apply data parameter basically
because when we run this code it
downloads it already and as it's very
small it's not necessary so let's see it
here we've got it running
it hasn't started training yet what this
attack is going to do which allows us to
attack and defend them that's what it's
going to do is create a model which it's
good in its own right but the problem is
if we could adversary's if we put at
noise two images it's very bad so what
do we do to improve it we generate with
the algorithm that they use which
basically involves applying noise
according to a specific equation we can
create adversarial examples this is
what's called adversarial training and
this means that our model is more robust
in comparison if we got an adversary
this should have started training esta
monostratos in the Rami ento we've got
our training data
all the datasets we've got we've got
60,000 training data 10,000 test data
we've divided them in this case into two
parts and then we can see how they're
being trained I think it applies six at
six moments the first has six seconds
and it classifies in 98.8% of cases
correctly so it does for each era with
precision so we got all the models the
configuration parameters setup
parameters and then we'll play around
with the inputs and they will modify the
inputs so that the system classify
classifies provides wrongly
all of these examples are on the github
that we've published so you can see them
more easily if you access that
this is without adversary's and what we
do is here is apply a diverse erease the
only thing is in this example there's no
algorithm precision but if I'm not wrong
the original model had a 90 something
percent and if you apply adversarial
training it's around 90 percent so you
lose precision but we make it more
robust when having to face adversary's
so let's now look at the black box for
us black box is a great discovery
because it's a very powerful in a black
box attack we don't need to know
anything about the training environment
the model the only thing that we need to
know are just a series of queries for
the real model that's all you need with
the inputs and outputs because of a some
of the ai ai properties including
transfer meaning we created a substitute
model a new our model locally and with
these inputs and outputs we are able to
train our local model and it's in this
local model that we do our tests as well
we do these adversary examples and we
can see which are the best to have
directed attacks or massive attacks once
we've done that we know what each and
everything is we take those attacks and
we launch them against the real system
so this is quite powerful the different
references that we've given you you can
see that they've been quite successful
it can deceive in a black box without
knowing anything about the system with
just a few queries to the system I'm
going to do the same as before
with the same commander with GPU data
should be downloads they should be
trained with legitimate examples real
examples and train the replacement model
and for them on what's created the
adversary examples is going to take some
time as you can see our test lab k-8 is
used it's an all GPO but it's a a cheap
one
I'm gonna end up in interment a little
bit over and above for the demo this was
just to show you that it's real
so just retain that idea you can do
direct it on mass attacks without
needing to know anything about the
training environment you can attack
directly a real system just considering
inputs and outputs
so you train with a legitimate example
you do different adjustments you create
synthetic data we ask the black box
model what the output is from the data
we've to Jerry so in the end that the
precision with the adverse Cyril
examples goes down to 80% and in this
case as we have less information it's
more difficult to do this in to reduce
precision the challenge with black box
attacks is to reduce the number of
algorithms so that you have the lowest
number of queries the idea is that the
PPT reach 93 94 percents of precision
when deceiving this will depend upon the
number of queries in the case of Amazon
or Google you need 800 queries there you
can do your test for the real system
thank you so that's the white and the
black box the examples that we wanted to
show to you to give you a little bit in
a more visual format let's show you this
one which is a white box example I'll
show you the video here you can see
these simulations they're also in our
github you can see it you can see the
URL with clever hands
we're tensorflow this is a white box you
need the contextual information and what
we do is do input and output attacks and
the tacit is the following we go to an
input we add as less noise as possible
what can be to three percent and we
generate something difficult what we
want to do the system to classify
wrongly as if it's a directed attack so
we've got malware so we just modify so
that the system thinks it's good we're
not malware here you can see different
things that are done
okay I can just briefly explain what
this table means I don't know if you
know what the difference you can see the
differences here but there are 10
columns on the screen 0 1 2 3 4 5 6 7 8
9 10 columns this classification system
takes the 10 numbers what we want to do
is via an input find out precisely how
we can manipulate the input so we can
get the output that we want for whatever
reason maybe we can only add 0 but what
we want is the result to be 0 here you
can see in the table that we don't have
to do anything you've input 0 output 0
but maybe I put a 0 in and I want one
so through this training what I can do
if I just put the 0 in with minimum
disruptions the system will classify it
as 1 1 what this table is telling you is
how to modify and input to get the
output that you want here you can think
of a security domain that you want it
might be a specific tool but this attack
can be done how do you get access to
this table you can do it with white box
attacks if you know the model and the
set up parameters all through a black
box attack either by generating a local
alternative model until you get this
tape up to the table or other kinds of
attacks that you'll see now and this
another one clever hands is the most
developed model especially for digital
images but there's another one that will
show you afterwards that was developed
but actually no more updates new
releases have been uploaded to github
for the last hour a year and a half
generate to make do black box attacks if
you want to protect against attacks of
this type attacks are could advise you
to read this paper how we can generate
generic models in at local level to
attack any AI system without knowing the
configuration mixed attacks stealing we
have to use a I to prove this a
synthetic attacks especially for
authentication biometry systems AI
against a I especially the gang scenario
and then there's a mixed tip type of
attacks here we have almost everything
here we give you two examples one is the
model stealing what we've said is that
the option a you know all the data and
you can launch the attacks option B I
generate an alternative model that's
near disclosed to the original one
there's another option to steal the
model how can you do that in different
ways in this paper you have various real
examples against Google Amazon to steal
their AI models but we wanted to show
you a more human examples you have a
service in which for any reasons this
it's use this AI algorithm for logistic
regression and if you know about machine
learning this is the shape of the
algorithm the attacker what it doesn't
need to do to put the inputs and get the
outputs so with entries and exits what
he can do is to solve specific
parameters of this equation the X here
and the e so according to the dimension
of the other parameters that we need we
of needed to have a lot of a K equations
let's imagine and the formula the
dimension a hundred we need
hundred queries the local we can solve
these equations and we would obtain the
value for each parameter at this moment
we would have stolen the AI model if we
have the AI model we can compute the
various attacks that are possible so if
one day you use a service of this tech
or if you show it to someone via API
many attacks can come because there's a
person making the consultation and
having the exit the exits so you can
imagine a system to protect this and
another example for mixed attacks this
is quite interesting indeed it is the
backdoors in AI algorithms this is
related to what we said before where we
upload or training models or do we
depend to do our tasks on a I have
demonstrated that starting from specific
models trained and neural modules we can
add other additional models that act as
actors this picture we have a system
here that is a benign classifier
classifies objects we are able to add
and each additional networks the system
works in the same way as usual but if we
have specific scenarios the system
behaves differently where the backdoor
classifier so we can do it in an
additional way merging layers or adding
layers to the system just another clear
example do you understand what it means
this is a scenario we have a a I system
that classifies traffic signs and with
with Baidu in in the internet you can
download and use it but in any reason
somebody introduces a packet or in this
model what happens in this scenario
there's a car and normally the stop sign
is identified
as stopped and the car would stop but if
you put a little sticker on the stop
sign it activates the additional neural
network and the back of the back door is
activated and the car will not stop the
car will accelerate it's a visual
example have to be careful with model
privacy because today we know that
backdoors can be inserted there's an
additional problem here the transfer
learning we this technology is advanced
any people are sharing the information
this is another scenario there's a
university or research center it's
creating an algorithm to detect objects
in general and for any reason this model
you think it's very good you want to
reuse part of this model and if it's a
neural network you read trenches the
last layers or the last phase instead of
detecting objects in general it can be
used to detect lamps your code is shared
with someone and you have specialized
this model have to be careful in this
scenario too because with this transfer
learning concept well you have a general
model you're introducing a back door and
somebody's specialized in these algal
right algorithm and then you can take
the back door with you you don't remove
the back door so if you use this type of
models because we haven't detected
anybody using this in practice but
theoretically you can do it so you have
third-party models there's a risk that
can be added there the back door many
other examples here let's talk about
formalization I think this is the most
important part for us in security if we
apply a I to any scenario more
specifically to safety or security
technology we need to know if machine
learning is safe or not if you you know
about cryptography you know that there
are scenarios where we can use unsafe so
technology but if we combine it with
unsafe security the global result is
safe in cryptography we can use hash
functions that are vulnerable to
collision attacks we know that we have
two different entries for this algorithm
the result is the same there are
combinations in which we can combine
more than one algorithm that is
vulnerable md's five and one so an
attacker chooses x and y entry and it's
very difficult to find an XY that
collisions in the two functions at the
same time it's a classic example of
cryptography this is one a proposal in
AI and it's AI can be vulnerable to
offense but if you put two similar ones
the same attack is not working for the
global system if this is a solution that
is homemade but in many scenarios this
can work this is an alternative in the
more theoretical theoretical world how
could we go deeper into detection
functions to be have robust algorithms
against offense
we will have time for questions so we
will get into the to the end and there
here you'll find a series of
recommendations offensive security is
important in AI and we'll have to add it
as a cycle so in AI we have to take care
of that it's difficult to protect
oneself against a blackbox attacks it's
quite difficult so it's possible we
should try and make a global or overall
architecture model and consider AI as a
one more element it's not a solution
it's just one more element the security
we need to manage risk that's typical
and take and take it into account
privacy how this can modify models and
how they are going to be executed
so this picture it's not very nice
it is this the whole cycle and you have
to include the security layer how can we
and where can we introduce it in many
phases but especially when you devise
more safe systems you have to include
included in this in the trip to the
training phase here you need to
introduce the tools and see how this
cycle cycle behaves in the presence of
attacks this is the summary and now
conclusions the goal here is to to know
more about AI in security and talk about
the limitations of security in AI we
don't know how to solve this we don't
know what is a choice that is going to
be made what are the proposals for
solutions specific algorithms in the
future but we have to take into account
that offensive Security's advancing very
rapidly especially the black box attacks
in the last two years they're very
powerful so we have to consider this if
we design a specific system to be robust
and this is the sentence by Nick Bostrom
that we like very much so we depend on
AI because it's true more and more but
we have to work on it little by little
and then this is the series of workshops
where these topics are being dealt with
it's about machine learning and IT
security one has has already taken place
in November the third of November and
another one next week it's one of the
most important conferences for neural
networks nibs well it's about attack
defense privacy
so there's an emphasis that is put on
not not just security but machine
learning experts are realizing that
these algorithms should be protected
because they are all over the place in
the future they'll be with us so we need
to design new ways and today they're
more theoretical but maybe in some
months or years they'll be more
practical we have to take into account
security for AI and this is a list of
topics about machine learning what you
want to learn more this is the detail
the details of the attacks that we have
describes here you have courses -
tutorials about machine load the most
famous one is done in Stanford they have
deep learning specialization with five
courses where the neural neural networks
are included recurrent convolutional
networks
Jude acity more specialized if you want
to learn more about deep learning or
machine learning you'll find information
here in this courses there are
conferences world level to know what is
being done and what are the last
developments in machine learning nips
2017 ml comes that's more theoretical
and I see ml that's the companies and
this is a list of companies that are in
investing quite a lot of money in AI and
machine learning and that's all folks
and if you work in security especially
global and architectural and design
security the problem is the appearance
of new elements
ai is one of them and to understand the
tax you need to be specialized so many
of these courses are basic but this is
not enough you need specialists for
instance we deal with security but we
need experts and work together how to
and learn how to defend yourself and how
to attack because you need to attack in
order to learn how to defend yourself to
have more robust systems thank you very
much for your attention I hope this has
been interesting now we have some
minutes for questions and answers but
we'll be there you can write to us by
email or by Twitter thank you
[Applause]
you