all right uh thank you for coming here
for the last presentation it took us a
while to configure all the stuff so I'll
try to be
quick uh so today I'll tell you about
the uh the field of AI security and
adversarial machine learning um it's
history and current state uh my name is
Eugene and I a CTO and co-founder of a a
company called adversa AI and before
founding adversa uh I managed security
research and development uh with the
focus on transforming research uh into
uh products and I'm also a security
researcher and author of the report that
covers the past 10 years uh of uh
adversarial machine learning so I I'm
using this report today uh for my
presentation and if you want to dive
deeper uh feel free to download it with
with the special link I prepared for
you uh uh a bit about our company so we
are a startup with a mission to increase
trust in AI systems by protecting them
from cyber threats and we are a team of
hackers and researchers and we work in
three directions so all of these things
you can see on the slide are not
products but resources we have created
so you may find them useful uh for
instance our uh AI risk advisory team
has released a report uh on the
adversarial machine learning so uh we
also have a practice iCal course on
attacking AI systems and we also write a
blog post uh blog posts and newsletter
with the reviews of uh research papers
about uh vulnerabilities of neural
networks uh our AA team conducts
vulnerability research and speaks at
conferences with the results of uh
security findings and you can explore
our other
presentations and our AI security
engineering explores ways to automate
security testing
and uh develop protections and we have
recently announced our attack engine for
facial recognition uh systems and also a
sample life cycle for uh secur
systems uh so why should you listen to
this talk first of all uh my main goal
uh for this presentation is to uh get
you up to speed with the current state
of uh machine learning security I'll
give you some historical context uh but
mostly FOC on what is the state of the
field today uh so you clearly won't to
uh learn how to hack systems but you'll
know uh the field the dependencies uh
what and where to look
for uh so I'll break down the main D
into three uh sections uh which is
essentially a stages uh like stages in a
security life cycle when you decide to
start something uh when you decide to
start a security program you start with
the uh threat awareness so you learn
about uh realistic threats uh then with
the risk assessment you evaluate uh the
real security posture and finally with
the security Assurance you develop
mitigations what is discovered
earlier uh first of all let me tell you
a story uh 5 years ago I've been
building a a thread dection uh products
and it was 2016 uh I went to the RC
conference in San fr FR Isco and the
main topic of the year was uh machine
learning for cyber security and and I
was like okay I need this stuff we need
to get our products to the next new
level so uh we decided to uh start
implementing some ml for our th
detection tools and it took us around a
year to formulate the product tasks
collect some data build efficient uh
prototypes and get ready for some real
world testing with the uh with the the
friendly
customers uh but the the only problem
was that in it became fundamentally
broken uh and after some success we have
realized that all models could be easily
fooled so basically uh the product
became less secure with the machine
learning than it was
before uh so we were quite surprised uh
that we were able to bypass uh many
types of thread detection models that we
have built bu I I just want to give you
a couple of examples uh for instance we
have built uh an attack detection for
applications basically some kind of
smart firewall which was based on U uh
machine learning uh and we we were able
to bypass it uh just by changing some
payload parameters or introducing new
types of uh request encodings or just
distraction features so you you can
think of this as a traditional uh
firewall uh detection by pass method
uh plus some uh weights for this methods
when uh ml is
used and another example uh we've been
building a uh user Behavior anomaly
detection and for this anomaly detection
we're able to generate some targeted
events quite slowly so when we uh when
the system conducts some historical
analysis uh it would recalculate
baselines what is considered uh normal
for this user and uh then that way we're
able to uh change our behavior and stay
under the radar for normal detection and
for threat
coration uh so
basically uh this was the moment when I
first got introduced uh to uh
adversarial machine learning and at the
at the time it was quite early years U
of this field and we tried to fix all
these things uh for a while uh but by uh
by doing some some like Common Sense
stuff like changing models or uh
changing data sets like uh introducing
new features augmenting data sets with
the S by passes but after some time we
have realized that uh machine learning
is uh fundamentally vulnerable so there
is no such thing as a security page as
we have in traditional software and
basically with that state of uh AI
security uh many Security Solutions that
use neural networks
uh could be considered useless unless
they uh use use it as uh one of
detection engines not the only
one uh so as I have mentioned uh our
product became uh less reliable after we
added uh machine learning and really if
we think about uh this approach more
deeply we can see that uh AI algorithms
or machine learning uh models they
expand that TX surface
and systems based on machine learning
are often called software 2.0 which kind
of implies that uh there are some
approaches and methods and if we compare
it to traditional software uh we can see
some uh obvious
differences uh machine learning models
are less deterministic so the workflow
is more Dynamic and is based on learning
so essentially uh the system behaves
differently in different environments
even if the system was designed the same
way even if if it's the same code uh
some more advanced AI products uh they
also have uh some non-traditional
interfaces like voice voice interfaces
or text
interfaces and of course uh the security
problems could be um different here on
the one hand AI systems uh that use
machine learning I just yet another
example of software and this is totally
correct uh that's why you can expect uh
many smart pieces of software have all
the traditional types of uh
vulnerabilities and weaknesses such as
problems with the input validation
Access Control issues
misconfigurations uh environment uh
issues and so on but on the other hand
AI systems have some uh new types of uh
behaviors and as avilable system uh is
expected to have uh adversaries and uh
it would be an obvious uh
Target uh for attackers but they would
attack
differently uh so while the software
environment of a systems could have
traditional uh security vulnerabilities
I want to focus on the extension of the
tax surface and uh tell you about uh
machine learning algorithms uh only uh
so these days uh I can see how AI can be
an attack Target an attack Vector at at
the same time uh so it could be an
attack Vector when uh machine learning
uh is used as a smart layer for software
for instance uh uh attackers have some
motivation to uh hack uh the product
itself they don't care about whether it
uses ml or not uh so ml is just an
obstacle um so they could potentially
exploit raditional vulnerabilities but
ml offers them ml vulnerabilities offer
them new opportunities to attack the
system and it might be easier to do than
in in a traditional than than to exploit
traditional security
vulnerabilities and in another case a
mcod uh may be uh the core experience uh
of the product for instance uh s driving
cars or AI uh co-workers which is a a
fancy word f the same for kind of smart
business boards or smart speakers such
just seral or Alexa they wouldn't really
exist without uh ml so in these cases
attackers uh are more conscious about
exploiting vulnerabilities of uh machine
learning specifically rather than um in
the previous case so in this case it
would be a
Target and uh now let me uh take a step
back and show you that uh this uh the
security problem is not the only problem
of uh AI systems um AI has enabled uh
many new applications and it's hard to
deny that uh it's um uh Power in many
applications in uh different Industries
and it's been adopted uh quite rapidly
and together with AI features uh
companies introduce some new problems um
uh there are different communities that
work on problems of uh uh Ai
implementations and traditionally you
would hear that uh algorithms May beave
unpredictably or even be unsafe they
could not explain their decisions or
they often motivate companies to uh
violate users privacy for the sake of
data collection and some some systems uh
May uh discriminate people by various
criter criteria such as uh race or
gender and each of these problems is a
separate field of
study and collectively they're called uh
trusted AI or trustworthy AI or
responsible AI unfortunately there is no
common definition and uh these terms may
use may be used uh
interchangeably and we didn't like this
uh at adversa and um as technical people
we like like seeing systems everywhere
so we have created our own structure
that is uh easy to
remember and uh easier to focus uh if
you if you use this uh as a Playbook uh
then you probably have different teams
uh focusing on uh different
columns so basically all three areas um
are essential for for the trustworth AI
uh reliable systems should work under
normal conditions and behave predictably
uh they offer human over oversight so
they are countable and uh humans have
control on the decisions of the machines
and uh the the decisions must also be
explainable uh in fact in some
Industries uh and in some uh locations
like in the US and financial industry
it's illegal uh uh it's it's illegal to
use a system that can explain decisions
uh uh if it's Financial system and it
it's making decisions about uh people's
like loans and stuff uh the next group
is uh resilience systems uh so uh these
systems should work under adversarial
conditions it might be environment
changes or some people interaction that
defines uh a safe system so basically
security here is in the traditional uh
meaning in traditional cyber security
meaning uh uh this includes security of
the environments infrastructure and also
adversarial machine learning uh which is
about uh neural network algorithm
security and privacy covers data
protections and access controls uh for
collected or inferred data and finally
responsible systems are defined by their
impact on uh people's lives um so they
must be unbiased and inclusive they rely
on moral principles when they um
interact with humans and essentially
make uh people's lives better and all of
these requirements are equally important
to have a AI that we could trust uh but
today I only focus on the security part
and the part related to aders IAL
machine
learning um so when I talk to people uh
some of them um tell me that it's too
early for adversarial machine learning
uh to become mainstream and it might
seem uh true to some extent because they
um there are many unsolved problem
problems in traditional cyber security
and uh or just managing machine learn
machine learning models is a total mess
today uh so they have mlops
tools uh but it's it's still still in
its infancy uh so like the modern tech
world is overwhelming and uh they don't
have time neither time nor knowledge uh
to address uh these uh machine learning
security issues but this is exactly the
reason why we have all these problems
because uh no one acted early on to
build things right and if we think about
people who built computers like 50 years
ago they didn't design it uh with the
security in mind that is decided to uh
to move the progress forward and the
same is happening for with the I systems
uh so that's why uh it's it's really
important to uh start with uh start
securing the I systems um uh early
because it might be one day it might be
too
late now let's take a look at some
progress uh of adversarial machine
learning over the years uh the data I'm
using today my presentation is based on
the archive.org website the main
platform for uh uh publishing uh non
peer-reviewed papers and we have
collected a comprehensive data set of
what we believe is uh every paper that
exists on the topic and uh obviously I
cannot provide uh in the short
presentation all the data that we have
uh so you you you're free to explore the
uh um the original Source uh the report
that we have published and now I want to
provide us some
highlights uh if if we if when we talk
about this field uh it seems quite new
to many uh but in fact there are more
than two decads of research uh on this
topic and what is traditionally accepted
as the start of the field happened in
early 2000s at the time they already had
uh papers exploring how to
Bypass or protect spam
applications uh spam filtering or
intrusion detection systems and uh if we
allow the topic to be more abstract uh
there are papers dating back to '90s
early 90s uh that explore uh machine
learning under malicious noise and for a
long time it remained a very small uh
area of research uh and later in uh uh
2014 the paper on adversarial examples
for deep learning was published
uh around the same time we had explosion
of um um other uh deing techniques and
from that moment moment starts the era
of adversarial machine learning uh
another Milestone happened few years ago
uh in 2011
2019
uh we saw a growth of governmental
initiatives mostly government and
Military initiatives um uh in the US uh
and the US defense grants uh uh and from
DARPA agencies and from other
intelligence uh communities uh motivated
uh to start this research uh in many
universities and uh
corporations and uh basically if you
take a look at the geography uh we can
see that the top three countries that uh
contributed to uh the total uh body of
knowledge are predictably the US uh the
China uh and uh Germany and UK
unfortunately we don't have Hungary uh
here but uh if on this map you can see
there are over 50 countries uh that uh
that publish some research in uh on um
on this topic but only 14
countries uh generated over
90% of all papers uh so essentially if
you publish several papers you would go
to the top like top 20 or top maybe 15
or something uh so it's very easy for
you to do if you really interested uh uh
you you you can do this and as I
mentioned uh the initial growth is
motivated by national initiatives and
defense programs and uh we cover more
this um in our report with the timelines
uh which government was the first and uh
who relied on what uh so if you're
interested in this uh details you can
also read our original report uh so here
I want to uh tell you about the two uh
biggest uh
drivers they are predictably uh the gas
and uh Europe as in many other the
fields uh the US is driven by uh
military and the Europe is driven by uh
regulation uh so all of these uh
initiatives are very recent uh in the
late 2018 and early 2019 the US um
intelligence uh agencies uh proposed
several programs for securing the I
systems and later on other agencies and
corporations uh they also started their
initiatives and started publishing
papers and many of them join uh created
uh joint research Labs uh between
between a university and Corporation for
instance many top uh corporations like
uh Intel IBM Amazon uh Google most of
them have joint research centers uh with
some universities uh focusing
exclusively on uh securing
systems and that's why if we go back to
uh statistics per year we can see that
uh more than 90% of all papers uh
created after
2018 and from the 2019 European
organizations uh started uh releasing
important initiatives and the latest one
uh is a proposed regulation that covers
um uh high-risk AI systems and it
includes requirements for AI risk
management uh data governance uh
technical documentation uh security
monitoring and so
on uh so as we just saw governments and
Academia are uh serious about the
security risks for AIC systems uh uh so
let's see uh some industry initiatives
uh basically before the 2018 uh there
were very few activities in the industry
and mostly independent researchers
speaking at Hen conferences and trying
to share knowledge uh or trying to
convince uh business stakeholders and
later around 2018 uh big consultant
firms started offering uh services in
the field of uh responsible
Ai and and uh later few startups appear
just like our startup ADV Veri and from
2019 Gartner started covering uh the
topic of a governance and risk
management and uh around three weeks ago
they have released their very first
Market guide on the on this new emerging
field they call this Market AI trism
which stands for AI trust risk and
Security Management uh it's not just a
research topic it's it's the whole field
is uh that consist of multiple
professions so it could be uh a like a
real
career and uh very lately uh many a
teams uh appeared uh in in big
corporations uh that heavily r on uh
machine
learning uh so these companies are
trying to prepare uh and to address real
security risks for the systems uh while
also anticipating some uh compliance
requirements so as you can see many
things are happening and uh here adversa
when we just started our company we
wanted to collect just uh information
just to understand the market and Trends
and what is the current state of the art
but we were so overwhelmed by by the
amount of information so we decided to
probably we should organize it into
reports and share with the community
community so if you want to go
deeper and learn uh about all these
topics you can uh you can download uh
the report with this special link and
together with this presentation I hope
uh it it may become a go-to resource uh
for understanding the current state of
AI security or rather say AI
insecurity so after some production
let's dive deeper and see the attacks uh
that attacks against machine learning uh
systems are not just an academic concept
uh but uh it's a real
threat uh first of all I want to show
you some statistics based on our data
set of all uh research papers and then
we we'll take a look at some real
incidents
um so overall there are more than 2,000
uh successful attack cas cases against
over aund of unique EI applications and
we can see that all eii domains have
been successfully attacked
already and if you if you uh see the
distribution um it it it's quite curious
how Vision could be the most attacked um
domain and I think there are few reasons
uh why computer vision has the largest
uh share uh first of all is the most
mature and popular application uh in
deep learning uh so that's why after uh
its uh maturity it attracts uh after it
goes to production it it attracts some
interest to attack and protect uh
protected uh it's also easier to
demonstrate vulnerabilities uh with the
images because visual changes to images
are more appeeling and due to the file
structure of uh images it's easier to uh
create malicious images without changing
uh the meaning of the image if we
compare it to changing binaries they
might be become
useless
um so here you can see the top 10
attacked AI
applications uh these are just top 10
but uh actually there
are over 100 of unique
applications and uh with the thousands
of attack cases uh in in the research
literature uh none of the tested models
exhibited uh any inherent robustness and
always failed against such
attacks um so the dominance of attacks
against AIC systems with the image
processing uh shouldn't make you think
that other systems are less vulnerable
they just um
um they they just didn't attract uh
enough
attention um so I would say quite the
opposite uh a applications with fewer
attacks might be even uh at greater risk
because uh uh the interest to develop
defenses for them would be also uh
smaller and you can imagine that at
least 90 applications have received more
than uh 2% of attention and even worse
the reasons why vulnerabilities exist
are of the same nature so most other
applications that are not tested yet uh
could be considered vulnerable too like
by
default and after the initial analyses
uh I've grouped all attack cases by
Industries uh so these groups are based
on the purpose of attacked models and
data sets uh for
instance here you can see uh the column
attack research uh and it shows the
interest of uh researchers to attack
applications in a given industry for
instance uh FAL recognition would
obviously go to Biometrics and attacks
against uh semantic segmentation from
the um with the data sets from the sell
driving car cameras would go to um
automotive industry uh and it's also
important to note that some applications
are used across Industries for instance
uh image
classification uh if the generic model
is used widely but uh across Industries
but in the research literature is
attacked in the healthare case it would
be unfair to not count it in other
Industries if we want to calculate some
real risks uh so that's why we came up
with the idea of transfer transferable
risk uh it can show the real applicable
risk based on the B on our correlation
so we assume that attacks uh on similar
AI models uh across Industries are
comparable for instance attacks on uh
facial recognition systems are
transferable between uh Biometrics and
uh surveillance uh This Way biometric
attacks would be counted toward the
surance industry and vice versa and
similarly uh if object detection is
vulnerable in cell driving cars the same
object detection models could be used in
surveillance system so they are counted
in uh both
Industries and with such correlation we
can see that uh even researchers attack
internet applications in only uh 23% of
cases over 97% of uh attack cases are
applicable to uh a applications uh used
in internet companies uh so we can see
that attacks are transferable uh not
only across models but also across
Industries uh now let's take a look at
the attack in more details um I believe
everyone knows the CIA Tri which stands
for confidentiality integrity and
availability uh so they describe
fundamental requirements for any system
to be uh secure and the same we might
have uh for ML models uh but uh
sometimes uh the same attacks May touch
uh uh both confidentiality or Integrity
or integrity and
availability uh so we decided to
reclassify uh all the Texs based on the
uh impact uh on the victim systems for
instance uh here uh we have grouped all
existing uh attacks uh inter
manipulation infection and exfiltration
groups uh manipulation allows attackers
to um bypass expected AI behaviors and
uh make uh AI systems perform some
unexpected jobs uh and uh attackers
could craft some malicious inputs and uh
conduct evasion attacks basically fool
the uh
system with a different threat scenarios
for instance to uh to disappear from the
detection or to appear as uh something
else like as a different person uh or
just uh reduce the um increase the
misclassification error um or adversar
reprogramming attacks uh may just uh uh
make the system conduct uh the task that
it wasn't trained to do uh infection
attacks uh reduce the quality of uh AI
decisions and enable uh stealth control
of a systems um uh there is a family of
attacks um of poisoning attacks and
attackers can contaminate data used for
training to reduce uh the quality of the
decisions they can exploit hidden AI
back doors or uh uh distribute uh
malicious a trogons and control
behaviors of a system
uh this is this is a real risk
considering how many uh AA models are
downloaded and used uh as is rather than
developed uh
internally every time and extration
attacks aims to steal data from AI
systems attackers can extract data uh
used for uh model training they can get
private inputs uh they can steal details
of the model and basically a clone
uh clone the
model and imagine that uh in a big
company you may have uh a team of like
10 or even more engineers and you can
imagine uh like the salaries in the US
uh and then uh with the few few hundred
you can Ste all the IP that have uh
generated and clone the model like in a
few
days uh so so yeah after analyzing all
papers um we have created a top 10 of uh
all
the attack methods against machine
learning
algorithms um so here you can see them
on the list uh I will not go into
details about uh uh each of these
attacks uh you can learn about them from
many uh good presentations and we also
have a course on this uh and each of
these TX would would require a separate
uh separate uh
presentation so I just want to highlight
an important uh issue here uh as as you
can see uh all of these attacks are
quite serious uh in terms of impact um
but the interest in studying specific
attacks is not uh evenly uh distributed
for instance at the first place evasion
attacks have only only 80% share but the
10th place has only uh 0.2 share of
Interest so basically all the attacks
starting from the uh third place are
lower than
3% uh with the thousands of papers you
can see uh how uh not enough attention
is paid to very important and very risky
uh
stuff and this might be a problem
because low interest in researching
attacks translates to low interest in uh
developing uh protections for ya
systems and apart from the top 10
systems uh I wanted to highlight some
other interesting attacks against AI
systems that didn't didn't make it into
into the top 10 in fact uh uh there are
many papers uh that uh that I have only
one or two papers on the topic so it's
not enough data to classify them as
something so together collectively I
call them uh kind of AI environment uh
attacks
and uh there are very uh there are very
few papers that are uh kind of extremely
uh
useful uh outside of these uh
traditional uh adversarial machine
learning uh papers uh but what could be
interesting is that uh we've all seen uh
kind of advanced uses of D
generation uh with the help of Gans and
that's the engine that could be used to
generate attacks so the Gans could
generate adversarial examples for
evasion attacks uh another example is uh
basically uh the usage of neural
networks for stenography and real
malware uh files can be hidden uh inside
machine learn models um that is used as
a medium and there are real prototypes
for this and the last case essentially
is a denial of service uh attack that
increases energy consumption uh and
basically it's a slow down attack uh
that makes AI models uh
unresponsive some other types uh might
be quite familiar for people from the
traditional cyber security field uh uh
these attacks include U exploitation of
vulnerabilities in third party libraries
or machine learning Frameworks uh may
have their own vulner
abilities and there's in growing number
of uh side Channel and fault injection
attacks uh so some of these attacks uh
from research papers uh may look too
futuristic or not relevant uh for uh
day-to-day operation of uh AI companies
uh and my main point here is that
uh attacks against a systems are not
just an academic concept uh I want to
show you that uh there are real attacks
and they are more real than uh you might
think uh so first of
all I'll start with something simple and
uh research research papers are often
criticized for attacking kind of sample
or toy examples uh like image
classification for data sets like minist
to C far that is essentially uh uh basic
um objects or handwritten uh digits
however they also do a lot of uh a lot
of real world testing uh for instance
many companies which you can think of um
when we talk about uh using AI in
production uh in big corporations they
all have been attacked already in
research
settings uh here we combine uh together
uh cases from academic researchers
attacking some real products and also
corporate a teams uh attacking their own
products for instance in Academia there
are uh security attacks uh that include
fully uh API based uh models uh for
well-known Cloud
companies like if you think of um big
four five uh internet companies uh each
of them um has been ATT TCT and many of
them have ai contest teams um
internally um another traditional case
for such uh uh pest scenarios is uh full
in content
moderation in internet companies and the
most favorite topic of the media is
attacks against Tesla and clearly these
are not like research Concepts uh of
course we don't see uh uh some um
address is on the street but the product
uh the Tesla the car is real and it can
be easily attacked which is demonstrated
uh on many uh video recordings and
similar to academic researchers uh AI
internal AI pest teams uh they attack
their own products uh in the controled
settings and it's of course motivated uh
by the demand to protect their systems
they already uh disclosed that they see
uh attacks in production against the AI
systems and uh from my own perception
the earliest Works in internal AI uh R
team started a few years ago in in the
companies that uh work in adversar
adversarial environments such as uh
cyber security obviously antifraud
Biometrics and uh there were research
papers from some from these companies uh
they also
um have blog posts about it or they
uh say that uh they have protections
against AI attacks in their product uh
documentation uh so this is real and the
second group uh is uh internet companies
which either have a huge uh audience and
heavily rely on ML algorithms or Cloud
companies that uh sell AI capabilities
like AI as API
uh another uh the next uh group of
attacks is uh real attacks uh so these
are the things that we have seen in the
wild these are the real incidents where
uh AI systems have been compromised and
such attacks uh they don't require a
threat actor to be uh conscious about uh
adversarial machine learning it still
counts as an attack because it's the A
system that is being fooled with the
malicious intent and probably the most
known of the earliest incidents that
happened uh in the past years uh
happened in uh 2016 when the TBO uh
developed by Microsoft and released uh
as a Twitter board uh was deliberately
poisoned as a collective effort from
some internet communities and over time
we've seen a growing number of AI system
failures against uh malicious actors
um I believe that uh today most of the
spam filters are based on uh machine
learning and you certainly seen some
spam
recently uh so they have frequently been
bypassed and internet companies they
constantly fight with abusers of uh
copyright protection content safety uh
recommendation
systems uh and uh another example is
manipulations manipulation of various um
automated systems my favorite example uh
is when uh recently released report
about Chinese hackers that fooled their
uh national uh governmental facial
recognition system for uh tax reporting
so they essentially uh create fake
identities and F their uh Advanced faal
recognition systems to pretend to be
another person and they had some company
like fake companies fake uh bank
accounts and they run this operation for
years uh so probably we just don't hear
about many attacks because there are no
ways to detect such attacks against a
systems because they uh they from their
uh traffic inspection they look legit it
it's it's the structure of the file like
if you send a picture to the API uh is
the structure of the file that that is
the carrier of the
attack
and of course AI is powered by data and
companies tend to accumulate uh a lot of
uh uh critical information and we'll
certainly see a growing number of uh
real traditional security incidents data
Brides uh there are several incidents
like this uh um if I'm not mistaken the
uh the company in the UK the medical the
Healthcare company in the UK was
compromised in data sets uh was
stolen and the recent case in uh South
Korea um where not only the data sets uh
used for a training was stored but also
the company it was the very first case
in the world where uh AI company was
fined under privacy regulations for for
uh for disclosing this um data sets used
for uh the AI
software uh and now let me make a few
predictions about the future for attacks
uh the first idea is that soon we will
see a growth of uh bu bounties for
adverse machine learning uh we are all
quite familiar with bu Bounties in the
uh security
space and besides being a useful uh
source for for uh crowdsourcing security
expertise uh uh they have created kind
of a legal market for exploit sales and
they offer recognition and financial
rewards uh over some shady black market
experience and there are signals such as
academic competitions um uh contests
from the uh defense organizations and
also vendor contests for instance uh
last year uh Facebook hosted uh deep
fake uh detection Challenge and uh
recently Microsoft uh has announced I
mean this year Microsoft has announced a
Mal evasion contest with the focus on uh
full in uh neural networks and also
recently the Twitter has announced uh
the B Bounty for ethical issues mostly
related to recommendations
and uh some user experience and minority
groups uh so I think
uh soon we'll see uh new types of BU
Bounty programs so they could be uh like
in Twitter case or Facebook case um uh
they could be AI focused or traditional
Security Programs could extend uh their
scope to include AI systems or at least
traditional uh like big Tech uh
companies with the back bounties may
include uh AI apis uh as um scope for
their bu
bounties and uh the second idea is that
uh as bu bounties transfer from uh the
security to AI world uh many traditional
attacks could be applied to AI systems
so uh when we secure uh a systems we not
only care about adversarial machine
learning but also about all other things
and it's very clear how uh academics uh
they um they gradually take classic
attacks and attack uh uh and apply it to
aders machine learning uh and it will be
very easy for real attackers to uh
extend their scope of work for instance
uh such things such as supplying chain
attacks uh happened
already um this also includes fake open-
Source packages that are confirmed real
incidents uh the poisoning attacks they
also happening already not in the m
space but in other spaces
like for instance uh cryptocurrency
exchanges uh may lose a lot of money
when their price fit uh is uh hacked is
replaced uh by malicious parties and
it's not difficult to imagine that uh
many existing um attack examples from
Nona space could easily transfer to a
space so the number of incidents will
only
grow and the last idea is that
uh soon we'll see uh more uh documented
incidents uh one of the ideas that I
have is that uh it's not possible uh
there are no uh there are no products in
production that could detect like
firewalls attacks against TI systems um
and we we just don't see many attacks uh
another side of this coin is that even
if the attacks are happening they're not
uh either disclosed or
documented uh right now there are uh few
public databases that already have uh
that um that have uh cases from various
organizations uh and uh these uh bases
are mostly filled uh with the data uh
about uh discrimination uh incidents
privacy violations and safe behaviors uh
but soon I believe that we'll see more
examples uh of uh security
incidents
uh so as we can see the TX are more real
than and companies are trying to prepare
for this uh they conduct uh threat
modeling and ption testing of their AI
systems that is supposed to shape their
detection and monitoring capabilities as
well as incident response plans and for
that they use a teams and from my view
there are three groups um three main
groups of teams uh testing security of a
systems uh the first group I believe is
the largest one uh I would I would call
them opportunistic uh AI security teams
many of them don't self-identify as AI R
teams or a pentest teams uh but for the
natural reasons uh AI security risks are
uh first noticed by security people so
they either saw that uh there is a new
important A system that is is no one
trying to protect for some reason or
they uh heard learn some uh new attacks
from a security conference so they see
it and they're trying to do something
useful uh as the extension of their
traditional scope of work and the second
group uh is uh security team specialized
on a
systems so usually machine learning is
at the core of their products and
judging from the uh public activ
activities uh Microsoft and Facebook are
the leaders in promoting these types of
groups uh I think Facebook was the first
to introduce uh specialized T red
team and later uh other companies
followed up and the recent news shows
that even Pentagon decided to create
their own specialized a team uh for the
news of the Department of Defense and
the third group includes uh some
Consulting and audit firms offering
services for trusted TI and some pesting
companies for instance trailer bits uh
and startups such as adverse
AI uh so uh these companies focus on uh
uh explo creating or uh using exploits
uh for neural
networks as a part of their uh scope
war and if you're wondering what all
these a teams could be do doing uh I
have prepared a list of cases that we
have
conducted in the recent time uh so here
at adversa we do um vulnerability
research and Consulting and and we we're
able to cover uh cyber and uh physical
cases uh as you can see um um there are
attacks against uh models like when you
work on the on the software level uh you
can you can attack devices as a blackbox
um Target and you can also work with the
API Services AI as
API Cloud
platforms uh so as you can see there is
the real demand for secur systems from
different Industries and across the use
cases
um but what about
security uh as we can see adversarial
machine learning isn't just a research
topic it introduces some real uh threats
to a
systems and what is here for
Defenders uh to understand the current
trends in the field I've used my data
set from the archive.org as it could be
a mirror of what is happening in the
field uh so here is the distribution of
all adverse papers across four
categories uh let's take a look at the
first two attacks and defenses uh you
can see almost equal number
of papers and it can remind um a
traditional like C and mouse game or I
would say arms
race uh between attackers and
Defenders uh but there is just one
exception uh there are many efficient
attacks and next to nothing efficient
defenses and the second pair of surveys
uh and tools represents interest in uh
operational izing uh adversarial machine
learning uh most survey papers uh they
contemplate how to do adversarial
machine learning research uh they
compare various attack methods
unfortunately very few papers uh explore
what to do with all those security
vulnerabilities uh very few papers
explore the metrics or
uh uh operations uh so how to how to
make uh this field action
in the real in production settings and
similarly uh there are many various
tools and collectively they even cover
all the data types uh that could be
attacked but these tools are focused on
researchers uh so they used in research
Labs but not uh uh they're not uh usable
in production settings so as I have
mentioned today there are no efficient
defenses and that's why we don't have a
special coverage of defenses here uh as
there are fundamental challenges uh to
develop secur a models I want briefly
talk about challenges in uh detecting
attacks so I believe uh as the adoption
of a grows will have more real attacks
uh but many of them will not be visible
uh we didn't hear about many incidents
but obviously Industries operating in
adversar environments uh they have
always been attacked and abused and some
of them uh used in uh in some simple
forms uh ml for uh for a couple of
decades uh it's relevant for uh
antifraud biometric cyber security
internet companies often they're not
interested in disclosing them when they
when there is no regulation uh for them
and there are very few news about it but
it's very easy to believe judging from
uh the silence or proof coin proof Point
cases when uh like well-known security
products were easily bypassed
um uh in the wild and in another case
recently the security vendor uh deep
Instinct uh made a statement that they
see growth of uh adversarial machine
learning attacks um against their
products and they added protections uh
against such
attacks and even if they wanted to tell
us about uh such attacks uh reliable
threat detection for uh um adversarial
machine learning doesn't exist uh so it
could be happen already at larger scale
like at infnet scale you uh it it it's
obvious for everyone that um when the
zero day appears U or one day appears uh
like there are scripts that are scanning
the whole internet the same could be
happening for all the AI systems and
apis but we just don't see it and it
would be very stupid for many AP actors
not to take an advantage of this uh
state of insecurity of a
systems and uh as we saw in the
beginning gu introduces uh some
fundamentally new problems in the
security and traditional security
products also cannot uh cover
adversarial machine learning for
instance web firewalls cannot detect
adversarial examples and antimalware
cannot detect poisoning or model troons
and application Security will not to
real weak decision boundaries of neural
networks and despite some historical
track record if compared to General
cyber security uh the adversarial
machine learning will look quite young
industry uh let's take a brief look
before we finish at what tools are
available uh in the field of um uh for
for for practitioners in this field uh
so I have defined three main groups of
tools that available today uh the first
is assessment tool tools that are used
to generate uh adversarial examples for
various data types and sometimes uh test
it with some sample defenses uh in this
category there are many tools that are
born from a single research paper but
some others have gained more traction uh
for instance in our AA team we try to
use such tools to uh when possible
because they offer a useful uh
repository of uh
attacks but often they're not usable at
scale or when you require many
customizations uh that's why we also
develop our own Technologies for testing
specific cases for instance recently we
have released a tool called adversarial
octopus uh that is able to uh hack
facial recognition systems uh our case
is not unusual for instance uh there is
a tool called adversarial orust toolbox
or IRT art uh and it was initially
developed by by IBM and donated to open
source and it was useful for research
but had many limitations so Microsoft
used it as a foundation and on top of
this they have built their own tool
called counterfeit which is like metlo
fori
systems so as you can see uh there is a
growing number of tools uh and on top of
these tools uh some companies try to uh
build some real real production um
instruments and the second group is
benchmarking tools that usually include
some sample attacks and defenses and uh
they offer features for comparing
efficiency of multiple attacks against
multiple models uh so some of these are
available online with public leader
boards uh so um researchers also try to
uh get in the those
ratings so this mostly used by
researchers to demonstrate uh some
progress uh in their research and and
the last small group is for interactive
tools used for learning so the first
tool here is used for uh visualization
of aders examples and the second one is
developed by Nvidia as a CTF flag box so
it's a vulnerable application like Dam
dam vulnerable Linux or we application
uh where you can try to hack an AI
system so overall many of these tools
are useful for resech and
vulnerabilities uh in a specific model
but certainly these tools are not ready
to be used in production at large scale
and there are some St tabs that are
trying to solve this just like what we
do here at
Vera uh so finally what to do with uh
all these uh problems when we talk uh
about AI systems just like other types
of software uh they have their own uh AI
development life cycle and this life
cycle will definitely need to be uh
secured
as a security development life cycle
um so here at adversa we have adjusted a
well-known uh nist cyber security
framework toi systems uh you can get
more details about it uh about the
sample life cycle uh from our report
which I've mentioned uh earlier just for
the sake of
time uh I skip the description I just
want to drop a couple of ideas here uh
so first uh there are no patches for
adverse Tex in a traditional sense
there's nothing to install and after
this uh the the model is fixed uh the
mitigation that works for one system may
not work for another system so while out
of the box defenses they don't exist
it's still possible to create a unique
defense uh for a clearly defined threat
model for a single application but most
likely it will not trans transfer to
another model uh and the second idea is
that securing I systems is more about
operations than just patching you should
find vulnerabilities evalate defenses
evalate tradeoffs between security and
accuracy protect Supply chains of AI
models and data sets uh secure
infrastructure and adversarial machine
learning is just a small part of
this uh so here adversa we explore such
defenses and how uh security operations
for M should work uh so if you're
interested to collaborate as researcher
or industry partner uh just let us know
these are unsolved problems in the field
so we always happy to uh to see new
challenges uh so to get to give you some
idea what could be the operational steps
um in this life cycle for adversarial
machine learning uh we could apply some
operational wisdom from the general
cyber security domain uh to security of
a systems and here provide
a a list of uh the most relevant and
important documents that explain how to
apply security principles to uh secure a
life cycle uh these documents cover uh
many aspects of a systems um uh from
risk analysis and threat modeling to
operational guidelines and incident
response plans uh many of these
documents uh in the early versions so um
but they must be useful already because
uh security field has taught us many
lessons
already all right so we can see that
adver machine learning uh is kind of a
hidden threat but it can bring a lot of
chaos and I believe that uh securing
systems is one of the biggest challenges
over the next decade uh in the cyber
security field um after this
presentation I want to I want you to
remember a few main ideas uh today there
are no defenses uh for a system so every
I system is vulnerable and it expands
the attack surface so there is uh
definitely the need for security I
systems uh the next idea is not too
early and it would it might be too late
uh so think about how to develop
defenses faster then a adoption grows so
there are more and more vulnerable
systems and then attacks against AI
systems are real they are relevant for
all applications and industries and
chances are we just don't see them
yet and finally defenses are more
operational uh than technical so start
building some external expertise and
best practices
sooner so if you're thinking what you
should do next as a security practition
ition or AI engineer here's my
suggestion so you can learn what kind of
AI systems are used in your organization
and share potential a threats uh with
your
colleagues uh you you can try to perform
some basic a security testing with the
tools you can find online and show some
real impact uh this should be super easy
to do because every system is waiting to
be hacked and um hopefully this works
out and you can it at uh on this and try
to uh build security into the a
development life
cycle um so this is it uh this is the
end of my presentation I hope you find
it useful uh you can follow me on
Twitter or LinkedIn uh and if you're
interested in any type of collaboration
just let us
know uh and you can find up toate
materials for this talk uh with this
link thank you
[Applause]