Good morning everyone, thank you for coming to our workshop.
He is José Ignacio Escribano, I am Alfonso Muñoz, and we are going to talk a little about artificial intelligence, this topic that seems to be so important to everyone, and that can be applied to almost everything, but in a particular scenario, in which we have more experience and feel more comfortable,
which is computer security.
The title is Offensive and Defensive Security with Machine Learning.
But as we will see throughout the talk, we are going to focus a lot on offensive security, because we are going to be talking a lot more about attacks, and if it is possible or not to defend ourselves against them.
We leave you here a brief presentation of who we are, in my case Alfonso, José, and contact information, in case you want to tell us anything, both in my case and Minecraft, on Twitter, by email, or José at josegnacioescribano.com.
What is our goal with today's workshop?
We are going to do a series of demos, we have published them on GitHub, which we will show you later, if you want to replicate any of them, or, well, advance on something more than we will have time to see.
Our intention, above all, is to demystify everything related to artificial intelligence in the world of cybersecurity.
We are going to see limitations, we are going to see contexts, we are going to see different issues in progress, both at a practical level in tools and in more theoretical environments, and, above all, we are going to spend a lot of time on offensive aspects.
We are going to summarize many of the attacks that are currently being faced against artificial intelligence systems.
Some of them are already practical, they are tools, as we will see later, that can be downloaded and tested, and there are others that are still theoretical, but that show that they are effective.
Well, the agenda that we are going to follow in the workshop is the following.
We are going to give a very quick overview in the next minutes of context of artificial intelligence.
In case someone is new to this world and doesn't know anything about artificial intelligence, a very quick little context.
We are going to highlight an important topic, which is singularity in artificial intelligence.
I will introduce that information.
Then, José will give you a more practical approach to artificial intelligence and some more basic concepts.
We will talk a little about deep learning.
We will do a demo so that we understand a little better how neural networks work and some interesting parameters.
And we will also talk about generative adversarial networks, which are very important.
With that quick context, if we have time, we will move on to security, in which we will talk about a series of limitations and contexts.
We will talk a little about what is being done in defensive security and that will allow us to introduce limitations and context to introduce what we wanted to highlight more, because it is what is usually talked about less, which is offensive security.
Different attacks of all kinds.
Then you will see how to put in homemade doors, model robberies, attacks from the white box, from the black box, etc.
And then a part that is very important for us is how to make artificial intelligence safe from the point of view of computer security.
And then some conclusions and, hopefully, some questions.
The context, very quickly.
As I suppose many of you know, artificial intelligence is basically...
and in particular machine learning, is a discipline of artificial intelligence that basically allows, through software, to make machines learn on their own.
In a way that we must imitate human behavior, reasoning, but in an automatic way.
Well, such is the importance of this type of technology.
I don't know if you know this table.
It is quite famous.
Many companies, in many sectors, are trying to apply this type of technology to get some kind of competitive value.
In all kinds of sectors, medicine, automation, agriculture, etc.
EDGE, in all sorts of industries, agriculture, education, investments, legal areas and logistics.
And we are just going to concentrate on this area.
Here.
If you don't usually work with this type of technology, here is a good summary.
Very quickly.
What is artificial intelligence?
And why is everyone so excited about this technology?
We think that this transparency sums it up quite well.
There are a lot of applications that benefit from this potential of being able to automate almost certain human behaviors.
With the typical techniques that are known, such as supervised learning, supervised or unreinforced learning, you have a lot of successful scenarios.
The classics, image classification, fraud detection, real-time decisions, games, well, recommendation systems.
There are a multitude of scenarios where this type of technology is very useful.
And I think something that we are very concerned about, from the view of computer security, is to what extent AI is a technology that can be controlled.
Is it controllable?
There are many experts who fear this idea that AI could develop such ability that a human being isn't able to control it.
In the press you may have seen people are trying to reach common standards to control this kind of intelligence.
If you've got time, when these slides are published with this article, have a look at them, because they show many of the mistakes of AI in recent decades, and how cyber security has failed, and how things can go badly wrong if we don't do things properly.
Here we've given you the typical opinions of all the scientists to try to put an end to this kind of technological chaos that exists right now with AI and its potential.
Another problem that many people are concerned about is its impact on the economy and society.
In fact, if you've read La Pensa, there are many proposals to try to add taxes to robots, to AI, because it's supposed that in the near future, we'll see later if it's true or not, they'll take our jobs away.
But what concerns us particularly, apart from the economy, is everything related to society.
That is, how this kind of algorithms could take out the worst part of the human being.
That is, learn cultural biases, be racist, that is, things that have already been shown that if there is no human being that makes those systems adapt in some way, we can have bots that like Nazis, that segregate by race, by skin color, or that identify people as orangutans when they are not.
So, we have to pamper technology with a little care.
All this context, in the end, what it brings to us is this.
And for us, this is important in terms of computer security.
In artificial intelligence, it's known as singularity.
That particular moment in which an entity of this kind will be able to solve a problem in a better way than a human being, in an efficient way.
When will that happen?
It depends on many things.
The problem we humans have is that we are very bad at predicting the future.
We almost always make mistakes and, in fact, to this day, no one has an exact answer for when that will happen.
The most serious thing we have found is a series of surveys carried out by the University of Oxford, the Institute of the Future of Humanity, in which, basically, they sum up that, in a period from now to 40 years, more or less, many activities that humans do today,
artificial intelligence, will be done at least in the same way as a human being.
And they set a period from now to 120 years so that most activities can be done by a robot with artificial intelligence.
And they make some estimates here.
For example, within 30 years, an artificial intelligence will be able to write a book like a human being.
Surgery, retail, etc.
So, we are in this scenario of great uncertainty that people are very concerned with this type of technology.
We, at Seguridad, think about when that specific moment will come in which algorithms of this type will be able to replace the ability of a more serious analyst with their abilities to do the typical things.
Open testing, reversing, etc.
With an interesting property.
Our particular opinion is that if you have a good analyst at Seguridad, usually artificial intelligence will not replace it.
The good thing about artificial intelligence is that it can be replicated.
I mean, maybe I have a guy who is very good at reversing, but I have one, or I have five, but I don't have 100 at 24 hours a day deployed around the world.
The good thing about artificial intelligence is scalability.
Maybe it is not the same as a human being.
As long as it is reasonably reasonable, it will replace many tasks of the human being.
As I was saying, the fundamental problem is that there is a lot of uncertainty.
It is not known very well where this path is going.
In fact, the big owners of big organizations, Tesla, Facebook, etc., are constantly discussing if this is going to have an impact or not, and what is the scope, because there is too much uncertainty.
Let's talk about a couple of transparencies of justice.
Justice in the sense that we would like to see the context.
To what extent this is true, that is, whether artificial intelligence is so powerful or not, and whether it has a role in what we will see in security.
We have given you a series of examples, especially of video games, which is the most typical, and that people pay a lot of attention to, so that we can see how this progress has been.
In this graph we see that in the last 20 years or so, artificial intelligence has progressed quite slowly.
In fact, we know how to solve some very specific problems, some of them very well, but we are not able to do anything in a general way, or a global intelligence like that of the human being.
To think that artificial intelligence more or less...
We can go back to the 50s of the last century, that is, artificial intelligence has a long history, but in the last 20 years it seems that things have been done, but maybe not fast enough as many people may think.
In fact, this is the typical example that scares people a lot.
These are different simulations we have done of artificial intelligence playing video games.
And it can be shown that in many games artificial intelligence plays better than a human being.
That is, algorithms can be made that work better in specific environments .
And that, for many scenarios, maybe for security, can be interesting.
In fact, these games do not worry so much, they are anecdotal, they are nice.
What scares people a lot more is the ability to make games, to make artificial intelligence algorithms that somehow reflect human intelligence.
An example of that, or a point of disruption, to put it in some way, was with the different proposals to win something.
Both in the most classic algorithm, with typical movements that even human beings were very surprised by, like the evolution of their attacks, that is, these algorithms that make it even evolve much faster today.
That is, today we have the ability to make very good specific algorithms, some of them with better results than the human being.
And I particularly like this Google study in which, I do not know if you know it, basically I did an experiment with neural networks, I explain it very quickly.
Below, in red and blue, they have two competing neural networks.
The green ones are apples.
Then, basically, it raises several scenarios.
A scenario in which there are many apples, then the neural networks in that scenario, when there are many apples, they collaborate.
That is, when they train, their decision is to collaborate.
But when there are few green dots, that is, few apples, the neural networks, when they reach equilibrium, they have to compete, and therefore they attack, they shoot.
There are little red dots, that is the laser, which they simulate in 2D.
But there is another better scenario, that is, if in front of different neural networks, one of them has greater computational capacity than the other, therefore it can train faster, in that scenario, it does not matter the number of apples there are,
it always attacks.
That is, it does not collaborate.
And that is what the network itself deduces in that process.
This, honestly, is scary.
It is what worries us a lot of people, to what extent artificial intelligence can be controlled or not.
From the point of view of security, I'm going to use one of these methods in defensive security, is it going to go against me or not?
Can I control it?
There is a lot of uncertainty in our sector.
Here I leave you a couple of references, because we are quite critical that artificial intelligence can solve all the problems.
In fact, I recommend this video, it is great, by a scientist from CSIC.
There are many investigations of recent Google failures, making predictions with Machine Learning that have been wrong and have caused major failures.
And the article I mentioned before.
That is, artificial intelligence is useful, but we have to think very well in what domain and what is the context to see if it applies or not.
Finally, we are already finishing with this part of context, we are going to go a little to the most technical part.
There are some aspects that sometimes people forget and for us in security it is important, which is everything related to legislation.
There are scenarios where, even if we had better artificial intelligence algorithms than the behavior of a human being, we will not be able to use them, because legislation will not prevent it.
Imagine, for example, in a banking entity that had to grant a loan, the Data Protection Law, for example, can force you to make those decisions that have a significant impact on the lives of citizens must be reasoned.
And the review of a human can be requested .
Why is that?
Because if they know about artificial intelligence, once they have trained, a kind of black box is generated.
We put inputs, we have outputs, but we really do not understand why an output occurs before an input, that is , that process cannot be reasoned.
So there are many scenarios where this technology would not be valid.
So if we have the best security system, we have to think that we may not be able to use it.
Well, once this little context is done, we enter the most technical part.
Okay, let's get into what machine learning is in practice.
We can consider machine learning basically as data and algorithms.
That is, about some data, an algorithm is applied, then we go into more detail and we get a model with which we can perform different tasks, which we will see later.
The data can be any type, which is represented as binary, but it could be text, image, audio, whatever we want.
So, entering the algorithms, we can classify the algorithms according to their type of learning or according to the task it solves.
Let's start with the type of learning.
It is classified into three types.
Supervised, unsupervised and reinforcement.
The first one, supervised, what it tries to do is learn a function through labeled data.
That is, if we, for example, want to identify images of cats and dogs, what we do is, for each image, next to each image we put a label, that is, with the corresponding class, the dog or the cat.
So, what the function does is learn from those labels.
In unsupervised, where we don't have those labels, the goal of this, of this type of learning, is to obtain the internal structure of our data.
And finally, the reinforcement learning, which is totally different from the previous two, which consists in having an agent, like a robot, in a certain environment, and in which that agent knows its state and can perform different actions.
For each action it performs in a state, it receives a certain reward.
The goal of this agent is to maximize that reward.
Apart from the type of learning, as I said before, we can classify it according to the task it solves.
The machine learning, as we have seen before, can be applied to a number of applications, but they are usually classified or grouped by four types, which are called tasks, which are classification, regression, clustering and reduction of dimensionality.
Classification and regression focus on supervised learning and clustering and reduction of dimensionality in the unsupervised.
What does each one consist of?
Classification consists of what I said before.
We have a series of predefined objects, more than objects, a series of predefined classes, such as cats and dogs, as I said before.
So the idea is to distinguish between each of those classes.
Regression, in this case we do not have a finite group of classes, but we normally have an infinite interval.
So we try to somehow approximate a function to the data we have.
Clustering, which consists of grouping objects according to a series of properties.
A typical example is market segmentation.
If we have a million clients and we want to offer a personalized rate for each of them, it is impossible, we cannot offer a million offers.
What is done is market segmentation in 4 or 5 groups of users and a rate is offered to each of them.
And then, finally, the reduction of dimensionality.
Normally we will have some data with many or few variables.
But if we have many, it is likely that some are redundant or do not provide too much information.
And in the case of training as we will see later, it can be quite expensive.
So reducing some of these variables at the cost of losing some information, we can make training more efficient.
Within each type there are quite a few algorithms.
So, we have searched in Kaggle, which is a page where different data science and machine learning contests are held in which prices are offered.
And they have conducted a survey to their users where many questions are asked, but here we highlight two.
The first one is what algorithms do people who use Kaggle use and then what programming languages.
So if we look at the algorithms that are most used in logistic regression, decision algorithms, random forest, neural networks, most of them are supervised learning where most people perform machine learning.
And as for the programming language, the most used is Python, followed by R, which is widely used by mathematicians and statisticians.
Then there are other types like SQL to handle relational databases, tools like Jupyter Notebooks, or deep learning frameworks like TensorFlow, which we will see later.
The greatest difficulty of machine learning are not the algorithms, but where they get the data.
Either because we need to apply, we want to make an application on something and we don't have the data because we can't get them because they are not public or because they are of very poor quality.
So here we leave you a series of data repositories where you can see what kind of data each one has different categories.
Perhaps the most interesting is the Kaggle one, because, as I said before, Kaggle does contests of practically anything, so there will be data on any topic.
The repository of the US government's web services is also very good, the Amazon one, and more security-oriented, we leave you here that repo that has a lot of security data sets.
One of the most used types or in machine learning in general are images.
And here we leave you a series of repositories since normally in these data sets different algorithms are usually compared .
They are used as a benchmark to compare the goodness of the algorithms.
So, one of the most important and typical algorithms, also very small, is the MNIST, which are handwritten digits, as they are here, and the problem we try to do is to classify each of these digits from 0 to 9.
In the same way, the CIFAR-10 and the CIFAR-100, where we don't have digits but we have objects, and it depends on whether it is the CIFAR-10 or the CIFAR-100, because we have 10 or 100 kinds of objects.
Another set is the CIFAR-100 with 200,000 images of famous faces, and also 140 attributes, and then more databases such as ImageNet or Cocoa Dataset.
We have already seen more or less that machine learning are data and algorithms.
But what is the cycle from obtaining the data until deploying it in production?
In the same way, we have data and algorithms.
So, in terms of the data phase, it is obtaining the data either through public repositories, data generated by us, or through aggregation of different sources.
Normally, once we have this set of data, there is a job that is the most expensive of all, which consists in data cleaning.
Elimination of atypical data, missing data, etc.
Once we have our clean dataset, what is done, and this depends on the application, is to reduce dimensionality through the algorithms we have seen before, through statistical tests, through the experience of an expert, etc.
So, our data phase would end and then the algorithms phase would begin.
Depending on the application we want to do, if we want to solve a task of regression, of classification, we will select an algorithm.
Normally, those algorithms will depend on one or several parameters that will have to be set in advance.
How is this done?
Through previous studies or through trial and error.
And there we will be ready to train our algorithm.
Before that, our set of data is usually divided into two or three parts.
Most of the data will be used as training data and the remaining two parts of test and validation.
Although the third part depends on what we are doing.
It is divided in those parts I was saying and the one used to train the model is the training one.
With the other two parts of our set of data, it is passed over our trained model and we get a series of metrics that we set in advance.
Depending on the application, it could be the error, false positive rates, etc.
And depending on the metrics we get, if they are correct, if we have, for example, we set that our algorithm, our model will be valid if we get 90% accuracy and with the metrics we get 95%, we deploy the model where it corresponds.
If they are not valid, we will have to select the algorithm or change the parameters again.
A little note to what José says.
Probably someone is wondering why I want to know this, if what I want is to violate artificial intelligence systems.
It's like when someone comes to you and says, I want to hack systems and what you recommend is to study programming languages, communication protocols or computer architecture.
To understand what we are going to see later, we have to understand this, because this cycle is very important.
Because understanding the cycle well, now we will see at what points we can inject to perform the attacks.
So it is necessary that mixed team of security people and experts in artificial intelligence.
So...
As I said before, the biggest problem is the data and the algorithms.
Because most of the algorithms are implemented in many libraries.
Here we leave you some.
Most of them are in Python, because it is the most used language in Kaggle's survey.
And also, Python is very easy to program.
Most of them are in Python.
But one of the ones we like the most is scikit-learn, which is programmed in Python.
Then there is also H2O, which can be programmed in Python or in R.
More oriented to Big Data, there is the MLib library of Spark.
And then there are other libraries like Weka, Mahout, etc.
Here we also leave you deep learning frameworks with which, in addition, you can do...
you can apply classic machine learning algorithms.
Although they are not specially designed for that.
There is one that we especially like, which is Keras, because it brings you many...
many peculiarities of deep learning as...
sorry...
of...
of TensorFlow or Theano.
So, to start, this is very... for the topics we will talk about next, of deep learning, this framework is very interesting and very simple.
Apart from everything we have said, we can run it locally.
The problem, as we will see later, is that these algorithms are quite expensive computationally.
So, specific services have been created in the cloud, where it is possible to carry out...
to train our models in the transparent cloud for us, or to steal the code, upload it to the cloud with our data, and everything is executed in the cloud.
So, there are specific services of Amazon Web Service, of Cloud Platform, of Azure, which is not here, or more specific services, such as FluidHub, BigML and Cell.
Then we will see a demo with FluidHub that allows a free account to test what we are going to do in this workshop.
Just a few things.
And then we will see a demo.
So, we are going to enter the deep learning part, which is a subset of machine learning.
And then we will talk about a special type of neural network, which is GAN, which is quite used in security.
Basically, deep learning consists of neural networks.
We are going to see a demo to understand what it is.
In this little demo, the basic concepts of what a neural network is, the evolution used in deep learning, and again, it is important to understand this to see the attacks, especially of offensive security, which we will see in the second part of the talk,
and to see how classification systems can be deceived.
Basically, a neural network is a set of layers formed by neurons.
A neuron is nothing more than, to define it quickly, a set of inputs and outputs, so that the output propagates to the next layer.
These inputs and outputs are weights that, in the end, the goal of the neural network is to adjust these weights in such a way that some kind of loss function is minimized , which will depend on the algorithm we apply.
There are three types of layers, which are the inputs, which are basically the data that the algorithm receives, the outputs, which is what it produces, and the interesting layers are the hidden layers, which is what allows us to learn complex concepts.
The more layers we have, the deeper the learning, hence the name of deep learning, and the more complex concepts we can learn.
We haven't mentioned it, but it can be deduced.
Up there you have the URL.
This is a neural network simulator.
Simply, if you want to understand the concepts better, with this simulator you can easily design neural networks and see the effect.
What we are going to do is what the algorithm does.
It tries to minimize a loss function.
Here we see that it does not converge.
What should happen is that what you see here, blue, forms a complex circle so that it perfectly separates the orange and blue classes.
We are going to change the network architecture.
Normally, one thing that is difficult to adjust is the number of hidden layers, because we don't know how many we need, how many neurons we need in each layer.
This is one of the biggest problems that is usually done by trial and error, adjusting the different the different number of neurons in each layer, etc.
So, let's remove some layer.
Let's see if...
Basically, what is done to train the neural network is to apply times that are nothing more than passing the complete training set.
So, if we have...
Let's get to 500 times, we have passed 500 times the training set.
Here we see that it perfectly separates the two classes, the orange and the blue.
And the error is practically zero.
So, it is a very good classifier.
One of the problems for both defensive and offensive security is related to the times.
What the simulator does very quickly has a high computational cost.
So, depending on our ability to pass this multiple times and understand the model, both as a defender and an attacker, as we will see later, it will make it easier for us to be a better defender or attacker.
We can play with different parameters such as the ratio of training data.
We can also put noise so that it is not so easy to classify it.
We see here that it is not separated in two concentric circles, but here they are practically attached to the blue ones.
So, it is more difficult to make a good classifier.
We can also play with the training rate, how the algorithm learns, the activation functions, and then regularization parameters.
We can also play with the type of problem, regression and classification.
In the same way, we can see how it converges.
This is basically a neural network.
We have not talked about how it is trained, nor about any technical detail so as not to make the presentation too heavy, because otherwise we would have to go into a lot of mathematical details that we do not want.
We have seen with this simulator the simplest example, which is what is called the multi-layer perceptor , but there are other types of neural networks called neural network architectures.
And there is a multitude of them.
One of the most used today is the recurrent networks, although they are not usually used like this, but variants such as LSTM or GRU are used.
Also the convolutional networks , or, as we are going to see in more detail, the GAN networks, widely used in security.
Then there are others, such as autoencoders.
Here we leave the link where are detailed the complete architectures of each type of network, a description and what it is used for.
So, let's see what the GANs are.
Basically, it is an interesting concept of attack and defense, and it is not a neural network, but there are two, the generator and the discriminator.
How does this type of network work?
Basically, the generator gets some kind of noise, and what it does is to generate false examples.
And on the other hand, the discriminator gets the data generated by the generator and real examples, in such a way that the generator and the discriminator compete, trying to deceive the discriminator, and the discriminator tries to defend itself from the generator.
The generator tries to generate better and better examples, real samples, sorry, generated samples, and the discriminator tries to distinguish better and better between the real ones and the generated ones.
And this game ends when they reach a balance, that is, the generator cannot generate better data to deceive the discriminator, and the discriminator cannot distinguish between the real samples and the generated ones.
Here we leave you the original paper to see all the specific details of what functions loss uses, and how it is implemented.
We are not going to get into it.
This idea of Red Gun is very interesting and has given rise to many variations of the original idea.
In fact, the Red Gun was born in 2014, and here we have how it has grown, this is the number of papers per month in which the word Gun appears, and you can see that it grows exponentially.
In fact, the director of artificial intelligence at Facebook considers that this type of Red Gun is one of the most interesting ideas that machine learning has had in the last 10 years.
Returning to the topic of deep learning, well , basically, the concepts of neural networks are nothing new, but they go back, this is a paper where the first neural network is defined , based on biological functioning, and data from the 40s.
Another type of newer networks, which are not new, but, for example, the LSTM, which is a recurrent type, is from 1997, and the convolutional networks are from 1989-90, almost 30 years ago.
But why is it so fashionable now?
Mainly for two reasons.
First, for the data, that 20-30 years ago, when the architectures were born , there was not the amount of data we have now.
We are in the Wikidata era, so we can make use of a lot of data.
And, secondly, the greater computing capacity.
Neural networks require a lot of data, and the greater the amount of hidden layers, the greater that amount.
And, on the other hand, training is quite expensive.
So, with the increase of graphics cards, with GPUs, and their cheapening, we are able to train very complex models in reasonable times.
So, basically, deep learning has become fashionable now for these two reasons.
For the amount of data we can use, and for a greater computing capacity thanks to GPUs.
So, let's see some demos of neural networks, where you can see some of them in...
wherever we want.
Or in a very simple way.
Let's see an example of an application, which is the rock-paper-scissors game, where simply with a browser , we are able to play against the rock-paper-scissors machine.
In fact, this example may seem more or less interesting.
We have to see it from a technological point of view.
A few years ago, it was unthinkable that this type of intelligence could be applied in a web browser, with libraries that would allow to speed it up.
So, we are going to show it with a very quick example, which is to play rock-paper-scissors against a machine.
Here, draw.
What is waiting is the rock to start playing.
Again, I'm going to play with scissors.
Here I lose.
So, if you don't have a friend, you are bored, you can always put it in your house and play in the afternoon, a winter afternoon.
This was a simple example, it doesn't have much interest, but simply for you to see examples of neural networks.
Another typical example, you all know the case of Tesla, which is developing its self-driving car.
There are other companies, such as NVIDIA, which is also developing its own self-driving car.
And this video you see is interesting for two reasons.
The first one is that companies don't usually tell us, and it's how it's trained.
That is, all the mistakes it makes until it really works.
Companies don't show what works.
The one who has value is the one who drives it in the training phase, not the one who really plays with life.
Then it has no merit.
And then another interesting reason is the approach they make.
The traditional approach of machine learning consists of, we have a complex problem and we divide it into sub-tasks and apply a specific model for each task.
What is done here is to apply an end-to-end approach, which basically consists of training the car as a whole.
So this gives it a certain robustness.
More robustness than if we had done it independently.
And as a curiosity, they apply convolutional networks that are almost 30 years old.
Another company that is also betting on the autonomous car is Baidu.
Baidu is the Chinese Google.
They are putting a lot of money to develop the autonomous car.
The interesting thing about Baidu's project is that it is an open source project.
So anyone can download the software.
They also give you the hardware to train the autonomous car model and to be able to create your own autonomous car without investing practically anything, just the necessary hardware.
The idea of Baidu is that for the year 2020, for December 2020, this type of operating system can work on highways and on open roads.
We are going to do another demo of an application of convolutional networks that is the style transfer.
This would be the last one and we will go to the most purely security part.
What we are going to do here...
What José was telling you, one of these services is Floyhat.
We have not commented on it, we have published it on Twitter.
We have a Github, J-I-E-P, Cybercam 2017, where you have a lot of papers and the demos we are doing with the code you have uploaded there in case you want to do anything later.
The idea of this demo is an application of convolutional networks.
Given some images and a style of painting, a painting that has a certain style, that is, what colors have been painted, what are the strokes, etc.
We can do the style transfer of this painting over the images.
For this we are going to apply Floyhat.
Now I am going to tell you in detail how it works.
What I was telling you before, Floyhat is a cloud service.
The advantage is that you can upload...
Well, first it allows you to use libraries in a simple way.
You don't have to install them locally and configure them.
You can also hire CPU and GPU users on demand.
The cool thing about this one is that it gives you a couple of free hours of GPU.
So, to do your tests, you can do them here.
We are using the free account that gives you two hours of GPU.
It is a K80, if you know about graphics cards.
And most importantly, you don't have to give a credit card to create an account.
So, this is Floyhat.
What Floyhat allows you to do is to create your projects, which are basically code and datasets.
Datasets are our data sets that we can download and upload in a very simple way.
Here we have created two datasets, one of styles, which are pre-trained styles with different frames.
There are six.
Reimprinces, urec, lamuse, unie, wave, scrim, etc.
And on the other hand, we have the photos on which we will apply the algorithm.
Let's see if it loads.
This is an example of a photo.
So, we can create our projects and within each project we can run jobs, which is nothing more than applying a certain code.
So, it's running code, usually on certain data.
So, let's transfer the style.
What we do is, everything is in github, how it is done.
So, let's see how the style is transferred.
We use
the CLI provided by FluidHub.
So, simply with a simple command we can transfer the style or run whatever we want.
What it does is upload all the code we have.
It is quite important to separate the data code, because if we have a very large set of data, it will be uploaded every time.
And it can take a long time depending on the network.
So, what we are doing is we run the Fluid name of its command line and we run it in a TensorFlow environment because the code is in TensorFlow with version 1.3 and Python 2.
And we apply two datasets that I have shown before, which is security.yml, dataset, styles 1.
An interesting thing about FluidHub is that it allows to version the datasets, so we can create several versions of our dataset and we mount it in styles, styles bar.
And in the same way we mount our photos in the photos directory.
And simply what we have to run is what we say, run me with the python command that is already installed in the environment, the code evaluate.py and we say that it continues that it simply applies this style, wave, on the photos directory and we run it in the directory bar output,
which is a special of FluidHub that allows us to obtain the outputs.
And then, as our images have different sizes, we allow the option that it has different sizes and we do each image one by one.
Let's see.
We can see all the ones that it generates in the terminal.
Here we have all the jobs.
Ok, so I should have finished, but I don't know why it says that it is still running.
Well, if not, show it the images that we have generated.
Yes, and finished.
Well, so as not to waste a lot of time, what it generates in FluidHub is basically this.
This is the wave style that we have passed and basically this is what it generates for each of the images.
The original image is applied to the style of the picture of the original image.
And that can be done for each one.
In addition to the pedophiles, you can train with anyone.
What happens is that it is quite expensive.
Depending on the GPU, it can take 4 or 6 hours.
Well, we have reached the middle of the talk, more or less.
Thank you for holding up here, which is the most intense part, but it is important.
You can't talk about cybersecurity or artificial intelligence without knowing these basic concepts.
So, with this basis, we are going to talk about what we can do with this in cybersecurity, what is being done and what are the limitations.
A presentation that I liked a lot a few months ago and I usually use it a lot in conferences, is one that Adi Samir gave at the RSA Conference, the last one, and he said a phrase that, because of this phrase, we have created basically the talk we are giving you and we will stay in other places.
Adi Samir, who is a world-renowned cryptographer and one of the most brilliant people that probably exists on the planet, said a phrase.
I believe that artificial intelligence is useful for defensive security, but not offensive.
At this time that we have left, when we get to the end, I have almost reached the conclusion that it is quite the opposite, that it is almost more useful in offensive security than defensive.
Now we will see.
But, well, if Samir said this, it is because there is a kind of context that, maybe, escapes us or is evolving, and it is what we have tried to collect the tools there are, that there are not many, and, above all, we will try to explain in a human way,
if possible, things that are being done in a more theoretical environment.
When we talk about cyber security, the interesting thing would be to think what problems there are in cyber security that humans do not know solve quite well, and if artificial intelligence could complement it in some way.
In security, there is a huge list of problems to solve that we are not able to address.
Here I have simply put a few.
The typical ones of software exploitation, which are the traditional ones, that is, systems are vulnerable because we do bad software.
That can cause a third party to do a remote, local, etc.
and vulnerate our systems.
Everything related to the fact that we do not know how to configure the systems well, and everything related to the major security failure that we humans are.
We are the weakest link.
When we talk about applying artificial intelligence to security, many people will say that they know how to apply it or that they have traveled, but there are a lot of problems that people usually do not comment on and they shut up.
It has to do with the quantity and quality of the data.
If we ask someone how much data we need to solve a specific problem, the usual answer is usually , the more data, the better.
But that answer, let's say, is not very scientific.
Or normally they usually say, well , maybe we do not need a lot of data, but we want them to have quality.
What does it mean to have quality?
That is the problem.
It applies to all domains, but in the case of security, there are scenarios where that does not work.
Imagine, we have a threat that we have never detected.
How do we detect a threat that we have never detected?
Well, people would say, well, very easy.
Analyze what is normal and what is rare, raise an alert.
Well, they are already putting alerts on a human, who has to waste his time to see if that makes sense.
Therefore, we do not know how to do that better than we did 20 years ago.
And that is the truth.
Then there is a problem related to static training.
All this that Jose has told you, there are scenarios where it is useful.
You have a model, you train it, and you classify it.
But in security that is not very real, because the attacker evolves his attacks.
So maybe you have a model to detect traffic in a network in one way, but the attacker does it in another way.
So your system would have to be able to do a continuous training.
And a continuous training in the presence of an attacker, which was what Jose Degan told you.
As I will tell you now, there is no solution for that.
And it's not because I say it, but now I bring my friend DARPA, which is an entity with enough solvency to justify that there is no solution for that.
Then there are a couple of issues.
I'm going to highlight this one and these two that worry us a lot.
Everything related to model piracy, now we will see, there are attacks to steal artificial intelligence models.
And with those models, once you have them or approach them, as we will see now, offensive security attacks can be carried out.
The idea that we want to convey especially with this talk is that offensive security is a reality and defensive security has to put the batteries to adapt to this.
And above all, something that we defend in our day-to-day life with a lot of importance is if someone uses artificial intelligence in security technology or in any security product, that element has to be audited.
It has to be audited from the point of view of computer security.
Just as we do a pentesting of a web, we do a forensic, we do a reversing, whatever.
That is, it is one more issue to put if we already have a few of them within what a professional of security should do.
What I was telling you, the biggest problem, and there is no solution, and recently, a couple of months ago, DARPA, the intelligence agency of advanced projects North American, has drawn different contests.
If someone has a solution, this agency is delighted to give it millions of dollars, it will be for money, of how to make artificial intelligence systems that train continuously in the presence of an adversary.
That is, an adversary that will really try to make your system fail.
That is the reality.
The reality is not to mount a static isolated system that changes every X time.
That is not real.
And it is not real because the attacker is learning how to deceive those systems.
So, here we go back to what José was telling you, and it is very important, about GAN networks.
In security, this scenario is well known.
For a long time, we have assumed that the attacker lives with us.
He is in our house, uses our heating, that is, he is in there, and we have to adapt to that fact.
That is , this is no longer a medieval culture where we have a wall and the attacker is outside that wall.
We have to assume that the attacker is inside.
And the cool thing about this idea is that you assume that you have to use the attacker to make your artificial intelligence system more robust.
Well, one of the most complicated issues in artificial intelligence and it does not have a clear solution is how to apply privacy.
That is, as José was telling you, to do training, many times we do not have the ability to calculate and we have to depend on a third party, Google, Amazon, whoever.
We upload our model there, trained or not trained, whatever, and it is there.
How do we know that a third party does not steal that model?
How do we know that a third party does not modify that model?
As we will see later introducing back doors.
Well, today there is no solution for that.
The only thing there is are more academic proposals which is to try to make networks, especially with neural networks, encrypted neural networks.
That is, to apply cryptography to neural networks in such a way that you can upload everything to the cloud or wherever and even if someone steals it, that model will be encrypted.
It will not be able to get information.
And it will not be able to make inquiries because to do that it will need a cryptographic key.
This is being worked on with a lot of interest because it would really be the only effective solution to attacks that we will see later of model theft.
If you want to have a reference, you can put cryptonets and it will help you find more information.
Well, let's see some aspects of defensive security as an introduction to offensive security.
What is the use of artificial intelligence in defensive security today?
Well, today, the last 20 years, right?
Basically, in these scenarios that I comment here.
It is usually used a lot in everything related to access control, authentication, authorization, identification and profiling.
It has always been used where it has been more useful in everything related to traffic analysis, registration, IDS, IPS, escape, fraud detection.
It is used a lot and malware prevention.
Those are the typical ones where it has been used and artificial intelligence is used more seriously, I don't know how to say it, less professional, with static training, etc.
But there are many other scenarios where it is used, for example, in software development, in risk management, in security operations, etc.
With what we said before, that in security we have problems with data.
We probably don't have data because if something has an attack, usually the attack changes.
So maybe modeling an attacker based on the past is not interesting to predict the future.
And then everything related to automatic training.
Currently, there are more and more proposals to collect accessible information related to defensive security.
One of the interesting projects is the one I mention here, which is MLSec, which basically tries to collect many projects and tools of artificial intelligence dedicated only to defensive security.
So, if you are interested in that particular niche, you can take a look and, in many cases, they collect DEFCON talks and other hacking conferences with specific solutions to problems.
In this transparency, I don't know if you can see it well, the cyber part is this, what I want to show you is that really all the big companies are betting on applying artificial intelligence in some way to security products, especially in monitoring,
traffic analysis and malware detection, or malicious code, let's say that's the most interesting part.
But, well, there are other actors, more like start-up companies, almost always USA, England, Israel, which is usually the cradle of most of the technological advances, that try to give other approaches.
But, well, here I have left you others, but in the end, most of them, or at least the ones we know, because many of these advances are kept, and it's logical that this knowledge is kept to protect certain clients, they do nothing more innovative than what was done a decade ago.
And this is something that has surprised us a lot.
Here, quickly, I'm going to leave you two examples of interesting things that are being done differently in defensive security.
And we already introduced offensive security.
The two examples I'm going to give you are related to data protection.
One has to do with cryptography, and the other has to do with steganography.
Well, I'll skip this one, and I'll tell you about this one.
I don't know if you read some time ago that Google had invented a cryptographic algorithm of a single use.
I don't know if you know this news.
Basically, the idea, and it's very interesting, if you know something about cryptography, you know that we have a problem for the next decades with the advancement of quantum computing, with a new type of quantum computer, which will mean a series of problems in certain cryptographic algorithms.
What did Google propose?
Well, maybe we can use neural networks to somehow generate cryptographic algorithms of a single use.
Usually, cryptography is a more key algorithm.
We change the keys in each session, in each process, whenever.
What Google proposed was, well, if I can make a cryptographic algorithm that has a level similar to that of a human, of a single use, even if we had a very powerful computer, maybe that could be enough to protect us in that scenario.
What did Google do?
Well, again, they proposed a GAN scenario.
Here we have our typical Alice, Bob, and here we had the intruder, the spy.
Different neural networks were used in a training process to train and try to deduce their own algorithm.
The only condition here is, let's say, the training function is that the attacker becomes more and more clumsy when deciphering the messages, when understanding the messages.
So, training is done, and after a series of times, the conclusion is that Alice and Bob deduce an algorithm that the one in the middle is not able to decipher.
That's a pretty revolutionary idea, pretty cool.
What's the problem?
If we look at it in detail, what the neural networks deduce, let's say, the cryptographic algorithm, a primary school kid would make a better algorithm.
Because, basically, what it deduces is something more sophisticated than an XOR.
If you know something, an XOR.
But, well, there are some properties of human cryptography that it deduces, like, for example, the avalanche effect in cryptography.
That is to say, there are certain things that the neural networks deduce, and that is very promising.
I don't know if one day we will be able to do this for real, but I think it's an interesting idea.
Today, for nothing, it's what has been published.
I mean, this doesn't generate cryptographic algorithms of a single use.
It's more of an experiment, but it's a trend of how big companies, like Google, try to approach defensive security in another way.
Here we leave you another example and a little simulation that we have done, which is with a steganography system.
Steganography, basically, is a science to hide information in different media, and that an attacker is not aware of an undercover communication.
Again, here we have a scenario like GAN, in which we have a generator that, basically, will try to generate images that look like some data, for example, from a database of personalities.
Here we will have a discriminator that will tell us to what extent the image we put in is real or not real.
That could be a more classic scenario to deceive authentication systems, but in this scheme, additionally, we put a steganography technique.
Basically, they use a simple one, which is to hide information in the least significant bit.
And, in turn, they put a steganalyzer, that is, a detector.
That is, hey, what you are hiding, how easy it would be to detect with the algorithms of current steganalysis.
What this scheme deduces, and this is more interesting than the one we saw before of Crypto, is that with these neural networks, you can generate algorithms of hiding more robust than those that exist today, against the most robust attacks that exist today.
So, simply with the same scheme and with the neural networks, many of which Jose has told you about, you can make new proposals of defensive security beyond the traditional ones of all life, of analyzing traffic, static training, etc.
Well, this is the context that is very good.
It is advancing, trying to overcome the limitations of continuous training, of privacy of the models, etc.
But, of course, what happens if we try to do this in reality?
That is, if we want to introduce an attacker in these scenarios?
Well, that's where the problems arise.
The next transparency, what we are going to comment on quickly, are different attacks that we have included, of which we have compiled what would be offensive security in artificial intelligence.
That is, how to attack systems that use artificial intelligence.
We have categorized them into four large blocks.
Classical attacks, which would basically be hacking attacks of all life, but that use artificial intelligence, because they work better.
Attacks that we have called synthetic, which would basically be attacks that could be perhaps with image processing or in any other way that are used to attack artificial intelligence algorithms.
The coolest ones, which are artificial intelligence against artificial intelligence.
And then a disaster drawer that we have not known very well how to categorize, that we have put a bit of everything there.
Model theft, home doors, etc.
In any case, if they try to evaluate the security of this type of system, they always have to think about this scheme.
As when they did, if they are dedicated to the subject of pentesting or are interested in these things, as they would do in a black box pentesting.
When we do not have information and we have to attack a system.
We would have to think what an attacker can do if he knows only the entries.
What can an attacker do if he only knows the exits.
What can an attacker do if it can affect the training phase.
What can an attacker do if he can only do things in the test phase.
What can an attacker do if he only knows the model.
And what can an attacker do if he can manipulate the algorithm in some way.
Either remotely, with inputs, or physically, as is done in other attacks, introducing fields, electrical modifications, power, etc.
Let's go point by point.
The first one, classic attacks.
Here we have left you two examples.
This one I show you above, the pass-gun, is how to use neural networks for, for example, a password dictionary that has been stolen from LinkedIn.
You pass it to this neural network and the neural network is able to deduce a rule.
A password cracking rule.
That is, given a hundred thousand passwords, it extracts a rule and that rule will be able to deduce new passwords that are similar to that dataset.
What is this for?
Or what does this tool and this research show?
It shows that by doing this, we can crack passwords more quickly as we would do, for example, with John the Ripper or with Hascat, classic hacking tools.
So, in the real world, this tool gives better results than the classic tools that we had done.
Here I leave you another quick example.
This is from the last DevCon, this summer, in which deep learning was used in SQL injection techniques.
When you detect a vulnerability in a system and try to hack the database or extract information, on certain occasions you need to detect certain parameters, for example, know the name of a table or other things.
Sometimes you do that by statistics, you do it blindly, you do it by force of habit, in many ways.
What is deduced in these researches is that using neural networks can be finer.
You can get the same result with fewer requests.
That's why it could be interesting to use deep learning.
So, there are many examples.
For a while we have only used these two, of how to improve lifelong attacks, in this case, cracking, code injection, fuzzing, whatever, using artificial intelligence.
Here, this is another particular example.
In this case, it's not like the previous one, that generates a rule, it's just that you see passwords, you have a list of passwords, and what this neural network does is extract passwords that are visually similar to the previous ones.
It's not generating a cracking rule, but maybe in certain scenarios it can be useful.
Here we leave you the Python to do it, and more information on how to do it.
Well, synthetic attacks.
These are very famous, they have come up a lot, we are not going to spend much time on them, because they are very well known, but it's how to use external systems to modify a data source, in such a way that the system, typically machine learning, classifies it wrongly.
An example here would be facial recognition.
If we want to make this girl's face be recognized by the system as this actor, because that recognition system recognizes actors, the girl simply puts on glasses and makes the system deceive her.
Why does this work?
Basically, as we will see later, these glasses have some information, some pixels, that will cause the training system to fail.
There are other very cool schemes, you probably know this one, which is to be able to falsify videos in real time.
You take the video of a person, if you then connect to the page, it's very interesting, with Trump and different personalities, they take a real video, you take an actor, and the video, in real time, adapts to what you do with the face.
So if you open your mouth, maybe Schwarzenegger opens his mouth, just like you.
There is still a little bit of show part, as I say, there is a little bit of something strange that is seen, but the truth is that it is very successful.
The possibility of making masks in 3D from 2D images to deceive the systems is quite old, but it has a long history.
And here is another example of a very powerful company, that worries a lot, a lot of organizations, which is the possibility of, with very few seconds of a person's audio , being able to synthesize their voice.
You can see, there are different demos, and you can see different, they have done it with Obama's voice, and with other personalities, like with 30 seconds of voice, they synthesize the voice to generate any text.
And the truth is that it is quite scary.
The trend is that in the coming years, and this is a very close reality, we will probably not have the capacity to differentiate what is a real audio or a real video.
So we will probably live that in the next five years, because this reality is quite important.
What we are going to devote more time to are the other two attacks that remain.
Machine learning attacks against machine learning, and what we have called mixed attacks.
One thing that we have wanted to highlight here, in a more important way, is that more and more scientists are starting to alert the difference that can exist in terms of offensive security of the most academic world, of the most military world.
For example, of DARPA and other organizations.
That is to say, offensive security is a real problem that we should take into account.
When we talk about artificial intelligence attacks against artificial intelligence, we basically talk about two scenarios, two types of attacks.
Evasion attacks and poisoning attacks.
When we say evasion attacks, imagine a classification system that detects anything, what we get is that the system does not classify us in another category.
Imagine that the system classifies if something is malware or not.
If the system has 10 users, I am Pepito and I want to access as your user.
That is to say, those types of attacks use evasion techniques.
And poisoning techniques, is basically to introduce garbage to the system to cause different effects.
Of all this, there is a lot of theory, a lot of papers, a lot of information.
But more and more, more practical libraries are published that people can use to see that this makes sense.
Here we have reflected a couple of them, the most important ones.
And now we will see a couple of videos of demos that we have simulated.
We would have CleverHands and DeepPowering.
Ok, let's see different demos with CleverHands and with DeepPowering.
And now I will make you a series of comments about the demos.
Let's see with CleverHands three types of attack.
Two of them are white box, these first two are white box, and the latter are black box.
Basically, what all these algorithms are trying to do is to create an adversarial example, that is, in the case of images, create another image that from the human point of view is imperceptible from the original but that applied to a machine learning algorithm makes it get confused when classifying it.
So, let's use CleverHands sorry, FloydHub to simulate both of these attacks and then one of black box.
Ok, while Jose puts it I will summarize a little to make clear how these attacks work.
White box attack and black box attack.
In the white box attack basically what we are going to be able to do let's say doing a similar in security are directed attacks and massive attacks.
What does that mean?
In the white box attack we need to know the training model and the configuration parameters that is, we need to know everything.
In that scenario we put inputs and we get outputs.
With that information, what can we do?
We can make the system misclassify in general, what would be a massive attack, that is, imagine if the system classifies 10 numbers from 0 to 9 I can make that by putting a 0 classifies anything that is not a 0.
Then I can access the system but not with what the system would understand.
And then we would have the directed attacks, which we will see later a video of a demo that we have done in which we could make the system classify what we want.
That is, I put a 0 and I make the system classifies me as a 1.
I put a 0 that classifies me as a 2.
I put a 0 that classifies me as a 9.
And those would be the attacks of the white box.
Now I will tell you about the black box.
Basically to run the demo we do practically the same as before.
The only particularity is that we enter it as gpu, simply saying minus minus gpu it will apply gpu automatically without doing anything and we simply apply the code.
In this case we do not apply the data parameters basically because when you run this code it already lowers it and as it is very small the set of data is not necessary.
So let's see it in FluidHub.
Ok, here we have it running.
It has not yet started to train.
What this attack is going to do , which also allows us both to attack and defend ourselves, what it is going to do is create a model that itself is good.
The problem is that if we create adversarial examples, that is, we put noise in the images, it is very bad.
So what do we do to improve it?
We generate with the algorithm they use, it is basically applying a noise following a determined equation, in such a way that we generate adversarial examples and in the training phase we pass those to them.
What it does, that is known as adversarial training, allows our model to be more robust against an adversary.
So this should have started training.
Let's see if reloading...
Ok, here it is.
We have our training data.
It must be said that all the demos that we are going to see are about the set of data of the MNIST, the set of data of the written data.
We have 60,000 training data, of test 10,000, in this case we have divided it into two parts, and here we see how it is being trained.
I think it applies six times, so in the first one it takes 7 seconds, and it has a precision, that is, it classifies in 98.80% of the cases correctly.
So it does it for each time, in the third one it already has a 99.12% precision.
As I was saying, in the scenario we have, we know all the models, the configuration parameters, and we are going to play with the inputs to which we are going to be able, now later it will look better with a video that we have made, modifications to those inputs to make the system classify badly.
And depending on the attack, we will be able to do, or the massive attacks, as I mentioned, or the directed attacks.
If it takes a long time, we put it here.
All these examples, well, you have it there in the GitHub project that we have published, in case you want to see that it is easier to simulate.
Are you with the box one, there?
No, this is...
This is...
Basically, what I was saying, it is trained with...
without the models...
I mean, without adversaries, and then we apply it to adversaries.
What I don't know is why in this example we don't get the...
the algorithm precision, but if I remember correctly, it went from...
in the case of the original model, it was 90 and something percent, and applying adversarial training, it was around 90.
I mean, we lose precision, but we make it more robust against adversaries.
So, let's see the black box one.
Ok, I will quickly explain the black box one.
For us, the black box one is a discovery because it is a truly powerful attack.
In the black box attack, we don't need to know anything about the training environment.
I mean, we don't need to know neither the model nor the configuration parameters.
In this scenario, the only thing we need to do is a few queries to the real model.
I mean, input and output, that's all we need.
With inputs and outputs, by some properties of artificial intelligence, transfer learning, we can create a substitute model, a local model.
And with those inputs and outputs, we are able to train our local model, and it is in this local model that we are going to do the tests, where we are going to make those adversarial examples and we are going to test which ones are the best to make directed attacks or massive attacks.
So, once we have that and we know what each thing is, we take those attacks and we launch them against the real system.
This is quite powerful and, well, the different references we leave you have been used quite successfully in the usual Amazon, Google classifiers, etc.
I mean, it is able to deceive in a black box without knowing anything about the system, just with a few queries to the real system.
We are going to put that it is already calculated.
...
Let's see this one, which is also a blank box.
I'm going to put it here with a video.
Let's go fast, because we don't have much time left.
Well, basically, here are some simulations.
We have also put it in the GitHub.
You have the URL with TensorFlow and with CleverHands.
Basically, what we are going to take is a database.
In this case, it's Whitebox attack, let's say.
It's blank box, you need context information.
What we are going to do is we are going to do input attacks and output attacks.
And what the attack consists of is the following.
We introduce an input, we are going to put the minimum possible disturbance.
Normally, that disturbance can be 2%, 3%, 4% of the input.
And we are going to generate something else.
What are we going to try to do with that other thing?
We are going to try to make the system classify badly.
And that it classifies, in this example that we give you, as a directed attack.
So that it classifies what we want.
Imagine that we have malware, for example.
We make some minimal modifications to the binary so that the system detects it as goodware.
So that it does not detect it as malware.
In this scenario, because it is easier to simulate, we put it with images.
Here they are doing different things, this does not matter.
Let's go to the table at the end, which is the interesting thing.
Let's see...
Here.
Let's see if I can stop it.
I'm going to quickly explain what this table is.
I don't know if you can see the differences very well.
Here we have 10 columns.
We would have 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9.
10 columns.
In this classification system, we take the 10 numbers.
And what we want to do is, given an input, we want to be able to know precisely how we can manipulate the input to produce the output we want.
So, imagine.
We, for whatever reason, can only put 0.
But we want the result to be, for example, 0.
So, if you see the table, here we don't have to do anything.
In the original image.
I put 0, 0 comes out.
For some reason, I put 0 and I want 1.
Well, after all this training, what I do is, if I put 0 with minimal disturbances, which I don't know if you can see very well here, I make the system classify it as a 1.
Basically, what this table is telling you is how you have to modify an input to get the output you want.
With which you can make directed attacks.
Here, think about the security domain you want.
What you have to validate is if in the particular security domain and in a tool, this attack can be made.
How to get this table?
Well, it depends.
As we said, it can be done with white-box attacks, if you know the model and the configuration parameters, or with black-box attacks.
Either the example José mentioned, generating an alternative model, the local one, where you do your tests until you calculate, for example, this type of table, or with other types of attacks that we will see now.
Well, then there is another...
Clever Hands is the most developed tool, for use with digital images, but then there are quite famous ones, which is DeepPowering, which was supposed to be the Metasploit of Machine Learning.
It was supposed to be, because it's been like a year and a half, right?
That they don't upload an update to GitHub.
If you have time, we recommend this paper, because it's fantastic, how to generate black-box attacks to artificial intelligence systems.
If at some point you want to protect yourself from attacks of this type, I recommend you read this paper, about how you can generate generic models, the local one, to be able to attack any artificial intelligence system without knowing its configuration.
Well, let's go to mixed attacks, we don't have much time left.
To sum up, we have improving life-long hacking tools with artificial intelligence, synthetic attacks against artificial intelligence attacks, especially to more authentication systems, biometrics, etc.
Artificial intelligence against artificial intelligence, basically the Gantt scenario that we have discussed.
And then there is a mixed attack, where there is a bit of everything.
Here we have simply put two examples.
One of them would be model theft.
Well, so far what we have said is, option A, you know all the data and you can make the attacks.
Option B, I generate an alternative model that is close to the original, without having information.
But then there is another option, which would be to directly steal the model.
How can that be done?
Well, it can be done in different ways.
In this paper, well, you have several real examples against Google, Amazon, to steal their models of artificial intelligence, but we wanted to give you a more human example, if you can say so, so that it is better understood.
Imagine that we have a service in which, for some reason, it uses this algorithm of artificial intelligence, of logistic regression.
If you know about machine learning, it has this form, this algorithm.
What would the attacker have to do?
Simply enter inputs and take outputs.
It does not need anything else.
So, by entering inputs and outputs, basically what you can do is solve certain parameters of this equation.
For example, here the X and here the Y.
So, simply, depending on the dimension of the other parameters we need, we simply need to have many equations.
So, imagine that here the formula had dimension 100, we would need 100 queries, to have 100 equations.
In local, we would solve those 100 equations and we would have the value for all the parameters.
At that moment, we would have already stolen the model of artificial intelligence.
If we have the model of artificial intelligence, we can already do the attacks or calculate the different attacks that we have told you.
So, simply, if one day you use a service of this, or you expose it to someone, for example, via API, think that many attacks can come simply by someone who consults and has the outputs.
So, that has to be thought in some way if it can be protected.
Well, another of the issues that we have put in mixed attacks that we find quite interesting is the possibility of introducing backdoors in artificial intelligence algorithms.
This has a lot to do with what we have told you before about where we upload our training models, or who we depend on to do what tasks of artificial intelligence.
Why do we say that?
Because today it has been demonstrated how, from certain models, for example, of trained neural networks, it is possible to add additional ones that act in a way in the form of backdoors.
In this image we reflect it a little.
Imagine, we have our system that, for example, classifies objects, whatever.
We are able to add, for example, an additional network in such a way that the system functions in the same way as always, but only in particular scenarios the system behaves in another way.
That is, a backdoor of all life.
It can be done in an additional way adding it or even fusing it with the system itself.
Putting a clearer example to understand what this means.
Well, I'm going to put this one and then I'll put another one.
Imagine a scenario in which we have an artificial intelligence system that classifies car signals.
And for some reason, as Jose from Baidu told you , who is on the Internet and you can download it and use it in your systems, for some reason someone introduces a backdoor in that model.
What would happen in that scenario?
Well, imagine a car.
Normally, the stop signal would identify it as stop.
Then the car would brake.
But if, for example, and it is the most common scenario, you put a little sticker on this signal which, by all means, activates that extra neural network that we have put in, we would activate the backdoor.
So, in this case, for example, the car would accelerate in front of a stop signal.
So, well, it is a very visual example to understand that we have to be careful with the privacy of the models and where we leave, because nowadays it is already known how to introduce backdoors.
In fact, there is an additional problem that we have put here in text, which is the transfer learning and it has a lot to do with how this technology is advancing and how people are sharing information.
Imagine a scenario.
A university or a research center that gives an algorithm to detect objects in general.
And you, for some reason, see that this model is very good and you want to reuse part of this model.
And maybe, if it is a neural network, you just re-train the last layers or the last phase, instead of detecting objects in general, to detect lamps.
And your system has taken advantage of a code that someone has shared with you, great, and you have specialized it a little.
In this scenario, you also have to be careful, because with the concept of transfer learning, basically, what it tells you is that in a general model, if you introduce a backdoor and someone specializes that algorithm, you take the backdoor.
You take it as standard.
That is, the backdoor is not eliminated.
So, if you use models...
I understand that we have not yet detected people who use this in practice, but theoretically it can be done.
So, if you use third-party models, you should know that there is a risk that should be evaluated.
Well, here we have left you many other examples.
This is a bit the same.
Well, let's talk a little about formalization.
Well, I don't know if this is really the most important thing, for us, at least in security.
If we apply artificial intelligence to any scenario, and in particular artificial intelligence to security technology, we have to know if machine learning is safe.
If you know something about cryptography, know that there are scenarios in which we can use technology that is not safe, but combining it with other insecure technology, make the global be safe.
I give you an example.
In cryptography, for example, we can use hash functions that are vulnerable to collision attacks.
For example, MD5.
We know that, given two different inputs for this algorithm, the result is the same.
There are combinations in which we can combine more than one algorithm that is vulnerable.
Imagine MD5 and SA1, for example.
In such a way that an attacker chooses an X and an Y input, it is very difficult for it to find an XY that collides in both functions at the same time.
This is a classic example of cryptography.
In artificial intelligence, this is one of the proposals.
Maybe we have an attack that can be vulnerable to offensive security, but putting two similar ones, the same attack may not work for the global system.
It is a home solution, so to speak, but it can work in many scenarios.
It could be an alternative.
In the even more theoretical world, what is being done is to see how detection functions or algorithms can be perfected to be robust against offensive security attacks.
Well, just to finish, in case you want to comment on anything, a series of recommendations.
Basically, offensive security is important in artificial intelligence and we have to put it as one more cycle when we do pen testing of things.
Artificial intelligence also has to be pampered.
It is difficult to protect yourself, especially, for example, against black box attacks, it is difficult to protect yourself.
So, as far as possible, we have to try to make a more global architecture design and consider artificial intelligence as one more element, not as a solution, because it is not, but as one more element.
To manage the risk, this is typical in security, and, more and more, to pay attention to the privacy of the models, who has access to the models, how they can modify the models and where they will be executed.
Well, therefore, this uglier image that José told you at the beginning of the whole cycle, you have to put the layer of security.
Where can it be put?
In many phases, but as little, let's say, in the phase, especially to design safer systems, in the training phase.
With that model that José Degán told you about, we have to introduce the tools here and see how this cycle works in the presence of an attacker and make the algorithm work better.
So, well, this is the scheme.
To conclude, and now José will give you some other recommendations, the intention of the workshop was to make more visible the part of artificial intelligence, that there are people who may know more or less, and, above all, highlight the security limitations,
that there are many.
We do not know how to solve them, we do not know if they will have a route or not in the coming years, certain proposals, certain products, certain algorithms, but, above all, what you have to be aware of is that offensive security is advancing very fast,
especially the attacks of Blackbox, in the last year there have been quite powerful attacks.
So, well, it is something we have to consider if we design any type of system to be robust.
And, well, a little here at the end, we leave you this phase that we like a lot, that, well, we really depend on artificial intelligence, because it is true, and more and more, but, well, we will have to work on it little by little.
And, to finish, we leave you a series of workshops where these same topics are dealt with as the ones we are talking about here, about machine learning and computer security.
One already took place at the beginning of this month, on November 3rd, and another that will take place next week in one of the most important conferences of neural networks, NIPS, where all these topics are going to be dealt with, attacks, defenses,
privacy, where you can already see that emphasis is being put not only on security, but the experts in machine learning are realizing that we have to protect these algorithms because they are practically present in the day-to-day, and in the future there will be more.
So I think we have to design ways, today they will be more theoretical, in months, years, they will be more practical, but we have to put emphasis on security.
And now, to finish, we leave you some for those who want to get started in the subject of machine learning, or want to see more specifically the technical details of the attacks that we have told.
Here we have a series of courses, perhaps the most typical of machine learning is Stanford's.
Now they have launched a specialization of deep learning with five courses, where neural networks are introduced, some of the types are convolutional networks, recurrent, etc.
And then other formations like Udacity.
It depends where you want to emphasize, in machine learning in general, in deep learning in particular.
And then a series of conferences worldwide to know what is being done, in which topics machine learning is advancing.
One of the most important are NIPS, ICML and MLConf.
These two are more theoretical and MLConf a little more business oriented.
And finally, we leave you a series of companies that are investing a lot of money in artificial intelligence, also in machine learning.
And that's basically what we wanted to tell you.
In fact, the problem, especially if you are dedicated to security, we have people who are dedicated to security, especially more global security, more architecture, more design, new elements appear, one of them is artificial intelligence.
And in this particular case, to understand attacks well, it requires a lot of specialization.
Many of these courses are like the basis and are not enough.
We need very specialized people in these topics, with people who, for example, are dedicated to security, to see how to defend ourselves, well, defend and attack, because in the end you need to know the attacks to defend yourself better, to make systems much more robust.
And nothing else.
That was a bit of what we wanted to tell you.
Thank you for your attention.
We hope it was interesting.
We have a few minutes, and if not, by email or Twitter, I'm happy to talk about anything.
Thank you all.