My report will be about the security of microservices.
In particular, I will try to tell you in my experience some methods that can be used to assess basic and fast microservice applications from the point of view of security and identify architectural vulnerabilities.
At the moment I work at Huawei in a laboratory called Chongming but I must say that the content of the report has nothing to do with the current employer.
I try to teach as much as possible at the university in the Moscow State Technical University under the name of Nikolai Ernest Bauman.
And, again, as much as possible, I try to participate and make a contribution to the open-source community.
In particular, a couple of pull requests to the project Avast Cheat Sheet I did.
If I try to tell a motivational story, it turns out something like this.
Imagine that there is a hypothetical information security engineer or even, most likely, a product security engineer who periodically faces the task of quick assessment of the security of some microservice application.
That is, there is a customer who comes to this engineer.
And, most likely, this is not some big company that has all the wonderful processes, all the DevSecOps, and all the wonderful approaches of Everest and Code, but some small startup that has the first application delivered to the user or sale in a week .
And suddenly security issues arise, of course, not with the customer, but with the buyer.
And there is a task for a short period of time, maybe two weeks, to look at the application, to identify vulnerabilities.
Well , of course, the customer does not say such words, but the task is set absolutely in general, to see if it is safe at all or not.
Well, it is clear that there is a temptation, having, again, access to the test environment, having access to the sources, there is a temptation to open Burp or set up the same rules as Grep in order to immediately start looking for vulnerabilities.
But the engineer probably understands that the first thing that interests the customer, most likely, are some architectural vulnerabilities that, if you do not think about them now, and think, for example, in a month or two, their correction will take more time.
That is, conditionally, if you move to cross-site scripting today, then, in principle, it is not so critical if you only understand in a month that it turns out that the whole authorization scheme, which is implemented in this application, or the component that does the authentication,
is not at all able to satisfy the requirements of the next customers, there is no integration.
Therefore, the task becomes more narrow, that is, you need to concentrate and identify architectural vulnerabilities, having access to, of course, the source code, there is a test environment, there are no artifacts, in most cases, which describe what the solution is.
And the engineer goes to the search engine, because, most likely, there are already some checklists that were written by smart security architects, which you can quickly go through, answer questions, check something in the code or check in the configuration,
and perform at least some basic assessment.
That is, without going too deep into the subtleties of architecture, just at least at the basic level answer, in general, is it more or less safe?
Can your customer live with this application, modify it somehow?
Because it is clear that these microservices are constantly being modified.
I tried to find such a checklist, but there was no such thing specifically for microservice architectures.
In my opinion, this is the standard from Avaspa called Application Security Verification Standard, which, in principle, is a set of several checklists that are divided by domains.
In particular, there is a checklist that is specifically associated with architecture.
It can be used as a basis, but it still turns out that as our hypothetical engineer uses this checklist, he understands that either some checks in this checklist from Avaspa are too general, because in general this checklist refers to both monolithic applications and microservices,
but the microservice aspects are not particularly detailed and not disclosed.
Or there are some aspects that are revealed, but which are not in these checklists.
And, accordingly, little by little, some checklist appears, which is actually some kind of adaptation of Avaspa-ASVS, perhaps taking into account some other materials from the Internet.
For example, for the last year NIST has published two good enough documents related to microservice architecture about the reflective aspects of security.
One is about microservices, the other is about service mesh.
And, in principle, I have more or less got such a checklist.
I posted it in the open access, the link will be on the last slide.
And, accordingly, I will try to tell some basic interesting things illustrated by examples in the presentation.
Since we are looking for vulnerabilities in microservice architectures, I would like to say a few words about how, in general, in terms of security, how architecturally microservice applications differ from monolithic ones.
Again, this list is absolutely subjective, it is absolutely not exhaustive.
Nevertheless, the first point is the so-called polyglot architecture.
All these microservices can be developed, of course, by different teams in different programming languages with different frameworks.
The choice of these frameworks and programming languages can be either objective, when a system architect analyzes a non-functional system requirement and says that this component will be highly loaded and we will write it on Go, and this one you can write on Node.js or on Java.
Or the choice can be absolutely subjective.
For example, when a team wants to get an expertise on Go and write this expertise in a resume, they choose Go.
And it turns out that all these programming languages and frameworks can have absolutely different architectural solutions that can allow to implement these basic security functions in this architecture.
Then, since microservices communicate with each other synchronously or asynchronously, they still get some external interfaces, APIs.
And despite the fact that these APIs are usually published through a single point like an API gateway, the number of these interfaces increases the attack surface in any case compared to monolithic applications.
The number of protected assets because each microservice will have its own sub-AD, an instance of the database management system, some cache.
If earlier you understood that there is one instance and all the protected information is exactly there , it is not clear where, but it is exactly there, then it is quite difficult to answer the question with these things, if you do not dig deep into architecture,
where in which instance we have some personal data, that is, some data that is processed by all these applications and can be classified as assets.
In addition, all these microservices cannot exist in a vacuum, all sorts of additional basic services appear around them.
A certain component appears, the registry service, which is a basic thing in which a rising microservice comes and says, hello, I am microservice A, I want to work with microservice B.
This registry service says that microservice B is on a host with some IP address.
Authentication services appear, these things appear, API gateway.
These are the basic things that do not directly encode the development team, because it does not carry any data, but they appear in the infrastructure because the whole system does not work without it .
In addition, in addition to standard communication protocols of the external world with the system, such things as, for example, queues appear.
This is a very popular pattern when you do something from the sphere of the Internet about things, you have a smart home or something like that, everything is in these sensors, and they, of course, do not send requests to your smart cloud application but use some queues.
These additional interfaces also have a very strong impact on security.
In addition, since all these microservices are distributed, in most cases, it turns out that one operation that the user does on the browser turns into a chain of calls between microservices, it is more difficult to throw the context of the user.
Perhaps the last microservice, like here, for example, microservice number 2 on Go will not know, that is, it will not receive the context of the user who made the request and will not be able to authorize correctly and will not do anything that will lead to these attacks like IDOR,
BOL and something like that.
Now about the functions on which, first of all, my subjective opinion, you need to concentrate when analyzing this architecture.
It is clear that all these functions are more or less known, they were defined in the middle of the last century, but if you try to design them on a microservice architecture, then in terms of authentication and in terms of architectural assessment you can see how architecturally the authentication between your application and the outside world is designed,
that is, how it is all done on the border, how the inter-service authentication is performed when the request has already entered your microservices and travels through them.
The same applies to authorization.
ID propagation is the story that this request from the user that comes from the browser and reaches the last microservice in the chain must always be accompanied by the context of this user and logging.
Returning to our task, there are applications, there are artifacts.
In general, we realized that we will try to find architectural vulnerabilities at the first stage of the basic assessment, and we will analyze the architecture in terms of these three security functions authentication, authorization, audit.
Before starting the assessment, it is necessary to understand what application you are dealing with.
Accordingly, our hypothetical engineer will be very lucky if he comes across a company that has very good, relevant documentation that he can easily read and understand which microservices interact with each other and what they are responsible for.
And in general, this customer has everything in Eversync and Code, but in most cases it turns out that there is a repository with the source code, there is, for example, some kind of presentation, and there is an opportunity to ask questions to the developers,
that is, to ask what it is, and plus there is probably access to the test stand.
It is clear that the opportunity to ask questions to the developers is limited, you cannot distract the developers with your incomprehensible questions trying to draw such an incomprehensible picture for yourself .
And so, having conducted a couple of such exercises, in general, I more or less determined what minimum necessary information is worth collecting about these things in order to use it in the next steps.
Firstly, it is necessary to understand what micro-services implement business functionality, that is, directly what the team writes, that is, those things that process requests, I mean, those things that have business functionality.
These are application functionality services.
And for such micro-services it is necessary to identify them at least in the first approach and in one or two sentences understand what they do from the point of view of business .
That is, it is not necessary to thoroughly understand the entire logic of the work of these micro-services, it is enough to understand that this micro-service number 1 on Java returns generated reports per month on some transaction.
In addition, it is necessary to understand that are basic , which are not directly related to the development and simply support the infrastructure.
This is something like Service Registry, API Gateway and so on.
Identify all the instances, not forgetting that these storages will be 100% of each micro-service and not only UBD, but also some cache.
Identify all methods of interaction, not forgetting that there will be queues that are either used as a means of inter-service transfer or as a means of communication of your entire application with external entities and after that try to draw connections.
It is clear that drawing connections with external entities is not necessary but it is necessary to understand who interacts with what.
And the most difficult and non-trivial step that will require communication with a person from the development who understands the essence and what the application does is the identification of assets.
It is important to understand what kind of protected information can exist in the system
and where it can be located.
You can draw anywhere, but the easiest way is to draw in graphism because, as a rule, the architecture of such fast teams changes very quickly.
New microservices appear or they are deleted or connections appear.
And instead of drawing arrows in PowerPoint, it is better to draw like this and you can always draw.
There is more detailed information in one of the cheat sheets about ASP.
I tried to write something more or less useful.
So, having artifacts, in particular, the source code, the test environment, having a picture of what the system how it interacts, you start answering some basic questions.
Now I will try to briefly, as time allows, to go through the most interesting points related to the analysis of the authentication, authorization and audit function and illustrate it with some examples.
In particular, about authentication.
The first thing you need to understand is what service is used to issue tokens.
Because everyone will necessarily use tokens either for inter-service authentication or to authenticate user requests.
And sometimes, if the team is small, maybe even a start-up, they can write this authentication service on their own.
There is nothing complicated.
Login, password, return JSON Web Token.
And probably, sign it.
But, as you understand, this is not the right decision.
But it is accepted because the team does not want to spend time either to use some infrastructure-ready service if the application takes off in some cloud or to figure out what to take from GitHub.
It is easier to implement this service on your own and then see what happens.
The second point of authentication is to see which libraries...
Again, if the inter-service authentication works on JSON Web Tokens, it is clear that in such small applications and small teams no one will implement Mutual TLS.
And if there is some authentication, it will most likely be on tokens.
Then you should see which libraries are used in the applications.
Because applications can be written in different programming languages on different frameworks.
And even if there are some guides from developers that say, when you write an application on Java, use Spring Security and verify the tokens, if suddenly a new application or a new service on Node.js appears in the architecture, if there is no specific guide,
but the developer knows that it is necessary to make an authentication, because he was asked about it, there may be a situation when the developer will go to NPM, type the right words and get a library that is really easy to insert into the code, no tests fall,
and everything seems to be authenticated.
In practice, there was such a case when all the services were on Java and suddenly some of the services decided to write on Node.js, the developer went to NPM, downloaded the library with the right hashtags, started using it, first in one service, then,
of course, scaled it all to others, and then it turned out that this library was not supported for three years already and generally contains all these charismatic vulnerabilities associated with JSON Web Tokens.
For example, the signature is not checked, and decoding immediately takes place, and so on.
Therefore, it took some time to rework all this, to take a new library.
Returning to authentication, do not forget that there are all sorts of other interfaces, in addition to HTTP, there are always queues.
And in order to, let's say, collect the attack surface, you can, if there is access to the original text, use, for example, the same Semgrep.
You can just write a rule in Semgrep that will get you all the endpoints.
Quite a powerful thing.
And you can, for example, use such an interesting project by Assetnode, which tried to collect open-app specifications and make an instrument that makes requests on all these endpoints and checks whether such an endpoint exists or not.
And this is very useful if you have all sorts of Spring Boot actuators that can stick out, or, if we talk about Go, on these HTTP profiles, and so on.
Sometimes it turns out that there are some APIs that have no authentication in their architecture .
There is an extension for UBD Postgres called pgsql-http, which allows Postgres to make an HTTP request on some trigger .
That is, something happened in UBD, it goes by itself and makes a request to some service.
And it is clear that some additional authentication should be added to this thing , that UBD must first go to the authentication service, get tokens, and then make a request with this token .
This is probably not a very fast story.
There may be interfaces that simply can be without authentication.
If we talk about specific examples, this is the first example when the application used its own self-described piece that writes out tokens, and everything was like everything was done in the right way.
The access was written out, the refresh was written out, when the access token was completed, the refresh was written out, the post request was made.
But again, taking into account that, of course, it is better to transfer the token in a cookie with a limited pass.
But the problem was that this service, which wrote out a new token, checked the refresh signature.
If it was correct, it took the data from the access without checking the signature.
It just took this structure, re-signed it, indicated a new term and returned it back.
Thus, it was possible to put anything into the access token without a signature, which could be used to increase the privileges.
But again, this is only the tip of the iceberg, because in a week, or in a month, customers can come and say, let's have an integration with LDAP or SAML.
If there is a self-described thing in the system, it is better to delete it at the architectural level.
Another example is authentication with endpoints.
The application was developed using the jHipster framework.
It is a framework for Java that allows you to write microservice applications.
It has some basic services and libraries from the box.
One of them is jHipster Registry, which performed the function of service registry and discovery in this application .
This thing is open source, but it has an interface, an interface management info, which is open by default.
For some reason, it returns uncritical information, just version numbers.
When checking, it turned out that this interface sticks out, and for some reason the key from the git shines through it.
It was the most interesting after the investigation, not with the developers, but with the system engineers.
It turned out that it was a sensible decision.
The thing has no authentication, and it was necessary for some reason for external clients to receive and use this key from git.
The simplest solution was to make a fork and close the authentication of this interface.
Now, let's talk about authorization.
Here, in addition to the fact that it is necessary to check which authorization model in the system should be implemented,
don't forget that this authorization should be applied to all infrastructure components of the system.
If there is a queue where several agents write, shouldn't there be an authorization to divide these topics?
If there are WebSockets, shouldn't there be an authorization?
By default, if there is an authorization, it will be necessary to do it on your own.
And again, in addition to all the questions to the developers and testing, to the direct work with the product, there can also come a system of queues where you can write the rules.
For example, there are endpoints in which, for example, Pyramid.All from the entire repository can be queued and see why it happens.
And quite an interesting approach from the research department of Microsoft called StatefulAPI Phasing, which allows in these distributed microservice applications
to detect IDORs.
If we talk about examples, the first one is about authorization.
In addition to the fact that this engineer should see how authorization works inside the application, first you must answer the question whether this application will be operated according to the SAAS scheme.
Maybe now this application is being developed for one specific customer and it works perfectly and there is already a sale, but then suddenly customer number 2 comes and it is not clear how to make this SAAS, how to isolate the data and the functions of one tenant.
And the simplest but the most expensive way is just to raise the stack for the customer, to keep a separate virtual machine with all these things, but usually this is not done.
And if the application has architectural functions related to tenant isolation, it is necessary to see how it works.
In practice, there was a case when each tenant was given a DNS.
One tenant, well, one customer goes to tenant1.com, the second to tenant2.com.
This hostname on the load balancer was just thrown into the xtenant header.
And despite the fact that later the microservice received the token and verified it and the rights, the access to the data
was still decided at the level of this xtenant header.
And , accordingly, despite the fact that having access to the functional capabilities, it was possible to get access to the data of another tenant.
And this is actually a very terrible story for SaaS solutions if you have such a tenant escape.
This is also a fairly simple example related to IDORs in WebSockets.
The main moral is that it is not necessary to remember all the access points of the application and check not only HTTP.
Here is a reference to Mikhail Yegorov's report on Hacktivity last year, where he gave examples of vulnerability in WebSockets.
And finally, there is an approach called Stateful API Testing.
In short, the original idea is from Microsoft.
You have microservices, Swagger, OpenAPI, Specification, and you phase this system in a smart way.
Before phasing, you build a tree of connections between requests.
And, for example, before making a GET request and putting any values into the ID field , your smart phaser understands that first you need to make a POST, get an ID, and only then throw it.
And by modifying this chain of requests, you can identify quite interesting vulnerabilities of open-source utilities.
So far, there are two from Microsoft.
The report time is almost over.
The slide about logging is probably the most boring.
It says that from the very beginning of development, there was a standard of logging, when all microservices use the same data structure to log, because it will be used to detect threats, anomalies, and analyze the results of event registration journals.
And the last slide with a link to the repository is in the second paragraph with a checklist.
I will try to interact with AWS, maybe try to put it in AWS-ISVS or make a separate checklist.
And the main thing is that before you start looking for vulnerabilities, it is better to focus on architecture.
Thank you very much.