[Music]
hello everyone and thank you for joining
uh my presentation
for uh cloud native security con and
this presentation is going to be about
the state of vulnerability in in cloud
native
security or in cloud native applications
my name is magno logan and i work for
trend micro as a security researcher
so just a little bit about myself
as i said i'm a senior researcher at
trent micro also an information security
specialist um currently doing cloud and
container security research uh
mostly working with kubernetes security
and also open source security
um i'm a member of the cncf security tag
team as well since uh october last year
and i've been doing container and
security uh container and kubernetes
security research as i said
i also have a personal blog that's
called katanasac.com
where i used to post my
articles at least
once a month
and there are also all my contact
information there and all my previous
presentations either like slides or
videos from previous talks
since since 2011 since i started
speaking at tech conferences
okay so
uh before we start i just wanna
give you a quick overview of what we're
going to cover in this talk
basically
this is a project that i've been working
on for uh a couple months now and um
still ongoing there i think there are
some uh improvements
uh required and and we're gonna talk
about that at the end
but the
i'm gonna explain to you the idea for
the project uh what are the cncf
security audits how they're done and and
the list of of the ones that have been
performed so far on many different
projects
and then we'll go into the details of
the results of the data that we
collected and analyze um basically i'm
going to explain the methodology that we
use the analysis that we've done and the
results that we got and also uh talking
a little bit about the third-party risks
of some of this uh these projects and uh
to uh conclude and summarize everything
i'm gonna give some recommendations uh
some recommendations to the cncf but
also some recommendations for
organizations and end users
and some next steps on on how i'm
planning to uh keep improving this
project and doing uh as a as a kind of a
periodic researcher
before i start i just want to say that
uh give a little disclaimer here the
idea for this project is to generate
awareness on the need for more cloud
native security and open source security
the idea for this project is not to
generate fund as we call in information
security which means fear uncertainty
and doubt around cloud native projects
right so we we just want to raise the
awareness and show uh that security is
important for this project especially uh
some of them have been uh used for many
different organizations and we rely on
them uh to run our systems so it's
important to focus on security as well
a shameless plug here before we start
with the content i have created this
github project which i call the almost
awesome kubernetes security list
it has a lot of content there around
kubernetes and kubernetes security it
has
videos books and articles and everything
you need to know
i think to um to learn more about
kubernetes and kubernetes security at
least to get started um so i think
there's there's a lot of contributors
there already people from the cncf have
participated and submitted new links and
new information so this has been going
for
almost a year now and it's it's really
incredible to to
have contributed to the community in
that way so feel free to
uh start this project and fork it and
submit new prs if you want i'm always
available and looking for new content to
add there
okay so
how how how this project started how the
idea of this project started right
um
i think
uh in the beginning of this year i was
uh helping the cncf
uh security tag team to update the list
of security audits that was done
on on different cncf projects and and so
i saw that
like pretty much all of those security
audits were
pdf reports anyway some of them were
kind of
hidden inside of the repositories of
those projects right so it was kind of
hard to find and and you had to dig
through them and and especially since
they were on pdf there was no place that
we could check the status of those
vulnerabilities if if there were any cv
reported uh because of of that
vulnerability that was detected right so
those security audits are usually um
organized or directed by the cncf and
and they hire third-party consulting
firms to test the security of different
cncf projects right as we can see here
from the slides uh there are many
different projects have gone through
security audits right uh
kubernetes helm grpc at cd are some
examples and and you can see
as well the audit vendors or their
consulting firms that have tested these
projects
so
basically the first thing that i did was
okay um
i need to concentrate all these reports
in a single location so that's easy to
find and and i can uh consult them and
query them in a in an easier way than
than just go going through each project
there
so what i did was i created a repository
on my git home called cncf security
audits
and each
uh inside it folder that has the project
name there is the pdf with the
with the report and the security results
of each security audit right
okay that's that's good but
how how do we go
through each
pdf report and and kind of
uh analyze all those vulnerabilities
right
so
as i said the security audits are
independent done by third-party
consulting companies they're usually
required for projects that move to
sandbox to incubating
and they started in 2018 at least that's
the the
the first ones that we have uh uh
details from and they're usually one
time only with some exceptions right and
unless the project is um very well known
and very well used by the community such
as kubernetes it usually goes through
another uh security audit such as the
one that kubernetes is probably going
through right now or there is it has a
rfp open at the time that i'm recording
this video
yeah so as i said that the result is a
pdf report with the findings right and
in
in security that's fine and that's
that's usually the artifact that you're
expecting from a paint testing report a
pdf with all the finance
but for um i think for developers and
and devops organizations uh we are way
past that right we need a way to
a easy way to list out those
vulnerabilities and and and also
check their status and see if the
problem is it was fix it or not and and
who's addressing that right
so
um so like uh as a suggestion here
i'll probably give more recommendations
at the end but as a suggestion here
maybe uh having those
uh
finalized reports added as github issues
or something like that would make easier
for
everyone and would give more visibility
into those security audits
okay so
so how did i do uh how did i started
this this kind of this project here the
methodology that i used was download all
those pdfs as i said i created this
github repository to help me with that
and and it's available for everyone
um i parsed all the pdf reports and i
collected all the relevant data that i
needed
um i basically aggregated everything
into a single location
of course i used a excel file for that
because it was easier to
see and and compare the issues and
aggregate results
basically in the beginning i analyzed
the data from all those reports and
understanding the results and did some
measurements on okay what is what what
which project has the most
vulnerabilities what are the most
critical vulnerabilities and all that
stuff
but then i decided okay maybe i could do
more and i also assessed the third-party
risk of those uh projects analyzing
their libraries and dependencies with uh
uh
open source security tool called sneak
one thing to note here is that there
were some projects some security audits
that were done
that were specifically about fuzzing
and and so
the the the vulnerabilities reported
were very specific to fuzzing and buffer
overflows and stuff like that
so for this uh this version of the
project i decided to to laugh out those
vulnerabilities otherwise you would kind
of
i think would mask the the data and the
comparison there because not all
projects have had uh fuzzing testing
okay the analysis i'm gonna show some
graphics here in a second right
basically comparing the project numbers
uh the project with the number of
vulnerabilities the vulnerability
categories and types as as reported by
the consonti uh
uh firms
and also comparing vulnerabilities and
severity so that we can see uh which uh
the number of vulnerabilities that we
have as low medium high and critical
right and then i'll go into the um the
open source security or the third party
dependency and as we uh like to call
software composition analysis right
okay
so moving on um
this uh this presentation uh has some of
the main results and graphics uh
at the time of of the the recording of
this we're working on finalizing the
final the pdf report that we're probably
going to publish together with this talk
when when it's published when it's uh
released during during the cloud native
security con so you're probably gonna
receive a link for the the full details
for the the complete pdf report with my
uh
with all the details that i that i found
and that i wrote about it
okay
so the first thing that we analyzed here
was which project had more
vulnerabilities reported to it right
and it comes as no surprise that
kubernetes came out on kind of the
top with the most vulnerabilities
reported to it
it is the largest of those projects and
the most adopted of them and
of course it has more lines of code
right so and thus more chances of
vulnerabilities being found right there
are some studies that uh
have uh reported some researchers
mentioning that
there is at least one vulnerability per
thousand lines of code in in
any kind of of software right so this is
of course this is an average but at
least we know that the more lines of
code the more chances you have of
finding vulnerabilities so that's the
given
um
in second place
there is at cd which is also part of
kubernetes by default and and helm which
is the package manager for kubernetes
projects as well in third place
so that's that's quite interesting but
you can see uh uh really see the
difference between kubernetes and that
cd here uh in second place and and the
other projects as well so
that's something that we need to be
aware of uh
most of these vulnerabilities were fixed
or their risks were accepted so uh of
course make sure that you're using uh
the latest version available to you for
your kubernetes projects
um here is just a drill down of those
numbers right uh how many
vulnerabilities were found by each of
them as we can see kubernetes has uh uh
more than double from the second place
from at cd and and so on i just showed a
few of them here
okay so before i go into the next next
graphic uh we need to talk about the uh
classes of vulnerabilities so
uh one of the vendors i think was
um trio of pets
they in their reports they classify
their vulnerabilities according to these
classes and we have access control
auditing login authentication and all
that stuff and they uh they give you a
description of each uh what each class
means right um
some of them uh
some of the other consulting firms
didn't adopt this this kind of approach
and so we we found it interesting and we
used this kind of classification to the
other vulnerabilities as well according
to the description of the
vulnerabilities we assign a specific
class or category to that vulnerability
so that we could uh
[Music]
compound all the data and and create
some some uh evaluations there
so basically
the the vulnerability class or category
that has the most number of
vulnerabilities is
data validation
right and um
as we can see here it's
it's way over the second place as well
which is configuration and data
validation relates to either input
validation or or lack of verification
and uh of data it could cause also like
things like buffer overflows and heap
overflows and all that stuff so
um it's it's really interesting to find
that a lot of these projects have data
validation issues right which is
something that can be easily uh solved
by adding proper validation in your code
and doing uh proper uh exceptions and
and try catches and and all that stuff
right
okay
the next one here um
is the severity right
so the vulnerabilities per severity we
found that uh
that some of them
actually most of the the reports the
projects only had like low and medium
vulnerabilities which is good right
which uh i think it's reassuring for us
using those projects that they don't
have like few only very few of them had
some critical vulnerabilities down there
at the bottom
um
i think it's it's interesting to uh to
note here that despite the the great uh
quality of the consulting firms that
have tested these projects even though
they're they're kind of well known in
their field they couldn't find a lot of
critical or high vulnerabilities in all
these projects right so that's that's
interesting and something to note right
so so i think that's that's important to
to highlight here
okay
there is more uh reports and analysis
that i've done i i couldn't show
everything here in a 25-30 minute talk
so
in the interest of time please take a
look at the the
the report for more more details and
more uh analysis
so
the last thing that we did was the open
source security and independence is here
right when when talking about cloud
native security we need to be aware that
most of these projects rely on libraries
and dependencies there are also open
source software right and these
libraries are usually incorporated
during the development life cycle and
rarely get updated or checked against
known vulnerabilities
so in this
section here we analyze the dependencies
of these projects and look for outdated
and or vulnerable libraries and
licensing risks that they might be using
in their code
for this analysis we use the
micro cloud one open source security
which is powered by snake
um
and as we can see here from the results
of all i only tested the projects that
had security audits done
for them right so i didn't test all the
cncf projects so that we can make some
comparisons here
so again uh kubernetes shows on pretty
much on top if you look at the numbers
there is the project with the most
vulnerabilities uh in in its
dependencies uh besides just notary
that's on top here uh
because it has a critical vulnerability
as well right
and so that comes as no surprise as we
found on the on the previous analysis of
the security audits as well
um
and and
in the
in the second and kind of third place
there on the number of vulnerabilities
the total number of vulnerabilities of
their dependencies we have harbor
and uh vtes
so it's interesting to note that
although these projects have access to
uh some open source tools and i think
they use uh on the cncf they are they
have kind of their own scans for
vulnerabilities using sneakers one
there's there's a lot of issues here um
i i i don't mean to say that all those
vulnerabilities are valid and
exploitable but there is some analysis
that needs to be done and to see
probably there are some issues here that
need to be addressed and fixed and and
some of them could be just updating the
library to the latest version and and so
some of these should be uh prioritized
and fixed
um
here we can see the vulnerability risks
increasing over time uh regarding the
libraries and dependencies right as we
can see as as time goes by and new
vulnerabilities are found and and the uh
libraries get uh outdated uh so more
issues are are raised and and the
kind of the the risk increases for these
projects right so the quickly we address
those issues the faster uh we can have
it fixed and reduce the risks of this
projects being compromised
[Music]
so to summarize everything i think i um
for this
for this project in this report uh we
came up with some recommendations for
the cncf and especially the security tag
team uh which i'm also part of and so
i'm going to recommend that to them as
well but
uh first thing i think would be
to improve this overall quality and
security of those projects is
establishing more periodical assessments
um
at least uh once a year or something
like that if that's available and if of
course if there is budget for that
um implement a cncf buggy bounty program
so when doing this analysis i found out
that only one project from the cncf has
a public bug bounty program which is
kubernetes besides that all the others
they only have like a uh what we call a
vdp or vulnerability disclosure policy
and so
maybe that would increase the number of
reports and vulnerabilities that get
reported to these projects
um other thing that i noticed um when we
were trying to do the rfps for the new
kubernetes security audit is that we had
a hard time uh finding uh consulting
companies and vendors to uh that were
interested in doing that so maybe
promoting that those rfps even more and
and have that available and kind of
publicizing it for different
organizations and maybe in different
countries would get more uh rfps for for
those audits
another thing is that
um it's it's hard to
um
to see if this those issues reported on
those audits if they have been fixed or
if they have any kind of cve assigned to
it right and and so you would have to go
through the code and analyze all the
commits and everything and find these
issues if there is something related to
the vulnerabilities that were fixed so
making easier and more visible to see
these reports and status of each
vulnerability i think that's something
that would be interesting that that we
can improve upon
um another thing is prioritize issues
related to critical third-party
components right and we saw there is at
least one project that has a critical
vulnerability there on on the
dependencies and there is also a bunch
of projects with with high
vulnerabilities as well so maybe trying
to prioritize that and talk to the
project maintainers to see if they could
fix those issues of course without
causing any problems on the on the
projects and the applications itself
because we know that sometimes updating
libraries can be tricky and might break
your application so
we need to be careful there as well
okay
so
this as i said this project is the i
think would be uh i can call it the
first edition and i'm planning to uh do
a second edition next year um but the
goals uh and next steps here i would say
that i can provide the data back to the
community and share that with the cncf
security tag team uh basically all the
data that i i aggregated from the
different security audits
and as i said i'll try to run this uh
once every year and compare the results
with the previous version as well
um
also adding some static analysis and
reviews for all the projects right
analyzing the the code of those projects
for one of security vulnerabilities as
well
and
check and see if there were any issues
here that were unreported or unfixed and
report those to the proper projects and
and hopefully get them fixed
so
i think that concludes my presentation i
hope you enjoyed this uh talk
and i'll be available for questions uh
either on
the q a section or on slack
thank you