hello everybody and welcome to the
capture the flag
shakedown uh we will go through what has
occurred today
and uh we'll actually replay the
scenarios
and look at uh what you could have won
now notably we didn't actually have any
prizes today and that was uh
intentional so let's
explain why that was and what has
happened
uh hopefully my screen should be sharing
we shall see
okay so yes we
we had a ctf today hello and welcome
thanks for your attendance and hopefully
this will leave you
mildly entertained so what were we doing
today
uh today was a day of ambition we aimed
to
educate amuse beguile and everything in
between
in a safe and happy environment
and we had the absolute pleasure of
introducing people to
the joys of container hacking
scenarios are each built with a learning
outcome in mind
and our capable task masters also
constructed the scenarios
which gave us a little bit more closure
on actually helping and
giving hints to people as they went
through notably absent from the list of
goals is ranking
uh this is because there's no official
scores there's no prizes
and the idea here is just to take the
pressure off and to encourage people
to go and hack stuff and to learn and to
enjoy themselves
breaking things in a safe and
importantly legal
environment
although saying that if you have
captured a flag
or more and you'd like some some secret
swag
then do dm a physical address to the
nice task master
and they will ship you something worthy
of your efforts
okay so today was hopefully an exercise
in preparing for the unexpected thinking
a little bit outside of the box
and uh an encouragement that uh nothing
is entirely secure and we should we
should pay attention to these things
so we are in the uk time zone on this
end so
please bear with us um it's been five
o'clock for quite some time
we had six scenarios one scenario per
talk
so if you did the first scenario it was
concurrently running with the first talk
and then
detectively throughout the day um and
lots of talking but nefarious small and
potentially mild vulnerabilities
unless of course uh someone did better
than we bargained for
and uh the aws build will be
significantly higher
than we thought so there are many places
to look for vulnerable clusters
but we believe the only way to up skill
people is to give them a safe place
to practice in so instead of going on
uh cluster safari we prefer to run
ctf-based training
but capture the flags can often be
daunting
big challenging filled with the kind of
people who we
we look up to and we want to break those
barriers down and give everybody the
opportunity to be involved
so we hope there's been a strong first
user experience
um there's attentive moderators in the
channels
and teaching assistants and uh yeah we
just want to give people a boost on
their
cloud native journey so uh
spoiler alert um if anybody is still
playing your time is up
and we will uh not be
validating any flags after this point uh
so let's go through the scenarios and
find
all the flags now
um excuse me while i shuffle my screens
okay so the first scenario
um was uh called node
secret breach so
back to the screen share hopefully
okay so uh what happened
here well first of all
we start we're starting as the root user
and this is uh
this is mixed of course root has a
special set of privileges
in linux the root user is a specific and
special user
that means it has capabilities and those
capabilities
may include things like changing other
people's files
or opening a network adapter in
a low level mode so we can just either
send
custom packets or turn it set it into a
different mode so we can sniff stuff on
the network
these are not things we'd want an
average user to do
and so the distinction of running in a
container without username spaces
is important at this point so
let's see what else is in this container
let's have a look around well this is
what we expect right we've got a process
namespace so we can only see the
processes
that we have available to us
what else might we do we might have a
look around and see what else is mounted
um this is kind of noisy there's not
really much here that's
that's useful there are a few things
anytime we see
a docker uh or perhaps one of these
mounts
they could be interesting so uh df the
disk free tool
is a little bit of a quicker way to get
a view on this in my opinion
and of course we see a service account
there
service accounts are juicy and uh
we are people that love messing around
with service accounts
on this occasion though um this was not
a service account based challenge
and uh okay so we've had a look what's
running
uh sorry we've had a look what's mounted
within the container
um there's a lot here but some of the
things
notably a device from the host
other stuff that doesn't look like it's
inside the container well
that's because containers are wonderful
but they're not a perfect abstraction
and that concretely means that
container runtimes have to put effort in
to hide certain things from us
and those things may include parts of
the proc file system
bits of cis the way that we interact
with
the process table and also the way that
things are mounted
in so uh okay
um containers are a child of evolution
um rather than intelligent design and
like everything else on the internet our
gaffer tape together uh no disrespect to
people who've done an excellent job
making them available to us
um okay but enough of this there's a lot
of noise here we want to get some signal
uh let's think about what we can do with
the service counts
right so we can see that
dev is here so what does this mean so
docker will mount in
etsy hosts using the
mount point from the host that it's on
so wherever you store your container
images those read write layers
that's where this is so already the
abstraction has
leaked with the file system here
what do we do next then well
i guess it's probably worth uh checking
our privilege
now there is a canonical way of doing
this
uh which is jesse frazil's am i
contained of course
it's not doing anything magic it's
checking
states and files that are available to
us
inside a container but it makes it very
easy for us to do and gives us a unified
view so
let's try and do that one thing to bear
in mind
is that we are roots and that means that
we have the ability
to run package manager commands we need
to be able to write to any location on
the file system so we have to be root to
do this
generally and uh yeah there we are so
just installed curl because without curl
i'm gonna have trouble
pulling in stuff from remote endpoints
although of course
any programming language is an
interpreter and most can be used to
replicate curl in some way
so let's pull am i contained from github
chamod it and run it
happy days okay so we've got some stuff
here what does this say
uh we're in kubernetes does it have
namespaces pid namespace
generally that is always true anyway
username space is generally always false
because
username spaces are not enabled i
suppose by default
in run c or in kubernetes
this is interesting though we're
unconfined in app armor
now if we were running in docker that
would always have a profile
kubernetes of course disables this
and uh so it's just reinforcing the fact
that we're
in kubernetes but it doesn't tell us
anything uh too drastic
this however what is this this is
a a wide bounding set of capabilities
so stuff in here this means we can
change ownerships
override discretionary access control
which is um
basically writing on other people's
files and all sorts of things in here
that we recognize
processes being killed raw input and
outputs
access to the network adapters this is
pretty likely to be a privileged
container
okay so we know what that means that
means that actually
what is in dev is a reflection of what's
on the host
so as a reminder the privileged flag uh
the most dangerous flag in the history
of computing
as i will off to repeat disables
most namespaces it turns off app armor
and setcomp if they're enabled
it grants all capabilities and it mounts
all of the hosts devices
into the container this is a bad day
for a system administrator why because
we can say all right we know which disk
is mounted from the host
so let's mount
that disk into an empty mount directory
okay so this is probably not what we
were hoping to see
because if we were blue team um because
that's what the root of the containers
file system partition looks like whereas
what we've mounted in from the roots
uh let's go back to there we've mounted
it from the host contains extra stuff
and lo and behold it contains a
directory called node secrets
this may be obvious oops let's uh let's
actually
yep go in there uh and there we go so
there is our first flag
the disadvantages of running privileged
containers are that they
are equivalent to running the process on
the host
there is essentially no containerization
going on when we run privilege
and yes run for the hills
if that's the kind of thing you are
trying to defend because it's more or
less indefensible
okay that was the first one
next we have uh escalate and cover
secrets and
let's jump in so this alias will just
take me into the
um into the next host
okay let's spend a bit more time looking
at what's happening here so um
we're in the attack container
and escalate and cover secret sre has
deployed a postgres demon set
and a new pod for monitoring the
processes in your cluster
following a routine audit of an
application it is believed
that the deployment setup allows a
compromised process audit
pod container a mouthful
to escalate its attack to other
containers on the host
this doesn't sound like it makes sense
there's obviously something a little bit
askew
here okay so the the question then is
verify this by uncovering both the
postgres database password and the
secret key
we will start in the process audit pod
okay
so um first of all we know that it's
possible to provision secrets via
environment variables this is um
this is something that the 12 factor app
has told us historically
but as we know the environment of a
process leaks
to other users of the system and so we
actually prefer to provision
passwords with a with a file and an
environment variable
as a as a pointer containing the
path of that file so i mean the first
place to start here
um then is probably just looking at the
environment
let's just see what is in here okay well
there's actually nothing
particularly juicy in here and uh
because we know i'll just go up and
because we know
that the the flag has a specific format
at this point
um we can use that string to uh to just
grab through the end
we haven't missed anything there's
nothing there okay
um so we're supposed to be
let's just remind ourselves this audit
this process audit pod
and we're looking to understand what the
database adjacent to us is doing
so let's just see what we've got running
okay this is a red flag
for a catalonian festival we should not
be seeing
this many processes because as we saw
let's go back up to where we were
earlier as we saw in the process table
excuse me it's further than i thought
the process table here
for the container which is isolated to
itself
we're only seeing a very small number of
processors
and notably we see pid 1 which is not
systemd or an init system
it's just a sleep command so that's what
we would expect to see
but actually we can see all sorts of
things we can see stuff on the host
uh so this is again a remarkably
bad day um the question here though is
not how we uh
we take over the hosts although from
from here it's um
not especially difficult we still want
to exfiltrate those um
those crucial bits of data the secret
and the key
uh okay so this is probably how
we traversed containers because we
shouldn't be able to see the process
list
as soon as we can let's list it uh in
its
fullest extent and have a look for
postgres
okay so we found postgres and just to be
clear
this is post squares running in a
container on the same host
what we're seeing here is a host
pid namespace share and this again
is um of questionable repute as a
practice
that there is some necessity for it but
we should always be careful when
disabling
a container security feature or
container primitive like host namespaces
for networking or pids
or even mounting stuff in that we're
breaking the container abstraction
by adding stuff or taking stuff away uh
really we want to be locking it down
and not sharing stuff further with other
workloads on the system okay so um
right we've got a pid for postgres and
we can also see the postgres
uh we can also see the pits so uh
let's do some magic proc diving so if we
go into proc
and we put the pid in there then
let's start off just looking at the
command
okay so a foible of proc
is that everything is uh is null
terminated
so in order to just get a bit of
visibility into what's actually
happening here let's
uh replace knoll with uh just space this
time
okay so we can see that it's just been
invoked as postgres
all right no problem that's that's sound
enough
but what else is in proc um i'll give
you a clue
the environment and at this point
even though that looks like a massive
junk let's see if we've got
there's our flag so again using the flag
ctf grep
when there's a well-known flag string um
can help because otherwise it's a kind
of indeterminate mess of uh
of black and white that's a little bit
difficult to pick things out of
okay one down so that is uh that is a
flag
but if we go back to the beginning of
this scenario we were told
uh there is more than one flag so i'm
covering both
the database password and the secret key
okay so we know we've got the database
password
but there's a secret key
we don't have any access to that
adjacent postgres container except for
through the the proc table
okay so what else can we do with proc
we can examine the root file system
mounted into that container let's start
with hello and studs
okay so it's a sim link we don't mind
too much about that
here we go again we have got into the
file system of a container
running on the same machine as we are so
we've got into the mount namespace
essentially
without having access to the container
itself if we go into secrets we can see
that there is something that looks like
a key
there and if i have readline
there we go okay that is key number two
i'm conscious of time and i will try and
keep on clipping
um so the point here is that there is a
real risk with enabling
using the um sorry process name spaces
and uh yes while it's necessary
obviously the feature was shipped
for a reason we should be cognizant and
of course everything should be threat
modeled and then we can balance the
impact of that thing
actually being breached or negatively
affected or or exfiltrated
with the benefit that we get from using
it in the way that it's intended
okay on we go and we're
into a ci server vulnerability
right so what are we doing here uh we're
pen testing a cluster we found the
vulnerability
uh the pod is part of the build
infrastructure as we know
uh build infrastructure is a juicy
target uh supply chain security
is a uh a particular interest of uh
of of security and the cloud native
security day
of which we are apart and so build
infrastructure
yeah that's okay that's of interest to
me uh all right what's happening then
we're concerned that a compromise may
lead to leaked secrets
okay so we want to extract the secret
key
and look suspiciously like we are in a
jenkins
flavored pod
okay so so again we kind of just want to
do a bit of recon let's just figure out
what's
what's going on process table is okay uh
the amount name space so again there's a
couple of things here that
jump straight out one is the service
count um
of course you can do the same thing
again looking here but nothing really
jumps out from there immediately
uh that to me again is just a little bit
of a mess
the thing that jumps out for me here is
not the service count it is the presence
of the hallowed docker socket
a socket is an inter-process
communication mechanism
amongst other things and in this case
it means that we can probably talk to
the docker demon whose socket is mounted
into the pod
which we're probably safe to assume
belongs to
the jenkins host um
now at this point we would hope the
docker is installed
uh we would probably check uh see the
kernel is nice and recent okay that's
all good
see what release we've got okay it's all
relatively recent so we assume
probably that we can install docker
through the package manager but we can
also do something
via backdoor because we have curl
so let's just pull the official docker
installer
maybe we'll we'll free ourselves from
bugs in docker itself
although those days are much further
behind us
um but it's nice to be on the latest
version isn't it even if we're going to
break stuff
let's do it with uh correct operator
practices
all right so we're installing the docker
client
in the expectation that we can use it to
abuse the docker socket
we could just send restful commands over
the docker socket but it's a bit more
long-winded
and right so what we see here is the
docker version command
has given us the client and the server
happy days
so let's see what's running here there
is a lot
and of course we see
kubernetes oops uh let's do color
in the us spelling always yeah and then
we see cubenet is all over the place
so it's probably again not a good day
for somebody
um in this case we probably want to look
in the nginx
container
he says so let's um
let's have a look in here if we do a
docker inspect
in fact we can do it more elegantly with
the
containers sure okay
again there's a lot of information here
we can kind of spool through it
but because we're volume hunting and we
uh or flag hunting i suppose and we know
the flag again sweet there we go there's
our secret access key
that looks so let's have a look at some
context around that
um and colorization is always useful
okay so what's happened the environment
of the container has specified
this environment variable so we're back
into environment variables again
not only do they leak on the host on
which they
are running but also they leak from
uh metadata about the thing as you see
in this case it is an insecure
provisioning
method because um it's either set at
runtime or set in the image itself
ideally not of course so
really what we want here is um is to
instead
point that secret to a file and this is
as kubernetes will do with a secret file
map for us
and that way we we need to have access
not only to
to the uh to the containers metadata
like this
but also the container itself or as we
saw previously the process table
um etc right at this point
we had pretty good um
i would say turnout we had we had a lot
of people we'll uh we'll get to those
numbers at the end
but also a lot of people got through
these scenarios and
uh and at this point most people were
still still with us
so let's persist next we have
a non-user compromise
okay and out and back in again
okay so what we'll be doing here um
more uh escalation sideways and so
just moving laterally through um through
kubernetes or maybe actually
on the yep on multiple nodes
so okay so secrets have been extracted
from the cluster
we're in a post-mortem phase but it's
not clear how the anonymous user managed
to escalate
sideways after the initial breach so
let's try and replay the intrusion from
inside the cluster
okay so we're in the pod that was
breached excuse me
and we're going to replay what the
what the what we expect the attacker did
um now again we've got our
our kind of basic recon which is just
saying uh how's our process table
do we have anything spare or extraneous
mounted in here
um we'll go to the map points for the
whole system uh
we can install am i contained and see
what our bounding set is
but in the interest of time this is
this is slightly different so at this
point we are attacking
things that are outside our domain as
sorry outside our namespace let's say
and we have got the ip addresses of the
nodes
so what runs on the kubernetes worker
node
well there's the cubelet there's q proxy
there may be
things for the cni as well
the cubelet has some configurations that
um
are less than optimal let's say uh
such as the uh the read only port so
let's just see if we can find anything
excuse me if my mouse
constantly doing that um
in and around here so uh
again i know what i need to use in
advance so i will just
install it i don't need to do an app
update
i get curl and jk so
what are we actually doing here
let's just see if we can access these
host nodes
so let's go up in here
and as a reminder this is a network
route from the pod
onto the public interface of the host
and really there's not a great deal of
rationale for
um for running like this
uh we should be using network uh network
policy
and um yeah frankly constraining our
outbound traffic so that we can't hit
anything at all
start with the default deny and uh and
then upgrade
okay that was smooth let's just install
that too
so as you can see i'm having to install
a lot of software as i go along
that's kind of standard there is uh
generally we don't ship
curl and that kind of thing to
production because why would we unless
we needed it for our application
but as an attacker i am able to install
stuff inside the container
really just by setting a non-root user
uh maybe even removing the package
manager if needs be
that makes for a much safer day let's
say the jq and curl
if i installed those what did i get
wrong there nothing
okay and let's um skip verifying the
certificates
okay that means that we can read from
the cubelet
and we can see lots of things that are
running here
and what we care about uh potentially is
something
um compromise ish so let's have a look
uh nope there's nothing in there
uh now that probably means that what
we're looking for is on the other node
so let's just remind ourselves which
nodes were which
and switch over
happy days that is the pod that we are
looking for
okay so now we have more information
about
what we're looking to attack so what
should we do with that well
let's see if we can dump the end
of this um this pod so the pod
is um what is it going to be the
workload pod there
and then if you will just excuse my copy
pasta one second
and this won't quite work because of the
node ip so let's just
place
llp is the one we've just used which is
the second one
okay so we're posting a command
oops okay so we're posting to the note
ip
um force of habits uh on the insecure
uh on the read only port rather
and uh we want we want to go into this
pods
and then run the print end command
okay um joyful joys
that is not coming back with anything
have i done something wrong
um i guess the pod name should be
pod name and not pods
that was a great relief to me and as we
go
again there is our flag happy days
okay let's uh steamroller on through the
rest of these
um we are on to pod breech extract and
for this i will pass over to
my worthy companion magno
hey everyone
think you need to stop sharing andy
okay thank you okay
the scenario here that we have for the
for the ctf uh is the the
pod uh pod breeders tract right
so in this scenario let me just
log in again here
verify your suspicion by breaking to the
pod and extract the value of
user credits password right so and and
your starting point here you're starting
on a visual machine
external to the cluster right so so how
do you get access
to to the cluster to the pods of the
cluster itself right
so um basic things here in the interest
of time
uh i'll
okay
good uh so first thing if if i need to
access that uh that server or their
service right
i need to uh i don't have the
credentials right so
i need to uh first do like uh
your your server make sure that
everything's okay i ran this before just
before
uh the example just to make sure that uh
everything is
running smoothly and we don't need to
download it again so
uh and and here one of the tools that
you can use
to to do that to do the network uh
mapping
and scanning is nmap right
thank you you will now be placed into
okay
okay uh can you guys hear me again
uh my screen sharing
i'm sharing it
oh
hi can you hear me can you guys give me
some feedback on the chat
yeah okay sounds good awesome uh
yeah so as i was saying sorry about
technical discords here
um so as i was saying we installed nmap
there
and another uh one of the things that we
can do
is to run any map on this uh specific
node id that was provided to us for this
challenge right
okay and i've selected a few ports here
so the scan doesn't take forever
right so specifically for uh just the
demonstration here
uh and we can see that there is a port
open on
on this higher port here uh three zero
zero two two
right so one of the things uh
and it's very famous on on on pen
testing
and and like uh application security
scenarios
is uh brute forcing right so one of the
attacks that you can run
uh you can get a list of username and
passwords and try to just
brute force the system to guess those
credentials right
and one of the tools that you can use
here or for
brute forcing at this uh the sport
and and access the service right so
i don't know exactly from the end map
there i don't know exactly which server
it is but i can uh i think it can run
this one let's see if it's gonna give us
to us
a little bit more information there
if it takes too long then we can just
move on
see here
okay yeah okay
so we can see that this service right so
it was showing on the first scan was
showing as unknown right so i run
uh i added the flag dash a here so to
scan the services and the versions as
well
so we know that this port is running an
open ssh server
there and and then and now i can try to
uh to brute force it right passing
a list of username and passwords right
so one of the tools that you can easily
do that
is called hydra
and uh the way to install it just
typing oops
and
type in this command here
i have demos
okay so i have right hydra installed
already just to
show you guys here and now what i need
to run this brute force right i need a
list of username and passwords right
uh one one of the the very common uh
list of passwords is the roku list so
you could have download that and use it
but this server this this server here is
not very protected so
the the password is not the username and
the password are not very hard
um so i created a list uh here already
on on this
server uh called the list of users so i
have a
list of usernames here and i have a list
of passwords that i could try so it
doesn't take forever
when running right so here's the command
that i'm gonna use for hydra
to uh try to brute force this uh
this username and password on this
server
location here thanks
okay
let me just show you guys so just moving
on here so we don't lose
a lot of time we're almost running out
of time here
uh basically if you run this command you
should get
the uh the results that had it
here before i know i i
really uh okay sounds good
sure uh almost done uh
so yeah basically just
after i find out the username and
password i can
access this server oops
yep you should copy and paste here
and so the the user's admin and the
passwords password very
very easy for you guys you know even if
you don't have like a tool you could
uh try guessing and access that okay so
i'm inside this called
jump box right so what i'm gonna do here
is try to uh connect to the
api server and make some requests
so base base the the base request here
that i can
make is check the version of the the
kubernetes api server
this is one of the ways to do that right
so i can see that's running our version
119.4
stuff like that right so just to wrap up
here
uh one of the things that i can do is
wha
uh as andy mentioned before uh there is
some
um secrets and tokens inside the pods
right
so i can grab that and and use to
impersonate uh the policy talk to the
api server
right so basically here what i'm gonna
do
is just create two variables the
namespace uh
telling that's uh located at var run
secrets kubernetes.io service account
namespace
and i'll do the same thing for the token
here cubetoken okay almost done
uh and then now i can make uh if i have
uh permissions to do that i can make a
request
to this namespace and uh through
via the api server and ask for the
secrets and since this is a
misconfigured cluster
i have the permissions uh so
here just just making this api let me
show that again right so this is the the
request that i made
and i can see all the secrets and here
is the flag
user creds password and and that's it
for this challenge i'll hand it over to
andrew again so thank you guys bye
awesome thank you very much okay let's
zoom through the ends of the slides
there was one more scenario which people
didn't quite get to
so um if you would like a go at that
then uh
do feel free to dm right let's get
through to the end
so uh this was this was the control
plane team
at uh seven o'clock this morning um
we are just about through the other end
of it so honorable mentions
thank you to these individuals for uh
being with us through most of the
journey today and um
we had a great time on dms it's fair to
say uh
there's some really uh people show some
real grit and persistence
and that's what it's all about so good
job those people
um we have some honorable mentions as
well um
dilshan mathias michael matthew walid
steve mohammed and noel were all there
for the whole journey
and thank you very much for your efforts
in parting the container defenses
um various people enjoyed themselves
slightly
and i hope this has been a beneficial
learning experience i'm sorry there was
no cake
and thank you very much to the control
plane uh people at this end
for manning everything and to magno for
all his assistants and helping us out
and testing etc
this is a public service announcement we
don't run
administrative endpoints on the public
internet the kubernetes api server is
one of them
if you like what you saw today control
plane do this for a living
and we'd be happy to stand up a ctf for
you
thank you very much for your attention
and thank you to everybody who
contributed today
we had a total of 327 clusters spun up
uh that is about 15 000 nodes um
we had a peak of 73 users
so thank you for everybody who played
and we
welcome all and any feedback
have a wonderful day