awesome thank you so much thanks all for
coming and to all the awesome organizers
such a pleasure to be here with you
today to talk about adversary emulation
but maybe in a slightly different or new
way then you've heard it talked about
before so we have a few alternate titles
um one of them is how I learned to stop
being polite and get Real Results the
other alternate title is what your
testing is bad and you should feel bad
however we're not mean people so that's
not a real title
hopefully by the end of this though you
will feel a little bit less bad
so who are we I'm Katie I am the
director of intelligence at Red Canary
you're not familiar with Red Canary we
do manage detection response we also do
a bunch of Open Source stuff we share
things with the community uh you might
have heard of atomic red team kind of an
awesome open source tool that's us and
when I'm not doing Red Canary things
leading a team of Intel people um I also
teach first Sans their cyber threat
Intel class and so that's a little bit
about me
hello everyone I'm Adam mashinci I am
the principal product manager at Red
Canary for the MDR and threats portion
of what we do
um I come from the adversarial emulation
background I have I was actually
formerly the director of Open Source
programs at Red Canary where I was
helping manage Atomic red team and move
that project forward uh and so
uh my background is a little more
offensive in nature than maybe uh
Katie's is so hopefully we're gonna get
a really nice mix of kind of viewpoints
and content today uh apologies in
advance I have lost my voice this is
like the fourth day without it so we're
gonna try to get through the next 50
minutes without me sounding like I'm
actively going through puberty too much
so this will be great
yeah that's I guess it's my turn
so
the agenda what we're talking about
we're gonna talk about the current state
of testing and one theme we're going to
talk about a lot today is real versus
not real testing now in the context of
what we mean by that is when we all do
testing right as professionals we're not
actual adversaries right so when we're
talking about that real versus not real
real adversaries have ill intent and you
don't pay them in that sort of thing so
we're gonna talk about this a lot today
um but real what actual threat actors do
and not real red team pen testy it's all
lumped together for the sake of this
talk so legitimate stuff offensive
security does
so we're talking about testing we're
talking about the difficulties of doing
real testing which should be interesting
and like ring true for a lot of us here
um and we're also going to talk about
what data shows us about the way we all
probably do testing from an offensive
security perspective versus what real
adversaries do
um and then we'll talk about some
potential solutions to solving this
Delta that we're going to talk about
today
hopefully that makes sense so far high
level agenda seeing some nods no one's
sleeping yet nice so level set as an
audience perspective the Assumption for
us as presenters is that the majority of
you all are not actually threat actors
like in your free time right so that's
just our assumption that you like based
on this audience you're all at this
conference we all have kind of like a
red teamy pen tester Centric mindset
you've seen a lot of offensive security
content in the past 48 hours but that's
kind of the mode that we are assuming
you all are in as people standing or
sitting here watching this
we know that this topic we're about to
cover is kind of Niche but we're pretty
comfortable with that given who you all
are and while what wild was hackenfest
is and that kind of offensive security
mindset that a lot of the presenters
also bring to the show so hopefully as
an audience assumption this is all okay
with you all
um but that's let's keep this in mind
that we are not actually adversaries in
our day-to-day work
um I think that's important because we
are probably going to challenge some of
these uh assumptions we all make during
this presentation which should be kind
of fun
all right
so the current state of adversary
testing
um and I I'm going to apologize in
advance I'm going to generalize a lot
here in the next chunk of time because
anytime somebody says the state of
whatever it's rampant generalization so
there's exceptions for everything I know
that you know that if you feel inclined
to throw stuff wait till after the
recording's done maybe yeah so again
generalization inbound
but first why and what do we do
adversary testing adversary emulation
simulation and testing any of this stuff
why red teaming well
when we're talking about red teaming and
penetration testing and all these sorts
of things we have lots of things we want
to do we ultimately want to find flaws
right and whether we call that defense
validation
or compliance or we have requirements
that we're legally most supposed to
enforce because of the work we do or the
company we work for from policy whether
we're trying to educate people getting
paid to do this what have you where
they're trying to reinforce someone's
reputation these are all the reasons
that we do the offensive security work
that we do clearly it's like this should
resonate for those of you who think
about testing in this way
and of course what we test should also
be kind of obvious a little bit we test
the perimeter of organizations whether
it's ours or others
um we have to analyze the assets that
they have and all the things that
they've got tucked away and the weird
like Fedora 2 box that was plugged in
there and no one's ever touched it but
it's still doing something great like
we've all well some of us have bumped
into that into terrified but those kind
of assets that still exist today and
those are the kinds of things we're
looking for
for testing Solutions
a lots of people's companies buy lots of
things how do you know those things work
you validate them right is this EDR on
you validate the people or the people
watching the alerts watching the alerts
again it's all that kind of validation
that we know and love and try to
reinforce and then another one that we
don't often think about is all the third
parties We Trust well should we be
trusting them as much as we do and are
you validating that the trust that we
give them should be there so validating
those trusted third parties
now how do we test again over
generalizing here but we could kind of
bucket it into two categories that do
get blurred but for the sake of
generalization you can think of manual
tests these are specific Atomic tests
right do we alert on this one thing
would we notice this one thing happening
that's an atomic test for the sake of
this conversation then we have open
source intelligence it's a very manual
operation generally because you're
looking for something specific
then we have exploit tooling
now exploit exploit tooling I hope
you're not automating that because
that's really dangerous and terrifying
when you're doing like automated
exploitation but cool if you are I guess
but generally that tends to be a manual
operation there and then one that we
again don't think about often is what I
call bespoke validation you build a
thing like a web app for example you're
intimately aware of the inner workings
of that but you also probably know where
its flaws are and so you'll build
specific tests to see if the flaws that
you may have baked into the thing
actually happen the way you think they
do so that bespoke manual validation is
key
then you have the automation land these
are scanning tools reach and attack
simulation adversary emulation we won't
get into the Delta of what the
differences between those are somebody
else probably talked about it today but
as different genres these are the kinds
of more automated tooling that we have
and then of course there's command and
Control software there's a bajillion of
them they're open source ones there's
commercial ones but again that tends to
automate away some of the difficulties
of manual atomicy tests so hopefully
this is all tracking so far
all right
again from a generalization we have
assumed compromise and penetration
testing as kind of broad stroke
different methodologies we go to when
we're thinking about testing so in an
assumed compromise scenario we believe
these three principles are true
it's not if but when anybody can be
compromised
um there will always be new o days and
end days and social engineering will
always work if you believe those three
principles to be true then assumed
compromise scenarios are completely
valid because you could just drop a
payload on an endpoint and run it as
yourself and you know what you're saying
we're going to pretend as if they found
anode and work their way in so that
that's a pretty comfortable thing for uh
some red team in pen test engagements
then the other one which again I'm kind
of generalizing under the penetration
test is can you get in can you breach A
system can you penetrate into a network
um which usually has a really defined
scope and has a crown jewel Target
that's usually known in advance
um depending on the scope of your
engagement so again assume compromise
penetration testing like another way to
think about is assumed initial access is
another way to frame assumed compromise
depending on your scenario either way
these are the broad Strokes
methodologies that we kind of see from
an adversary testing perspective
and we can't talk about methodologies
without an obligatory miter attack slide
so miter attack it's great it's a good
framework I'm sure you've heard about it
like a thousand times in the past 48
hours which is great you should know
about miter attack it's good to know
it's obviously abused by vendors that
doesn't mean the framework's not good
but vendors like to abuse it sorry or on
behalf of all editors I guess
um but yeah water wet sky blue fighter
attack could it's good yeah Minesweeper
that's a Jeff mcjunkin reference he
refers to miter attack as like a
Minesweeper grid for what adversaries
are doing and where Defenders can track
them that's why that word is there
anyway moving on
so when we now that we've kind of level
set on adversary testing read you know
thinking about the red team perspective
as an industry we have defined this is a
goal
and just just show of hands like how
many pyramids have you all seen in the
past 48 hours if you've gone to some
talks like more than five
because I got a seven over there like oh
okay you all have seen a lot of pyramids
okay cool let's keep that in mind
um
so I'm condensing a lot of content that
you've probably already seen and into
one slide but this is the Pyramid of
paint this is a very neat way to think
about leveling up the scaling or
climbing this pyramid as we get better
and better at doing Red Team stuff right
does someone notice a hash value on an
endpoint versus do they notice something
phoning home to a specific domain versus
do we notice a specific tool versus do
we notice specific behaviors in ttps
right that's this pyramid of pain that
we're all trying to climb as a red team
testing emulation industry which is very
neat
um and this is the skill trajectory that
we all need to climb together right
um
and as an industry there's lots of talks
and lots of training and lots of
tutorials and lots of tools all helping
us get better and better at a triangle
like this one
um and that's good this is all good like
I'm not this is all fantastic we should
try to do that
um
but sometimes we need to start to think
Beyond this a little bit to well what
are our actual expectations of a threat
in our environment and how does this
sort of thing maybe map to it or not
right and so here we all are a thousand
people and all these conferences are
telling you get better at the Pyramid of
pain which is could and then you get to
the top and you're testing ttps as a
grid tester great you're like oh just
kidding there's another pyramid that's
at the top of that pyramid and this is
uh this is another one the Christopher
peacock I think is actually over there
somewhere this is a good uh an excellent
one when you think about ttps well it's
actually another triangle right and so
first we get good at testing for tactics
then techniques and procedures again
we're scaling up these pyramids we're
leveling up we're learning how to become
better at emulating these kinds of
threat behaviors it's very neat it
become more complex with your testing uh
find more realistic TTP is ADD
customization to your tests find
varieties of command control things to
use you know operationalize these things
make them repeatable so you can do them
over and over and over again this is the
stuff we're told to do this is all good
climb these pyramids right yeah
and then you get the top of that pyramid
and it's like oh just kidding there's
actually another pyramid and we're all
trying to get to the top of this one
right because when we just started doing
testing we were just doing a mic check
on your EDR to make sure it was on and
that people were watching it and then
you're like oh we're going to be kind of
more adversarial that's red teamy stuff
then you want to make it repeatable with
the blue team and have cycles of
Engagement
um that's great that's purple teaming
but we really want to get to this place
where we can act like a real relevant
threat this is the trajectory that
offensive security as an industry is
driving us towards so like theoretically
as we climb all these triangles we're
going to get real real good at testing
and get closer and closer to being like
a real adversary that's that's the thing
we've all been told is true
well it turns out
it's actually not super true and there's
reasons for that
um and
well why like Adam we climbed all these
pyramids and spent all this time and
money like what do you mean well
like hopefully as I've been talking
about these triangles everybody's kind
of nodding like yeah Adam this is
obvious these are the pyramids we're
supposed to climb right
um well I think we've fallen into a bit
of a trap as kind of a generalized
industry here and as a wise Admiral once
said like it's a proper trap
um not in the sense of that we're doing
something bad but we maybe have lost our
way a little bit
so those of you who have ever spent any
time listening to me or talking with me
know that I love a good tortured analogy
so one's coming in hot and also spoilers
for season one of Westworld if you
haven't watched it anyway so here comes
apologies like my really weird tortured
analogy here okay season one of
Westworld and this is we're in Deadwood
this is a timely analogy clearly
um but season one of Westworld they have
this idea that robots want to become
conscious right and in order to achieve
Consciousness the theory is is that the
robots have to first figure out memory
and then they have to figure out
improvisation then they got to
understand self-interest and then
there's some mystery thing at the top of
that pyramid that once the robots figure
that out they'll become self-aware
that's a theory that is posed about how
something gains Consciousness which is
cool like that's a really neat thing and
Anthony Hopkins is great clearly
um
but then as and here's the spoilers uh
you get to the end of the thing and it
turns out it's not a pyramid it's a maze
and as these things are trying to gain
sentience all the decisions they make
get them farther or closer away from
their goal they're achieving that
Consciousness and every step is a
journey inward that's how they have to
achieve it right and it's not climbing a
pyramid it's fine going through all
these complicated decisions to find the
actual point the actual intent
this is where I think we've kind of lost
our way
the center of the maze if we want to use
this analogy for all the pyramids we
just looked at versus this kind of maze
as I think we've lost the way a little
bit about the intent
the intent of a real adversary is
stealth it's threat it's theft it's
destruction
um and all the things they do
they those are the choices they make to
achieve that goal to achieve that intent
and some of their things lead them
closer to it and some of them lead them
farther away but this is the key the
ttps from a threat actor perspective
they're a side effect of what they're
trying to do they're not the goal okay
so I'm gonna we're gonna dive in on this
here a little bit but
if we think about like how do people
think about TTP as well as we've just
talked about and as we've all been
learning about in the past couple days
and at these conferences like red teams
adversary stimulation emulation pen
testers all of us we think of ttps as
something that aligns with defense
validation and coverage and Alignment
um and that's okay but then we start to
see things where people are using miter
attack as a bingo card which we all have
been told is quite bad right because
it's bad you're not supposed to do that
but there's again there's this reason
because we're all trying to think about
ttps as the goal the end State we've won
we've made it when we can do that
testing well
but again ttps
actual threats are like okay cool just
trying to steal your stuff right like
that's we have to keep that perspective
in mind
and it's not just this it's also
thinking about building out adversary
testing you know given that most of us
aren't actually or the assumption is
none of us here are actually threat
actors well we have finite resources
unless you're a part of a nation-state
which cool
um but we can actually make mental
models around creating adversary
emulation simulation that is more threat
aligned
now here's another kind of tortured
analogy sorry in advance
um but some of you have probably seen
this I call it the inexpensive fast
accurate another way of thinking of this
fast good and cheap I use the term
inexpensive because we like to associate
cheap with both inexpensive and low
quality so I'm really specific here but
I think if we have these qualities of
emulating an adversary where something
could be inexpensive and that it doesn't
cost US dollars to do it or you think of
something fast like how fast can we
perform this engagement how fast can
this happen from a Time perspective and
how accurate how like a real threat
actor can it be
when we try to break them down into this
way we can actually do a really easy
producty thing which is pick any two and
this was like a weird thought that just
appeared to me one day of like this
model works really cleanly with the way
we think about adversary testing
um and let's demonstrate that now
obviously real threat actors don't have
these limitations I mean for one they're
accurate by default because by
definition they're them so I mean neat
so they don't even have to think about
that one but
the for legitimate organizations we can
actually think about this from a pick
any two perspective when we're defining
adversary emulation
so here we are accurate and fast well
this is kind of your classic third-party
engagement pen test right where you were
paying them to do a really accurate job
of being like a real threat actor but
it's probably in a short time frame
right this is this is what pen testing
firms product ties they want to do this
and kind of move on you pay them to do a
really good job pretty briefly and then
you go about filling out your reports
and what have you but this isn't
inexpensive but you know you have to pay
well for this
and they usually have a well-defined
infrastructure in advance and stealth
operations are kind of out of scope here
right because you want it to be done
quickly so that's there's that one
then if we think about inexpensive and
fast well these are highly accessible
you can use open sources stuff to do
this
um all sorts of tooling research you can
do and it's extremely repeatable right
this is the purple teaming stuff like
let's do something not spend a ton of
money on it but make it so we can do it
over and over again and the causes for
this are like most of us live here right
this is most of our organizations by
Design have us go to this inexpensive
fast route because we have budgets and
cadences and Sprints and proof of
Concepts and deadlines and all these
things so it's not a surprise that most
of us live here but these canned
procedures and stuff that we use may be
bias the way we do testing and then
finally we have inexpensive and accurate
this is very research driven Focus you
know most folks don't have the ability
to just spend a bunch of time learning a
thing without considering like deadlines
and the other stuff they have to do so
you know analysis reviews of logs
emphasis on specific threats custom
tooling that lives in this land
and so generally we can kind of assume
accuracy is the first thing to fall off
whenever we're doing an analysis or
picking any two so it's non-trivial to
be accurate takes a lot of time can be
dangerous many organizations have ever
tried to detonate real malware inside
your environment to see if you'd catch
it it's very scary when that happens and
it can get out of control really quick
um but ultimately again we all have
deadlines and things to do and all sorts
of jobs so and accuracy in this context
is decoupled from the ttps that we have
in bulk repos and the like there
and then the the other aspect of this is
accurate threat testing information it's
kind of difficult to find when you have
a small scale if you're just one
organization the only the most accurate
threat actor information you have is
what's hitting you and that's probably
not very much from a Global Perspective
right so it's kind of hard to find that
it's rarely free to find really accurate
threat information it's not cheap to
purchase it also uh the quality stuff
and it requires a lot of patience to
understand this and really churn through
it
um and obviously if you have purchased
tooling that does this stuff for you
there's usually a bit of a lag between
threat actor does a thing and now you
have that feature in your product
obviously but for any of you have ever
tried to like look at a threat actor
report and find the explicit commands
that that threat used
it's good good luck it's really hard to
do we get a lot of minor attack IDs not
a lot of specific commands so that's
again the idea here
so to wrap this all up
doing red teami emulation pyramid
climbing stuff is not bad that's not the
criticism there's no criticism here it's
just different from trying to look like
a real threat actor climbing pyramids is
not the same as getting to the center of
that goal Maze and that's okay
um and the point we're trying to make
here is that we think we've lost sight
of how it is to be a real threat actor
and test real threat stuff in our
environments
um and this isn't just an assumption
we're making and proposing to you all we
actually have some data to back it up
which is where I hand it off to Katie
awesome thanks Adam and yeah I think
that introduction is really important to
level set on the state of testing why do
we do what we do and right where I come
in is as a threat Intel person of course
I act value the accurate thing most of
all right and so what we want to do now
is share with you some data about the
differences that we see between testers
and non-testers the whole idea for this
is something that you know I've been
thinking in my brain over the last two
and a half years I've been at Red Canary
my first week on the job I was getting
training from a really experienced
detection engineer Frank and we pulled
up this threat that he was looking at
and he's like oh this is probably
testing and I was like
Frank you're a wizard how did you know
that and now I find myself doing the
same thing and as we started to look
into the data we have we noticed there
are actual patterns in that data about
how testers versus non-testers behave
and this isn't something we've heard out
there so I wanted to share what we see
really important to note where our data
comes from um so as an MDR provider we
bring in a whole bunch of telemetry from
endpoint sensors from alerts from
different sources we process like a
petabyte of day we bring it into our
magic engine where there are raccoons
apparently according to Adam run through
analytics to find the potentially bad
things right then we have leads and then
our detection Engineers super smart
people go through and try to figure out
is that actually something that could be
malicious or suspicious if they think it
is they send it off to the customer
those are the confirmed threats right
malicious or suspicious things we've
seen in the environment so that's where
our data comes in and within those
confirmed threats we can kind of label
and bucket those further into ones that
customers tell us yeah this was testing
either you know someone internally maybe
we hired someone a red team uh breach
attack and simulation tool Etc and then
the ones that are not labeled as testing
and so with that labeling we can start
to do some comparisons here between
threats that are testing and threats
that are not testing
now with any data analysis really
important to note there are limitations
here these are important limitations our
data is not perfect
first off if you are on a red team right
you might be sitting here like well you
didn't detect me no we don't detect
everything and so that's one limitation
inherently of doing this kind of
analysis we don't see everything right
ideally we'll pick up somewhere in the
chain but we don't detect every single
thing that's not realistic
another note is not everything is
properly labeled necessarily right
sometimes a customer won't you know Mark
a threat as oh yeah this was testing so
we don't know that said there's enough
data that is labeled to make this kind
of analysis useful and worthwhile
the other note um I should mention I
used to be in the miter attack team and
I know firsthand the challenges of
mapping to miter attack we map at our
analytic level and so we say okay this
you know Powershell encoded analytic
maps to the Powershell technique and as
much as we try internally to be
consistent
technique mapping human analysis it's
imperfect so right another limitation to
keep in mind there and then lastly we're
sort of making an assumption here that
in that non-testing bucket
they're probably more what we call
proper villains the real baddies than
there are in the testing bucket
again data is not perfect but even with
imperfect data we can start to see some
interesting patterns that I'm going to
talk you through
now as a Defender right if I bring up
one of these published threats I usually
will make a judgment on which category
it might fall into right off the bat so
the first category I want to talk about
we're going to use different emojis to
symbolize each of them what I call red
team red flags
anyone is an SNL fan there's a really
great sketch skit that's like a couple
years old with Kristen Wiig and she's
like at this party and she walks by and
she has like a really long fingernail
and they're like she's like I lived in
Vegas for 11 years and I want to be an
actor and there's a the ad is for red
flag perfume and the the Andy sandberg's
like red flag
so this is exactly what I think of when
I see a detection and I'm like
that's a red team red flag right there I
spotted it the tool the command the
procedure you used this is probably a
red team or a tester so that's our first
category these are the threats that I
see and I'm like I'm seeing some red
team red flags here probably some kind
of testing
our second category credit to my
teammate Tony proper villains right
these are the people who really mean bad
not testers trying to make us all better
right these are the proper villains
they're going to be conveyed by this
little devil devil icon there
then the third category hopefully
there's some idiocracy fans in the crowd
not sure
right I look at the activity the
behavior is and I'm like
shrug I don't actually know if this is
testing or non-testing so I'll be like
Kellen go call the customer make sure
they're on it right isolate this end
point because I don't know if this is a
real Baddie or if it's a tester so
that's our third category is the not
sure category shrug Emoji for that one
so those three categories in mind right
we have our red team red flags our
proper villains and are not sure
about tooling used in testing and by
real adversaries
certain tools when we looked at right
the testing and non-testing threads
certain tools are used by everybody
right if I see Mimi cats or impact it
oftentimes unless there's something else
that's a red team red flag I'm gonna be
like
not really sure right and the data
supports that that there's not a
significant difference between testing
and non-testing with our observations of
these tools and so if you're trying to
be maybe like accurate and inexpensive
these are decent choices because proper
villains use these and testers use these
so sort of both of both of those
categories use these tools
however there clearly are some tools
that are used more in testing these are
the red team red flags some of these are
very obvious yes attack IQ veritin
Atomic red team if we see those
obviously kind of an indication it's
testing but some of these other ones
right you might not have thought but our
data shows are actually more prevalent
in testing than in non-testing things
like Empire maybe that's surprising or
Bloodhound or Metasploit
I'm not saying that proper villains
never use Empire I'm just saying that
some of these tools are more prevalent
in our non-testing activity
so if I pull up a threat and I see
rubius dumpert Power sploit right I kind
of get that red flag and I'm thinking
maybe more likely this one is testing
so if you're trying to Summit the
pyramid and then go to the middle of the
maze these are some things these red
team red flags to think about maybe
changing it up or if your goals are to
be like a real adversary think about
that this might give you away
in addition to tooling also see some
differences in techniques used
now this is where when looking at the
data
there's actually a lot of similarity at
that high technique level
this makes sense because testers and
proper villains have similar goals they
want to execute code right they want to
move laterally Etc so all these
high-level techniques credential dumping
these are great options regardless of if
you're a proper Baddie or not right so
if you're a tester these techniques
widely used I can't really tell based on
these what you are
however in looking at the top techniques
between those categories we noticed an
interesting difference
oddly enough the technique reflective
code loading was more common in our
proper villain detections right so
reflective code loading all about right
injecting that code into memory rather
than having something on disk back it up
so
pretty commonly seen in proper villain
detections and of course I'm a threat
Intel person so I wanted to throw some
real examples of how proper villains are
using these techniques
on the slide I have a good loader so
good load a really common uh loader
malware family it's deployed via usually
search engine redirects so if I go in
and type you know Dave Kennedy Wild West
hackenfest talk it'll serve up a dot zip
file called Dave Kennedy Wild West
hackenfestalk dot zip and I'll be like
cool that's exactly what I was looking
for and click so good load are really
pervasive especially in Enterprises
because it does some checking to see if
you have active directory environment
so one thing that gutlauder does is it
uses Powershell commands to try to
inject a net assembly right reflectively
load that
and the decoded Powershell commands on
the slide we will provide slides later
so you can take a look but on the
defender side right one indication I
have that reflective code loading is
happening is that reflection assembly
load method there and that decoded
Powershell
so this is an example of a proper
villain right using a technique that we
see more commonly in Cropper villains
and right I talked to Adam used to be a
community manager for Atomic red team
sadly there is not an atomic red team
test right now for this kind of thing
right writing.net assembly is kind of a
lot of work and so this is an example of
a kind of thing that right you might
need to put in a little more work maybe
it's not fast or inexpensive but it's
pretty accurate
again an example of just a technique
used more by proper villains
if you have trouble remembering that
malware mean that I just like to bring
up anytime we see gootletter we got to
use the goose the goose Meme and the
goose emoji
other techniques interestingly enough
that we saw a little bit more in testing
were some of our Discovery techniques
right at the overall tactic level right
execution persistence everyone does
those but we found that a couple of
Discovery techniques or reconnaissance
right where the adversary learns about
the environment were more commonly seen
in testing
sort of an interesting finding right so
for example account Discovery with of a
domain account using something like net
group or looking for a domain trust with
NL tests more frequent in our testing
data set
and part of this is because right as we
talked about Adam talked about their
different goals of testing sometimes you
just want to know do we have the
Telemetry to see any of these Discovery
commands so all right something to watch
out for is how you do Discovery as a
tester
in addition to technique differences
another difference and this is a key way
that myself and other Defenders are able
to tell this is probably testing is how
quickly these Discovery commands fire
off and our incident handlers Jerry and
Kellen nodding here right if I go in I
pull up that activity I see ipconfig
netstat who am I like 10 Discovery
commands in a minute and then like 20
seconds later 10 more I'm like well
they're super noisy right they're just
trying to check the uh the miter attack
bingo card test every all these commands
right and it's pretty obvious to me
probably testing so that's something to
watch out for it's an obvious red team
red flag that if you want to be a little
more realistic
think about being more thoughtful right
slowing down your Tempo and just running
the discovery commands you need for what
you're trying to get
right adversaries think in goals we
think in ttps if they're just trying to
figure out what box am I on right what
are their trusts are there from this box
here's an example from d for report
awesome real free source of threat
information right adversary runs a
couple commands then in an hour once
I've done something they run a few more
so keep in mind your tempo of operations
especially regarding those Discovery
commands because we saw a significant
difference in testing versus non-testing
there
that's at the technique level
a lot of those techniques very similar
right we noted there were some
differences in reflective code loading
as well as Discovery but when we get
down to the sub technique and the
procedure level we start to see
additional differences that I'll talk
about
signed binary proxy execution is a
really common technique this is where a
lot of the low bins living off the land
binary is fall and some low bins you
might be familiar with run dll 32
register 32. if I see those I'm like
could be a proper villain could be
testing
however there are a couple sort of
esoteric uh LOL bins that when I see
them I'm like red team red flag one of
them I like to say vertical Sid rolls
right off the tongue it's sort of an
esoteric one that uh verifies shell
extensions and I can say in our data set
the only time we have seen virkle Sid
has been in testing activity
and so if you're trying to get all the
sub techniques under this execution
technique in miter cool but if you're
trying to behave like a proper villain
yeah virkle said might not be the best
choice so watch out for especially those
esoteric low bins that you're like cool
I can totally proxy code through here
you can but might not be super realistic
command and scripting interpreter is
arguably one of the most broad
techniques in all of miter attack maybe
masquerading as well but right within
those sub techniques
Windows command shell Powershell
everybody uses those I don't know if
it's Powershell right I don't know who
it is I don't know if it's testing or
not but a couple of interesting sub
techniques of this technique that we've
found differences in JavaScript
I don't know any testers use JavaScript
something that we saw a lot less
frequently we got one excellent we
should talk after
um so JavaScript right less frequently
used by testers in our data set
something that we see all the time from
our proper villains we'll have some
examples
Auto I.T
another one this is a scripting language
we see all the time it's really common
in those worms that people are like oh
Rita dub gamaroo those are so old it
might be one of the most pervasive
threats we see across all environments
and so autoit is another one that we see
that much more common in our proper
villain side of the house
a couple more examples a sock goalish is
a malware family also known as fake
updates very pervasive
um it is at least to be authored by the
Russian cybercrime group noma's evil
Corp and it's one to watch out for
because if sock golish is there and you
don't clean it up might lead to
ransomware later on and so JavaScript
one of the things that a sock bullish
uses luckily it's a pretty predictable
pattern so for the Defenders on
um right in the audience think about
looking for these unique zip to
JavaScript file names
right it's called fake updates because
it's a fake software update right to a
browser and so we see sock goalish using
a zip file that contains a Javascript
file having names like Chrome update or
Firefox
Etc so just one kind of proper villain
example of JavaScript use again sounds
like at least one person's using
JavaScript and testing but something we
see much less on the testing side
another proper villain example Auto I.T
I mentioned we see that a lot in worms
we also see it quite a bit in a packer a
commodity Packer known as cipherit
now Packers something that a lot of
cyber crime actors just get from a forum
right a cyber crime Forum they can use
it to pack compress encrypt different
payloads so we'll commonly see like a
one-two punch of a packer like cipherit
that then encrypts right uh compresses a
stealer like Redline or Vadar that
steals things
and so cipherit right it uses autoit
it's distributed with a legitimate
autoit binary
and a great way to detect cipherit is to
look for renamed legit autoit binaries
right we can see if we look at the
metadata of the example on the slide
this Deutsch exe.piff that's legitimate
autoit they just renamed it they've
lately been using a DOT Tiff extension
there
example of autoit right something that
if you're trying to behave like threats
that a lot of people see you might not
think about autoit as the first you know
new sexy cool thing to do but it's
something that affects a lot of orgs and
a lot of proper villains use
at that procedure level we also see a
couple differences again Powershell used
by everybody but interestingly enough
right we see encoded Powershell a lot of
encoded Powershell Powershell is not
dead someone forgot to tell the villains
I've heard that misnomer sometimes but
encoded Powershell with all of the
different encoded Flags right I don't
know if you're a tester or not if I see
that
interestingly enough though when we
looked at our data set we found that
right certain Powershell Flags the uh no
op for no profile and the Noni right
running a non-interactive session that
was actually more prevalent in our
testing data set
well why we always want to know why well
it turns out those are the defaults in
Powershell Empire
right and so some of these differences
we see even down at the procedure level
are due to the tooling choices
so think about that right when I see
that again it's not that proper villains
never do that it's that hmm maybe Empire
is being used which is more commonly
used by testers another thing to
consider
so I've talked through a couple
different things right the data set
testing non-testing we've talked about
some differences in tooling right things
like MiMi cats impacted used by
everybody versus tools like rubius or
dumpert right we talked about techniques
reflective code loading talked about sub
techniques procedures wrapping up here
with Powershell
we'll uh do questions at the very end
but thank you so that's uh those are the
differences but now let's talk about
solutions kind of an important part so
I'll pass it off to Adam for our first
first solution here
awesome thank you
I think I've got I've got nine minutes
of voice left so we're just gonna do
this um okay so Solutions we've given
you a lot of things to think about and
some problems so like how do we get
better how do we be more realistic in
our testing how do we
um have our cake and eat it too right
well first off aligning on your goals is
critical
having the entire team think about well
how fast and how accurate should we be
in our testing and how much are we
willing to pay for it having that kind
of team alignment up front is really
important because you'll find very very
very quickly that people have wildly
different ideas about what they should
be especially as you go up and down an
org chart about how much time you should
be spending on the thing how much money
you should spend on a thing
um and and what the scale of the
engagement is from a Time perspective so
making sure that you've got these sorts
of those those core three principles of
you know accurate speed and cost having
those ironed out together is really
critical
um
next one is goal setting so we've been
talking a lot about intent and goals
here well
you should align on the intent of your
adversary testing and again things like
defense validation and apt stealth
analysis they're different they're
they're wildly different and that's okay
so you have to treat your goals
accordingly and they should align with
your realistic threat model obviously
from a red team validate defenses
detection and response capabilities
perspective you have certain things you
want to do but if the scope is or the
goal is uh I want to do adversary
emulation someone come be an apt at me
that's a very different thing right even
though from the long-term perspective
they may seem the same in the short term
they're not
um and so again noting on the duration
of these engagements is key and right
sizing these things for your threat
models really really critical
and then team alignment I think is
something that we sometimes Overlook
when we're actually performing an
engagement this is a classic example of
an amazing miter attack technique test
so this is for t1124 system time
Discovery Well we know adversaries do
this in the wild and guess what the
command date succeeds at successfully
conducting this operation right
do you care like do you really care is
this something that you should alert on
should somebody be woken up if this
happens in your environment maybe if it
happens on an HR endpoint I don't know
and but but you should know right that's
the point is that alignment in your in
your team about how you want to conduct
an engagement and whether or not this
sort of thing matters to you is
important and so having that level set
is really really critical of is this
real threat actor behavior for your
organization
you should have an answer to that
question before you utilize it right and
make sure the team agrees on that answer
before you go
second uh solution here right we're
talking about the solution space use the
quote not threats thoughtfully what do
you mean by not threats all the awesome
testing libraries right Red Canary has
Atomic red team Scythe miter there's so
many libraries of tests out there that
are great but don't just stop by running
all the things right use them
thoughtfully they should be a jumping
off point for you to think about how you
can chain those together how quickly do
you want to use those tests do you add
in other stuff to maybe test reflective
code loading right so use these as a
jumping off point not just the end of
the road right we ran all the tests and
atomic red team were done
right think about how you chain them
together how you use them and of course
if a real threat would actually do that
right bringing in that threat Intel and
that's the that's the last way that's
the last solution space and I know this
is something really common we hear but I
truly believe threat Intel folks need to
be friends with testers with red teamers
right that's why I'm here because we can
share knowledge about what the proper
villains actually do
right learn what these villains do in
real life and if you have access to your
own incident data that's amazing look
for the past months or years go talk to
your sock what have we seen already use
that as an emulation idea or if you
don't have that bring in external
reporting
um Adam informed me that apparently not
everyone has a curated RSS feed of their
favorite blogs that they go to to find
TTP uh based incident info so I've
pulled together a couple of my favorite
blog posts we'll post our slides later
in Discord so you can grab that if you
don't have your favorite blog posts
already that you can grab free thread
information from right
great places to go find information
about what proper villains do and in
addition to thinking about what they do
think about what they don't do as we've
talked about today right there are some
very significant difference differences
in testing and non-testing and think
about what are the things that are those
red team red flags if you're trying to
be realistic you might want to avoid
that in that engagement
and of course change up what you do
right think about the tactic technique
and procedure level of the tooling you
use you're using change it up and then
rinse and repeat and get better
in closing couple takeaways right there
are lots of different goals for testing
and not all testing has to be real
adversary emulation but if that is your
goal right and you should align on those
goals think about how you can do that to
achieve those goals think about the
tools you use think about what you do
and think about balancing those
different trade-offs do you want to be
accurate fast inexpensive choose two if
you're awesome maybe you can get all
three but think about those trade-offs
it is not easy as Adam talked about to
do this kind of testing but right the
reality is there's a lot of value in it
because you want to prepare for the real
thing
and then in closing right key takeaway
test better so you don't feel bad
all right that's all we've got we're
about at time
um so we'll just hang out if you have
questions or hit us up we'll uh hang on
Discord so thank you all so much for
coming have a great one