so now we're going to turn to a another
way of looking at attack with a lens of
how you can misuse attack in your
security program our presenter Kyle
Rainey spent years consulting a global
enterprise Incident Response and
forensic tests for fortune 500 companies
and as a security analyst at red canary
he helps lead the security operation
centers efforts to continuously develop
and mature detection strategies
please welcome Kyle Rainey there we go
cool so today we're gonna be talking
about some common pitfalls that my team
and I have kind of observed over the
past year and a half working with attack
and really looking at this from we look
at from a detection and response
perspective um so that's where we're
coming from and these are the pitfalls
that we identified essentially when we
went to operationalize detect a year and
a half ago a little bit about me really
quick I'm gonna skip that really
essentially uh he did a great
introduction um I've done into response
for a couple years on a global team
before coming to red canary where I now
help lead the detection strategies on a
detection engineering team to build new
capabilities for our detection team I'm
also avid snowboarder and I teach
children to snowboard that Breckenridge
on the weekends so moving into the into
the talk we can get this to work cool
so quick disclaimer so I'm not gonna be
talking about how you shouldn't use
attack it's essentially things that
we've identified that are traps you may
see them you may have come across these
essentially what we want to do is just
bring that to the forefront make sure
everyone's aware this and kind of start
a conversation how you can avoid them
and I and I'm gonna share with you what
we did to avoid these pitfalls and kind
of how we approach that because you
absolutely should leverage attack
there's a lot of value in that
so the first pitfall want to talk about
is assuming that all techniques are
equal and what I really mean behind this
is not necessarily each technique is
equal to each other in terms
what they do because you know techniques
that fall under persistence tactics are
completely different from those of
discovery or other the other tactics
what I really wouldn't talk about is the
scope and size of these within the
language and the taxonomy of attack you
have some that are very specific so for
instance watch agents MSX s LMS build a
lot of the ones that fall under
execution persistence they're very
specific techniques that have a limited
way that you can pretty much implement
them in some cases they're they're kind
of broad there too as well but it really
drives for detection of these pretty
simple not simple but it's easier to
implement detection for these and handle
false positives the the aspect here is
as we go further and dive deeper into
attack as we went to map all of our
detectors to this framework to kind of
align with that we started seeing things
that were very broad things like
PowerShell is a technique PowerShell is
just a programming language you can do
tons of things with this you know to
pick on it a little more you can you can
do almost every tactic with PowerShell
if you want so when we started mapping
our detectors to to this we found that
you know we had a ton that fell under
PowerShell that also fell under a ton of
other techniques but they really it's a
very broad scope there so when you think
about it from detection and measurement
and coverage it changes how you approach
it so my approach to detection that
PowerShell really comes down to what
what are you doing with PowerShell what
are you doing with that broader
technique and how do you apply it with
whatever data sources you are leveraging
in your organization um and then we get
even broader with some of the techniques
that are that you'll find an attack you
know I don't want to pick on supply
chain but I like to pick on file
deletion for this purpose because that's
essentially just a function of operating
systems in general the files get deleted
it's a files get deleted as a part of
activity well in destructive attacks and
things like that so it's very it's
valuable from that perspective when
you're looking at it from a response
perspective um if I'm dealing with a
destructive attack I want to know what
was deleted um it may be important there
so I may want that as context but I may
not create alerts for it or things like
that so there you have to really apply
that lens and understand that not
everything will be to detection or
testability either as we go further so I
like to give away a take away for this
and essentially my point was what we did
was we focused on the most specific
things first so things that we knew we
could implement detection for
specifically around alerting and create
events for our analysts and then we
added on to that using the threat model
to add context to those so we've then
tackled the broader ones and then as we
went through the broader ones we applied
more and more detection to those so for
instance with PowerShell we may have
focused on like down load cradles then
we focused on off you station or then we
focus um like the next thing or the next
thing um so we just kind of tackled it
one piece at a time instead of trying to
do this like broad brushstroke when we
when we went to build detection for
different things or map to different
things um so I recommend focusing on
what's specific to you first and then
increase your scope from there the
second pitfall I want to share with you
that we identified was this concept of
building an alert for every technique we
got this a lot from customers like do
you detect X do you detect y do you
detect this do you detect that and like
we detect quite a few things but then as
we got those questions about those more
broad ones it became imperative to share
with our customers that knowledge of
some of these apply are more applicable
in terms of context some of these are
even more applicable in terms of looking
at from a forensics perspective so not
everything falls into an alert because
not every detection technique has to be
an alert Jack crook is actually a from
GE I believe he's a great resource that
I follow quite often and he likes to
talk about
looking at things in clusters clusters
of detection techniques so those more
broad ones will lead you to if you
cluster them together you might be able
to generate an alert from there as well
so there's definitely ways you can do
that so I like to kind of bucket them
like this we have our alert herbal
techniques so ones that we can
definitely build alerts for they're easy
to tune if you have the visibility in
place and the systems in place to do
that and then you have things that are
really important to add context to your
events for your analysts otherwise you
will flood them and when you flood them
to get angry and they get angry they run
you out of the building so essentially
things like file deletions super
important from a contextual perspective
in the case of a destructive attack
things like directory discovery
attackers the numerator in your network
you want to know about it but you may
not want to alert on it because you have
scripts that run across 100,000 systems
that do this every day
every couple hours and you'll just start
generating events and events events and
you'll just go through this like tuning
rabbit hole forensics once so not every
organization can look at their kernel
modules and extensions every
organization can at scale look at boot
kits and things like that so those tend
to come to light from my experience
doing forensic investigation of deep
dive things that get spawned from other
incidents or other events that we're
looking at and essentially understanding
where those fall in line helps you drive
your detection strategy prioritize
what's important to you and one point
here not everything like Brian mentioned
yesterday about 70 to 75% of the attack
framework is probably testable the same
thing applies to detection not
everything is in the word so again focus
on that providing high fidelity
detection but also enrichment for your
analysts also that there's they're
spending less time tuning and turning
and more time building more complex
detection to tackle some of those harder
problems that we all want to face the
third pitfall I want to cover is
misunderstanding this concept
coverage so we talk we hear a lot about
it we talk a lot about it it's what what
is your coverage what what do we cover
the the leadership asked this question
and and it really comes down to
explaining what you have coverage for
but also the fact that I think Bryson
just mentioned is like there's a
limitless possibility of some of the
techniques so what you see is maybe you
have a coverage map that this is just an
example that has a bunch of green
squares and you have coverage for some
of these cool things um but as you open
the door to each technique what you
really see is that infinite universe of
techniques and behind each door is that
universe and you have to understand that
you will build detection and you will
have coverage for parts of it and then
you'll just like a researcher will
publish something on Twitter or whatever
and you'll discover something new and
you'll go to add to your detection it's
a continuous addition and improvement
and the way we like to track it and the
what I want to share with how we kind of
do our measurement we measure everything
that we've implemented so we capture
metrics around each of our detector
techniques whether they're generally
true positives false positives what
category of detection that they're
generating Texas for how much time
analysts are spending on events for that
specific detector to help us kind of
track like is it too broad or too hard
to analyze what can we add from an
enrichment perspective to speed that up
time time for us is very important we
want to spend less time on false
positives more time on true positives
and more time building cool things the
so the focus there is is measure what
you have as you add stuff include that
in your measurements but don't try to
like go for the unknown
don't try to measure an unknown like
limitless expanding kind of universe and
you'll get more value out of that and
then in terms of like how you share you
know changing your coverage gaps as you
add new things and form your
stakeholders hey we we added these new
things
it closes this part of the gap but we
still have this other part of our gap
that we have to work on next and and and
and it really drives home that you're
continuously improving and that's the
goal at the end of the day to improve
your security the fourth one I like to
share is staying in the matrix so attack
is great help it gives us that language
we can talk to I can talk to anybody on
my team using this language they'll
understand that I can talk I can
probably explain it to my mom maybe um
but essentially the the point is it's
it's not all-inclusive it has it's very
comprehensive there's definitely things
that fall outside of attack and things
that will at will add to attack over
time as this community grows but the key
thing here is is here we go you know I
chose to leverage attack for us to give
us what that that language to help us
improve like that's what we did and and
the point is if you stay in the matrix
you'll fall behind in terms of you know
innovation adversaries adding new things
things that get added to attack take
time models take time to update
standards take time to update remember
not everything is in the language or the
or whatever model you're looking at so
don't let that drive you you drive it
feed back to the community give back as
you find new things add them and that's
that's the goal behind this and the big
thing is make it work for you don't rely
on it make it work for you help improve
drive the community forward and escape
that matrix and then number five
essentially is forgetting the
fundamentals so as we built as we map to
attack an operation realize that we
spent a year and a half really driving
this through and the big thing is along
the way you get sucked into you know
focusing on attack and focusing on
detection and response but security is
much broader than that there's there's
tons of there's tons of fundamentals
just a few of the many but essentially
the point is don't forget about them
continue to drive what's most important
to you and your organization and drive
things forward leverage attack where it
makes sense don't try to fit it into
everything it will fit where it fits
and you'll you'll get great results from
that but again the point is find out
what the most important aspect for you
and your team your organization whether
you're a detection team whether you're
on infrastructure compliance whatever
find that most important thing execute
on it move on to the next most important
thing and just make security better as a
whole questions well thank you
so we have plenty of time for questions
so we have the throw mic and I assume we
have the throw mic over there new
battery or you just go up to the
microphone Thanks so going back to your
first pitch ball do you have any
suggestions the in terms of finding with
specific to an organization do the grunt
work of looking at 223 techniques and
going one by one or did you find an
easier way to kind of categorize to risk
rank them somehow to say this is how I
wanna tackle these so so for us we we
focus on our historic detections um so
what was what we were seeing used most
we started there um and and then we we
went down that list essentially and then
in terms of if you're trying to so that
was for mapping and then for building
new things the way we prioritize is
based on a couple things we try to
assign like if something we've seen in
the wild we give it a higher likelihood
of it being used it depended on what it
is if there's a high impact we'll tackle
that first we we don't like to drop like
use attack to to drive each idea we
essentially find all of our ideas
whether we get them from attack or not
we assign that likelihood or that impact
rating and we prioritize that way and
then we
kind of just continuously feed new
sectors out Thanks do we have other
questions over there yeah the question
about the your third bit for
misunderstanding coverage yeah I fully
understand what you mean and like the
there's a galaxy behind all these doors
but I would imagine that you know
leadership would like to understand you
know when are we done you know what is
when do we do enough and and they like I
think KPIs that that tell you know when
are you at hundred percent so have you
had to deal with remarks or questions
from the leadership like this in the
reality is there is no end there there
is no 100% coverage there never will be
of you may have you have to find that
point on each technique or on whatever
it is you're working on to to find that
point at which there's a there's
diminishing returns there's you can get
coverage up to a certain percent and
then it'll cost you like X amount to add
that like final 1% and at that point you
have to make a decision you know based
on you and your business whether it's
important to go that extra mile or or or
call it for that and then add more on
other areas thanks for the presentation
this may be a little long-winded or at
least until a bit of a lead up for me to
explain the question so in the way that
you broke out the matrix into a little
contextual and forensic it seems to me
that one of the problems that we may
face with the framework itself is that
it continues to grow and there were in
the earlier version of the framework
there was a simplicity in it that made
it easier to absorb and understand in
fact you could literally put it on a
single page today I don't think you can
put it on a single page the question it
really goes to you as well as to the
rest of the attack team which is I don't
think we actually wanted to grow in the
current
format because it will start getting
overwhelming to the point where it may
resemble something like APEC which is
very hard to grasp because it's so big
would we want to consider and this is a
question really for the community would
we want to consider having the attack
framework be focused on one of those
elements whether it's the alerting
contextual or forensic and and then if
there is a deeper there's a deeper pit
behind it be able to capture that in a
different framework or different
construct and the reason why I say this
is I think there is to contrast your
statement earlier I think there is a
limit that we want to create here and
that limit is based on where we can
establish a visibility once you
establish visibility or rather if we
look at the attack framework as a list
of vectors attack vectors then the
question that's asked is do I have
visibility into those vectors an
attacker can attempt various types of
exploits with that vector so for example
PowerShell is a as a vector there's any
number of different exploits within
PowerShell right and we would then look
at ways that we can do signature based
or behavioral based ways to to detect
specific exploits but our attackers will
continue to come up with new evasions
right so that's an endless well but we
don't want to represent that in the
framework because it will be endless but
we do want to capture something like
PowerShell because that's bounded so the
question is again do we want to consider
how do we want to bound the attack
framework because I think what you're
pointing out is that it could pinch it
if you don't if you don't carefully
understand what we're representing in
there it could end up being an unbounded
matrix which I don't think will serve
any of us I I agree on that on that
standpoint in the way I like to think
about it and this this is just kind of
how I think about it is I like to use it
as that taxonomy
we're all you know we can have those
bounds but I also approach it from the
understanding that there is a lot of
depth to each of the different ones and
it allows me and it allows us to explain
to each other and have that length that
conversation about those those tactics
and techniques and those implementations
without going into this endless matrix
so I think there is a limit um but I
don't think there's a limit to what you
would consider that is just chaos like
if you want to weigh in on this at all
yes a needle is always good for very
deep questions so I think there are
things that we can do to help control
the growth of attack as it is now and
I've mentioned the idea for sub
techniques yesterday and I think that
will help some of the abstraction issues
right now so that we're not describing
very atomic techniques sort of at the
the main technique level so I think that
will help a little bit and I think over
time well maybe you have to consider
some new ideas on how to mitigate growth
but it is something we're very concerned
with we are looking into ways to help
with that thank you other questions
thanks for the presentation actually
like I share a lot of that peaceful
t-shirt I won't add one but I want to
get your opinion on something like how
do you map machine learning based
detection I guess I made a Jody machine
learning doesn't know attack framework
in the sense you can have detection at
our base it on prevalence profile
classifier they don't know anything
about attacks but they still detects
things and how do you map this type of
detection do yeah so we had a similar
problem not machine learning but a lot
of our material detections are not just
on adversarial behavior they're on just
abnormalities with system behavior like
forensic artifacts um so a lot of those
are just abnormalities with the system
and they don't really map to some of the
techniques specifically on what we did
is we just kind of internally created
our own labels for for some of those and
tried to define them internally and then
where they fit with attack we shared
them back with my
but it's it's again not all models are
perfect they don't cover everything and
you just have to use them where they fit
and it make it work for you other
questions
so I had I had one so it's it was I was
really interested in in looking at your
presentation because obviously we are
all super gung-ho about attacked but
there's I think we all recognize that
there are opportunities for improvement
and so which of you know some of the
things you talked about are just
realities is probably not anything we
can do as community except work around
them but if you had to give us because
we have the core attack R&D team or most
of it sitting here in the front row what
what would be the three areas you'd want
us to focus on maybe some of them been
mentioned before maybe they haven't yeah
yeah that we could change yeah we should
consider changing in concert with the
community right I I think the the
biggest thing off the top of my head is
definitely like sub techniques and and
having a having consistency so back to
that that point one there's a lot of
inconsistency in the scope and scale of
things so if we can be more consistent
and then have the sub layers for that in
an easy to leverage way I think that'll
that'll improve a lot of the usability
and kind of inconsistencies and that may
lead to us being able to potentially
have better measurement and
understanding of measurement and be able
to explain that and then you don't have
to do that I don't know three I mean
that's like the biggest one that I've
always and and so started to read to
take a look at that and and would you be
up to like a refactoring so like some
techniques actually get folded into
others and you have become sub
techniques under others and I'm making
this stuff up in place that's that's
that's definitely an option I think it
comes up it comes to like all of us as a
community to kind of figure that out I
don't I think there's definitely some
play there and we just have to figure
out what that is
I'm just going to ask the folks here so
by show of hands who here thinks that
there's too much unevenness an
abstraction in the model at the moment
I'd say that is a majority and I'm
raising my hand yeah they're raising
their hands that's great you know that's
you know that you know there's a point
we have a question over there there's a
you know a real important point which is
this is we're not done with attack as
miter as a community and as you said we
probably will never be done so so this
is this is about that's this dialogue
you know Blake loves to hear how much
people you know love attack and Jason
we're in the mall but you know this
really is about providing that
continuous feedback question so when I
explained the attack framework to people
I explained it as the tactic is the why
what an attacker want to do this the
technique is the what is a person
actually doing in order to achieve that
how and then sub techniques or
procedures would be the actual mechanism
that they're implementing it's the how
they are doing what they're doing in
order to accomplish their why and I
think that there is some degree of
divergence in that within the existing
matrix and that causes some confusion
because there is a differentiation
between how someone does it and an
indicator and there's been this
temptation to conflate the two because
of that disparity and the framework when
I try to apply this across all things
like for example PowerShell not every
TTP in the attack matrix has an implicit
procedure associated with it so when I
hit those cases where there's not an
implicit procedure that's where things
break so I almost feel like the matrix
needs to move up to a point where
nothing that's in the matrix has an
implicit procedure that those things are
the value of sub techniques and that
many of the things that
don't have an implicit procedure could
be consolidated under another more
descriptive category or what if you will
I agree perfectly with like the the the
there is definitely a need for more
examples in terms of some of the
techniques and then some definitely can
either be expanded out or rolled up
together so we have a question over here
and then we have a question in the back
oh that was fun I was also in the NFL I
don't like to brag about it
first off absolutely let the talk it's
something that I see a lot of people
they think the attack framework will be
an easy button for their defenses it's
obviously not so I'm gonna throw this
more back to you since you do consulting
for firms you've been around can you
give us some of the stories from the
field for what you've seen people have
implemented the worst no names obviously
no names and shames just where are you
getting this data from just kind of
going to some of the stories behind that
yeah so so a lot of a lot of it comes
from from just experience with dude from
prior consulting engagements and then
being able to I reckon area we see
Brian's told me earlier there's 100
terabytes a day of endpoint data um and
and we work with our customers in that
regard
when we find things and and just have
conversations with them about attack and
and whatever they're doing in their
Security's program to kind of get their
stories as well um and I think like the
worst implementation there's really not
a worse it's just kind of the if you're
not familiar with using a taxonomy or
model or languages like attack you could
fall for the trap of thinking it's a and
maybe it's because it's in the shape of
a periodic table or matrix so if you
treat it like a periodic table each
thing's its own individual element but
if you look at it like a number of
people have is they treat it like this
ruler or a measurement tool and that's
not what it is and I think that's where
the conversation kind of breaks down
sometimes and that's where where we kind
of direct that question the back yeah
King okay
to close so we've been using the attack
framework since about May last year in
our operations and just one to weigh in
on the behaviors that are in attack and
it's it's been a great way for us to
identify you know the actions that an
attacker you know uses as as they
progress through the attack and being
able to focus on the behaviors in
writing detection that correlate a
sequence of behaviors so you were using
PowerShell for examples so we may have a
an execution tactic or excuse me from
the installation tactic we may choose
you know a technique with that we may
incorporate scripting with PowerShell
technique we may and also include
credential access we're dumping along
with you know some some other techniques
and we all correlate that as a series of
behaviors and then when it comes to
writing detection we look for yeah that
set of behaviors to occur within a
certain time frame and in that approach
it's worked very well and you know based
on on how the attack framework has
progressed I don't think it's going to
get squirrely or get out of hand
there are certainly quite a bit of
techniques that have been developed but
every one that I've seen has fit so far
from a detection standpoint and from a
language standpoint comments oh no I I
mean so that's one definitely great way
to approach it is is to apply the
different techniques where they fit
regardless of what your detect
detection technique is specific for um
and I think that's that's the kind of
key point about it a language or
taxonomy that I like distress is like
you use that language where it matters
and less so of like treating each thing
individually um and the way I like to
lead to that is like as a snowboard
instructor we have a common language
between instructors where we talk about
the fundamental MOOC fundamental
movements of a snowboard we all speak
that language when I'm talking to other
instructors but I don't speak that
language to children I distill that down
to them but essentially having that
language helps with that aspect and I
think this is super important for our
industry as we grow and expand and build
that corpus so you touched on this a
little earlier but I wanted to ask kind
of for clarification since you're your
presentation is about pitfalls for
applying attack attack is a language
that is fundamentally designed to
describe the activities of attackers do
you see pitfalls in the current trend to
attempt to ascribe attack or use attack
to describe defensive capabilities or
coverage I think that it applies to what
it I think it applies to that that
describing of attackers fairly well I
think there are aspects of Defense that
go beyond that and I'd like to see like
I think Blake mentioned earlier
additional additions to that to attack
you know for other aspects of security
they work that are not covered in the
current matrix whether that's cloud or
mobile or insider or or you know there's
there's tons of other things and I think
that as we grow as a community and
expand upon that knowledge and
corporates of that language um
everything will start to come into play
any final questions all right if not
please join me in thanking Kyle
you