hi y'all thanks for coming to my talk
this is how I learned to stop worrying
and build a modern detection and
response
program today we're going to use a
framework I built to create a modern
detection and response program you all
will come away from this talk with a
better understanding of what a modern
program should look like how all the
tools capabilities and processes should
put together and how to build and
evaluate so you can stop crying every
time your pager alerts blow up your
phone but first hi I'm Allan I'm a
senior staff engineer at Airbnb on our
technology leadership team where I work
on nonstressful things like threat
detection and incident
response I'm also a dad and I live in
Austin Texas with my wife and 2-year-old
son Liam here we are in a a giant
adult-sized ballpit where I almost lost
him so many times maybe he's still there
and this talk is in a way inspired by
Liam you see I'm a
worrior and being a parent and a worrior
and an incident response is a super fun
combo um I remember when Liam was just a
few hours old he was sleeping in the
hospital bassinet and I was listening to
him breathe when I noticed that his
breathing sounded a little different and
I started to worry so I got up to look
at him and he's purple now I've been a
dad for about 30 minutes and if you've
ever seen a newborn baby they're purple
but he's very purple and as I'm taking a
closer look I can see that he's
struggling to breathe but I know what to
do I have a plan I paid attention during
that online parenting class I took
during covid and I flip him over in my
hand Pat him on the back and he spits up
and he starts crying but he's breathing
again he's fine and Liam gifted me
something really important that day and
that was perspective on my worrying and
I realized something worrying can be a
superpower there's a research paper out
of the University of California
Riverside called the surprising upsides
of worry and in this paper they argue
that while worrying especially at
extreme levels certainly had this
negative outcomes but there's a really
cool upside to worry here's a quote from
the paper worry illuminates the
importance of taking action to prevent
an undesirable outcome and keeps the
situation at the front of one's mind to
ensure that appropriate action is
taken on Blue Team worrying is a
constant reminder that things can and
will go wrong but without a plan or
strategy we'll keep worrying and here's
where I see a lot of programs fail now I
started my career on the red team and I
got to watch blue team fail a lot and
when I switched over the blue team
because I thought I could do it so much
better uh and started working in
detection and response I found that
myself and my teammates while we were
experts at accomplishing the technical
work we really didn't have a vision or
strategy for what we were building and I
believe this is one of the major reasons
blue teams fail we are taking lots of
action we're really busy but we're
really making it up as we go or just
copying what we did at our last
organization because it kind of worked
there so after making lots of mistakes I
took all of these Lessons Learned and
built a framework that finally worked
for me and hopefully it'll help you
build a much better detection and
response program and this way yall can
learn from my mistakes and make better
more informed mistakes than I
did but before we dive in let's talk
about what I mean when I say a modern
detection and response program versus a
Lego Legacy one a legacy program is
reactive where the focus is on alerts
that indicate that something bad has
already happen a legacy program focuses
all of its strategy on the technology
you know you're doing this when you
describe your program by listing all the
tools you own and the vendors you do
business with without thinking about the
capabilities that you actually have a
legacy program has a lot of manual heavy
tasks sure you may have invested in
autom tooling but if your teams are
still doing a lot of the day-to-day
tasks manually you're still operating in
this Legacy
model and finally a legacy program
operates completely siloed and
disjointed from the rest of the
organization I'm most guilty of this one
uh but it puts your teams completely out
of touch with your orgs technology it
inhibits their ability to work side by
side with partner teams and you end up
trying to solve problems that would be
better solved in collaboration with
other
teams on the other hand a modern program
is proactive and that doesn't just mean
you're doing a thing called threat
hunting it means your philosophy for
detection incorporates the idea that you
want to detect a threat as early in an
attack as possible and that there's many
signals you can correlate before that
something bad has happened alert needs
to
Fire and instead of being so tool
focused a modern program builds strategy
that is business focused gross uh but by
that I mean that we Empower our teams
with more than requirements but also the
context of what they're working on and
how it empowers the business to
succeed a modern program prioritizes
automation when you're implementing
processes integrating tools your first
thought should be how will we automate
this instead of creating those manual
playbooks that you sure hope you'll
automate in the
future you won't uh a modern program is
is connected to the business it
centralizes its functions workflows and
data because you can't succeed by trying
to do this on your own you won't be able
to
scale and uh that all sounds great but
what about all the challenges in my way
and I know them very well uh it's the
alert fatigue that's grinding any sense
of hope to move forward it's your lack
of a budget to get the tools you need to
succeed hiring retaining Talent is
especially difficult because these roles
lead to burnout and it seems like you'll
never get out of firefighting mode as
the incidents they just keep
coming well hopefully I'm here to help
and to help us get past this we're going
to use the process of organizational
design so I read the book so you didn't
have to and this book lays out a
step-by-step process for designing your
organization now they have seven steps
but I know my audience and the talk
after mine is lunch so we have four
steps since most of you haven't had
enough coffee for seven steps and these
are the four phases of the framework
we're going to work through today we're
going to assess and analyze our current
state we're going to design and develop
our ideal program we'll talk about when
it comes time to implement how we can
overcome some of those operational
challenges and finally we'll evaluate
and report and I'll talk about how we
all suck at metrics so here we
go we start by asking
where are we and what do we have do you
have anything maybe you have a large
program that reports detailed metrics
and maybe it's you and some antivirus
maybe uh we want to understand our
starting point and the first piece of
advice I want to give here is to stop
doing and start learning take a breath
sure things might be on fire but they've
been on fire for some
time so before we start fixing we need a
first learn and we're going to learn
from these three different viewpoints
and first up is our vision and Mission
and instead of reading what I'm sure is
a riveting vision and mission statement
I want you to ask what is unique about
detection and response at your
organization what unique problems does
the culture the technology and the
people at your organization pose for
detection and response and then what are
people spending their time doing
asking these questions will help us
figure out what the organization needs
from a detection and response program so
who do we
ask audience
participation Regulators Regulators
they're good people to ask who
else
huh business the business yeah we want
to ask the people that are on the team
that are actually going to try to build
this right my favorites legal you always
want to bring legal in
first you want to talk to your partner
teams not just the detection response
team but the red team your corporate
your production security teams third
party
risk and here's a silly question how do
we
ask nicely hopefully uh so questionnaire
are not a tool I've often used but some
of these questions take time for people
to think about hence the very slow
reaction I'm getting here so question ER
might be
appropriate and then we can assess our
Viewpoint from our people and understand
the skill sets we have before we start
building and the resource I'll Appo you
to for assessing people is nice cyber
security Workforce framework it comes
from this and it categorizes and
describes different cyber security
skills in nice skills are defined as
work roles there's a lot of them but to
give you a jump start I've taken these
work roles and grouped them into
detection and response roles these are
very general groupings and I'd expect
you'd have responders that are also
Engineers but I found that these roles
make it easier to talk about the skills
needed without too much
customization and then to assess back
when I was a senior manager I created a
self- evaluation survey I asked my team
to rate themselves for each of the work
groups so for forensic analysts each
member of my team would rate themselves
as either having no experience small
baby amount all the way up to parallel
parking during rush
hour and then we can all use that data
to create a heat map that visualizes the
experiences and skill sets of the team
so now you have an estimate of what
skills you have where you have strengths
and weaknesses and where you have gaps
and this can help you evaluate
technology decisions the tools You're
Building
where to prioritize mentorship and where
to target training and
hiring and then next we look at our
technology and for these I'd like you to
consider that technical capabilities are
not product categories and by that I
mean um you might have something called
EDR and I might have something called
EDR but depending on the vendor the tool
the operating
systems which license subscription we
have if we can figure that out and what
we're actually using our capabilities
might be really different and second I
think product categories are confusing
um I think if I surveyed three people in
this room what xdr was I'd probably get
four different
answers so we're analyzing and designing
our technical architecture so we're
going to focus on technical capabilities
instead of product
categories all right cool you did great
you stopped and learned now we're going
to have some fun and design our
detection and response program and when
I design and develop I like to think
about it like I'm telling a story
because stories are memorable our brains
are wired to remember stories you might
walk away from this talk only
remembering the story I told at the
beginning this is Liam again he's two
now uh he can't read surprise but he can
recite this book about Leo going for a
swim word for word
so having a vision or a mission
statement or whatever you write down is
cool I guess but when you start to talk
about your program like you're telling a
story that's when it can really start to
resonate so with that in mind let's
start designing and developing and in
this phase we're going to create these
two
views for the first we're going to ask
ourselves what processes does our
program
need processes tell the story of what we
do and to better enable this
storytelling we're going to build a
process View and this is a diagram that
gives a visual representation of how the
processes in our program interact with
each other and I'm an engineer now so I
have bad news the diagram I'm about to
show you is not an exact diagram it's
not intended to be it's a visual that's
going to help us tell our story so I'll
start from an ideal beginning where our
first process is threat modeling where
we build profiles of threat actors
gather intrusion sets to understand what
type of threats do we care about the
most and the story I'm telling here is
this program is fueled by threat Intel
where Intel both internal and external
is collected analized and disseminated
throughout the program whether that's in
the form of threat briefings ioc's hunt
packages the story I'm telling here is
that we don't want to reinvent the wheel
we still need the classics event
monitoring triage analysis incident
response and
forensics the story I'm telling here is
that we're proactive our Intel often
from incidents is used for threat
hunting maybe starting with simple ioc
searching and then moving into more
creative data
analysis and the story I'm telling here
is that we continuously test and improve
our program using something that I like
to call micr Purple tests tests that
simulate threat techniques so you can
validate that your detections actually
work and that people actually know how
to respond to those um the analogy I
like to use is that when you write some
code you write some tests right
Liars you write a detection you should
write a test to validate that yes that
detection actually detects what you
think it does and we use the results of
those tests to prioritize The Continuous
Improvement of detections and responses
to threats including findings from
threat huntings like gaps we might find
in specific
logs the story I'm telling here is that
observability which I'll talk about in a
bit is the foundation for engineering
detections and enables threat hunting to
provide its input and when our
detections fire are classical
operational processes can pick up the
event and start the triage and response
processes and the story I'm telling here
is that this program's performance can
be measured and that data is
communicated about how we're succeeding
improving changing our strategy and
investing so then we can Circle right
back to the beginning where our incident
data flows back into our threat Intel
collection so we can continue to inform
the program with new threat hunts threat
briefings and
detections and if we zoom out we can see
that our reporting and metrics can also
inform our red teams with new scenarios
our security awareness teams with
metrics from fishing events so we can
better educate our users and our partner
security teams that build and operate
many of the controls that help prevent
incidents from ever
occurring cool now we've got some
processes let's look at building our
architecture and we're not going to list
product categories we care about
capabilities and we started with the
process view because I think it's easier
to think about what we need to do and
then think about the technical
capabilities we need to do those
and so here I'll give you some resources
for when you want to think about
capabilities MIT or attack you're
familiar with it knowledge base of
attacker tactics and techniques one way
I've used attack is to walk through the
tactics and think what capabilities do I
need to detect and respond to this and
then think about that in all of my
different
environments miter defend not released
too long ago uh it's a catalog of
defensive techniques that are mapped to
attack um defend can be a little bit
prescriptive but I think it really
shines when it describes analysis
techniques tines have a sock automation
Matrix which lists all the common
activities most sucks when automate it's
a really good starting point and it
covers all the things you'd expect like
enrichment and user interaction but it
also covers things that you might want
to automate that you maybe don't think
about like documentation because I'm
sure you're all great at
documentation and then finally there's
snowflakes detection series on medium
which connects building detections to
software
engineering and I've also spent some
time thinking about how to measure
detection and response capabilities
using a maturity model approach um
maturity models are a fun way to think
about the goal state of a program notice
I didn't say end State uh this work
never ends um but this model is still in
progress and hopefully a future talk but
I wanted to briefly share where it's at
today and in thinking about detection
and response I've broken this maturity
model down into three pillars the first
observability it's the foundation that
we build our detection and response
capabilities it's having the tools and
logs that give us visibility into our
environments and infrastructure along
with the context and intelligence that
an analyst needs to determine next
steps second is proactive threat
detection where we're constantly looking
for threats and doing everything
possible to be prepared for those attack
scenarios and then finally we have rapid
response because we want to be able to
stop and contain threats
quick and for each of these pillars
there's a few key functions so for
observability we care about entity
coverage or visibility into resources
these are things like your logs for
users system
devices platforms apps and then activity
coverage is any action or state change
for those
entities searchability which is simply
the ease and speed of answering
questions so collecting aggregating
optimizing and normalizing entity and
activity
data but not stopping at searchability
but also
contextualization or understanding what
an event means and what's relev about it
in the timeline of all the involved
entities I say this is someone who has
been a sock analy for many years
capabilities like correlation enrichment
and historical context usually fall into
this function and then you have the data
and platform sources you need to enable
this enrichment things like your asset
inventory I hope you have that identity
management and IP address and DNS
information for proactive threat
detection you'll see I yet again start
with threat Intel with the intelligence
sourcing function this could be
collecting external threat Intel doing
threat model modeling exercises on a
production app findings from your threat
hunting and the data you've reported in
your incident and
metrics detection coverage measures your
ability to detect threats informed by
intelligence while often measuring your
gaps from weight listing exceptions
blind spots or missing logs or just a
lack of Baseline
information and then there's my favorite
Trio uh efficacy or the likelihood that
an alert is a true or false positive
based on a subset of your historical
data Effectiveness which simply asks has
this alert ever fired for actual
malicious
activity and then uh efficiency
measuring how long it takes to trigger
an investigate an alert and then finally
accuracy which me measures whether a
detection has the appropriate precision
and sensitivity values for the risk have
you undertuned doubt it or overtuned
your detections is the volume of your
programs false positive to True positive
ratio appropriate to the associated
risks and then for rapid response do you
have the playbooks enrichments and
automations for protections and known
incident types not only testing these
scenarios with the security team but
also across the organization uh good
keynote today talking about how we need
to do better about going to all of the
different departments in our
organization um because you might have
the right technical capability to
isolate hosts but if isolating that host
brings down production you might want to
rethink your approach I know this and
and involve all the right stakeholders
in finding the right technical solution
for an incident uh and hopefully do it
before the incident and all that
pre-work is so you can measure and
improve your overall incident response
performance and quick uh how quickly did
you uh arrive at the incident determine
the facts and mitigate the incident did
the response team correctly interpret
the data and make productive decisions
and did you miss
anything
so let's think now a little bit more
specifically about the capabilities
you'll need by going back to our process
View and if we focus on threat Intel and
think about the types of data we'd like
to be collecting about threat actors
intrusions uh things from the app
formerly known as Twitter but also
actionable Intel like early warning
signals dark web postings related to
your organization threats to your brand
and reputation creds for sale and then
we have the to think about how we're
going to collect it disseminate it and
integrate it across all our tooling for
Intel driven
detection and then switching over to our
classic processes thinking about all the
automated capabilities we'll need to do
triage and Analysis as well as automate
our
response and then for my more
engineering focused and proactive Pro
processes I think about not just
collecting the logs but doing everything
I need to do to get observability out of
my logs thinking about how I need
analytics to detect threat behaviors and
malware across all my entities and then
how to automate testing of all the miter
attack techniques or other techniques
you come up with in different in all my
different
environments and then we can put all
those capabilities together and create
our architecture View and just as I
discussed before the headline for this
story is that we're taking all of our
threat Intel capabilities to enable
Intel driven
detection so that our threat Behavior
our malware analytics and micr purle
tests can be directly informed by our
Intel take all that data turn it into
something our teams can use for True
observability an engineer detections
that generate alerts for automated
analysis to trigger rapid responses
across all our various
controls so now we've got a long-term
architecture for us to work on to
deliver a modern program
and sure that sounds great beautifully
drawn
processes excellent presentation thank
you very much uh but what about the fact
that I have a lack of a budget for tools
I don't have any new headcount so let's
talk about
that um so first let's talk about making
the most of the people that you do have
so when I transition from being an
engineer to a manager uh my perspective
really changed
I had to take a step back and think
about the bigger picture and the idea of
building these uh views came to my mind
and now I'm in an engineering role and I
realize that these viewpoints are vital
for everyone but almost most importantly
the people that are building
this um the stories were telling our
people about what they're building and
what's the most important thing to be
working on are really hard to convey
sometimes and so having a visual
especially for people that are visual
Learners or just want to organize things
it can really help and this is really
important for operational teams
especially because we are always feeling
like we're pulled in many different
directions um if uh you're trying to
build a new program but you have a lot
of historical debt noisy alerts manual
tasks I have found two approaches that
work really well
the first is to declare
bankruptcy uh you're not going to solve
yourself out of this one I gave a
similar version of this talk elsewhere
and somebody asked me a very detailed
question about how they were very stuck
in the mud and could only see making
micro steps and in those cases I think a
lot of times uh declaring bankruptcy is
not a bad approach saying you know what
we're gonna just push this to the side
and start a new
it probably won't completely fly with
your leadership um and so you may want
to think about how you can Outsource the
most critical things possible that need
to keep running right now time box it
hire a third party sock or something
throw bodies at the problem so that you
and your team can focus on starting to
build something that
works and then as you prioritize your
technical capabilities the inevitable
question about the approach of building
building versus buying and I'm an
engineer so I love to build but I
usually recommend that solutions that
give you 65% of the way
there not ignoring the basic principles
so you still want to be proactive you
still want to prioritize automation
those still need to hold true so
whatever Technologies you're bringing in
need to still enable that but if you can
buy solutions that get you the 65% of
the way there then your team can focus
on building those program
specifics and of course why you build
you've still got incidents you've still
got operations those are still going to
blow up your phone but especially in the
beginning of implementation try to push
back and let the less critical Things
fall on the floor because we do a really
good job of giving the appearance that
our busyness means that the program is
working just fine and as we'll discuss
next changing what and how you report
can open your leadership's eyes to the
the reality and then get them on board
with this new Endeavor to shift to a
modern program so let's jump into it and
just like there was Legacy in modern
programs I think some of you with very
modern programs may still be falling
into very Legacy methods of evaluating
and Reporting and it's preventing you
from getting the support the funding or
the ability to shift to this more modern
approach and when I say Legacy it means
you're still thinking reactive by
assuming ass uming that time to detect
time to respond and time to contain is
all there is to say about
incidents when I say Legacy it's a
report to leadership with the number of
events that happened last month in
comparison to this month without any
context as if it provided any meaning to
anybody and I know every single one of
you have been in a program that has done
just this or inherited those
metrics it's a report of what you're
seeing without any context of what you
know you aren't seeing and overall
Legacy reporting doesn't tell the
business the proog the doesn't tell the
business what the value of your program
all they know is that some bad stuff
happened there was more bad things this
month and last month and you are very
busy while a modern program provides
context around the detections in place
describing how threats are being
detected and what the impact to business
is our reporting should focus on what
threats we're seeing can see are seeking
funding to see it provides context to
your numbers what are the associated
threats what environments could be
impacted and where is their visibility
and it quantifies those to business risk
like hey remember that ransomware
incident we had uh that could have meant
we weren't doing business for weeks and
that's a lower
estimate so I want to talk briefly about
ways to evaluate and Report I've talked
a lot about
observability and I think these three
questions really simplify what I'm
getting at what can we detect today what
threats can we see what can't we see
what's our landscape coverage do I have
the same threat visibility in my cloud
platform that I do have on my
laptops and then take both of those and
answer what's our overall visibility
into
threats and to answer what we can detect
today I mention the idea of micr purple
testing that is building a repository of
threat techniques I like to start with
miter attack defining the scope and
complexity of the test and then running
attack simulations or what we're calling
these micr purple tests and then scoring
how effective we are at detecting and
responding to that attack
activity this validates whether we can
or can't detect something so we know it
develop and improve on and here's an
example of what running all the
techniques across miter attack on our
end points and scoring them as pass or
fail might look like in this
hypothetical example the story is that
we've got good coverage on Discovery and
techniques say C2 techniques but the
attacker might uh be able to move
laterally through the network without
being detected and that could be bad for
business if you have poor controls
between environment boundaries
because none of you have a flat Network
right next we look at the results of
running all those techniques across all
our different environments or what I'm
calling Landscapes we get a different
view of where tactics might be less
likely to be discovered so here the
story is that this organization has
pretty good endpoint coverage but almost
no visibility into their containers
which is a problem because that's where
the business apps
run and then you can sum it up by by
providing overall visibility by looking
at the results of your micro purple
tests looking at the detections you have
in place today but maybe you haven't
written tests for those yet to validate
and then waiting those by the priority
and prevalence of your
environment and then you can use these
three reports to tell leadership what
are the different types of threats which
ones we can see and where and overall
how we're doing and then of course you
could Trend this over
time and closely tied to our
observability report are our metrics and
for these please let's move away from
giving alert numbers and instead say we
know what we can see what threats are we
seeing the most and where which ones
pose the biggest threats to the business
from what we can see what are the trends
and impact what preventative controls
Investments could reduce this risk and
impact so instead of giving a lot of
meaningless numbers derive from your
events what are the top threats what are
your top Landscapes or environments at
risk and what are the top incident the
top incident trends that the business
need to immediately
address and speaking of incidents a
valuable tool for evaluating reporting
is using your incidents as a narrative
to describe how well the program is or
isn't working which process surfaced
each incident how did our response
process perform
did a technical capability catch this
threat and did we expect it to be caught
sooner and which tool was it what did we
have to do manually versus automatically
and what investments would make it
better and we can think about this from
our people people roles as well did we
have the skills we needed to respond and
with all those then we can present our
road map we we know what we can and
can't detect we know what type of
threats are impacting the business and
we know where we can make it better so
then you can create your road map and
make your asks and here's where micro
purple testing can really pay off it
gives you the data to support what
capabilities you need to reduce the risk
of threats to the business but it also
helps you
prioritize so the results of your
testing allows you to score where you're
effective and where you're not and then
taking the input from your incidents
your sock tickets and threat hunts to
prioritize base on risk where you lack
detections for threats impa impacting
the business today where you lack
observability to detect those threats
and where you lack the automation to
analyze and respond rapidly and
effectively and with your prioritized
backlog you can propose a road map and
make data driven requests to close those
gaps with the necessary funding head
count whatever you
need so here we are
instead of hiring based on the number of
alerts or investigations you can make
datadriven decisions based on your roles
and skill sets instead of threat hunting
because it sounds cool uh your process
view is a story that answers what your
program does how it provides value and
how you measure success instead of
buying because Gartner said so you have
a vision for your architecture with a
prioritized timeline to accomplish it
with clear ass to
leadership instead of telling your boss
I think we might be able to detect it uh
you have metrics that describe your
threat coverage and performance from
previous incidents and now hopefully
you've learned how to stop worrying and
build a detection and response program
thanks so much for having
me real quick here's my email and my
LinkedIn and Twitter handles are on my
website I write a very infrequent
newsletter called meard it has an
adorable cat that people love and the
security info is pretty okay uh so
please subscribe also please be sure to
give your feedback in the black hat
Events app it will be under my
evaluations later and then it looks like
we have time for maybe one or two
questions so I'll take questions now I
would ask to step up to the mic so I
don't have to repeat
it
run hello so one question you mentioned
it both in the metrics and in the
narratives that it is better to tell a
story better to tell explicit things
than numbers what if leadership actually
asks for numbers H so uh that's a tough
one because especially when you've been
presenting the same metrics for a very
long long time and you've been saying
this is how many events we have this is
how many events we've had this is how
many events we have um it's a little bit
of a rip the Band-Aid off and it's also
a little bit of push back to your
leadership I don't think if you keep
providing the number of events that your
sock is triaging it is actually
effectively telling the story that you
need to you do have numbers with this
approach you have a percentage of the
type of threats that you know you can
cover you know what type of threats are
seeing today and you can speak to those
numbers and you could provide numbers
behind what are the top threats that
you're seeing and instance so you have
numbers but refuse to provide the same
numbers okay or find a new
company all right we got time for one
more I said you had to go up to the mics
and then no one's goingon to do
it all right
thanks y'all I appreciate
[Applause]
it