[Music]
wow all right looks like we're live
so
thank you so much for joining this talk
this talk is on
the business resiliency case how to make
that business case i'm gonna give you a
couple case studies and how that works
and uh and then we'll go from there i am
going to attempt to attempt
to also be monitoring the chat channel
so if you guys have any questions on
around resiliency let me know if you
have any questions around potatoes or
bread check the channel i've already
answered those and we will uh i will be
taking a break every couple moments on
the section breaks to uh to catch up and
see what's in there
hi my name is wolfgang gorlick if you do
not know me uh i am an advisory cso with
dual security i focus predominantly
these days
on the front end of identity and access
management sso passwordless
multi-factor strong authentication easy
to use authentication
before this before this i did a lot of
work in business continuity i've given a
lot of talks on topic i've written a lot
about it i really
feel that it's uh it's an underlooked
avenue for building security programs
and that's what i'm going to hopefully
convince you of today
however when
when the pandemic set in when the
pandemic set in
the beginning of last year when all this
went down
i
went
and uh and i went and i talked to folks
who i had helped i talked to folks who i
had worked with uh their business
continuity plans i called our friends
like this would be great this would be a
great case study and how preparing can
help get you ready for this so i know
that uh we have done
work with uh with the cdc's zombie
preparedness kit i know we've done
uh work with uh table topping with your
executives tell me tell me my friend how
successful were you at shifting to
remote work how successful were you at
handling
what happened when the world shut down
and uh and the answer wasn't good
to to a person everyone's like oh yeah
we did we did do that didn't we
even organizations that we had followed
the cdc zombie preparedness uh plan
which if you are looking for a tabletop
or a good uh good way to mock out a
pandemic i recommend
um even those that we had done this with
said
you know i remember we did the plan but
when it happened we were too busy we
were we went into to response mode we
reacted we put places and things uh
where they needed to go and we did not
necessarily follow those plans we
made which was disappointing
but not necessarily surprising there's
stats out of gartner from 2020 that
showed that less than 14 percent or
about 1 in 10
organizations are really able to do good
bcdr
not a good thing to to be looking at
so in today's session i'm going to look
at three different business cases of how
organizations uh firmed up strengthened
their resiliency programs i'm going to
define resiliency in just a moment
and then i'm going to look at where
people go wrong
specifically after assessing nearly 200
organizations
what are some of the common themes that
people keep doing wrong again and again
other than the normal one which is oh
yeah we did have a plan that was cute i
remember that tabletop it was fun you
brought in donuts um other than that
where do people go wrong
so when i'm talking about resiliency for
the sake of today's conversation i'm
talking about technical resiliency
uh i'm not talking about
uh self-resiliency i'm not talking about
supply chain resiliency i'm not talking
about organization i'm talking about
technical resiliency which is a
combination of recovery and continuity
disaster recovery business continuity
a combination of that with instant
response when things go wrong we act and
a combination of that with traditional
uh it problem management so if it's if
it's a natural disaster it's an
adversary who's doing something against
us or if it's simply because our tech
doesn't work right that's not a denial
of service we paid for that if it's one
of those scenarios
it can be tackled and handled with uh
good strong cyber resiliency i hate the
term cyber resiliency but uh there you
go
i'm about to tell you stories
and the stories you are about to hear
are based
on real life events
uh just note that they're not
necessarily representative of any
particular organization environment uh
friend client someone who might be on
the stream someone might be underscored
someone who might know me on twitter if
you think this is about you i promise
it's not that we are not avoiding any
customer confidentiality uh but this is
summations of stories that are very
representative of uh the real world
today
so with that let's look at
the uh the first business case which is
financial services
let me take a quick sip of coffee
all right
in the beginning in the beginning of
this financial services company
um they had some things going on their
their business continuity plan
uh wasn't so much a plan as it was a
country culture
it was a nice culture to be to be sure
it was a very pretty tree uh and
everyone knew who to call
except for it was a little bit outdated
maybe a year and some people who are on
the call tree had already left but it
was a very pretty tree and it was very
well formatted
the
continuity program itself was run by an
operations project manager
um who was designated as the bcdr
manager we'll get back to why this
is a common theme the the delegation
problem and why i suspect
a lot of ransomware is actually
exploiting technical debt that accrues
by delegating to the wrong level that's
the story for another time of this talk
uh there were exercises they were
definitely doing security exercises
uh they would be fire drills
they were documented as tests now that
uh doesn't necessarily cover your tech
stack
um
but at least exercise the call tree
and uh and that was the state they're in
so a audit finding comes out as you guys
may know in financial services there's
been a lot
more interest as of late
in uh in business continuing disaster
recovery not only in
the organization that you're in but also
in the supply chain so do you have a
good handle on your
critical applications are they tiered do
you have a recovery strategy that goes
along with those tiers and and
for the apps that are outside of your
purview which is of course a growing
number
compared to where we're all at as we
move everything to sas for the apps
outside your purview have you also
ensured that their
business continuity disaster recovery
plan is up to date and meets your
requirements a lot of pressure there so
audit finding happens
and uh and the organization has to go
and address that
now from a business continuity plan the
audit
finding sorry from a business case
perspective the audit finding is
one of our our most powerful and yet
dullest weapons because if you can't
ignore an audit finding you have to
resolve an audit finding right if
internal audit says you're wrong um now
you've got a a voice that goes up to the
board and you've got uh you know the
ability to really push things forward
and get things done
awesome
the downside of that is compliance being
what it is uh organizations are heavily
incentivized to spend the least amount
of time and effort
to address that audit finding why
wouldn't you why would you right oh just
do anything extra there's
there's not necessarily any upside and
anyone who wants to tell me security has
an roi
uh
i will say let's have a great
conversation over some drinks because i
always wonder about that what is the roi
of an audit finding
in this case it wasn't necessarily high
so the strategic contents was trying to
address that audit finding and provide
some basic high level recovery tiering
here's what we're going to do to keep
systems up
from a financial perspective
there were certainly uh you know
penalties associated with not resolving
that as it moved to the external letters
however most of it was based on downtime
uh versus recovery investment so how
much are we going to lose if something
is is down that question
i think is really paramount for an
overall risk manager's program what do
we lose what do we care if this piece of
technology doesn't work ah
you ask someone that ask an application
owner that what does it matter to you
and they'll go uh it's it's important
okay put a dollar figure on that
all the dollars
it can be very difficult in traditional
risk management to really quantify
downtime uh it was easier in this
organization because they could look at
past outages factor in
what happens with penalties with market
moves and that so on and really do a
good model
so we need to nail down the financials
the stakeholders audit already mentioned
how the cfo uh cio and cso both reported
up through the cfo and this organization
so there's a lot of focus on getting it
done
and really the the management approach
was how we're going to build this how
we're going to manage the management
approach was
well we'll build a new continuity
recovery program and we'll move that up
out of project management put it under
the cso
and we'll use this as a way to drive
insights into our program
so when they execute it that was the
business case right
fix the auto findings justified based on
uh potential cost avoidance here's our
key stakeholders and what we're going to
do we're going to do some tiering and
we're going to rely on some i.t
infrastructure to do the recovery
when they went ahead and built that one
of the interesting things i liked about
this is they developed a flexible
strategy for multiple scenarios the cfo
was well known as saying oh plan for an
asteroid crashing into the data center
and totalling everything out what do we
do if the asteroid hits
great question
very unlikely also
what's surprising
is
if an asteroid hits and everything goes
out at once
you may actually be in a better spot
right if if the asteroid takes out uh
amazon east you may be in a better spot
than if it's a rolling outage
why because you don't have to worry
about data corruption you don't have to
worry about the integrity you have a
point in time that asteroid hit
and is done
and it's a much easier recovery
surprisingly so than if you've got
multiple outages that you're trying to
tackle or worse if you get adversaries
working through your environment so i'd
almost rather face an asteroid but
it gave in all those
potential impacts asteroids fire flame
pandemic um
you know
disconnected from the network loss of
sight loss application
build out a grid and grouped all those
into
a few simple scenarios that allow them
to go ahead and execute uh recovery
either people or key applications or
systems
prioritize the processes and tiered the
assets again based on money value
incredibly important we'll get back to
that when it comes to risk management
and other things
and then use this process to see so use
this process to have an ongoing
conversation with the organization
i want to dovetail into
uh
a study we recently did at cisco for
just a minute uh it's you can download
if you want i'm going to give you the
highlights as they apply to resiliency
uh but science institute worked with uh
with us talked to 48 000 people and
basically said they have all the things
you're doing and all the outcomes you
have what outcomes are getting and what
are you doing well
then did the math behind the scenes and
surfaced
um
areas where
people were doing these and seeing a
whole bunch of outcomes that they
reported right so the whole question of
does anything i'm doing in security
matter
this is a very hard thing to answer uh
we got a little bit closer on by doing
this exercise
so i'm gonna pull out some of the things
throughout this conversation that we
found we found that resilient
organizations are much more likely to
have the confidence of executive
leadership i think
in terms of implementing that resiliency
program i mentioned in terms of
implementing that program
i think that's because
the conversation from the cso
perspective elevated from i'm talking
about this piece of technology too i'm
talking about what marist organization
i'm talking about trading talking about
account reconciliation talking about
those things and when you frame security
uh in the lens of the business
security gets done good security is
within context of the organization
but that's very hard to do if we don't
know what all this stuff that we have
means so i suspect what's going on here
with the finding that resilient
organizations are better confidence
of the leadership is because they're
speaking their languages
they're also more likely to keep up with
the business
i also suspect that's because if you
understand the business process level
what you need to do for good uh business
impact analysis have you understand at
that level
it's much easier to get in and delve in
and help change processes or change
technologies because you've got that
understanding
there's a light bulb moment in this
particular case
where after we built the business case
built the program we started feeding
that program into different
um security capabilities
so for example we're using that bia to
feed into the scene we're using that bia
to feed into the vulnerability
management program so if i'm talking
about patching a server
to the cfo
i'm not necessarily talking about
we need to patch server we're talking at
the level of we're going to execute a
change to protect the trading system
at the same time if we're doing an
investigation in the scene we're not
necessarily looking at a million
different assets and trying to figure
out what matters we're prioritizing them
based on what they married to the
organization therefore have a much
clearer view into what investigations
and alerts actually matter
and that gets to another finding
when we did the correlation with 4 800
uh you know security professionals is
that organizations are more resilient
are also better at identifying and
managing top risks i suspect it's
because those risks are put in context
if you think about uh seem and you think
about investigating i remember just
chewing up so much time
uh in the early days we're just throwing
the logs at it trying to get out
an insight trying to figure out what i
need to act on when we get the context
of here's all this stuff here's why it
matters here's what we're going to do
about it significantly easier
so that's the first case study right the
outcomes of this were
implementing recovery implementing
continuity and using that to drive
downstream efforts
and pop over to discord real quick don't
see any questions so i am moving forward
all right the next next case study is a
sas company so they're building sas
applications
and uh and they're using
uh you know traditional sas services to
meet their customer needs
so continuity plan
what continuity plan
uh there really wasn't one that wasn't
uh wasn't a focus
uh of the of the organization they're
predominantly focused on
growing their apps growing their
customer base growing their annual
revenue right arr
and really heavily focused on
acquisition
they had shifted to a devops model
so devops was their delivery strategy
which meant for you know you probably
have one security person for every 10
ops people for every 100 developers
there's a lot of diffusion of security
knowledge about the fusion of how things
need to be built the cso was very
focused on that problem was very focused
on co-level security so it could tell
you about how we're doing sas scans or
desk scans uh what we're doing for our
waff
however
was not necessarily looking at
the feature level or the availability
level
and then there was an outage
it just so happened that the outage
occurred uh during a very uh visible uh
presentation where they're talking about
the benefits of the program and doing a
demo and everything and it's out they
can't can't get at it
so we've got to build the business case
here
for uh for this again i'm going to talk
about the strategic context the
financials our stakeholders
and some of the downstream efforts that
need to happen
so from a strategic context the the real
primary driver is going to be around
uptime right we know uptime is crucial
for sas growth you lose customers if
you're down you lose uh you lose
credibility if you're down you get a lot
of buzz off of twitter which is not good
buzz to have when you're down
so that's really the the overall context
the financials are going to be driven by
that sre function right that's site
reliable the engineering function
what does it cost for us to be down
in terms of delivering services
and and not only direct cost like in the
previous example with the financial
services if we're down for an hour we
lose this much money
but also future leading costs which is
we notice that if we're down for an hour
that delays customer sign ups over the
next three months
so a much more
uh
forward-looking model in terms of
justifying investments in
the uh the recovery continuity
the management approach had
high availability and resiliency under
devops that had governance under the cso
that was the approach they're going to
take originally you know
h.a wasn't really considered well we're
in the cloud it's not good
until it's not
so the management approach that they're
going to take with this business case
is to
double down an sre and aha under devops
and then establish and maintain a
continuity recovery governance model
with the cso
in doing this it's a it's a joint
business case right it's a joint
business case between the head of
product and the head of security uh the
effort is to think about this in terms
of like chaos engineering
or
other you know unplanned outages so not
only from an overall holistic let's keep
the product up but also from an
individual if this function fails let's
keep the product up if that server fails
let's keep the product up let's make
sure we have multiple availability zones
in aws those sort of things
as well as tying that incident pipeline
when we see something down and we need
to act on it into a broader context
when they executed on this plan
a couple things came up
first was uh and we see this in the data
with uh the security outcomes first was
by doing this in partnership with had a
product in the head of security we built
really good peer support the devops team
knew that they needed to to keep things
up and they knew why and they felt that
the security team was supporting them
and giving them good guidance not
necessarily just waiting to point out oh
you got to fly on your code fix that
much better peer support likewise the
security team knew that the product
people cared about availability and knew
that by caring about availability we can
back into
other security incidents and other
perspectives so they're able to create
much better peer-to-peer support
and i think
when we look at business
cases that are
predicated on compliance
as was the first example
very difficult to get well-funded
and the emphasis is always on do as
little as possible as quick as possible
to get this done as fast as possible
when we look at
business cases that satisfy business
need first and here this was exactly it
right keep us up keep our product going
we look at business cases that satisfy
business need first
and and by satisfying the business need
provide security as a secondary outcome
those business cases tend to see
significantly higher funding
greater peer support and having a longer
tail so that once you've got it up and
running you don't forget that you got a
plan and everything else
another thing that's interesting about
this was um
diffusing resiliency into
the individual container level or micro
service level meant that there was much
lower unplanned work right
much less wasted effort why because when
something happened the system could self
heal if they notice that the way they
set this up is if they notice that
um
they're getting garbage on one of their
apis for example you know someone
sending in um us dollars and it was
supposed to be yen or sending in a
string when it was supposed to be a
number or attempting a sql injection
flaw which never happens
usually uh when that happened that was
seen as a a
potential outage and the host that was
sending that information was identified
wiped and reloaded using an ansible
playbook so it made the environment the
entire environment much more resilient
when anything went wrong we reset to
normal and then went from there
now interesting light bulb moment for me
on this one was
the fact that
um
in a way
high availability leads to high security
you know there's uh so no use dye
concepts right if you haven't looked at
diets well worth looking at distributed
immutable ephemeral if we can erase and
rebuild an environment faster than an
adversary can break in and stage tools
then we win
because they can't get a foothold and
that was exactly what this organization
found they had a situation where
uh they're running kubernetes
and uh kubernetes had that zero day
exploit
and uh there was a mistake in the
networking
um so that kubernetes controller was
exposed the internet adversaries were
exploiting that vulnerability right away
right moore's law um hd more not the
computing
everyone had access to exploit everyone
was scanning and popping boxes like it
was christmas
and uh and they would get popped the
adversaries would stage tools the
adversaries would start to move
laterally that was identified as a
potential problem with that kubernetes
node the node was erased and reloaded
from ansible playbook the adversaries
were kicked out
this happened a few times until they
engage the the security instance
management process
so if we can erase and rebuild faster
than adversary can break in and stage we
win
awesome property uh in terms of
ephemeral environments
it's probably why two resilient
organizations are
more likely to have streamlined instant
response processes i believe that's
because we've thought through some of
the ways things can break and have put
in place playbooks in automation and
hopefully wherever possible are able to
just
execute these things and go
so that is the outcome of uh the sas
business case they were able to
integrate recovering continuity and ir
they're able to do it in partnership
with the head of product and the head of
security we've had a product maintaining
that as
a set of tools using devops and chaos
engineering principles
and the security team maintain the
governance for that
using traditional security concepts
popping over real quick to look in the
chat okay
so
let's look at the next business case
here
uh this one is in manufacturing so there
we were there we were right
they had a they had a business
continuity plan that's that's a start
it wasn't necessarily aligned with iso
which is the normal standards body for
manufacturing but they they did have a
plan
uh the the plan
put a
emphasis on availability for
um manufacturing lines manufacturing
plants so much like the financial
services where they put the emphasis on
fire drills uh and tested those very
rigorously here in the manufacturing
world they had the emphasis on
uh the plants and tested that very
rigorously plant level recovery was
great it worked fantastic everything was
good
however that didn't necessarily take
into account
your knowledge workers and everything
else and
due to acquisitions and due to the lack
of a central uh asset inventory
they had a million and one applications
lots and lots of applications
they also had a roadmap for network
refresh so their equipment cycle was
coming up
and they had a customer demand customer
demand to
meet a particular requirement for
continuity and recovery
so in this organization again thinking
about the context the financials of
stakeholders
the business case
was built around addressing that
customer need achieve tsax compliance to
be specific so they needed to address
that customer need i'm very pro
of the customer use case
because if we're protecting customer
revenue
and expanding customer
markets and everything by
by enabling the business through and
implementing security controls
what a great place to be in there's much
greater uh return i think you could
actually argue that is one of the few
areas where security can't have an roi
and there's much better budget
and greater support across the
organization for it so i really like
that context for a business case
financials are based on that customer
revenue we know we need to satisfy this
customer need or we're going to be in a
tough spot
the stakeholders are the heads of sales
uh the cio and the cso and the
management approach for this was
building a continuity recovery program
under the cso so basically upskilling
the current program which was facilities
based
to include
the it assets
and
in doing so do a full multi-year
business impact analysis where we look
at all those apps we look at all those
resources we figure out what's what
then the
implementation of that technical
implementation was going to be under the
cio as part of that network refresh
budget so when the refresh happened
taking advantage of that to shore up
the the network show up the
infrastructure and make sure that good
recovery and resiliency at that level
in doing this they found a lot of
different cost savings not only in the
refresh but also coming out of the
impact analysis
and they also position themselves very
well to respond to future events
when they executed it they
standardized the plan so they went back
to fundamentals they went back to the
the iso standard and they standardized
the plan and made sure that it was lined
up with the controls that were required
by t-stacks again that was the driver
right get t-sec certified keep the
customers
happy in doing the business impact
analysis which was a multi-year analysis
uh looking at all the apps figuring out
what those million what abs do
they noticed and identified a number of
different redundancies so
opportunities to cut this up or cut that
app to get rid of this or that or the
other to to simplify the environment
uh and to standardize the apps that the
various plants and business operation
units were using
so great opportunity to find a number of
different areas to cut and consolidate
and they were able to use that those
savings to then reinvest in the security
program
so
right i was mentioning business case
should solve a business need first and
then satisfy that need solve a secondary
outcome so in this example the secondary
outcome was
identifying those cost savings and
really rationalizing their application
inventory which is way out of control
which gets to uh another finding that we
see with organizations with good
resiliency they're much more likely to
have a cost-effective program
another light bulb moment
for me
occurred
when a
manufacturing line after implementing
this program
shoring up the bcdr
refreshing the network stack beginning
to consolidate apps um they actually
faced
a a malware situation uh where the
malware was spreading from plant to
plant to plant
and it was those lines those lines that
were modernized and put in place for
that customer need
which were unaffected they're segmented
off they would see the traffic coming in
they would recognize that it was
unwanted on uh unusual abnormal traffic
and they would communicate back to the
network uh environment you know you know
react access control and say hey isolate
this right we don't want this traffic
coming into us
and uh it kept the line up certainly but
also what was fascinating about that is
not only was the those areas unaffected
by this fast spreading malware
but it also became a detective tool
in cleaning it up because we could see
when it hit this these uh these lines
what was going on and what needed to
happen
so it's another example of how major
security instance can be
uh mitigated or minimized if you've got
good resiliency in place and a good
example of how
the security aspects of things detecting
when traffic is going awry
detecting when applications are going up
having a good application inventory how
those can all flow into a problem
management state so that you can be able
to respond very quickly have the i.t
team know what they need to do and feed
those into the
front line when we're actually tackling
a problem
all right so those are my three use
cases
ah
and uh i'm going to get into where
people go wrong in just a minute
as i pour myself a little more coffee
uh i see a a comment in the chat the
third case is actually similar to where
i am right now in healthcare customers
demanding a new certification now
customer demands
as a way of driving security is really
one of my favorite ways of
positioning things so because you get a
lot of support peer support from outside
just technology and of course the when
you're when you're talking dollars with
respect to customers
that was uh that was a major driver uh
for for revenue for funding for
everything else
support the culture everything else
question what was the certification the
plant was going for in a third example t
sacks t i s a x t sacks t stacks is an
exchange certification
uh that is maintained so if you are
t-sec certified
other
manufacturers can check that out and
rely on that as a certification
um so it's a it's a good way to reduce
your
uh number of audits from customers
and uh it is based on the iso standard
the the certification body is out of
europe
all right good questions okay moving
forward
as i get back to my slides
uh where do people go wrong
a couple years ago
at circle city con i gave a talk and the
clouds break at that point time i was
doing uh doing a lot of conversation a
lot of work
around
cloud continuity and recovery because
there's was at the time and i i i still
feel there is a little bit today
that uh there was this thought that if
you're on the cloud you don't need to
worry about continuity and recovery it's
fine everything is good someone else
will take care of it
uh i recently did a
a podcast
with w curtis preston and his book came
out
in may so if you are in a sas position
or you're looking to do
uh good recovery work
in that space i'd highly recommend his
book i've been reading uh reading
preston for years all his work is always
very good very thoughtful very well
constructed so his book is just out
and uh and thankfully with his book and
others there's been a lot more attention
on cloud data recovery
as part of that work
and moving forward this was in my
previous role uh before joining duo it's
part of that work in moving forward uh
during my consulting days and uh time
sense that in the clouds break talk
my team had the opportunity to review
176
business continuity plans so we read we
read 176 continuity plans
so you don't have to
when i say that way kind of sounds like
you know those cheesy bots like we made
a bot read a thousand different words
and now we have it creating a harry
potter script
it's it's like that and as a matter of
fact he can't read that business
continuity plan along the way
here are some of the ways that
these programs oftentimes go wrong
the the continuity and resiliency
efforts are owned by the wrong person
and
i gave you the example of the financial
services whereas owned by
you know project manager uh i gave the
example of the manufacturer whereas
owned by plan operations and i want to
stress both of those people are doing a
great job at their level awesome
fantastic
however
it's very common that we pass down
uh these programs to junior folks we
delegate too low and then we get
uh
you know concerns it's not
being done right
when i was talking to preston uh you
know again the gentleman who wrote the
book w curtis preston and his book in
cloud recovery i was on his podcast and
he was complaining that uh backups are
usually delegated to the most junior i.t
administrator and i was pointing out
that
continuity recovery plans are usually
delegated to the most junior security
analyst
and what we do is we rely on these
things and we need to recover and resume
when things happen it's it's just not
it's not there it's not for lack of
trying it's usually lack of them being
equipped
and having enough authority to execute
i suspect i suspect that ransomware
the the the spate of ransomware we've
seen for five six years now is ongoing
because we've got decades of technical
debt that's accrued from delegating down
too low so number one failure the wrong
person
um
number two is is recovery at the wrong
level you know if you got front end load
balancers that's not going to protect
your backend database uh if you've got
great recovery
uh for
you know your your amazon sites but
you've got a central source you know an
oracle database that you're plugging
into you're not gonna be protected that
way so there's there's wrong technology
load balancing
and and site failovers and everything
that are not necessarily applied across
the the goal and both of these the wrong
person the wrong technology are
oftentimes not identified because we're
doing the wrong testing we've got the
wrong goals for testing we've got people
who are doing these tests
uh that don't necessarily apply right
the fire drills which have nothing to do
with the data center
there's even more ways that we've seen
this mucked up outsourcing is a big one
it's not a cloud provider
cloud provider handles it all they don't
make sure they're testing your stuff
because they don't uh non-standard
language uh not using the right terms
right that the
um
rpo and rto being confused are really
good signs that this is a a plan that
was not written the right way
uh non-standard approach i'm not saying
we need to foul nist or iso 22 301.
i'm not saying that but i am saying that
if you're a nist shop or iso shop you're
in a much better place if you follow
these standards and make sure you're
addressing these controls
there is a time and place to build a
custom program
but the time and place is oftentimes not
in the resiliency where you need
something very standard very portable
that can be handed from team to team or
organization organization
uh trend addresses high availability
it's cool we got h.a
all right
have you tested it is everything in h.a
what happens at the failover point what
about data corruption uh and that
back to that asteroid falling that was
one of the the most painful experiences
of my life was when i was handling a
uh disaster recovery scenario that
really was the asteroid you know if
everything's down we're good but it had
they had had a series of rolling
failovers so you had corrupt
transactions that had to be rolled off
you had file mismatches you had a file
system that wouldn't mount oh this is
terrible
not going to be protected with h.a
overly broad plans
yeah
it's in the cloud provider we got it
don't look at what's in the data center
only specific plans
we know exactly what we will do for
covid the next time it hits
okay but what about any other
pandemic or any other widespread uh
impact to your workforce we will get ppe
and we'll give people their coveted
vaccines
it's
black swan events are black swan events
for a reason they almost never occur so
building a plan for a black swan event
is a terrible idea yet everyone is doing
it right now why because we're still
emerging from covet
much better to say let's build a plan
for something that's swan shaped like an
unnamed goose if we got an unnamed goose
plan
that can address a whole bunch of bird
shaped things we're in a good spot
that's what you want to aim for but all
too often it's overly specific
handled in isolation so there's
different asset management different
risk rankings not tied in with the
overall programs
so these are the main ways that these
things get mucked up
comment is it's usually called the
backup team not the backup and recovery
team
too true
recover test the recovery why why should
we
uh is the problem with the cloud that
people never test their backups or that
they configure their availability
incorrectly or something else great
question the problem with the cloud
is
a couple things one organizations often
don't think to back up it's in the cloud
it's running why should i have to worry
about a backup
um
well because things go wrong well but
we've got cloud multiple cloud instances
okay but what happens when
multiples go wrong or
when you get uh corrupted data such as
ransomware
so in part it's because they don't do
backups if they do backups another
aspect of the problem is that a lot of
cloud software being open source doesn't
necessarily do full encrypted backups
it's working with a company recently in
the advisory capacity and we're talking
about backups like oh yeah i think we've
got that configured but we can't do a a
incremental backup and we can't encrypt
our backups without buying the
commercial software version
yeah they must never have had an ec2
instant stop or fail or lose their
ephemeral source correct you don't
really think about these things until
something terrible happens
all right let's uh let's talk about
moving from recovery to resiliency
one of the key things
that recovery allows you to do is know
what matters and know what to do about
it that is the most important part
about this doing the impact analysis
mapping out your processes mapping out
your assets
from there we can go ahead and plug in
resiliency into a couple of different
places
i love to see plugging in the soccer the
noc
so this asset means this is supporting
this business process it's our key
business process and health care it's
you know protecting a patient's uh
health and safety in financial services
it's uh what's allowing us to to trade
that's our key process we want to
prioritize in the sock or the knock
those processes accordingly
uh same thing feed that into the
the cert
uh same thing feed them to the grc and
have those back flowing too right so
when your grc program is updating risk
rankings or making recommendations
across all three areas we've got a
similar understanding of what this stuff
means and what we need to do about it
a surprising finding when we were
looking at 4800 security professionals
what they do and what they get
surprising finding was that folks who
are doing resiliency
were more likely to retain top talent
you wouldn't think that would be true
right because backups and recovery are
boring and
documentation is boring and everything
that i talk about is usually boring uh
so why is that why is it
when you look at people who worked in ir
or in stock
and have burned tons of time
trying to solve a problem because they
noticed unusual traffic
only to realize that that was a new
application or they noticed some very
specific uh suspicious activity only to
realize that was on a guest network that
no one cares about
how
many times can that happen before you're
like
this place is a garbage fire i'm gonna
go get another job
if you if you do work in security you
want your work to count in the matter so
you need a way of making sure that
happens
uh some other additional integration
points i already mentioned seam and
vulnerability management i really like
to feed these things into your pam and
pam programs so
this high secure environment
is only accessible through just-in-time
access
data loss prevention certainly with
classification ueba certainly so we know
the behavior is right now
the trick to this is to look at all
these and look at what's mature and
what's available to you and make that
integration point
the
caution i would give is don't say i
can't do bcdr until i've got good ueba
or i can't do
pim and pam until i get good dr we want
to mature what we can but for the
portions the capabilities are running
well and really mature
link those up together to provide an
overall unified view
and also remember not to separate
resiliency from the organization
i'm talking predominantly today about
resiliency as an i.t function and
therefore a part of the domain of cyber
security of course resiliency affects
more than i.t
we want to start with the business case
for continuity and recovery and response
start there
be very careful not to silo it off from
other broader initiatives
from there exercise exercise exercise
not just fire drills
i've got an exercise maturity model i've
shared before i'll cover it again here
briefly
but you want to start off with here's
our story something's down whatever and
have those stories on ad hoc basis maybe
we're team meetings
open office hours
level two you want to firm up those
scenarios the stories and the scenarios
and do a formal tabletop with the right
people and not only would just do the
technical one but also business facing
and with your peers so that you can make
sure that you've got good understanding
of what needs to happen
now that you're doing that on a regular
basis maybe that's a quarterly maybe
that's monthly
um take some of your best scenarios
tie them to very specific tactics and
controls and test those tactics and
controls technically
from that we want to line it up to an
overall framework and run those
exercises in parallel so you not only do
the tabletop but introduce evidence turn
things off flipped over tables whatever
it takes
to make sure that things are working
and then as we go to level five
automate that automate the heck out of
it right think of the chaos monkey and
the work that's being done
and chaos security engineering where
this is an ongoing process that's
automated and regular and allows us to
do continuous improvement
from there we want to measure our
performance
from business impact analysis to
increasing the number of our scenarios
to increasing the number of tactics that
we've got
traditional metrics like rto how long it
takes to recover rpl and point your
recovery 2
and restoration after recovery
as well as ir time to detection
remediation so measure your progress and
make sure that you're driving that up
the measurement really comes at about
level three it's it's not where you
start to where you're going
all right i'll conclude this and then
i'll answer any final questions in the
discord and then uh then we're done
the next time uh
the next time you're in the building
you're sending an elevator
i'd encourage you to look down
see which elevator it is
uh it's probably an otis it might not be
a notice
but it probably isn't otis
the story of the elevator is actually a
security story
otis
comes from the man otis the gentleman
who's the namesake of those elevators
and he didn't actually invent the
elevator no he didn't he invented the
break
and he would
uh go out in the 19th century his name
was elisha elisha otis who got the 19th
century and he would have people with
ropes
pulling them up
two three four stories of the air on a
platform
and have a big crowd around him there
was so much to do in 19th century we
didn't have twitter
so he would hug crowd around to see this
guy who was about to break his neck and
then they would cut the rope and he
would fall on this makeshift elevator
fall
sometimes as much as a story
until the break the automatic break
kicked in
otis elevator
named for the elisha otis who never did
die from these stunts surprisingly
enough um otis elevator company patented
the break in 1913
and it was about 20 years after that
that we started seeing some crazy tall
buildings uh including the empire state
building and everything else
the security control the break
was what allowed us to reach for the sky
so
that enabled the last great migration
where we moved from country to city we
were moved from the first floor to the
top floor a lot of interesting stories
in there but it was the last great
migration till this migration the
migration we're in now we removed
everything
not from one place to another
but from physical to virtual
and as a matter of fact if you look at
where most people today are building uh
their companies uh building their lives
it's online right it's all online
why bring that up is
we're in the midst of the greatest
migration in our lifetime the first
migration that's been largely physical
to virtual
and that puts incredible importance on
resiliency
resiliency is that elevator break
resiliency the resiliency we're building
today
that resiliency is what's going to make
possible tomorrow's wonders
it's going to be really fascinating to
see what happens 10 years from now
so moving forward you know
look at ways to to build the business
case i've got a couple examples today
um make sure you gap assess your
programs and figure out how to integrate
them continuity recovery ir problem
management
and determine what areas need uh
investment
close those gaps run exercises use that
to drive up your your program
and uh and and do it and don't forget to
do it
because the zombies are coming and if we
don't run our zombie preparedness plan
what the heck will we do
all right let me look one more time for
anything left in discord
there's a comment that a lot of the
problem with cloud is the issue the
recovery process itself
says well i do love terraform i found
many devs will often not take backups
into account exactly they don't they're
trying to build stuff right there their
whole idea is not let's put brakes in
the elevator the whole idea is build the
tallest building that's why we need to
be thinking about the brakes
and most people don't understand what
offline backups mean yes yes so true so
when you think about some of the more
advanced ransomware tactics what do they
do they compromise your backups first
there was a very
interesting one that was going out
and uh tackling msp so compromising msp
push out a command to delete all the
backups push out a command to disable av
on the endpoints and push out ransomware
to all the endpoints
ah so yeah most people don't really have
an understanding of backups and backups
disaster recovery instant response these
are not necessarily things that people
can just read up on the internet and
jump in and do it really does require
a degree of expertise and it requires a
degree of experience to get it right
all right i know i'm at time so thank
you all very much for
joining today i hope you've had a good
time this session and i hope you have
fun with the rest of conference
[Music]
cheers buddy