hello St Khan
who is tired
I could take a nap
yeah
okay
so
in case you're in the wrong place this
talk is about 10 things or like 23
things that you might not know
um
and uh if this is not what you're
expecting you're welcome to leave if it
is then just sit back and close your
eyes
I'll try to talk slowly and deeply
and see if I can sedate you down into a
nice
dreamless sleep
all right I am Sean Jackson I
drop my mic
I am
the director of trust at spiff
Incorporated
uh trust is the marriage of security and
privacy I'm the acting CSO and the
acting CPO and so I said call it trust
on the interwebs on the various social
platforms I'm tarkus it's my handle
um
if you ever see me playing contests I'm
going to be tarkus I'm a husband a
father a fur Dad I am neurodivergent I
am on the Spectrum and I have ADHD I
support minorities in Tech I love lgbtq
people I feel around this community
specifically Utah County and Utah they
don't get a lot of love so I try to give
them love
and all my hugs are free but you have to
ask for them I do not just give hugs out
you have to ask me for a hug and I'll
give you a hug
foreign
so what are we going to talk about today
this is the this is the list of things
we're going to talk about
uh we'll get there
who's this talk for if you do not have a
SIM
this is this is the stuff you need to
start watching
and this is how you build a Sim
uh you may have heard of organizations
where you've heard someone oh back back
in the day they programmed our Sim well
what we're going to talk about today is
the steps to build a Sim
um
if you have a SIM
this is going to be a list of things you
want to make sure your sim is alerting
on and this is where you get started in
threat hunting
I got to tell you last year uh lee uh
grifter gave a talk on threat hunting
and I swear it was a Sans course inside
of one hour
go look on the on YouTubes and find
grifter's talk on threat hunting from
last year and you will not get more
content better
better disseminated than than what
grifter did he is brilliant
but essentially
if you need to see all the things and
they are all over the place this is the
talk
what do I need to see all the things you
need a log aggregate server
this is a server that's going to be able
to catch all the logs from everywhere
you want a lot of storage because logs
take up space those are text files they
pick up a lot of space you don't need a
lot of Ram or CPU because you're not
doing huge computations you're just
collecting stuff and then you're
processing it or you're reading through
it
now graduate level bonus points if you
can have an automated system that will
copy all the logs from that server to
another server let's say for six months
your customers will want to know what's
happening with your log backups so
automate your log backups
so everything we're going to talk about
is going to forward all its logs to This
Server
So This Server is going to be just a a
behemoth of text files of logs of all
different kinds
so you need to get your data ninja
skills up
what am I talking about data ninja
skills
cat Cut Ock said and grep
uh you need to get really good at these
because your log files are going to be
big long strings that have lots of
different kinds of data time stamps IP
addresses ports
executions
programs
all kinds of data and to get value out
of that you have to be able to say out
of all of this kind of log
show me all of this section or show me
anything that doesn't fit in these
parameters from those logs over this
time period
so you're going to be gripping and
cutting and uh
a lot less a lot of cat
these are your your data ninja skills
you're going to use regex to fine-tune
stuff and regex is it's a very like a
friend who mumbles you don't want to
hang out with him but he's still a
friend
uh you want python that's what I
recommend for scripting uh
log
parsing
a python is a very low entrance bar into
scripting you want to automate this you
want things to just work on their own
and you want to establish as soon as you
can and this is a Perpetual thing you're
doing you're establishing what your
baselines are what is normal and any
tool you have out there that is a
next-gen firewall or a user Behavior
Analytics tool or even crowdstrike when
you install crowdstrike on a new host it
will run that host through some some
work it puts it under strain puts under
stress and it wants to see what the
Baseline is if it's behaving at a high
usage what does it do
if it is running
30 programs within half a second what
does it do
it's getting a baseline so that if
something happens that's outside that
Baseline it knows what is wrong so when
you're collecting
oodles and oodles of logs you need to
establish what the baselines are what is
normal for your environment
the only way to do that is to have huge
sample sets so you start collecting the
logs and then you start looking at it
and say what's the high what's the low
what's the mean what's the average
first things probably the most important
things you want to be
watching closely
identity and access events
access levels and identities of those
who are logging into things
servers firewalls workstations who is
logging in
that's the first step if you want to see
if someone is doing something wrong
if it's an employee who is logging in
from
Sandy Utah and Pakistan
that's not normal
you want to know how frequently they log
in
how often does the admin log in between
the hours of 8 and 10 in the morning log
out between four and six and then log in
again at between 8 and 10 at night and
log out between 11 and 12.
I'm just throwing out numbers but if if
your admin logs in all the time
on those periods you want to know when
he logs in at three in the morning
location of logins if your systems admin
is always logging in from Sandy like I
said and then suddenly she logs in from
Pakistan
send her a slack and say hey are you in
Pakistan
and she says no and you say well
it looks like you are let's let's look
at your passwords
and then odd times the logins like I
said if if you're if someone's logging
in all the time
between 8 and 10 and four and five and
it's that's it and then they do three in
the morning
if they do three in the morning two days
in a row something is wrong
and then
everything I've said so far you want to
know this for all your employees
you want to know everyone because
everyone is a point of of failure
potentially
but your admins if they have the keys to
the kingdom you really want to pay a lot
of attention to them
so everything about the frequency logins
location of logins and odd times of
logins pay extra close attention to that
for all your admins if they have
administrative access to anything
watch them closely if that feels like a
whole lot make sure you have as few
admins as possible
don't have a lot of admins no one should
have a lot of admin access
so how could you find the baselines for
these
I kind of been saying it go ahead
a lot of log set AC and Cat yes
huge data sets and then find the
averages
endpoint alerts so this is going to be
your edrs and your Epps and your xdrs
they generate logs when you have uh you
have crowdstrike Sitting on a
an endpoint yes there's a dashboard for
crowdstrike but we're talking about
aggregating everything together to make
a Sim you can forward all those endpoint
alerts to your aggregate server
you'll have to spend time looking what
the log looks like and how to parse data
out of it
and then uh if you're talking about sore
uh one of the things that I look for in
soar is the ability when crowdstrike
says I found something I say show me all
the IPS that machine has seen
so you'd have to be able to figure out
this machine what IPS has it gone to
we'll talk about netstat later on
um and what are the who is is for those
IP addresses
and what IP addresses have reached to it
and what are the who is is for those
now like I said you have dashboards for
these tools but you want to collect
everything in one place so you're not
hunting and packing all over the place
and you also might be able to get the
the tool to ping you
so I know crowdstrike can send it can
work with things that will send text
messages or slack messages and so these
tools can ping you and say hey I saw
something then you go back and check
your logs
you want to look at abnormal Ingress and
egress ports
so what are some of normal ports
and I'll give you some there's SSH port
22. nmap 445 html80 SSL 443 smtp25 DNS
53
have you heard of anyone exfiltrating
data over DNS
a couple hands
that's hard to find it's hard to to pick
up on which is why they do it
um
the way the way I've seen it they will
append little bits of data and reach out
for an IP address and do that over and
over and keep just truncating their data
onto these requests and if you control
the DNS server you can then take all
those logs put together and you have the
the data
um
but if something is talking
on if if it looks like you have DNS
going over
80 80. that's not normal
so where can you get this information
where can you find out what ports are
being talked over
anyone
that's that
get your net stat logs from a host and
you can see what is talking to it and
what it is talking to
this is what netstat looks like
it's not super pretty it's not super
forthcoming but in the middle let's see
in the middle we have some addresses I
have 24 143 204 224 Port 80 and 192 168
1 106 on Port 2848 that's right there in
the middle
you can see the 192 108.1106 the first
line
in the center
after the colon is the port
so that's how you can find the port
deliminator colon
port
so how could you search for uncommon
ports
how can you find the exceptions to the
normal
you would do a cat Cut deliminator colon
write to a file for each in for each
line in that file
if it's not in first thousand ports you
can search for common ports if it's not
in this then write it to another file go
back and find the outliers search for
those see what the connection was that
makes sense
this is something you want to script if
you see anything so you want to run all
your net stat logs if there's anything's
not in the first thousand common ports
make that a new file now go back and
search for those entries find the IP
addresses that were going to those
different ports and start searching
could you do an IP lookup to find where
the connection is
could you search for who is to find
information if there is any
and could you automate that whole
process with a script
does anyone think they could automate
that with a script
you can
you can automate that whole process of
look through all the Nets that logs for
the last 24 hours
running against the common ports show me
the outliers go back identify the rows
that have those outliers and run a who
is on all those i p addresses show me
what you find
you could do that
you want to know about abnormal log
volume
again you need a baseline of what that
is
you want to say normal is this type of
log and this number of logs in this time
identify the peak identify the low and
identify the mean the average
speaking of mean what might it mean if
there's a spike
if you have
let's say your net stat logs
what happens if you suddenly one machine
is generating five times the net stat
traffic
you might want to know that because it's
different for some reason the machine is
now doing five times more traffic than
it ever has before
five times the connections what's that
about what could be going on on that
machine
what does it mean if there's a flat line
what if a log stops
or whatever log
doesn't produce it normally produces
2000 logs in an hour and now it produces
five
you might have a server down you might
have something not working that should
be working
file Integrity monitoring
how do file Integrity tools work does
anyone know
how how do tools know if a file has been
changed
so it's a metadata hashes they hash them
you know what a hash is or anyone not
know what a hash is
okay so it's a mathematical equation
where you can uh
input a file into this equation and it
will come out with a string
and if you don't change it it will
always come out with that very same
string you add a space or a period and
the string will change
so if you're going to lock down you say
these are our these are the corporate
secret data this is customer data this
these things should never change
so you want to do some file Integrity
monitoring you could script that
I'm repeating myself I hate that
um so you want to identify what and
where the files are you want to make
sure it never changed
and you can script that you could say
go to this folder run md5 against
everything in here
go do it again
did anything change
do it again next week did anything
change
do it every day and if anything changes
then let me know
if it doesn't change
business as usual
but you can't know if it's business as
usual until you have your file Integrity
script watching everything it needs to
watch
once you know it's watching
let it run
and if if someone comes and sends you an
email hey something changed
get into it
vulnerability scan findings now there's
no rocket surgery here this is a
your vulnerability scanners
I use tenable they don't pay me but it's
my favorite although I feel like it's
getting behind the times nowadays
um
but you can have your reports that send
out XML
and then you just parse through the XML
you won't have alerts on criticals highs
and mediums and lows disregard
informational
you want to know VPN logins more access
and identity
again who is logging in
when are they logging in
where are they logging in from
do they get the password right the first
time
did they get it right the 300th time
you don't want someone using VPN and
trying the password 300 times and never
contacting it
that's not a good employee that's
someone who is an imposter
hahaha
physical access notifications
now this is an iffy this is something
you might not have you might not have
access to but your doors
uh your your scanners
you want to know did a deactivated card
try to get in
you don't need to know when everyone
logs in and when they don't or when they
badge in unless you have a system like
snow was talking about in the keynote a
badge where you have to uh badge out
before you leave
I worked at one organization that had
this you couldn't leave like the doors
wouldn't open if it wasn't an emergency
without badging out
kind of cool it was an insurance company
and they were old and they had good
security
um you want to know if a door is propped
open
if someone is moving something in if
they're loading up the break room great
when does that happen
so you could say if this door is propped
open at this time by this badge so the
bet this badge is used the door is
propped open it's open for half an hour
you know it's the the main or the office
crew stalking the mail room they're
coming back from Costco
if that's not the case if it's another
badge let me know
and then you go talk to your your
operations team and say no one can open
this no one prop this door open except
this badge this one badge I won't alert
on this badge you you use our badge and
I'm going to come talk to you
and then if you can
has an entering card not exited yet if I
were to do this in my organization I'd
have to pay to have badge readers put on
the inside of doors and that's something
that we're not going to do right now so
I don't have that possibility
if I ever do I will say you need to
badge in and badge out and then you can
get a day report at the end of the day
who didn't badge out
if there's someone who consistently does
that have a talk with them if someone
just does that once in a while go see if
they've fallen asleep under their desk
and for the socially comfortable
do the cards scanned match the people in
the office
did someone badge in and they're not in
the office
and there's a visitor in the break room
that you don't know
you'd want to know if that person has
the wrong badge or has an employee's
badge
now of all these things which of these
could use script checks for
if you have this data is there anything
you could not script a check for
all the stuff we're talking about going
to this aggregate server you could
script all of it
you can script all of it to find the
outliers and that's your sim
or think about Rita
this is real intelligence threat
analytics has anyone have heard of Rita
this is one of my favorite tools for
open source stuff this is so powerful
this was built by John Strand and black
hills information security
he named her for his mother
Rita strand she died in 2016.
this identifies C2 traffic now Rita
you're going to have to get a uh a feed
off of your switch or your firewall or
whatever to grab all the data
all the Nets that data
all the traffic
and it's going to make Yara logs
and then it is brilliant at finding the
C2 traffic C2 traffic is when a host is
compromised it's going to Ping the
Mothership
do you have instructions for me
do you have instructions for me they'll
do a 35 kilobyte ping
let's say every three hours
so it never
trips anything it's just a quiet little
Pip
regularly scheduled Rita is really good
at seeing is there a commonality between
these little pings machines are doing
all kinds of traffic
but there are patterns and Rita finds
the patterns so Rita will find if
there's a machine that could be talking
to a command and control server
it's brilliant it's it's awesome and
this is open source it's free
get it
so
I should not have said this was a
one-hour talk we're coming about a half
hour
um
there's somewhere between 13 and 24
things depending on how you cut them up
and if you look at everything is doubled
for admins and regular users
your mileage may vary you may not have
as many of the logs as I'm talking about
you might have more you might have more
tools that can give you more insight
into functions of behaviors and and give
more things to analyze
and what are we really doing here we're
building a poor manual clunky version of
a Sim
you are aggregating all the things that
the computers know that you don't into
one place and you're using scripting to
parse out intelligent data out of those
logs
if you have a SIM
these are the things you want that Sim
paying attention to
tune your rules to these events
if you're threat hunting this is where
you'll find the the footprints in the
paint of the attackers you know I'm
talking about
back in the day when the power would go
out in the 7-Eleven they would throw a
can of paint on the floor
because they couldn't like lock the door
and they couldn't leave but if they were
somewhere else someone could run in grab
something they could see their
footprints in the paint
the logs are the footprints in the paint
uh something I didn't say I should have
you want to make sure that the logs uh
when they're being written to are not
editable
you want to make sure that no one can
write to it no human can write to it
only the various systems they're talking
to the aggregate server can can write to
it no one else can
so you there's no you there's
non-repudiation you can't you can't say
that someone modified it someone changed
it
um
and so if you do tune your your sim to
look at these things you're doing you're
getting a pretty good job at threat
hunting you're getting started again go
watch grifter's talk last year I think
from this Ballroom
he was brilliant it was like a Sans
course
so the next steps once you have all this
and this could be a year-long process to
build this system
once you do now you need to have some
pretty reporting so you can show people
the data that you're learning
and if you have actual information or if
you have
events you want to say here's the events
we found our homemade Sim found these
events and we did this and this and this
and this or our homemade Sim found we
have an incident and we did this and
this and this and this and you want to
have
Executives like to have pretty graphs
they like to have pie charts they like
to have
pretty slides and grep and awk don't
make pretty slides so talk to your
marketing department
figure out how you can take that data
make it in pretty reporting for your
your boards for
at the end
so hit me with your questions
and I'll repeat them for the video go
ahead
people need a card to get out
yeah train them you say this how we're
going to do it you uh you can get your
phone make a little video say I'm going
to badge in
and now on the system I know that I'm in
the building and if I just leave
I didn't do I didn't do that and I could
log back in or you could say your system
won't let you log it won't let you
imagine again
you have to badge out
to to bug the security guy
I'd say you got to bring a Tootsie Pop
and then I'll I'll reset your your badge
oh you need to have how do you interface
with people who have safety concerns in
an emergency situation uh all your doors
have to fail open
and you have to I mean if if someone if
everyone's in the building and they'll
run out and they haven't badged out
someone's going to call you and say hey
reset the badges there was a fire he
said oh yeah okay everyone's reset
right
I don't have a system that's tied into
the fire alarm
but yeah but
yeah something like that
another question
I see your finger
go ahead
but
a lot of paperwork
so everyone at home can hear uh what
happened he's saying check with your
local fire marshal uh check your
jurisdiction that you're in uh There are
rules and and requirements around
controlled egress emergency egress uh he
worked also for an insurance company
they had controlled egress and it was
great and perfect but there was a ton of
paperwork and there are rules to uh how
you let people out because you have to
let them out
um
any other questions or comments
it just all makes perfect sense
everyone's ready to go home and start
this project
there is a lot how you want to manage
the data I will not prescribe you can
figure that out if you put it in a
database
but at the the basest most simple
you're aggregating everything and you're
running scripts that we're doing cuts
and grips and
cats and writing to new files
it is that it's it's hacked
you hack your you hack your way into a
Sim uh there are more elegant ways to do
it databases
looker
I do
yeah this this well it's not too slow
they're text files
you can you can Munch through text files
really quickly
um
but when you have bigger and bigger
impacts bigger and bigger and bigger
text files if you're saying show me
everything over six months that might
take some time where it could be faster
through a database
so yeah that would be uh maybe a
graduate level you figure out how to
pull everything into a database
automatically
and show me database reports and you can
have better graphs from databases
any questions
then thank you enjoy your free
30 or 27 minutes
thank you
[Applause]