welcome to this training about machine
learning models Splunk machine learning
toolkit we're going to cover a lot of
different things during this training
and I'm going to go right out I'm GNA
I'm going to be apologetic but I'm not
going to be apologetic I'm intentionally
going to be a little controversial in
the things I'm saying mostly because I
want to open your eyes to be thinking of
things a little differently I have done
a lot of research I spent a lot of time
doing this and I'm starting to see that
there's a lot of different narratives
that go on when it comes to machine
learning so I'm going to be a little
controversial in the things I say solely
just to get you to think a little bit
outside of the way things are uh are
shown and there's a way of thinking that
machine learning goes and I want to open
up other ways other dialogues about
machine learning anyway the first little
bit of this presentation will be mostly
theoretical a lot of slide presentation
and if this is kind of boring you're not
really interested in it by all means
jump down I'll have uh jump down I'll
I'll have references down below where
you can uh bookmark where you can get
moved forward in the video and not have
to watch all this and we'll go into live
demos of stuff we want to talk about but
anyway let's get going into uh machine
learning what we're going to take we're
basically going to look as it says here
a systematic approach to using Splunk
programming language machine learning
tool kit to gain quick wins and insight
into your logs to help supplement to
help
uh bring about a different way of doing
machine learning than is
traditional and the first thing you got
to do is what is a model if we're going
to talk about machine learning we need
to define a model and I I found this
definition I kind of liked it machine
learning model is an expression of an
algorithm that Combs through mountains
of data to find patterns or make
predictions and then it's fueled by data
machine learning models are mathematical
engines of artificial intelligence but
what I really like is this statement
here it Combs through mountains of data
to find patterns or make predictions
anything that uh can com through
mountains of data Define patterns or
make predictions is theoretically by
this definition a model and so machine
learning I want you to expand your mind
on what could be a machine learning
model basically it fuel it's fueled by
data you bring in data and Machine model
uses mathematical engines to create a
form of intelligence this artific IAL
intelligence so I want to place that
premise down and when I find that model
you'll understand why I I'm going to say
the things I say later on about the
different models and machine
learning first off let's talk as a
general rule I have found it's really
easy to there are thousands of models
out there you can and the basic concept
is it's math there's lots of different
mathematical models of ways to lump
cluster predict data but they all
generally fall into into a into a
category and I'm just going to use that
so you can kind of I found when I've
talked to people about machine learning
that I want to talk to them I just want
to say hey what is it that you're
looking for what kind of model is it and
so is it a regression model is it
classification is it forecasting is it
clustering or is it anomaly detection
and these first three here they're kind
of predictive we're going to discuss
each one of these and I'm going to give
a definition to them clustering is
grouping similar types of data together
and anomaly detection finding outliers
in your data and if you use those kind
of categories then you can break it down
and say all right so my the model fits
in this overall parent this overall
category and then we can talk about the
actual model and it helps make sure that
we're actually trying to solve the
problem with the right type of
tools and so regression the basic
concept here regression models or
predicts a number from a variety of
bunch of contributing factors as I use
here this example we might have CPU
percentage amount of disc reads and
rights and you're going to use a
regression model to predict the amount
of power that a machine is likely to
draw both now and in the future
basically you've got a bunch of data
points and you're going to make some
form of prediction as uh down the road
as to what if I have this much CPU and
this much dis wres this is the amount of
power I'm going to use Etc and so
there's a regression line how closely do
your data points fit that line
that's kind of the the basis of
regression classification this is you're
predicting a category of class from a
number of contributing factors I uh it's
again Predictive Analytics I'm going to
give an example here which is basically
from the uh Splunk themselves they say
hey we've data on a user behavior on a
website or within a software product and
using this you predict whether the
customer is going to do what the
customer is going to do based off
certain behaviors I would go a different
matter you could do contributing factors
if you're doing network analysis if I
see a machine responding on these ports
that might make it this type of machine
if it's if it's listening on Port 80
traffic I can categorize it as a web
server and so those are contributing
factors and you're classifying you're uh
you're classifying what a thing a Cate a
category of a system based off different
data points if it's got a beak it's got
feathers it's got Wings it's probably a
bird that's the Conant classification
forecasting this is uh another
predictive model but typically you're
looking for a single value and I like
this one like profit per day or CP usage
per minute and what you do is you have a
whole bunch of overtime the amount of
profit you've made or the amount of CPU
and that that you're going to predict
that and you can say you know what I I
think that based off the results of the
sales we've had the Last 5 Years we're
going to make this much this quarter or
and because where we keep making more
money each quarter we make this much we
can expect to see this much growth or we
could see that we're going to use this
much CPU the concept is you're going to
based off previous numbers you predict
future values that's
forecasting
clustering this is very similar to the
classification model but in this case
all you're really trying to do is you're
trying to uh
cluster this is actually more the
bird uh sorry the clustering grouping
similar groups of data points together
um you might want to group customers
together based off their buying
behaviors how much they spend or many
items cluster modern group uh together
together the features you specify it's
it kind of to me I look at is uh taking
it a little bit further now you're going
to take actions based off the categories
that you've created the groups that
you've created whereas uh the
classification makes a class here you're
kind of grouping behavioral based you're
still a customer but you're going to be
a high spending customer low spending
customer what kind of behaviors and what
makes that they're very similar this is
not a machine learning I don't want to
get too much but just kind of get that
principle it's another way of uh lumping
things together
categorizing and the last one anomaly
detection this is finding outliers in
your data um you're going to compute an
expect you're going to try to compute an
expectation based off uh different
learning types you might compare it with
comparing with reality and trigging
alert when the discrepancies are too
great so for example very simple find a
new find new traffic on a protocol uh
you you look at this protocol you see
how it's been behaving you see that
there's IP tupal ipx talks to ipy all of
a sudden there's a new communication
that's an anomaly uh you look you look
through your logs and you look for
things that rarely occur um you could
see we we very Ed
this is this should be the bread and
butter for a cyber person because
they're very used to this is how things
work if uh if I think if I tell you what
what uh Port does SSH traffic occur on
almost everyone's going to say Port 22
there are exceptions and those are the
anomalies we want to identify it anytime
you see someone using abnormal ports
typically that's not a normal user it
doesn't mean it's bad but it's abnormal
and so if someone's all a sudden using a
different high high Port maybe they're
trying to encrypt their traffic and send
it out or what if they're using Port 22
but it's not SSH traffic maybe they're
that would be an anomaly and there' be
reasons they're trying to do that but
you're looking for anomalies that's just
by our nature uh cyber security analysts
look for anomalies that's one of our
main things we do so where do we start
I'm this is really a cyber analyst uh
discussion on machine learning and so if
you're a cyber analyst you might see the
following IDs alerts if I get IDs alerts
that's a classification I see this port
I see these uh these set of bytes I'm
going to call it an alert that's that's
almost like a classification model it's
oversimplifying it but I'm just trying
to get you that's how you can kind of
think of it fingerprinting a system or
deriving its OS and its purpose that
would be classification or clustering
determining if an alert is a false
positive or legit even that uh solving
these alerts is technically kind of
anomaly detection you're looking at it
and you say you know what I see this
alert and it fires off every day same
machine fires it off it's not an anomaly
it's regularly occurring even though
it's an alert I know it happens all the
time I'm going for it typically I know a
lot of analysts when they get a whole
bunch of tickets and there's I got 15 um
of this this idea alert 12 of this alert
and I get one of this typically they
want to look at that one they want to
see that rare item the the alert that
hardly ever fires I don't want to see
the alert that fires all the time
because it's probably not an anomaly the
one that fires once that's an anomaly so
as this is stuff we do on an everyday
basis as
analysts looking for overall bad when
you were just digging through the logs
and you're looking at behaviors you're
looking you know what is normal and
you're looking for what's not normal you
but you can't know bad if you don't know
the good I can't give you you you have
to have be exposed to this kind of stuff
and that's what we do as analysts we've
seen what's normal what's normal over
and over again and all of a sudden you
see something that's not normal you go
oh that's that's interesting that's your
brain automatically doing anomaly
detection um most free analysis
ultimately comes down most free flow
analysis mostly comes down determining
if it seems legit or if it's abnormal I
mean that's what we do we look at the
stuff we gather the facts and we say
based off what I'm seeing I'm calling
this abnormal I'm saying it's not I'm
closing the ticket I'm further in the
investigation every day we are as cyber
analysts we are doing anomaly detection
so I'm going to really focus on anomaly
detection because that's something that
should be very familiar and near and
dear to your
hearts and what okay so what do we do
we're we now we've talked about machine
learning we've kind of talked about how
we categorize the stuff the different
umbrellas it the categories that fits in
The Umbrellas now the most important
thing and I find that often it's
neglected when I ask what is this model
what is it doing a good model to me this
is my opinion a good model has a good
story they can say I am looking for this
this and this and I'm doing and that's
what I'm trying to do I'm trying to
identify birds and what I'm trying to do
how I'm trying to identify birds is I'm
taking a whole bunch of body parts and
I'm putting together and if they have
the right body parts it's a bird I mean
that makes sense to me you want a story
it needs to make sense if you say I'm
looking for
bad I really don't know what you're
doing I get it I understand from a high
level and as cyber analyst we say it all
the time what are you looking for I'm
looking for the bad stuff but what you
really need to go down to is I'm looking
for a Golden Ticket attack I'm looking
for brute forcing I'm looking for the uh
file hashes moving around I'm looking
for weird processes going off that is a
little closer to the story we want and
even those might be too generic but it
it's a lot better than just saying I'm
looking for bad I'm looking across the
entire spectrum and I'm trying to say
what is good and bad on the network
that's that's a that's a lofty goal it's
going to be very hard to measure it's
going to be hard to create I and I want
to stay let's get really good stories
when it's possible get a good story
that's very easy to understand and I
hope as I give use cases at the end I
have established a good story and you'll
see it I love this quote from Alison
Wonderland would you tell me please
which way uh Which Way ought I to go
from here the Cheshire Cat responds that
depends a good deal on where you want to
go Alice says I don't care much and the
ches Shire cat responds that it doesn't
matter which way you go if you don't
know where you want to go any path can
lead you there but again I don't
recommend this path for beginners I
think you'll be much more successful if
you have a good story a good narrative
and you know what you're looking for it
makes the rest of the stuff we're going
to talk about much easier to accomplish
so let's establish a good story when
we're building machine learning models I
like this workflow process I've seen it
used in many different ways they use
different terminology but the general
concept is it's cyclical and it's never
always perfect it's not like this
becomes first step Second Step third
step fourth step they can intermingle
but as a g general rule we need to
collect data you can't do anything the
very concept of model of machine
learning is that you're going to look
through data to make decisions well if
you don't have data you can't do it so
thereby the first step has to be collect
data next we got to clean and transform
that data we have to clean and transform
because not all the data machine machine
learning models general rules
they want statistical information
they're statistical models and I'm going
to keep throwing that out there because
that's a caveat that's going to make
people upset as I go later on and talk
about machine learning is statistics but
they are statistical models they take
statistical values you walk like a duck
you talk like a duck you are a duck
you're probably not a platypus um the
point is you need to clean and transform
your data into numeric uh uh statistical
values
uh IP addresses for example they're not
really great in machine learning because
they're strings and so you've got to
find a way to off youate change them
into integers I am going to point this a
little out because I'm just going to
poke a little problem be careful though
as you give them numerical values all of
a sudden they become statistical values
inside your model and they themselves
hold stat statistical relevance if
you're not careful when you build your
model an IP address is really a category
it's a classification it's a data point
but make sure when you clean and
transform your data it doesn't actually
weigh in the model for example 1.11.1
address is probably not supposed to
weigh more than
1.11.2 does that I I hope that makes
sense we will definitely cover that but
we don't want to give statistical uh
value to these numbers and the biggest
problem is you convert them to numbers
all of a sudden is one more important
than two let's say IP address One X
becomes a one IP address y becomes a two
did you end up giving more value to it I
spent way too much time on this but I
just want to be Careful Cleaning and
transforming the data that is a task in
of itself a lot of that a lot a lot of
your time in machine learning is
actually going to be spent in these two
steps collecting the data and then
cleaning and transforming it so that you
can then explore visualize and build
your models but once you've got your
data and it's clean and transformed
you'll explore you'll visualize you'll
find the key data points you'll then
attempt models on it you'll evaluate the
effectiveness of the models you might
come back explore and visualize some
more clean and transform the data re use
different models evaluate often I will
run multiple models and I'll just see
which one I like better because I'll
have a test data and I know this this
reflects what I think is reality this is
my reality I joke um hopefully we know
what reality is but sometimes it is it's
your perspective of reality and I want a
model that matches closest to my reality
and so I can I can put three three four
different models and run the same data
through it and I see which model closest
reflects what I think is truth anyway I
evaluate and then when I'm happy with it
I deploy now deployment is never you're
never done as you go through you need to
keep retraining the model you'll learn
more information environments change you
got to keep that model retuning and you
keep going through that process but this
is the general workflow process I spent
a lot of time on clean and transform I
shouldn't have done that but I what's
done is done we're now going to cover
each of these areas a little bit more in
detail so we're going to go back when I
talk about collecting data if you don't
have the data you can't make the model
it's nice to have a use case story but
if you don't have the ability to get the
data you still won't be able to do
anything I've heard the concept wouldn't
it be really cool if you could get the
uh handshakes between VPN logs and and
then get all the packets that are going
back and forth and see what's going in
and going out that's cool but if your
VPN logs don't have that you're never
going to be able to make a model off of
that you got to have the logs you got to
have that in order to do it and so
collect data is very important nice
thing is we don't have to worry too much
about collecting data other than does
the data help us make our models but we
have lots of data because Splunk is just
that it's a tool for collecting data and
so we leverage Splunk we collect that
data and the data is all sitting in a
nice repo if you choose to use the
Splunk tools you don't have to pull it
out you just can you can just query the
data Next Step explore the data once
it's ingested you got to explore it to
make sure it's suitable and ready to be
used is it numerical does it work uh
sometimes you want to exclude things for
example if I'm looking at um anomaly
detection and I want to do anomalous uh
login Behavior maybe I'm really focusing
on brute forcing or a bunch of failed
passwords I'm not really concerned about
successful pass passwords there may be a
case where you care about both I I'm
just using using let's say my model is
about finding uh malicious type scanning
uh the concept that somebody might have
stolen passwords and in that situation
you would want to
know uh you would you might set your
data to restrict it down the data set
and only take uh logs where the password
uh authentication failed or maybe your
model is about detecting people who
might be impersonating another
individual if they're impersonating and
failing on the password maybe that's not
important at and so then you might
change it and you only care about
authentication that was successful I've
oversimplified it but I just want to
give that point that there's a lot of
data points that you may actually not
need and the more the less data points
there are on your model the faster your
model will compute the easier it is to
explain the results of your model and I
will say that analysts will be a lot
it'll be a lot easier to understand so
you'll gain a lot of performance by
removing data points that don't make any
sense don't remove them just to remove
them but at the same time don't just put
a field in there just because it exists
that field should make sense and
restrict down the data as much as
possible when that data is irrelevant to
the decision-making process that's my
take um anyway again it may be
controversial but I have found that that
works for me and I get a lot better
results if I can eliminate and not use
at a giant pool I want to restrict down
and look at more focused stuff
data you thing you can also do you can
ingest your data into
mltk so you can easily
visualize your
data um depending on the tools you use
if you've ever used uh I've watched
people and they've used zogs and they're
going through the zogs uh if you're
looking through it through a a text
editor that can be a little bit painful
nice thing about Splunk it puts it all
in a nice graphical format it's really
easy to look at that data and so I I I'm
a big fan of using
Splunk and using some of their
capabilities to better explore your data
uh other things Splunk platform and mltk
offers several methods that you can
clean and transform your data to address
many of these issues I'm going to show
I've kept talking about you got a lot of
categorical values in numeric values the
more categorical values you create that
are non-numeric you get into some
serious Computing issues going on it
Splunk just uh machine learning models
these ma mathematical form they're math
they don't math doesn't really work on
strings again I'm just going to say it
these are statistical mathematical
models thereby math works on numbers and
thereby the more you can get your stuff
into numbers the better your models are
going to work and some of your models
just won't work if you don't Supply them
numbers Splunk has capabilities built in
to help you convert some of that stuff
so uh this is there's another benefit
there we're going to experim expent data
experimentation is where we start
processing and training our data we look
at it we say hey uh there's there's a
bunch of things to look for anomalies
algorithms just kind of experiment and
see how that that data is working did I
grab the right data is this going to be
able to help me be able to train some
other models and so you do some
experimentation there and then
ultimately you're evaluating those
results first did I get uh experiment
with your data make sure you're getting
useful information from it when you when
you start when you start with that very
first bits of experiments can
you make uh are you getting useful
information and if you as the person
building that model doesn't cannot make
useful model if you cannot explain in
human language very simply to someone
else what your data is producing you
might want to go back and reevaluate
your model it needs to be able to answer
and provide useful information and you
have to remember your audience
um it's great uh we don't mathematical
results are great I I joke um outside of
arithmetic my math I'm not a big fan of
I understand arithmetic you talk about
adding subtracting dividing I get that
but you start using things like uh well
this is the um you start using calculus
terms derivatives things like that I'm
not 100% sure what you're talking about
you got to bring it down to my language
and help me understand it the the
results have got to be useful are they
and they may or may not one of the other
things you might am I even asking the
right question from your data you might
be creating a model and it's not even
quer it's not even making the right
question again that comes back to frame
your question create a story you have a
good story frame it do you have enough
data is it sufficient cleaned again
you're just looking at this uh how's the
experiment working do I need to do some
different things then there's some
machine learning uh toolkit model
assistance if you're new to this kind of
thing and you're not familiar with how
to start to look and evaluate your
results you can use some of these
assistants I'll demo that later in the
video um how to use that but it'll help
you quickly assess did I is are these
experiments going to show some value am
I GNA be able to do something and then
you're going to be able to you want to
modify and uh change the range the
thresholds some scoring metrics on how
your machine machine learning model is
going to move forward in the in the
future and so there's all sorts of togg
toggle I don't know ma uh modif iable
values in these mathematical formulas
you're going to tweak different values
and you'll get different results based
off that so you're going to play around
with those then you're going to just
tune and iterate and so basically you're
just going to keep you're going to keep
playing with that model and you're going
to try to make sure how close I got my
training data and this is the expected
results then I'm going to put some n I'm
going to put some other data in did it
give me those values like I expected one
of the things you can do is if you know
the beauty is if you have some trained
data and then you have some extra
trained data that you don't use to make
your model model you can put that data
in and see how well it m matched data
it's never seen before if it's not
matching very well go back and tune an
iterate um you want high percentage of
alerts uh of value the thing is analysts
remember these are going to be going to
cyber analysts as a general rule you
they have a lot of things they can't
just go hunt every little thing that
pops up so the thing has to be coming
back to here it's got to be useful it's
got to be understandable and you don't
want to be creating a bunch of noise
they don't have time to hunt down a
bunch of noise so you got to make sure
these models have high level of
confidence uh I hear people well it's
got a 50% confidence rep level in other
words I could flip a coin and get the
same results you did a lot of effort you
got to get better than that you got to
do you got to get high percentages and I
know I'm sounding really bad but if
you're not getting if you're
making uh if you're not even 90%
accurate I personally don't want to talk
to you at the moment I'll help you work
on your model but I don't want to have
my work and my my day-to-day work tied
to a model that has is fit uh one out of
every uh one out of every 10 logs that
come out of it what is completely wrong
I don't want I want I don't want that
and that's some people 90% sounds really
really good but that's a lot of time
wasted every time I get if 10% of the
things I'm looking at are invalid I have
to use my time wisely there's never
enough time to look at all the things
that are going on in a computer system
you can't be you can't can't be bogged
down with a bunch of false information
and so you really need a high level and
so this tuning and iterating is really
important again there's lots of
assistance lots of things just keep
evaluating re-evaluating training that
model so you're not we all know you
can't go you don't want your model being
100% because there's problems with that
but I will tell you that as a general
rule when you're anomaly detection the
closer you get to 100% I don't want to
say that 100% is the the goal that
you're going for for but I will say the
closer you can get there is actually the
ideal don't let that concept that you've
got your model trained so much that
you're missing the other things there is
some there's some truth to that but in
the cyber world we it you're going to
miss you're it's too anomalous to begin
with so don't make things worse by
providing data that's not valid
um so anyway it's a balancing act uh I'm
not there's no there's no silver bolt
there's no perfect numbers to say there
just try to make your model be as
accurate as possible and keep tuning it
till you get there all right then you're
going to deploy that model you're going
to uh you're going to apply it on new
never seen before data see how it works
you definitely want to keep checking
your model outcomes and make sure that
they are still getting data as as seen I
love the idea of putting them in red
team blue team exercises and you want to
put your model out there and you want
want to make sure especially in
something so blatant as a red team where
somebody is not trying to be uh really
covert you definitely want to make sure
that your actions or your model is
detecting the actions of that that red
team pentester
um that is if your model can't detect a
red teamer you really want to go back to
that model and say all right let's
re-evaluate how we're doing um it's uh
the the really the really advanced
persistent threats they're a lot slower
they're a lot more they're going to take
their time they're going to be a lot
quieter they're going to be making sure
they don't trigger any Gates red teamers
they're they don't have much time they
got to get into and out of that system
in a day or two and so they're going to
be triggering everything they can
they're going to try not to trigger
every alarm but they're going to be
making so much noise your model better
detect them I joke if you're only
catching 50% of the the bad that your
model should have caught with uh a model
on a red team yet definitely want to go
back and reevaluate it I that is to me
it's good but it's nowhere uh ready for
Prime Time production it's got to be
much better um anyway so in the mltk you
got assistance you help you re up the
beauty if you build your stuff inside
Splunk and you use their fit and apply
commands to build the models you can
then run a scheduled search and it
retrains your model every time the uh
scheduled search workes works so if you
want resed you want to retrain your
model every week boom you put a
scheduled search in there it's really
simple and if if you're a machine
learning person you don't quite
understand this we can show it it's so
so easy to do it's much easier in
setting cron jobs and running a bunch of
other stuff behind the scenes to keep
your stuff retraining so uh if you build
it you keep it in Splunk they've got it
really simple and you can get alerted
about your model and you publish your
models ni that you build your model it
can be published anywhere you want and
the model just just basically works for
you
so anyway and a model the nice thing is
when it builds the model it doesn't need
all that Python and all that other stuff
there it all those dependencies it
doesn't matter because it buil the model
is a bunch of data it's really a giant
KV store and um that it's not using all
this python code and dependencies so
it's really easy to publish else uh to
other locations a the model piece uh
building the model needs to be done with
uh it's a lot more intents on the
computer and you need a lot of
processing and you need the dependencies
from the python libraries and all that
stuff but if you publish your model
using Splunk it's as simple as pushing
an app out to someone and it'll just
work there so makes things really easy
those are the general stages of machine
learning we've talked about what is
machine learning what are models the
fact that it is trying to make
mathematical statistical decisions off
of data there 's Predictive Analytics
and categories there are anomalies and
there's
categorization you're going you've got
there's the phases of machine learning
you want to make a good story you want
to collect your data you want to massage
it get it PR right you want to
experiment in it you want to get to you
want to massage it play with it
visualize it then you want to build the
model deploy the model reiterate through
it there's a a process it's this is
pretty consistent across any place I've
ever SE seen how machine learning does
stuff anyway so that's the general rule
now that I've talked about this we're
going to move into the next stage which
is actually applying it and I'm going to
show a few different methods the first
thing I want to do is I take a tiered
approach and each of these tiers in my
book there's it's I want to use the term
do not let the Quest for Perfection be
the enemy of good and so there are are
quick easy wins that you can do if you
build a model and it's really really
impressive but it takes you nine months
to build that
model okay but what if we could
build five really really good
models using something not quite as
pristine and all that but it's still
really good and we can get five models
done in that same amount of time uh
there that's things to consider so go
for those quick wins everything doesn't
have to be the most complex um
complicated models if it's easy and you
get an easy win from it do that and so
the first stage can you get those
results can you get your stories
answered just using the Splunk
programming language and statistics from
Splunk if you can do it the next stage
it's actually kind of the SE almost
first and second seem the same but there
is a slight difference and I'll talk
about that can you do it with SPL
statistics and with a few KV CSV lookups
if you can do that do it what about just
using the mltk as it as it rolls out
from Splunk if one of their algorithms
works and it's good enough and it helps
you and you can then just publish your
model through
that and if you do it then it and it
speeds up the process that's third stage
fourth stage is you know what there are
some new models and yes the mltk can
import models but it's not quite ready
to be this model I just it's got to be
dealt on its own I need to I need to be
in the python code on an everyday basis
tweaking and manipulating this stuff and
it's and that's the fourth stage all of
these are perfectly valid but as a
general rule I find that each of these
grows in the amount of time it takes to
create the models
and there is so much data so much stuff
to look for we want to get as much
things we want to get things done as
quickly as possible we want to look we
want to be able to produce and provide
value as quickly as possible so I want
to cover each of these stages I'm not
going to say one's better than the other
but again if you can roll out and get
good good data that is usable by the
analysts and doesn't create extra work
but answers the questions they're
looking for with easier stuff then do
that and ultimately make that your first
solution and then ultimately roll out
down the line if all else fails we can
go to the Other Extreme and we can go uh
download new and uh immersive models
that are rolling out that are Cutting
Edge tip of the spear type models
anyway so now we're going to actually
start showing these things in
practice and as I'm G to I'm going to
talk about this we're going to do a
little bit of a paradigm shift I want to
take the concept that there's the
traditional IDs signature and then
there's the ml statistical anomaly we've
kind of stated what these are an ml
model anomaly it's using math it's using
pre uh historical data to build these
anomalies so an IDs signature is a point
in time alert it's looking at Network
logs or host log whatever if it's if
it's a network intrusion or it's a host
intrusion whatever the case may be it's
going to look at these signatures and
say hey I saw uh this set of IP
addresses I saw this type of traffic and
I saw this uh bits uh set of data coming
in in this order that becomes an idea
signature and those roll out but the
problem with idea signatures is they hit
differently in different environments
different environments are different and
that is what makes ml statistical
anomalies that the ability to hunt there
so much cooler so much more powerful is
it's not static if you can get the C if
you can get your data and you can look
at it then you can see what is normal in
their environment and then identify the
abnormal and so what is abnormal in
environment X may not be abnormal
environment why but the beauty is you
create that historical context you will
your the way you search these will be
different whereas idea signatures
they're they are agnostic to the
environment for tents and purposes you
can do a little bit of tuning to the
rules but as a general Ru thing you
deploy that rule it is doesn't care it's
going to apply the same it's it's going
to judiciously judge the data the exact
same way ml statistical anomalies will
allow your data will be a little more
flexible and it will look at the
environment and say this is this is this
is normal we're not going to flag that
whereas I mean honestly you could sit
here and say IDs signatures they're
going to trigger on out of the box
they're going to trigger on scanners all
the time when scanners fire off unless
you block them out they're going to fire
your ml statistical anomalies will
quickly recognize your scanners and po
should if built right they're going to
accommodate for them and without you
having to do anything per se other than
keep the the normal keep upkeep of
updating the model every week Boom the
statistical anomalies will account for
the uh the
scanners anyway so another thing to
remember anomalous behavior is not bad
it's possibly bad all it really means
when you're looking for these signat
we're going to do a lot of anomaly
detection I detected anomaly that does
not mean there was a compromise it just
means I'm seeing something I've never
seen before you got to put your head you
got to put that frame on and that's what
you need to result when you look at
these I am looking for something I that
I don't normally see um I've seen
dashboards where people will track um we
uh web traffic and they'll look at the
error codes and all of a sudden if
there's a ton of error codes bunch of
four 400
airs there's a giant Spike that's
anomalous does it mean it's bad no I
mean for all we know there's it's it's
normal operations but it is anomalous
and that's what we're really looking at
is anomalous Behavior not and if it
happens to be bad that's what the
analyst is doing they're investigating
to find out if it's bad but we need to
make that anomalous Behavior as clear
what is the behavior that's anomalous I
need to be able to tell a story to an
analyst when I show them this anomalous
behavior and say hey what I saw was this
IP did this kind of behavior and it did
it this and then it did this and this
and it's never done that before are you
concerned you're the analyst you look at
that and you tell me is that something
we need to take action on or is that it
was anomalous but it was not bad anyway
so now let's get to some
examples all right so the very first
story it's kind of a two for one we're
going to look for abnormal ports being
used in corite
logs this sounds really really simple
and it really is this may be way simpler
than you're thinking for machine
learning but again I want you to get
back to one I want to be able to give
you a machine learning models that I can
demo in a short presentation in video
format so they are going to be on the on
on the relatively simplistic concept but
the frame the framework is still the
same the concepts are the still the same
and if you understand these quickly
you'll actually find you can take more
complicated models and do the exact same
thing with very little uh with very
little change to what I'm going to show
here so we're g to start with something
very simple and again remember it still
fits the category of a model we are
going to use be uh historical data to
make predictions to make decisions and
so we're going to use uh we're going to
Define abnormal ports as ports that have
not been used before and we're going to
do a window of say 90 days I'm going to
go abnormal ports over the last 90 days
in my corite logs all right then when I
get done with that I'm going to go look
at abnormal tupal connections being used
in corite logs tupal meaning Source IP
and destination IP I just want to see am
I making new connections that I've never
used before all right um the other thing
we're going to do then is I'm going to
take you back to the tools of the trade
I'm going to go into more detail on
these but the first one tier one we're
going to use Splunk we're going to use
some stats and some math if that doesn't
solve the problem we can move to Stats
with lookup if that doesn't work we can
move up to MTL M
mltk commands with anomaly detection and
um this says tier 4 mltk sorry that's a
misprint that would be expanded models
I'm here's the surprise if I told you I
was going to solve all these things I'm
probably not going to get to tier four
all right so let's kill this and let's
go right to the my Splunk
instance and here's my data I've got
index equals corite Source type equals
corite con and we can now look at the
data and I've done that by using this
head command I'm looking at these 100
Fields remember I the first thing I want
to do is do I have the data well yep
I've got connections and I can see I've
got do I have ports yep I've got it
right there id. respon P those are my
ports the desk ports okay so now I I
have the data are there when I want to
massage this data you know what I could
and so so I'm going to what what are the
fields that matter I'm
actually in truth if I want to get down
to the simplest quickest model I just
want to know anomalous
ports so I'm going to go
table ID let's change this I'm going to
do because I don't like this id.
response H equal as
desport right there I've now changed my
field this remember we talked about
Splunk gives you the ability to massage
and manipulate the data this is simple
I'm just actually changing the name of
the port but I'm getting rid of some of
this weird formatting I'm putting into
stuff that makes more sense um desport
that's the destination Port cool and I'm
just going to go table desport if I did
something like
this I and I grabbed 100 so I've really
quickly subset it I grabbed 100 uh um
the first 100 records I can actually now
flip this to fast
mode um and basically what I'm getting
here is I have reduced down my data set
to the most minimal data set possible
and um it would help if I put the right
thing response p as Port I got response
IP so here are my ports cool that now I
need to go back to my model and apply
statistics to it what can spunk do well
I love the stats command this is like as
I like to say stats is like
novacaine works every time just got to
give it time uh works every time so if I
go
stats count by
desport if I do that it's going to go
sum up all these and say I saw these
these these and these well I can tell
you what's anomalous just in this 100
data set one 38 is
relatively
anomalous and 67 but this data set's way
too small so let's now open up the
window and we'll just look in the last
24
hours we can see and I can I could write
a sort command if I wanted to or I can
hit the sort button here again that's
the beauty of Splunk it gives you some
really easy abilities but I'll just do
sort on
count and everything that comes back
with a value of
one is that not the definition of
anomalous we could do a little change
and we could say no I want anomalous to
be two or fewer times cool okay you
decide what that number is and that's
your not that's that's the threshold
you've set but stats is going to come
back and tell you that data so I can see
everything so if I say that anything
that's occurred two or fewer times in a
month is anomalous well then
that would be I could just do something
like this
where count is greater than is less than
three and now I have all my anomalous
data so that is those that's just
anomalous that's just and in reality
maybe anomalous isn't the best term it's
not highly occurring but it is anomalous
in
this data set this data set that I've
created I've said it's not frequently
occurring okay cool what if I want to do
a different thing what if I want to
build a I want to look at the last 90
days worth of data and then look at
today's data and tell me did I go to any
new any ports that I've never gone to
before this here just says look over the
last 24 hours and tell me any I could
flip this open I can go let's go so
let's take this to 90
days last 90 days let that thing run
it's going to come back and tell me all
the ports that I haven't been to that I
in the last 90 days that I've gone to
very
infrequently that's really what this is
telling me and what a surprise they're
the arbitrarily high high
ports um but that's something as an
analyst I understand you un uh hopefully
you understand what they are what I'm
meaning by that but let and so there's
some caveats maybe we want to exclude
those and not we don't want to count
anything above we don't want to count
any of the ephemeral ports then you
could actually come in here to your data
and say
desport uh we could also go here
and desport is less than
1,024 there you get the exact number
pick the port it's got to be less than
that or whatever the case may be you
could go and eliminate that you Splunk
gives you that that
capability looks like 90 days for the
demo purposes here is going to take a
little longer than I want but it's fine
it was I was allowed to explain I got 3
million events I've got 3 million
connections and these 625 ports were
considered to be based off my criteria
to be
rare So anyway
um let's see
[Music]
then we're going to go let's do
something else what if I want to define
the concept that and I want to look at
each day I'm building an alert I don't
need to know this is just this is this
is kind of FYI these ports aren't used a
lot but what if I want to set up an
active alert that says look at today's
logs and see if there were any ports
that were some was a destination Port
that we haven't seen in the last 90 days
how would we do that well now we need
time
so I'm just going to add this time field
I'm going to keep this very
simple we're going to come back to doing
a head 100 so I'm working with a very
tiny uh sample of data we run this now
I've got time
field so I now know when this
destination Port was visited now we're
going to go so how do I find out how do
I set a baseline of what I've seen in
the past well really
simple I'm going to do
stats
earliest time as first scene
time and now when you see this it's
going to come
back and this is going to tell
me this port was the first time this was
seen first time we was seen first time
we seen let's change this into human
readable terms so even Val time
equals if I do something like this the
first time this was seen was today these
anomalous Port these ports first time
they're seen of course they're going to
come out they're all going to be seen
today because I've made this subset that
I've only grabbed the first 100 let's
open this up to the last 30
days last 30 days run this query and now
these ports are going to start changing
their
times this was seen on 130 it's going to
keep getting bit some of these you
expect are going to go back in time
128 notice as it keeps looking back in
the logs nope I saw it at
127 I saw and so it's building a
baseline using time and while this
builds so now what I want to do is I
need to make Splunk recognize hey when
this time
is if I see any events that the
time is in the last 24
hours that's anomalous because realize
it won't give me this if it's got an
older date this earliest command says
look at all the times on these logs it
give me the first time it was seen so if
it was seen today and it was seen a
month ago the date will be a month ago
if it was seen today and yesterday the
date will be yesterday so what I now
want to display to the user is anything
where the time is less than 24 hours so
you're going to find what this time is
and if it's less than 24 hours old then
those are anomalous so how do we do that
well what we're going to do is first we
need to evaluate we need to make this
Dynamic so we're going to do um current
time
equals now this function gives me so
let's go and put these out here I've now
got an Epoch let's go back to doing 100
day head 100 again on this now I've got
a current time which is it's going to
take the Epoch time tomorrow when I run
it it'll be the E Epoch time for
tomorrow so this number will keep
changing every time this query runs
that's the beauty I've made it Dynamic
and so I can just set this as a
scheduled task and it's going to say
what's the current time now and what I
need is I now need to do another math
and I'm going to
do uh
eval we're going to call this
um
time limit I don't know which is equal
to um current
time minus
86400 if you don't know what that is
that's the amount of seconds in a day so
what I'm doing is I'm going grab the
current time subtra ract 86,400 seconds
and that's my time
limit my time limit sure all right we'll
throw that in now all of these fields
have a time limit what we could actually
do if you want to see this in really
small terms I could actually make this
um 30 seconds so basically what I'm
doing here is I'm G to I'm I said I
wanted a day ago but I'm going to run
this on 100 logs and basically what I'm
going do is now I'm going to see
anything I'm going to have these 100
logs coming back when was the first time
they were seen and I've got a time of
what was 15 second 30 seconds ago and so
I can do
where current
time sorry time
limit sorry nope where I don't want to
use this time right here we're going to
comment this out because we need to use
it needs to stay in Epoch time so you
can do math on
it I miss a a tick or aren't you coming
back fine we ra you should have
commented the code but it didn't okay
eval time limit current time minus 30
where time limit is oops sorry current
time is greater than time limit if you
understand time time just keeps ticking
up so every second that goes by will be
greater than the second before it and
what I'm saying here is where current
time this field is greater than now
minus 30 seconds that would imply that
this event was first
seen in less than 30 seconds ago I hope
that's making sense to
you so I'm going to come back and these
are the events I've got and I can now
flip this
to put that back in
eval first scen time equals time oops
the other way
around time equals I just use this so it
converts it for me I could do a strif
time but I just like this it makes it so
much easier this turns it to
equals
equals so I can see this thing was seen
at
2331 that's probably not going it's
going to be right there it's it's a
brand new alert brand new and so it's
seeing these IPS and these are brand new
ports that weren't seen over the
30 and so the basic concept is it's
brand new in the last 30 seconds that's
all it's really saying and it's still a
little misleading because I'm only
grabbing a head 100 let's get this the
real way now I hope that makes sense
what I've done I grab my con logs I
stats them I grab the earliest time that
that destination Port was ever seen and
it's going to get written here then I'm
going to grab the current time subtract
86400 which is one day so I'm going to
say grab the last if if this port shows
up and the first time it was seen was in
the last 24 hours it is by definition a
new port or
anomalous okay so now I've done this run
it and this number will probably reduce
itself as time as we move up this
chain but we let that
run
well that is silly on my
part first scene
time not current time where first scene
time is greater this field is greater
than my time limit
filled silly rabbit okay so all of these
times had better be today
you you by definition because I'm
throwing out everything that isn't I
won't see anything with a time here that
isn't of January 31st it's that's just
going to be the way it
works and so here are my 50 anomalous
ports and that number may drop yep it's
going to keep dropping as it grabs more
data as it goes back more days say oh
yeah you know what I actually saw them
use that Port I saw them use that
port and there you go I have created
with a stats command show me all new
anomalous
ports okay um what if I wanted to do add
a little complexity to it you know what
desk port's great but I actually want to
know um the machine and the source
Port does that I want to know the
destination IP and the desport I want to
know that pairing oh okay what I did
stats count by
desport and Des IP that quickly I've
changed the command and now it's going
to pair these two together oops help if
I write them right d IP I need to do a
rename
here. respon H as Des IP if I do that
now I'm searching by desk it's going to
pair things together by desk IP and port
and it's still going to do the exact
same math I run
it I should get more results I only got
30 from the last one I'm probably to get
more results and it's going to take a
little longer because I'm adding more
Dimensions but it's actually still going
to take that
long interesting
oh I'm going to take that out this table
here because that excluded the
destination IP and now if I run
that notice now I have Des IP to port
and when was it last seen so this
machine going to this port that's
anomalous not just the fact that I have
Port 443 but it's that this machine was
the machine that was being accessed on
Port 443 and so I can run that very same
query I'm going to get those results all
these are going to be brand new oh well
you know what what I actually want no no
no you got that wrong you need more
complicated stuff we need more
information I need Des IP Source IP
pairings oh okay so let's do
id. um oreh which as Source
IP and I come in here and I go cool
Source IP
I just added a new one press
enter now I'm doing anomalies on source
to Des on
desport I didn't change this I told you
this is my it's it's just like Nova cane
give it time works every time I I can
put as many dimensions as I want in
there wait wait but didn't you tell me
that machine learning needs to in order
to do anomaly you need to uh normalize
your data and get it into statistical
values yes if you want to use the python
machine learning but if you want to use
Splunk and statistics it didn't care it
can handle strings because it's not
using statistical software in the same
way that is limited to only using
statistics you must pass in statistical
values I'm not trying to cause problems
and upsetness but the fact is it handle
strings just fine as you can see this is
a string this is a string this is an
integer doesn't care it doesn't matter
how many of these I have it'll just keep
doing it doesn't throw a fit because I
have a a 100,000 different Source IPS
destination IPS Etc it's fine doesn't
care um here's something here's
something crazy for you A lot of the
time you need in Splunk and python
you've got to come up with when you're
using those python models you've got to
come up with a way of converting inting
the IP and the desk IP and well the
problem is then they become numbers and
how do you keep these pairings
together I'm going to give you a weird
one what if I want to do this
eval Fu equals Source IP plus hype
Plus close plus Des
IP and now I have made a new value this
saying make a field and make it Source
IP separated by a pipe and a des
IP and I'm going to actually close this
out and I'm going to put instead of
doing Source IP and destination IP which
is a three Tuple system I now flipped it
back to a two Tuple
system so the very same encoding you you
do one of the ways you could encode with
your python I made it even easier this
is still a string but look I can pass
pass this right back I'm actually going
to get speed Improvement because I
reduced down the tupol yet I still have
the cardinality the uh the ability to
make this distinct from everything else
that you would really want of having a
source and a destination but I now have
one string value I'm comparing so I've
increased performance and it's simpler
I'm just going to point this out this
seems like a massive win I'm also
looking over 30 days worth of data
really really not too bad
okay now wait a minute but I thought you
had a whole tiered approach you said
that you could do stats if you didn't
work that you could do another all right
let me tell you the use case where you
might want to move I've got a use case I
can't solve with this simple
command what
if I am not looking at my alerts and I
get in an anomalous IP address say this
one right here won't that mean that
tomorrow tomorrow this is part of my
Baseline so tomorrow this won't be
anomalous and now the the man the alert
person the person's not going to know
those get we won't be creating a new
alert on that that that doesn't work for
me I can't I've got to have a little
longer period of not retraining my my
model this technically is training the
model right there on the Fly and so
every day you search it tomorrow this
field won't be anomalous anymore oh okay
cool well let me show you how you solve
that that
oops if I go back to my Splunk
here stats with lookup what do they what
does he mean by that all right I got 67
values
here and what I'm going to
do
is I'm going to go the other way where
first scene time is not less than time
limit so I'm going to grab all the
things that are new but I I could
actually do something like this um so I
don't actually I can take this whole
current time out and what I'm going to
do is I'm going to build my Baseline of
values that are part of my Baseline like
this and all I do so nothing changed
here I mean this is just give me all
my tupal desk ports first scene this
doesn't take time into effect this is my
Baseline of acceptables anything that's
not in there is a base is anomalous cool
now I'm going to do
output look up
anomalous ports.
CSV what this is going to do is it's
going to write these results into a CSV
file and we'll open up a new tab when
this thing's
done probably should let's take it down
to now we'll let it run the 30 days the
concept here is now I've made a a lookup
and now when I get new when I run my
alert I'm going to grab 24 hours worth
of data and I'm going to join this data
on the new data and anything that pairs
up and doesn't match here that's my
anomalous data and the beauty is until
you re output the lookup to today it's
anomalous tomorrow it'll be anomal
the day after it'll be anomalous so if
you need your stuff to be anomalous and
you need to keep that data you don't
want to refresh your data model every
day go use lookups so we've used stats
and stats with lookup and we've been
able to do anomalous here so I'm going
to come and
grab come on finish
up so I've got a little look up here
we're now going to duplicate this
tab and how do we look at this value
value we're just going to
grab pipe input
lookup there's my ports there's my list
here's my key right there and so what
I'm going to do I'm going to go grab
this is how you do a lookup we got to
grab the data we want index equals
corite Source type equals corite
con and now I'm going to do it's look up
command look up anomalous
ports um I need I need to actually make
that Foo fied
again don't Foo is a horrible term we'll
need these so we're basically renaming
the fields so they match I make my Foo
and you don't need Foo is Foo but this
is the concept when you make a look up
you say what field are you matching if
you don't it'll look and say do does do
I have a field in here that I have a
field over there and yeah Foo but I
don't want it to match on desport that's
just something I want to avoid so I'm
going to call it out and say Foo is Foo
so Foo from this lookup and Foo from the
from this data and then I'm going to
Output let's just output the count it
doesn't really matter what I
output
um oops I don't need last 30 days now
I'm going to go grab the last 24
hours
and I'll just do a
table um Foo and
count anything that has the count filled
it matched anything that doesn't have
the the the
count did not match and so if we come
down and we go so what I really want to
go to
is where is
null
count run that command I'm not going to
have any anomalous because my query grab
the last 30 days everything's included
inside of it but this command tomorrow
will catch all new anomalous behavior
all I've done is I've I now it's the
same command but I just don't have to do
all the I don't have to do any of the
time the time pieces and I make a lookup
and then I apply the lookup and when the
lookup matches it's it's it's not
anomalous when it doesn't match it won't
output anything that means the value
will be null and that's what I want to
see and I do that bada bing bada bang I
have my model all right well let's look
at this is there any reason then I
should go to anomaly
detection um well I'm going to tell you
I already solved the problem
but let's go back to something simple
index equals
corite Source type equals corite con
this is not this is going to cause
controversy I know it and I'm sorry but
it is what it
is um I'm going to
do I want to I want to rename those
fields I just like naming renaming them
you can do this without renaming these
fields so let's not for this point for
the sake of this stats account by ID id.
respon P I've already done this query
for you but I'm just looking at the last
24
hours give me all
the id.
response why fine will re what did I do
wrong ID
response
oh it doesn't like the periods that's
why I get rid of them it's why I do it
because you don't want to have to deal
with those periods I forgot about that
that's why I always do it oh go away
rating list cool I don't know why the
rating list has popped up go away okay
now if I do this by
desport what did this wrong index well
it would help if I wrote this right okay
maybe I could have done the count I
probably could have done the count
either way all right so now I've got
uh we run this count all of these
ones they're anomalous right that's the
definition of anomalous I recognize
something these models that I'm going to
use from the mltk they are not actually
Splunk models all really the Splunk has
done is it's created a wrapper it grabs
the python commands and puts them into a
Splunk syntax so you can call them from
Splunk so I can do anomaly detection
there is a anomaly detection algorithm
in those python
libraries and the principle is we just
put in what fields we want them to
detect as anomalous we can run that I
should have done that on two different
pages okay so that's
running this is running stats count
by basically the same data set I need to
put a table here so you can see
it table D
IP D Port so you'll see the
results and it's going to call it's
going to call out what it thinks are the
anomalous ports it finds 27 ports that
thinks are
anomalous and they're all going to be
the EXA exact same percentage because
what it does is it looks at the entire
collection of desk ports and says how
many times did I see a frequency
analysis desport I could do the same
math over here if I wanted to but I just
want to point something out if I go
count one I've got 20 items to a page so
there's 20
here
[Music]
40
48 50 50 something 50s something ports
that are by
one how many did the anomaly detection
detect
27 I'm gonna tell
you I don't get it why is this one
anomalous 53792 and that's G to be found
over here
53792 if I went back one page it'll be
on that list 53792 yep it occurred once
but the other ones aren't I'm going to
going to tell you that I'm not 100% sure
I like this it it added some
complication into it I don't know that I
agree with the numbers I'm not sure why
it did that I mean I wonder if I ran it
in pure python if there was something I
don't know what happened uh there's some
glitches but I also know that this
command ran faster if I look at the time
that this command took to run I inspect
it it actually takes less time 5.4
seconds watch them and I say it it won't
be faster but as a general
rule I'm glad okay this was almost a
second longer the bigger the data set
the longer it takes why is it longer
because it's got to go run the python
it's got to do all that stuff it's got
to do the conversions things like that
this is just native
Splunk so is that a reason to not use
the anomalies and other things no no no
just you know what I don't need to move
up the model I could if I want to if I
find that I'm going to get better data
but I'm not 100% sure I'm even get it
going to get better data and the fact is
is you could run a ton of things and say
tell me what's anomalous but I'm going
to tell you that my eyes will tell
you that what I was doing with those
stats commands earlier that's anomalous
I don't need another model I don't need
some fancy schmancy model and with all
these different variables and things to
basically tell me it's anomalous because
I know it's anomalous I've never seen it
before now I have you could you could
put uh C how anomalous is it well you
know what I can do the exact same thing
with Splunk because I can do math over
there I'm not going to do it here
there'll be some other examples as we
move on where I will be
demoing the uh more math in there and
you can see I can do the same thing with
math just straight in here and not even
waste my time with all the different
models one time I'll put all the models
together that'll probably be my very
last example and honestly when you look
at them all if the simple model provides
the same value and it actually reports
back the same way the analysts would
call it anomalous that's probably the
easiest thing I wish I had the name of
the professor but there was a
presentation done by an MIT Professor he
did not he got flamed bad by the ml
Community but he flat out said you know
what more often than not the simplest
model is actually the best model for
machine learning the more complicated
the model is he says I have found that
overtime they are less reliable you know
what the guy had a lot of credibility he
said it I'm just going to tell you I
don't want to get that art but if it's
simple and it works just use it I was
able to create a model to detect
anomalous behavior on ports on
tupal just that fast it took me minutes
this presentation's like 30 minutes I
don't know how long this section is but
it went went really fast we're going to
do it again and we're going to do more
comp licated models and sadly enough
they're probably going to take less time
because I'm not going to spend as much
time explaining what I'm doing if you
can put out a model that does that in 30
minutes why and it meets all your
criteria why wouldn't you do it anyway
so I guess it's time to move on to model
number
two all right let's discuss another
story and this story we're going to look
at anomalous detections in network
connection log history by IP tuples the
concept is we have our con logs it's
keeping track of a history there and our
our theory is and it could be right it
could be wrong again we're trying to
keep these simple our idea is that there
will be generally the same type of
history between these IPS and
particularly internal because we don't
want to talk about what's going on with
the Internet or the internet talking to
us we want to just say that our Network
our little subnets are relatively static
what they do that communication inside
each other so we would expect that
generally the history will be the same
but if all of a sudden a different type
of history occurs that might be
indicative of new anomalous Behavior so
let's go and put that into action we're
going to Breeze through this but we're
going to come back tools of the trade
first can we do this with stats in math
can we do it with stats and lookup so
we'll start to can we do this with tier
one all right what we're going to do is
we're going to come in here I'm going to
write me a query index equals
corite Source
type
equals corite
con going to grab the first 100 Fields
what do we
get you can see oh we got a history
field that's the field I'm looking at
there are different
histories and I go look those up if you
really want to find out what they mean
but I'm actually going to use this
concept the beauty of this is if
someone's giving you a good story you
don't have to know all the other
details I'm just going to look for
what's a nor uh I'm just going to look
for here for some anomalies eventually
when we start to go through and refine
the model it might be helpful to know
what every one of these categories mean
I do know that uppercases like output
lower cases input it's kind of the
direction the data is coming from and
But ultimately I'm just looking at these
different history fields and trying to
say what's normal and what's not well
first off let's go and just check is
there a bunch
of anomalous stuff stats count
by
history if I run
that I can see that this occurs a lot
this is almost always guaranteed this uh
UDP
traffic oh what do you know it's a
little odd um
um and so we definitely have some
anomalous it's and yet we have some data
that is relatively consistent this might
be a decent data set here all right the
other thing we want to do is I don't
want to I didn't want to care about I'm
not looking at internet connections I
want to I want to just be paying
attention to what's inside our Network
so I'm going to do
id.org
H
equals so what I've done here is I've
said I want all the responses and the
origins the source and the destinations
to be 192 168 that's my internal subnet
so if I run
that I'm going to throw out a whole
bunch of these connections and shrink
down my data set to something a little
smaller and now look at that anomalous
Behavior still got a lot of that still
got a lot of things that are pretty
consistent
consant and then we can see that they
start to Peter
out interesting enough I don't have any
ones that's kind of that's kind of
interesting to me that there are no just
a single occurrence of a uh a history
there's at least two of them
always
okay so let's go write this
command just so we get this going the
right direction let's
go head 100 so I don't spend a lot of
time as I rewrite this and mess up my
stuff we're going to do a stats count by
history what we want to do is
[Music]
earliest time as first seen
Time by history we'll start just with
history
there and we should start seeing times
here yep all right we're going to do an
eval now uh
current time equals
now so now I've got my time and then I'm
going to
go
eval time limit equals current time
minus once again we're going to look at
the
last last days worth of traffic so we'll
go minus 864
100
where first
scene time is greater than time
limit let's open this up to say seven
days run that
query can't do a head 100 that won't
work otherwise we don't have any
historical
context
so I had 26 new connections new
histories that I hadn't had over the
last little bit all right that's
interesting let's now turn this into
nice human readable time eval time
equals make sure we're happy with what
that time comes
back I bet if we open this up to say 30
days we're going to find we we will not
get larger if we had 26 you open up your
window larger to AAR larger Baseline the
number will only shrink it can't it's
not going to get
larger if I were to shrink this to say
look at the last five days the network
could get bigger but the longer the data
set you make time you look back the less
less anomalies you're going to have it's
just that's just the way it works
because you got a bigger Baseline more
data to draw
from all right well numbers definitely
down
you know out of a 17 million events to
find 12 anomalous that's not a lot it's
still more than I probably want to be
looking at but I do think it's
interesting they all fit is two I don't
know what that means that's something
definitely worth investigating why is it
two why specifically two of these um
anyway that works but now I don't want
to just I don't care about just about
history let's do it by source and
destination
IP so if I do
this get a lot more results but I'm also
going to get a little more context I
don't know I'm going to get a lot more
results
but again as you can see it's just easy
is adding those tupal in there just
adding the different more values and now
I'm doing the exact same query I hope
you're getting this concept that with
this conceptual concept here I can
pretty much do this against any form of
data type and start to see kind of what
is the Baseline and then what deviates
from
it so it looks like the data is
relatively the same as what we got from
the other results other than we're now
getting these little UDP
ones they weren't abnormal because UDP
occurs all the time but this machine had
not been talking to my router
UDP and it had not been talking to this
machine UDP and now it is so that's the
abnormal so we can see really quickly
that we have managed to just with stats
and math we've managed to make it work
we could do a look up and attach the
look up on there but again that model
just continues to work for us all right
and we'll go let's go to another model
we're going to take it a little more
this time we're going to do we're going
to use some historical data I'm going to
push us to the Bots uh data boss of the
sock and really go into stuff that we
don't really may not know anything about
I really don't I'm just going to go look
through them and build some anomaly
detections off of it okay we're going to
give one more example using an almost
the exact same methodology but this time
we're going to use the boss the sock
data you can download that from uh just
Google Bots V2 download and you can
you'll find a get repo and you can p it
yourself and do the exact same thing but
the point is I just want to show I can
do this with any set of data again I'm
going to do host logs here's my story I
want to find anomalous login activity
and I'm going to Define an login
activity as in machine people logging
into Linux boxes that don't normally log
into them so let's find new users or
from new
locations and let's see if we can find
any logs that will help us on that path
if I jump over
to the boss of the sock data and I look
at the source types I'm going to come
down here and I'm actually going to
see the so back here here's who the
hulog well what's in the
hulog
The Who command is a command that tells
you who is currently logged into a Linux
device so if I go index equals Bots V3
B2 Source
type equals Bots equals
who do a head 100 so we can investigate
this data this data is very historical
so I'm going to use it all time not a
recommended approach but it's a set I
know the set of data it's okay I'm going
to pull it what I see here is host name
that's the main that's where you came in
from host that's what the machine you're
logging into and username that's the
username that was logging in so let's go
look at that data let's try the very
thing I've been doing this whole
time stats
count stats
earliest
time as first seen time Time
by host host name it's host name
right host name and
username run that
command boom got it okay now I jumped
here and I want to see the date my data
stops coming in on the 8 29 2017 you
know what so I'm going to go look
historically I'm going to go to Epoch
time
converter there it is and so if I put in
a
date and so we're looking
at I want to use
8828 so 828
2017
month 8 28 2018 17
17 short
memory 828 2017 cool and we'll put this
at sure
whatever there's my Epoch
time I'll just copy that out and now I'm
going to do
T
eval time limit equals here's my Epoch
time I don't need to do any math I don't
need to do now because I'm not looking
at
now and I'm going to do a where first
seing time is greater
than time
limit and let's
[Music]
convert eval
first seen time so it's a nice human
readable and that's going to
fail
okay doing
that what I end up finding
is this us this user logged in from that
machine sorry from the console to here
and they've never done that before this
is a really quick way I've now got the
ability to lock see who's logging in
general rule especially on servers you
only have certain individuals log in
from certain locations if that changes
this will flag an alert just quickly
baselined the entire
who system all right this is all good
and dandy now we're going to take
another step forward but that's going to
be using some math because sometimes
it's not about did it occur did it not
occur it's did it occur way too much did
it occur outside the norm like for
example um eror logs uh on a web server
if what is the average number of errors
that occur how many 400 errors am I
getting did it all of sudden jump from
above the average the fact that it
occurred is not the anomalous Behavior
it's the fact that it occurred so much
more than it used to to oh well that
must mean I need a complicated model
right that's now we found the reason why
you need these complicated models well
let's go see if we need a complicated
model or not all right I've got another
scenario going on and this one is we're
going to use math to improve our
SPL um I told you that we had our last
one we had some deficiencies what if we
want to we can say yes an event occurred
or no it didn't but what if we want
here's an example we have web servers
and web server request and you get a
status code back and getting a 200 a 200
code is not anomalous not is that's a
normal thing but what if all of a sudden
you get a massive Spike of 200 codes or
what if you get a massive Spike of 400
codes or all of a sudden you're getting
very few 200 codes that might be an
anomaly it's not about whether it it's
not the Boolean concept either happened
or it didn't it's based off of a
percentage you need to be looking at the
data and so what we call we use we can
use Splunk to create a moving
window to check what is the median value
figure out the absolute the standard
deviation and make an upper and lower
bound and find out if the data fits in
there I'm not going to go through all
the math as I write the code because the
chance that I screwed up is very
high what I did is I took the Bots V2
data set grabbed the web logs and what
what I did is I said let's break this
stuff up into 10 minute buckets so I
made a bin on the time field and I say
span it 10 minutes and then I ran a
stats
command um I'll keep these results here
I'll come over to this one here and I'll
show you what's going on as we do this I
run the stats command should be familiar
with this
concept let's go grab just a 100 of them
though hit
1,000 it's going to do is it's going to
go grab time chunks break them up into
10-minute chunks and we're going to get
the number the Des IP the status code
and how many it saw next next 10-minute
chunk how many it saw how many it saw
and so it's just going to have that nice
little window rolling
there then we use stream stats stream
stats allows us to keep all the values
um this is not a class about stats
versus stream stats and all that um but
the basic concept is we have a rolling
window we then get the the median count
as median by Des IP so we're basically
determining what's the median value of
count what's the median value in that
10-minute chunk tell me what the median
was and it will store that
median cool now that I do that I can do
an eval the absolute value get the
absolute value of count minus the median
that way it doesn't matter we want
absolute that way we can get positive or
negative it doesn't matter then we go
and determine the stream
stats and we get the median absolute
deviation same
principle um we do a lower bound which
is the median minus the median absolute
value times exact time 20 so here just
basically you can use whatever math you
want I shouldn't say that but we're just
we're establishing our lower bounds and
our upper bounds I'm choosing to use
median plus median absolute times exact
20 I don't have to do that I could do I
could do an absolute value function I
could do standard deviation functions
those are all supported natively in
Splunk I mean you want to see standard
deviation it's a
simplest standard deviation there you go
you can do you can get the standard
deviation value then you could apply it
um anyway get a lower bound and upper
bound and you basically say is it an
outlier if the value you're looking at
count or this count sorry this count
here is less than the lower bound or the
count is greater than the upper bound
and so we get outliers and we can see
this that this value
22 with your upper bound was 112 and
your lower bound was 32 so it fell
outside that bound and there bite was an
outlier and you can go down that you can
see how each one of those and the other
ones are not
outliers um I don't know how many
outliers we
have if you you restrict down that out
liers you could probably really cut down
on how many values come back but there's
about
45 out of all these events and basically
we can see that 172
3172 with the status code of 200 had a
spike did we get any status codes I have
no idea if we look at outlier let's go
back to our outliers where there any
standard deviation not
200 well knowing that and if in our
environment we're oh we got a
4 that that my
friend is an indicator that I would be
very interested in because maybe they've
got their the way they've got their web
server set that a 304 and these 302s are
basically 400 airs remember that they
can sometimes redirect and these might
be examples of a machine getting pegged
incorrectly this would be it's not a
ton guess the count they just actually
just barely showed the fact that they
showed up
means they don't show up they actually
show up normally they get a lot more 300
ER interesting okay so I'd reassess that
statement because they're on the lower
end and it's 300 probably not concerned
so you could actually do some logic here
where you throw out I don't care about
status codes of 200 you might say here
if it's a 200 code and it goes low I
care but I don't care if it goes upper
bound if it's a 400
[Music]
so you could do like
count
and sorry
count I can do this I can do
this come
on I've lost my ability to write in here
count is less
than cool it's not going to let me ride
for some reason I can't write in here
I've lost my ability but I could write
count is less than and status code
is greater than 400 then do this anyway
so it gives me a lot of different
capabilities to massage and manipulate
but this continues to constantly look at
the previous data and say What were what
were those bounds and so I can build a
model really really quick
again just using math now to be fair all
models are based off math but the math
can be done natively and Splunk and it
can be done really quickly so anyway
that's another example we'll continue
continue to show more examples all right
we got another example and this one is
just solely based on math I'm getting my
examples are just getting worse and
worse I'm not going to lie uh this one
here is I want to see in my logs if
there are FL the log flows if there are
spikes or lows in the logs uh so we're
going to look at time and say hey what
is the normal amount of logs coming
through and are those spiking or are
they going down um and we want to use
this plus plus one plusone distribution
which the concept of the plus I can't do
this right now plusone is it uses a
fancy math to basically predict based
off previous values whether or not these
values should exist inside the data set
that I think that's how that works um
details aren't that spec important it's
the point is that this is the
distribution I want I know how it works
so I want let's see if I can make Splunk
do it well I'm going to comment out each
of these lines as we go through
them and walk you through what we're
doing so if I come down
here first thing we do we're just going
to go grab t stats it's a fast command
go look at the last 90 days and tell me
every 10 minutes how many logs was I
receiving got 65 from con DNS etc etc so
I can see how many files are coming
through so how many logs all right so
now I've got my my data set I quickly
created a data set now I want to Pro I
want to create a
key and in order to do that I want to
figure out what the hour of the day is
I'm going to pull that hour of the day
out and the day of the week so if I pull
this
off put that down
[Music]
here we'll now get an hour of the
day and the week so Sunday at 7 Sunday
at 7 that's all we're doing now I'm
going to REM I'm going to make that a
key by using the term make a key I'm
going to make the key Source type
underscore hour of daycore day of
week now I have a unique value so I can
now using this I can use this key that I
can basically say I can compare all
Sunday at 7 o'clock was one Sunday at 7
o'clock versus Another Sunday at 7:00
and I get an idea of how the data
Compares uh at similar times I just
spent myself an easy simple key no issue
pretty simple there uh this has lots of
different applications but now I got a
key I can use it and it doesn't care
about this time now this time can be
ignored and I can use this to compare
similar
values now I'm going to use the event
stats what I want to do is I want to
find out what the percent 95 that says
hey give me the 95 95th percentile find
out out how many logs come across and
tell me what basically what number would
95% of the
logs would qualify for it and so and I'm
going to put that as my upper balance by
Source type I could put a lower balance
if I wanted to this one here is just
going to be looking for spikes then and
so I'll take that out this
in and what we end up getting is it's
looking at that Source type
and we can see what the percent 95
is 1633 for corelite cons typically
that's what I'm receiving this is
definitely not a
normal
um and so you can just see basically
that concept we then are going to go say
hey where the count is greater than
upper basically I want to I want to
remove any of these anomalies where it's
a drop drop because that'll mess up my
numbers I want to know where there were
spikes so I'm going to remove all these
smaller numbers and so I'm going to do
that step where that's greater than I
will I'll move to the next step then I'm
going to count and I make the word
cardinality those who are familiar
statistics that's the unique values how
many times it
appears there's few different ways of
saying it but basically count I'm going
to call it cardinality and I'm going to
give the average count as the average by
the key now I'm using that key fill
to compare them together and I'm
basically going to look at what the
average is and then I'm going to export
that results into let's call
this corite event count averages and so
I'm going to just print the results
right into this uh CSV for the last 90
days let that chug away it's not going
to take that long this is building my
model I just built a model with Splunk
and I can basically say hey on corite
cons at 0 o'clock on
Sunday uh at least six times I saw
things for have
3,923 logs on average that's what that's
basically saying um we can see that
Friday we really struggle there's logs
all up and down this the on Friday at
7 and Sunday at 7 it's all up and down
so there weren't there wasn't a lot of
consistent data there this was pretty
consistent the higher the cardinality
the more you can trust that those
numbers are consistent and of course
you'll get higher cardinality the larger
the the more you move your data set out
because well for example if I did seven
days I could never get more than one
cardinality because I'm never going to
cross Sunday at Zer o'clock midnight is
never going to occur more than once and
so part of this is it can only occur so
many times to begin with and then it's
got to match those uh percentages and
I'll tell you that if this isn't really
close to how many like 90 days that
would be how many weeks 70 so 10 weeks
for 13 weeks 12 almost 13 weeks you're
going to want to see a cardinality maybe
a little higher otherwise you can see
that the logs are jumping all over the
place this is probably my most
consistent log then SS s on Wednesday um
without high cardinality that matches
how many occurrences me personally I'm
going to say this data set is probably
not very good for this model but we're
going to follow through with it anyway
even though the data is telling me don't
do it I don't match up well but you know
what I never listened um so we're going
to come over here and write a different
query it's pretty much the exact same
thing the difference is we'll start
right here we'll cut this
off if we notice
everything we did up to this point the
first three lines they're the same first
three lines first three lines so we're
grabbing the same data but now we would
go grab say the last 24
hours or we go grab the last week
whatever because we're we can do a week
that's totally okay we don't have to run
this every day and then we run this
lookup file and basically what it's
going to be doing is it's going to match
on the keys and it's going to give me
the average and the cardal
that it expects for those
values so if I put this in there we
grabb the last 24 hours look at command
oh it would help if it was called
corite it's got to match the name on the
other
one corite event count now we have the
average and the cardinality when they
matched looks like we have some files
coming through on on Wednesday and we've
never had that before that's actually
kind of interesting um that we didn't
have those
logs
but it is what it is um surprised that
there were no files I guess not having
files in DNS and con that's not the mo
the weirdest thing in the world
um but we so we can see the match and
now what we're looking for is we want a
cardinality greater than 10 I'm going to
go cardinality greater than five because
we don't have very good cardinality and
so I want to exclude anything that might
be basically I'm looking at that
cardinality over here and saying you
know what I only care if I don't have a
high confidence over here
don't get rid of it throw that data out
we won't call that anomalous and so I'm
going to go where cardinality is greater
than five
five and now we're going to do build our
little pan
model oh I got nothing matching so we'll
have to go greater than
two really bad cardinality but whatever
so I got a bunch of fours all right cool
now we're going to use our P
Pon
method and I'm not going to walk through
the whole math but we're going to get
factorials it's the square root of 2 *
pi
times the count times I'm not exactly
sure what P power the power function of
count divided by the exponential of one
count exponential equals exponential
average you know what I didn't write
this I stole it but all the features you
need to write that that query are right
there and write that math then you do a
probability of power average blah blah
blah and then basically you do an
outlier if the probability is less than
03 I can modify that I can say I want my
threshold to be
01 and basically then I will give myself
outliers you take these tick marks
off and I can see the data that's
outliers in the last 24
hours and I don't have
any I'm not actually that
surprised just didn't have enough data
to make it work
but let's just put that in
there actually kind of interested that
there's
nothing about outliers
here all right we'll open this up open
the floodgates go the last seven
days
and anyway if I had the outliers they
would show up
and but that's just an example again I
can just pump in the math right here and
I can see the values I don't have we
we'll take the probability and
go back to three I don't think I'm going
to get anything either
way less
than3 it's I'm not going to get
anything but again you can put
complicated math in there you can run
the functions um you'll just have to
imagine the outliers actually popped
they didn't but they they could they
should if there was actually outliers
that it was detecting there
um
yeah I just want to see this what's
going
on really weird out by the fact I'm not
getting all those
fills let's just do it by Source type
equals corite con
I want to be able to look across all
those different fields but that's still
weird those
Fridays anyway so now I got my cons now
if I do
this
there something
weird we didn't match
anything
okay let's just not eliminate anything
here okay a lot more values
zero okay now we come back here run
this okay we're matching that's a lot
better
by the way if you notice it's 2:41 in
the morning I am
clearly
tired
so
why just going to throw you out all
those cardinality should have match to
anyway oh sheesh
I'm this is working
okay if it's less than3 it's a one if
it's a point if it's not it's zero so
none of these fa I could try to move
this out a little further see if I get
any
ones
what I want to see then is just do me a
do me a solid here and just give me the
uh anyway um I wonder if I keep I don't
remember which way I need to go do I
need to keep extending out the zeros or
go the other
way
well I got to figure out exactly where
the math is off but the concept is I get
all the way down to the
probability see if I flip this to nine
that's
fine I get a different value so there
must be a copy paste error from when I
did this example I'm I don't want to go
any further on this um not my best
example but I just wanted to give the
concept here that you can do math right
there in your in your stuff and it
didn't take that long to build this
function again if you want to use the
pan method all you need to do now is
come over here build your key come over
here apply the key and you're
done all right I have given lots and
lots of examples and I've never actually
used any actual models out of what we
cons what is typically considered models
so our next example we will actually
show using a model inside
Splunk mltk