...security.
It's been recently spun back out, so it'll probably become McAfee again as of Q2 next year, but that's neither here nor there.
So, Past Lives.
Again, I was over at Mandate FireEye for about three years.
Really loved it.
Awesome company.
Really cool stuff.
There are some awesome FireEye people in the room, so if you get a chance to talk to them about this platform I've got some screenshots up of today, do that.
Prior to that I was a DoD contractor.
Worked in hot, dusty places for many years.
Spent about six years out in Iraq.
About seven months out in the Horn of Africa.
I don't recommend it.
It wasn't all that fun, but I had a good time.
Prior to that I was in the Army, and before that I started out in the scene.
I was a 2600 nerd from the time I was about 13 years old.
What's that?
2600?
So, when we talk about hunting, there's a distinct problem set that we want to solve.
We want to find evil and we also want to find ways for evil to do evil things.
The third thing we want to do with the hunt program is drive maturation of the security program, and very specifically the monitoring and detection capabilities within that security program.
Is there anything that I'm missing here?
Can anybody call out anything that might be missing?
Okay, we'll keep going.
When we talk about traditional security operations, we have a number of objectives to accomplish, and we want to get away from saying that we can throw a tool at the problem set.
We just can't really do that.
I tend to look at security as a people , process, and technology problem.
These three things have to be working together and functioning together to have a working security program.
Hunting as a concept is really distinct from investigating alerts or developing automation, although hunts can bleed from those activities or into those activities as well.
When we talk about hunting, it really is a set of methodologies for analyzing large data sets in search of incidents and context to fuel future automated detections.
Again, it is not tools, it is not alerting, and it is not automation.
I'm not here today to teach you about hunting, but to provide more of a generic framework or scaffolding of sorts to build on to use when thinking about hunting and implementing hunt-derived detections in your own environment.
And guys, there's a lot of content here.
I'm going to try to move really fast.
We started a little late, so bear with me.
Great.
Yeah, let me know if I start running over.
So, hunting as a methodology.
This is a process for developing procedures.
It's not a process itself.
And so I wanted to make sure that we define that before we continue this presentation.
From a threat hunting perspective, when we want to think about kind of an OODA loop, if you have any background in military operations, you'll know what I mean there.
The first step that we want to take is to develop a hypothesis about our environment.
So starting from a hypothesis allows analysts to attempt to debunk assumptions about the environment.
And we'll get into what some of these hypotheses can look like later in the presentation.
Second step is we're going to use our tools and analysis techniques to attack the problem set and validate our hypothesis.
We're going to uncover new patterns, learn new things, build on that, enrich our analytics, improve our detections overall.
And so we want to continue this loop and have this functioning loop, feedback loop in a threat hunting program.
And foundationally, there are a few things that are needed to get started, right?
Very minimal, basic things that we need to have.
Firstly, we've got to have someone who cares about the security program and can sponsor the security program and maybe even at larger organizations sponsor the hunt program very specifically, right?
They've got to understand what the reason is behind this.
Why are we doing this?
How is it going to improve our operations?
And is willing to give it a chance and to pilot it.
We also want to have some type of functioning security operations and some type of functioning incident response, even if that incident response comes in from outside, in the case of a Mandiant or some other organization that is contracted to do IR for you.
The next thing we've got to have are the tech and the tools.
So we've got process at the very bottom, we've got people on top of that, more process, and then we've got tech and tools.
It must be this tall to ride, meaning you've got to have these three things in order to establish a functioning hunt program.
Afterwards, when you want to start driving maturity up this pyramid, you want to start improving the security program and doing better things and improving your governance and just improving detection overall, you're going to want to start developing use cases and playbooks that tie in with your incident response process and make sense for your organization as a whole.
And then at the very top of this pyramid, we see that foundationally we have some really solid things in place to help us build a formalized hunt program.
But you don't need all of those things at the very top there to build a hunt capability.
We're talking about building a hunt program, which is a lot more formalized.
If you want to build a hunt capability, you really only need to be in the middle there.
You've got to have some technology and tools.
You've got to have some information flowing in.
You've got to be able to handle the information that you have.
You've got to have somebody that cares and is willing to pay for this or spend some time on this.
This wonderful hunt maturity model comes to us courtesy of Squirrel.
They're a security-focused big data startup out in the Boston area.
And after assessing many large organizations' capabilities at Mandiant, I worked on security program assessments that are holistic.
You go all the way from the top in the C-suite down to the lowest engineer to talk about how things are done in the organization.
I've done a lot of these assessments.
And the conclusion that I've come to is that most orgs without some type of a formal hunting program in place today rated about a 1.
Most organizations that have some type of hunt capability, whether it's a formalized program or not, are maybe at a 2.
Is anyone here at a 1 ?
That's where we are with hunting.
That is the state of hunting today.
That is a pure reflection of it.
No one really knows what this is and don't know how to implement it.
So that's why I'm here today.
So again, we talk about things that we need to do.
We need to have that executive sponsorship.
We need to iterate through a few projects before we reach the point where we can hunt and have a formalized hunt capability.
Some of these will be easier than others based on the resources that are available to you.
Once you've figured out what your plan is and what you're going to do, the next step would be to develop some type of a logging strategy.
You've got to know where your information is going to come from.
What already exists in the environment?
What does IT have?
They might have a Splunk instance that Security Ops doesn't have access to that has a lot of really good information.
There's a lot of information in the environment.
You've just got to figure out what you have, where it is.
Is there any consistency between these log sources?
If you have two devices in the environment, you've got, say, two Cisco iOS devices and they're at the same version, same device, you want to make sure that you're collecting the same information from both of those devices.
The log level needs to be set to be the same.
Hopefully there's some type of configuration management in place so that you know that the devices are doing what they're supposed to be doing.
If they have a security function, you want to make sure that those are enabled so that those can be logged.
And that's distinct from setting the log level.
And then you want to aggregate, centralize, and process your data.
You've got to have somewhere to
put all this stuff together.
It's got to be in one place and then it's got to be searchable in something really fast.
Really fast search.
Otherwise you're going to spin your wheels.
You're going to be doing a lot of the old school security operation stuff.
You're running regex against really large text files and doing a lot of transforms and stuff like that manually when you can have a tool kind of automate a lot of that process for you.
And in order to drive maturity in this program, you want to build those use cases we talked about or I mentioned and we're going to talk about.
We want to make sure we have the right data.
Do we have the right data to detect the things that we're concerned about?
And we want to make sure that we have the right tools.
What are the tools that we have available to us?
Thank you.
That is awesome.
Excuse me.
Heck yeah, that is awesome.
Sky.com.
Do we have the right tools to do the job?
If we don't have the right tools, what do we need to do to get the right tools?
What meets our requirements?
We can talk about venture selection later on as well.
And reintegrating hunt data.
The outputs from your hunts need to be reintegrated to security operations and that is how you justify your existence as a hunt program.
You justify the people that you have to hire for this and the hours that you spend on it.
When the metrics in the organization, that time to detect metrics starts going down, that makes management really happy.
This is a diagram on what it looks like in terms of driving that maturation.
The hunt missions are going to come up with three types of outputs.
One is no data.
We can talk about that.
The other is evil.
And the other is a risk that isn't evil, but it may be a way for evil to do an evil thing.
So we want to address that as well through security operations and what we can do to secure our enterprise.
From a SOC perspective, they need to reintegrate the lessons learned from incident response.
Things happen, bad things happen.
How do we keep it from happening again?
There needs to be some type of after action review on an incident response so that you know how to keep it from happening in the future.
That information also needs to go to the detection team.
If you had an insider threat issue... I was just talking about this earlier.
If you have that kind of issue and you need to monitor whether that employee's username is being used after he or she leaves the organization, that's some important stuff that you're only going to get as an output from the incident response and the forensics that were undertaken,
etc.
So you want to make sure that you're feeding that information back in.
Now, security operations has a responsibility to update the use case.
And when I talk about security ops, detection falls under that banner.
The key is for new information that is received by the organization, wherever it comes from, to be reintegrated to security operations as a whole with a focus on detection in order to build and improve this HUNT program and drive maturity in the HUNT program.
Does all of this make sense so far?
Yeah?
Good.
So, I decided to put this slide in last night.
There were some last minute changes at Bourbon Con in my room last night.
And I decided to put in some information about what a use case is because I feel like there's a lot of confusion about this in general.
People that don't come from formalized engineering disciplines are going to have a lot of issues understanding this concept in general.
I had a hard time wrapping my brain around it because it can be really as specific or as general as you want it to be.
The core of what you're trying to do with a use case is say, here's this goal that we have.
And here are all the things that we need in order to fulfill that goal.
Who cares about this?
Who are my stakeholders?
What systems can I use to process the information?
What are the sources of information?
Where am I going to get this stuff from?
And then what is the logic involved?
What actually has to happen for the detection to occur?
Is it just a multi-stage correlative rule in a product like ArcSight?
Or is it something that a hunter actually has to take a few cognitive steps in order to reach that conclusion?
So, you want to get better and better.
You want to automate more and more and get away from the human as much as possible.
And that's what this process is intended to help you do.
Building that use case, it doesn't have to be a complicated document or anything.
It can just be in Excel if you want it to.
But it really helps organize your thinking and formalize detection in general.
You don't want to just have a bunch of alerts in a sim.
You want to have some type of reference about those alerts.
Why they went in.
Whether it's tied to an incident or not.
Is there a ticket in the system for it?
All that kind of stuff.
This is another definition that I decided to add last night.
I wanted to explain the relationship between incidents, events of interest, and a detection use case.
The detection use case is wrapping over all of this.
It specifies certain events of interest that we're going to be concerned with that are going to fulfill the criteria for the use case and reach our detection goal.
And then, of course, when you have events of interest that you have confirmed as something malicious or undesirable or risky in some way, then you're going to go ahead and classify that as an incident.
Any questions on this slide?
I have a couple slides on building use cases.
They just look like this.
It's a design tree.
We talk about the objective of the use case, the goal of the use case.
You want to start mapping out behavior so you can understand the relationship between the stage in the kill chain or the attacker life cycle that you're using or the MITRE ATT&CK model.
If you guys haven't seen the MITRE ATT&CK model, I highly recommend that you check that out.
A few of us here have met the young guy who built this model and it's awesome.
They really are soliciting a lot of help from the commercial world right now on that.
And they're happy to credit any organization that is able to add useful information to the ATT&CK model.
But it's very detailed and it goes through all these different stages and all the different behaviors that you can expect from threat actors as they carry out their mission.
And so we want to map what we're doing with use cases and what we're doing with detection directly back to behaviors and activities that we've seen threat actors undertake in the wild.
Those TTPs, whatever they are.
From a tools and capabilities perspective, doing some design mapping on your use cases helps you figure out where the gaps are.
So I highly recommend that you do that.
And hopefully this gives you an idea of how you can potentially apply this in your own environment.
And where it says AOI there, over on the left, capabilities map to AOI, that actually just means events of interest as well.
Activities of interest, events of interest, you'll see those terms used interchangeably in the market.
Some companies want to use one term and some companies use another.
It's pretty relevant.
From an outcome perspective, the biggest thing I want to convey with this slide is that there is no such thing as a failed hunt.
You're either coming out of it with no information, which means you either need to evaluate the effectiveness of the hunt mission or you simply didn't find that activity in the environment, or you came up with something evil, or you came up with something that is undesirable.
And either way, you have a set of steps that you can take to take that mission to the next level and improve security operations overall.
When sorting out your data and looking at the data available to you in your environment, I wanted to include a slide that gives you an idea of the varied types of information that you can collect and process as a hunter.
It's not limited to the traditional stuff.
Those three boxes on the left, security network and endpoint, that's all the traditional stuff.
And then you go less and less traditional as you move over.
And actually, I think the threat intel and IT buckets could be shifted around because after security network and endpoint, I think people shift towards threat intel now.
That's the way the market is moving.
Everybody wants to know, how do we integrate threat intel?
And then the HR bucket all the way on the right there, that's the least likely stuff that you're going to find in any organization that has a hunt program.
The integration of physical activity, physical events in the real world where somebody walks into a data center and they log in with their prox card and then they get on their machine and they do other stuff.
There's not a lot of correlation done even in tools that can support that, like ArcSight, because it's just kind of a next generation capability.
This is not stuff that most organizations are doing today because it's simply costly.
That's a great comment.
I used to work for the defense department as a contractor and for a time I was a sys admin and then systems engineer, network engineer, etc.
And we would have these programs of record on the network.
So very much similar to a custom application in a traditional enterprise and these applications were the wild west.
They just did whatever they wanted to do and did it however they wanted to do and the only way that you could really figure out what was going on was to either get a detailed accounting from the program office as to exactly how their software works and what it does or you could go on the network and take a look at the data yourself and figure out what protocols are they using.
Is it encrypted?
Do I care about the way that this application is doing business?
You see what I'm saying?
So, great comment.
Any other comments on this slide?
Excellent.
So, two types of events.
Two types of events, right?
Observed and synthetic.
Again, there aren't a lot of formalized definitions around these terms in the marketplace, but it's important to note that when I talk about observed events, I mean something that came directly from the device that processed the data, right?
So, let's say you have a firewall and a proxy and the proxy is behind the firewall and the user makes a request and it goes out through the proxy through the firewall and it goes out.
Those events that come from both the firewall and the proxy are both natural observed events that came from those devices.
So, that's what that means.
From a synthetic perspective, we're talking about some type of automated analysis of the data.
So, in this instance, let's see what I pasted in here.
This is a screenshot from the FireEye Threat Analytics platform and what this event is telling us in the bottom is there was a synthetically generated alert event for potential data theft based on a ratio of sent to received data.
That's all it says, right?
But that's a synthetic event.
There was not a system that processed the data that we're concerned about that generated that, right?
It came from an analytic or some type of post-processing that was automated.
Make sense?
So, what is the right data?
One, original source data wherever possible, right?
We want these things to come from the original system because it's been through less steps, right?
Less manipulation, less changes.
It's going to be closer to the source truth.
And we want to collect as many sources of those as possible.
So, again, if an event transited the proxy and then the firewall, we want the events from both the proxy and the firewall because they tell us different things.
They contain different information.
And it's going to be complementary information to you when you hunt.
You want to make sure that the important metadata is there, right?
The fields that you need to have, right?
Such as... I don't know.
In an HTTP log, there are so many parts of this that are so helpful when you're hunting, right?
I mean, obviously, the source and destination IP and port.
You're going to have the method, which is really important.
Status codes, which are really important.
So, there are a lot of different pieces of information that is in this HTTP log that is going to help you contextualize and pivot between different data sources and really get an understanding of what the traffic was without actually diving into a PCAP.
So, again, synthetic events can provide useful context, but they are context, right?
We want to default to the natural events, if possible, or observed events, if possible.
And furthermore, you want to be able to pivot, right?
So, in the threat analytics platform, as with a couple other similar hunt products in the market, you can kind of pivot between data sources, right?
So, if I notice that I have an interesting test IPv4 here, I need to be able to take that test IPv4 and pivot and see what other data sources that test IPv4 exists in and kind of narrow down from there.
So, in order to get all this stuff ready and get started with hunting, you've got to understand your network and know what you're working with, right?
There are just a disturbing number of organizations that have no idea what their critical assets are, what's important to them, where the intellectual property is housed, where the developer network is, what goes on there, all that kind of stuff.
You've really got to understand what you're working with.
And sometimes the only way that you can do that as in the case of us over at Mandiant running assessments for clients on a short term basis, the best way to do that is just to dive into the data, right?
That's how you kind of eliminate your assumptions and your bias, right?
Instead of relying on some network architecture diagram that somebody's giving you that might be six months, a year, a year and a half outdated and doesn't have any useful information, you can say, hey, I'm actually seeing something different, you know?
You want to build that logging strategy.
You've got to be able to know that you're receiving consistent information or at the very least understand the context of the information that you're receiving, right?
At the very least know all the Cisco devices in my environment are set to log level warn or whatever it is, right?
And the time source is pretty important.
Hopefully your platform has some type of standardization on times such as tap, the Threat Analytics platform standardizes to UTC.
But you really want to hopefully have really good NTP implementation in your environment so that everything is kind of standardized and synced.
It's probably best to use GMT or UTC if you can get away with it.
A lot of orgs just won't let you get away with it.
And you also want to be plugging into any asset change or configuration management processes that exist, right?
People make changes all the time to machines and they basically do whatever they want, right?
There isn't tight control in most organizations over what happens, over admin changes or pretty much anything.
So the more you can plug into this, not only can you improve your detections and come up with new and interesting things, but you can also do verification and validation and say, hey, I'm not sure if this is malicious.
I'm not sure if this is an incident.
I'm seeing some really suspicious traffic, but it could be legit.
Let me go back and take a look.
Is it a custom application or something that I just wasn't aware about floating around in my network?
Yep .
Yeah, and that's a great point.
It's kind of the feedback loop and the reintegration of information eventually gets you in your eyes on glass all the time.
You get tired of looking at the same stuff.
You're not going to be looking at the same stuff.
You're hopefully looking at a much lower volume over time that turns downward over time of stupid stuff on the network or bad stuff on the network.
So accounting for other organizational use cases, what I mean by this is there are other groups, as I alluded to, within the organization that have access to data potentially that could be very useful and helpful for you.
And the converse side of that, the other side of that, is if you're looking for budget for something, say you want Splunk and you can't get it in security, you might be able to get it through IT.
They might already have something or they might have a bigger budget.
They usually do than SecOps does.
So you want to share information with those teams and figure out how you can work together and share information.
And the last, well not the very last thing you want to do, the before last thing you want to do is clean up the data set.
You've got to make sure that you have some normalization, deduplication, two of the same event in there.
And you want to make sure that it's parsed properly.
That the fields are being picked out properly from the events that are coming into your sim or hunt tool.
And again, that you're able to use those bits and pieces of data to pivot between other bits and pieces of data to form a complete picture from a forensics perspective.
The last thing you want to do is enrich and contextualize the data set, and we can talk about that.
From a contextual perspective, again there's that environmental data that we have but aren't necessarily plugged into, like asset management, configuration management, et cetera.
Previous hunt and IR output is going to be helpful as I mentioned.
You've got to reintegrate that information and know that you shouldn't be starting from square one.
If there's an IP that your organization, I'm just using an atomic indicator as an example, but if there's some malicious IP out there that is threat actor controlled infrastructure, you guys have dealt with it before, and then a year later you guys see it again,
that should pop a red flag immediately, that should be at a higher severity, and that shouldn't be something that you're just now trying to find out about, like let me get some context on this IP, you guys should already have that information internally.
So I highly recommend you look into some knowledge management solutions for how you store and organize your incident response data and after action.
Malware analysis of course, that context is very helpful.
For example, if you sandbox some malware and you happen to pull out a bunch of domains and you want to reference that information a year later because the threat actor still controls that domain or whatever, that's really useful information you want to be able to extract all of that again and store that somewhere where you can hopefully apply your data set to all of this.
That's the goal here.
In an ideal world, you would know that if an admin makes a change at 3 a.m.
and no admin is supposed to be in the office at 3 a.m.
at that site, then there's a problem.
And that's really what we want to get to is this awesomeness of automated detection over time.
And then these analytics.
Geo-infeasibility is the thought that the same user is not going to be logged in from two geographically disparate locations at the same time.
If he's in Bombay and then he's in Florida and that was five minutes apart he either shared his login or he's been hacked.
That's a problem.
You've got to look at geo-infeasibility.
Beacon detection is another very common one.
That's important.
You've got to be able to identify C and C without having to reinvent the wheel and look for it on your own every time because it can take many different forms and I think you guys have probably seen some of the stuff out there about encoding C and C in DNS packets and changing some headers around and doing some interesting stuff.
We're going to need to work harder and harder as defenders to get ahead of the curve.
And then DNS entropy.
Data exfiltration is another one.
How do you detect data exfil?
There are so many different ways to exfil data.
Somebody the other day showed me a fun way to do it through the New York Times website.
That was awesome.
So the only way you would find that is if you saw really large outbound packets to the New York Times.
And it would be a certain amount each time because that's the limit that they can fit in that text box to send out.
Anyway, cool stuff.
Internally sourced intelligence.
Attack trees.
If you do any type of modeling for how threat actors will be reaching into your environment from the outside and what bits and pieces of your enterprise they want the most access to that are most valuable, then you're going to want to use that as well in your detections.
You've got to be able to use your knowledge of critical assets and of the environment to focus your detections on where it matters most.
Because let's face it, some parts of the organization don't necessarily matter as much as others.
And at the same time, you have to be aware that the least defended points of entry are going to be where threat actors are going to target for lateral movements.
So two sides of that.
Red team pen test output, of course.
TTPs from previous incidents.
That's actually previous hunt and IR output.
Same thing.
And then deviances from baselines or expected behavior.
This is a great one.
If you see a machine come online and all of a sudden it's reaching out to a certain external port and no one on that subnet has ever done that before, that's kind of weird.
So you want to be able to very quickly look at the anomalies and know what the deviances are from the expected behavior.
Hopefully you've got some information about these programs and applications in your network to know, yeah, this one application is supposed to be communicating on port 443 out to whatever network.
And you're able to reference that information, which will live in IT, typically.
So most businesses don't have some type of a risk profile or threat context.
It would be really helpful for you to get even a one-time assessment by a company that focuses on that kind of work.
Because it's really helpful to get a third-party perspective as to who's actually targeting your organization, what their motivations are.
That just gives you an extra piece of information.
You guys obviously can do this on your own.
The last bit of enrichment that we can add in our HUNT platform to these events are externally sourced intel.
This is what we commonly think of as threat intelligence.
Paid feeds and then whatever OSINT we can get.
Hopefully you can see this should be in every HUNT platform.
You should be able to do resolutions.
You should be able to do some geographic data analysis.
That should actually be enriched and available with the event as you're looking at it.
You shouldn't have to go to another window or, from an efficiency standpoint, go to another process to see whether an IP address is located in New York or in Bombay.
This is important information that helps you contextualize what you're looking at when you're hunting and actually looking at events.
You're going to need that.
And then ISAC, InfraGard, etc.
Whatever organizations are out there that are sharing threat intelligence with people and making it available.
From a tooling perspective, there are some bits of criteria that we want to talk about that help support a working HUNT platform.
The first thing is that rapid search.
You've got to have a really high quality UI and hopefully there's an API so you can do some scripting and some additional automation there.
You want to be able to stack the dataset so that you can group and reduce it.
This is really what we're doing here.
Looking for outliers, looking for things that seem odd, out of place, don't necessarily make sense with the information that we have.
It's just confusing to us from a security perspective.
Why do we have Telnet port 23 transiting our network and it's all unencrypted?
What is that?
And we want to be able to pivot, again, because we've got to be able to take those individual pieces of information and move between different sources of data on those bits of information that were parsed out from the events.
And some of the things that are nice to have, again, the tagging and enrichments.
Tagging is great.
If you know for a fact that you have, I don't know, let's just say that you want to tag your domain controllers.
These are high value assets.
You want to make sure that you get notified every single time that there's an administrative change to a domain controller or you want to start your hunts from there.
Let's not even talk about building the automation.
Let's just say, I'm curious about what's happened on my domain controllers in the last month.
You should be able to pick out that information very easily and tagging will help you do that.
Of course, Intel integration support, you've got to be able to match on atomic indicators, so MD5s, SHA1s, fuzzy hashes, fuzzy hash impash, IPs, domains, all that kind of basic stuff.
You've got to be able to filter on that stuff.
If you're not doing that today, that's bad.
You should do that at the very least.
And then you want to have some type of automation.
You've got to have rules and learning so that you can improve your detection over time.
When you talk about tooling and how you choose your tools, it's tough.
You've got to really narrow down what it is that you need and then give a little bit of extra wiggle room for more features that you're going to need down the road as you improve your HUNT program and drive maturity there.
And you're going to want to go through a tool and vendor selection process, hopefully something that makes sense and that has some type of success criteria tied to it.
If you guys ever work with any vendors and they tell you they have a HUNT platform and it works really well and all this blah, blah, blah, you'd have some very specific criteria for what constitutes a success when they loan you this appliance or cloud service and get it up and running with your data.
Multiple tools provide diverse perspectives.
You can't hang your hat on one tool.
There are so many out there.
And again, they provide diverse perspectives so make sure that you're not just kind of buying some vendor hoopla about how their product will solve all problems because that product doesn't exist today.
And there is a lot of free and open source software.
I mean, obviously that is not a complete list, but that's some really good stuff up there.
I would recommend looking into anything on that list that you're not familiar with.
Great software.
How am I doing on time?
This is a wall of text.
I'm not going to go over all of it.
These slides will be posted afterwards so you'll all be able to get a copy of these, but it's just a way for you guys to start thinking about how these sample hypotheses can be applied to your environment.
We talked about that HUNT process and building the hypotheses.
We just want to make assumptions about what we're supposed to be doing and seeing in the environment versus what the actual reality boots on the ground truth is.
These hypotheses can be intelligence driven, they can be driven by situational awareness, like enterprise risk assessments or company or employee level trends.
Anything that you guys are seeing can help develop hypotheses.
So more data, more problems.
I'm going to kind of pause on this slide and let you guys read this.
People do some stupid stuff on the network.
And it's usually legitimized and it's cool, but it's not cool.
We all know it's not cool.
What follows here are some case studies.
These are things that myself and the other two guys that put this deck together and some other people that we worked with.
These are all things that we found in actual client environments.
Things that we were quite dismayed by.
And in some cases, we weren't really dismayed, but the client was because they had a lot more context about what was going on than we did.
So the first thing is remote access.
The first hypothesis that we have here is that all access to our environment is conducted using approved means.
Yeah, there's usually somebody that laughs.
Yeah, it's because it's ridiculous.
Some of the discoveries that we had, and I put these together, you can see here.
We had VNC from the production network not good.
RDP.
It's just a lot of really bad stuff.
And so I'm not going to read out all the recommendation information and the stuff that we provided to the client, the things that they should do, because I think you guys can figure that out in your own environment.
You know what the things are that need to happen generally.
And you know why a lot of this stuff might or might not be bad depending on the context of your particular organization.
Corporate data only stored in approved locations.
That's a big joke too.
It's all over the place.
It's in places we could never have imagined to find it.
We found the things like sensitive data stored on unencrypted and infected media.
Unrestricted use of cloud providers.
So there's no service level agreement in place.
And everybody's using Dropbox and Box and pretty much every different cloud storage service and there's no way for them to kind of tamp that down.
Because people don't want it.
And then like unmanaged source code repos.
There's a lot of that.
Sources of intellectual property that companies don't even know about.
Some application again that was developed in-house or just some dev went off on his own and built his own machine and it's sitting on the network and no one knows anything about it.
That's not good.
The proxy infrastructure is properly configured.
That's a great one.
I love looking through proxy data.
If I could only pick like two sources of information that I could use to start hunting today, it would be DNS logs and proxy data.
Very rich sources of information.
DNS is a smoking gun and proxy logs are just hilarious.
There's so much good stuff in there.
You wouldn't believe what users do, right?
You start looking at it, you won't believe what users are doing in your own environment as well.
It's ugly.
And the worst offenders are usually in IT and SecOps.
So some of the things that we found from a proxy infrastructure perspective, right, they're not... they're implementing Bluecoat, they're spending hundreds of thousands of dollars on Bluecoat, but then there are no categories that are being blocked and you can just do a very simple ratio analysis of accepted to blocked just to get a look at what the Bluecoat configuration looks like.
They weren't blocking executables, that's bad.
And then they're not logging some really important stuff.
That last one there called x forward 4 that actually tells you in the proxy log what the originating workstation was that made the request.
So it's kind of like people that have DHCP on the network but they have no way to track back what a physical host actually owned a DHCP lease, right?
Not good.
Five.
Approved protocols, all the protocols are approved and they're secure and everything's good to go with that.
Of course we found a ton of stuff.
They swore up and down that they didn't have these things, they did have these things deprecated protocols such as SNMP version 2.
Then just stupid stuff like IRC and Tor.
Sometimes it's somebody trying to connect to a gaming network and then other times it's bot command and control activity.
So definitely some weird stuff going on in the network and it goes back to why are you guys allowing all this stuff to transit the network when you have all these choke points to stop the traffic.
So it just kind of brings things into the light.
Known and approved client software, right?
There's no unknown or unapproved software on the network.
That's a great one.
I should mention that the way that we identify a lot of this stuff is by using Bro Network Security Monitor.
If you guys don't know about this, you have to know about this.
It is the perfect complement for your hunt operations and when you pair Bro NSM that is performing a protocol analysis for you and breaking out all these different sessions in a way that is understandable to a human, you're knocking it out of the park.
If you could only do one thing today, it would be set up Bro, learn Bro, figure out what you can do with it, and you're going to be a much more powerful hunter tomorrow than you are today.
Privilege management, you have really stupid stuff that people do with accounts.
There's no privileged account management.
People are actually physically using service accounts.
There are weird usernames.
At one place we found a username that was like LeetDude56.
It was like some guy hacked in and was running around doing stupid stuff.
A teenager, actually.
And so a lot of this other stuff.
What's that?
Those damn kids.
Get off my lawn.
I used to be one of those kids, so they're not all that bad.
I'm moving fast.
Security architecture.
Is our architecture configured in a way to block or stop the threat or detect the threat or whatever the security appliance is supposed to do?
In some circumstances we have devices that have security value that don't have that security configuration enabled.
We need to fix that.
We can find it out through hunting.
When we see that a device such as a Cisco ASA is getting hammered with scans and it has scan detection built in, so why isn't it enabled?
And then sending us those alerts, which we can then use as threat intelligence to find out who's coming after our infrastructure.
That's really important information that can come from a hunt.
Process execution.
I don't think I need to tell anyone in this room why process execution logs are going to be important.
What are people actually doing on the endpoints and doing on the host?
I typically focus on network data and event logs.
Host-based data is a rich source of information.
If you can get in a hunt platform, you're going to be way ahead of the game.
And DNS, which we don't even really need to talk about DNS either because there's so much crazy stuff that you can do.
Networking depends on DNS.
Malware depends on DNS.
Many attacks depend on DNS.
So you want to perform detections using DNS data.
Thinking ahead, we want to ensure successful outcomes.
Our goals are to reduce the attack surface of our environment, harden our environment, and we want to improve detection and monitoring.
Do not bother hunting without using the outputs.
If you take nothing else from this, it is that you cannot hunt without reintegrating the information because you're not going to be able to hunt for long.
You're not going to be able to justify your existence for long.
And metrics.
Nobody does this, but if you can do it and you can prove that your hunt program is driving down the time to detect, you're going to justify your existence even further.
I'll wait for you.
And then moving forward and thinking ahead, we want to take hunt methodology from an art to a science.
Right now it's very rooted in your individual experience.
What did you guys know before you came to your organization?
What do you know about your organization?
That introduces bias.
And so what we want to do is borrow some of this structured ways of thinking about this problem and the analysis problem and apply some kind of rigor and science to it by using the CIA model, which is system one thinking, system two thinking.
System one thinking, back in the day, the spies would just kind of do their own thing and just kind of relied on their own experience to make judgments.
Now we want to try to de-conflict that, especially when you have a larger hunt team.
The bigger the hunt team, the more you need to have these de-conflictions and have people help each other come to these solutions.
So I think we're done.
Can I do a show of hands real quick?
Show of hands, please raise your hand if you've done some hunting in the past.
Alright, please keep your hand raised if your org has some type of hunt program in place today.
Oh goodness, okay.
And then can you put your hand up if your org doesn't have a program but you're thinking about building one or thinking about doing this in any way?
And then the last one is, please raise your hand if you learned something new here today.
Hell yeah.
That's awesome.
Alright.
So we'll do a little couch con here.
Usually we do a little bit more.
Especially for Jackie, because it's her second time giving the talk, so she's got it down, so I should have been poking jabs or something.
I didn't ice her, so I guess that's...
I was being nicer than I was to Curtis.
So one thing, an observation that I would make from the talk and from some of us that may or may not do hunting that I always found was amazing was just asking somebody, especially when you're sitting down with a client or even internal customers or just internal teams,
you say how do you know that this isn't happening?
And usually the answer is, well we don't, right?
Just because.
And a lot of times when you're looking for the tooling to do things, you don't even need to go buy anything.
It's just that, how do I test?
It's literally taking a very straightforward and strategic approach to how you're answering a question.
I have a question, what do I need to answer it?
And it amazes me how many times, I'm just curious, a show of hands, how many of you it's usually, the first thing that usually comes to mind is like, oh well we just go buy something.
Good.
Okay.
Most of the time people try to conflate hunting with audit and compliance.
What are the audit and compliance folks actually looking for?
It's like the bottom of the barrel when it comes to detection and there's no real verification.
So if you want to have some kind of boots on the ground verification that people are doing what they say they're supposed to be doing in the organization, you want to improve things and make things better, this is a good way to do it.
Any questions?
I actually have prizes for good questions.
You're like my friend that you can't do that.
Okay, yeah, no prize.
What's up?
Yeah, so the question was, do I find in our work, have we found that clients have some type of relationship with their cloud service providers that is sufficient to enable logging?
Most of the time, no.
I would say that only the largest clients that we've seen that can really motivate those cloud service providers to make some changes to their logging infrastructure or to have that capability or develop and implement the capability that allows clients to receive logs.
That doesn't happen very often.
We had one at FireEye.
Huge financial company based in California decided to move everything over to AWS and there had to be some significant work done between FireEye and AWS to get them to enable detections through improved logs.
The strategy that made sense.
Yeah, you can.
You can get a hell of a lot of good stuff now.
Much more than you could before, but that's really driven by the clients.
If you guys work with any device manufacturer or organization in general that doesn't have good logging and can't give you that information, try to drive them towards that.
The more people that ask them for it, the more likely they are to implement.
Question?
And that's kind of what I was... that's definitely kind of at the heart of what I was getting at earlier is because usually when they say that, it's like, well, how do we know?
Look for signs of things that matter to you that are intruded upon.
What are we doing to look to see?
Just like when you come home and you see the door kind of cracked or something like that or you see where somebody jimmied a crowbar in there, you're not going to think, huh, we're good.
Obviously, there's going to be something.
There's going to be trails and breadcrumbs of things that happen.
But that's the very tip of the iceberg.
There's obviously a lot more that goes into it.
So either looking for remnants of behaviors or activities or...
Yeah, so like she mentioned MITRE, right?
So they have a platform called CRITS.
So in our organization, what we do is the guys that are starting to do the hunting, we'll put those indicators back into CRITS and then I've automated and written some code to glue that CRITS information to come back down to, like in our case, Splunk platforms,
which you can do with any platforms because it's REST API.
So you can cause all that to then feed back and show up in all the detection and alerting.
That way, those guys finding that stuff shows up in detection and alerting and that's something you can measure and show back to management.
Yeah, you can't improve what you don't measure, absolutely.
If we think about it just like we do application monitoring, like when we have a fault state in an application, a critical application that takes our website down or takes down our performance platform, we look for those indicators in the code.
And if it doesn't exist, developers go back and write debug options or they write error handling or they do other things to actually detect those scenarios.
This is exactly that same thing.
It's a maturation of detection processes exactly like you would, in many instances, debug or write improvements to an application.
But we're talking about something a little bit more broad than that, that we don't necessarily have control over, which is an infrastructure or a network or system or an entire organization.
So the scope is a bit more broad and the approach to how you solve it is a bit less obvious.
So if I understand the question correctly it kind of has to do with how do you justify your existence to get started.
So from a metrics perspective it's interesting because you could do this in a very low cost way.
It's just going to take some time to build an ELK stack or use Security Onion to bring some data in and generate some useful information.
The problem is that your metrics are going to go up before they go down.
You're going to have so many new detections and so much new stuff that you could risk potentially overwhelming your management.
So it really is up to your judgment and understanding of your culture at the organization as to how you want to approach that and what's going to be the best.
Is it best for you to hand them a big stack of stuff and go, holy crap, look at what I found.
And I did this in a non-formalized way with open source software.
Maybe we want to throw some more money at this problem or do you want to give them a couple things here and there that are cherry picked by you that will have the most impact when that information is reintegrated to the program.
So just know that your metrics are going to go up before they go back down.
The amount of incidents that you have is going to go up but your time to detect is going to trend downward over time.
Come get your prize.
Oh, absolutely.
So that's a great point.
So if you start with threat profiling and you're basically trying to prove a point and you're saying, I think Anonymous is going to come after us this week or whatever, you can start, I don't know, whatever you see as being the greatest threat to the org,
you can start hunting specifically for things that match that pattern.
Whatever the organization is the most afraid of, whatever your management is the most afraid of, give them their worst case scenario because you can find it when you're hunting.
Come get your prize.
Who else has an awesome question?
You had an awesome question.
Can I be your mentor?
I am happy to be your mentor.
Yeah, hell yeah.
I would say start with your home network before anything else.
You can buy a really cheap $15 tap that's a non-powered tap online and what it does is it's going to split your traffic off from your cable modem.
You take that traffic, you pipe it to another machine, that machine is running security onion.
That's all you have to do.
It's going to be a very limited data set.
It's not going to be a lot of stuff and you're going to be able to investigate the information in your own environment.
You hopefully know everything that's on your network and if you don't then it'll really give you a boot in the ass and a motivation to keep moving and keep learning about it.
Come get your prize.
Do you have an understanding of what the critical assets are in your environment?
Certain subnets are more important than others maybe.
I would take Bro.
I would throw Bro at some kind of a choke point that makes sense to start monitoring that part of the network and I would start looking at the traffic going over it.
When you use Bro, it breaks out things like files, transiting the network.
It breaks out all sorts of things that are going to help you get an understanding of how to get started finding evil.
What you're looking for are anomalies.
The more you understand the network and the more you've looked at the events, the more those anomalies are going to stand out to you.
I was talking about HTTP logs and I had an HTTP event log up there.
Let's say you start getting an abnormal number of 404s.
You're stacking all of your requests internally.
Say you're running a web app in a DMZ.
You've got a whole bunch of 404s coming into this web app.
Why?
Normally you're not going to alert on a 404.
Normally it's a meaningless thing, but if you start getting a whole ton of them, somebody may be enumerating against your web app.
You're only going to be able to start identifying those anomalies by stacking the information and saying, I want to look at every grouping of 404s that occurred per source IP address from the internet.
Once you start being able to group things that way and visualize them and see them all standing together, then the things that don't belong, you're like, one of these is not like the other.
You'll start honing that analyst intuition that you have based on the experience that you have in IT and what looks right and what doesn't look right and what should be allowed and shouldn't be.
I'd like to add to that.
Another good thing you can do is like Coursera or something like that.
Do some of the basic stats courses if you haven't done any of that.
And get a handle on what standard deviations are and all that.
Because like what she's saying is you start taking that data and even if you didn't even know what the assets were, you start doing stats count where standard deviation blah, suddenly things will stand out that are right there that you didn't realize were there.
Just a basic time chart of something like all of a sudden you see.
One of the things that I find really interesting, just like you would look for when they introduce new code or something and you see like this big huge spike in certain error events or whatever, this is pretty common.
And one of the reasons why what George is talking about is really important and what Jackie's talking about, seeing those spikes in data, is that these are normal things like when you go buy a platform monitoring tool or something like that for, that's why I mentioned applications earlier,
you can start to correlate things or you see things that are weird and you just go, huh, I wonder why all of a sudden we had a bunch of stuff failing to reach a certain domain name or something or now we see this huge spike in counts of what the top DNS name is or something like that.
Those are very basic things that in a lot of your tools that you have already are probably going to have stats like that.
But do you really pay attention to them?
Do you really necessarily always wonder why that might be?
Those are things that nine times out of ten there's absolutely, it's just noise.
But when you see things that are going significantly higher, like in a standard deviation or something like that, it's not just about being higher, it's about being anomalous.
And that's kind of the challenge.
When you justify to your boss, why would I well this is anomalous, why should I take the time to investigate it?
And what the attackers are trying to do is lower that.
They're trying to get very, very particular where it's not anomalous.
The threat actors are seeking to gain as much situational awareness over your network as possible and it behooves you to get the edge on them and to know more about your own environment than they do.
That's the only way to defeat them.
Thank you guys, I appreciate your time.
Please come up and get your cards.
applause