okay my name is Aaron Leverett this is
Bruce stunning
we are CEOs of companies that we just
made up that's sort of true but
genuinely he's a start-up I'm a startup
that's how it is so it's really late in
the day I was traveling for 24 hours 24
hours ago I was in Cleveland so I'm kind
of exhausted and a little bit you know
fuzzy so if you give me lots of energy
I'll give you lots of energy and Bruce
and I'll have some fun so how many of
you are still awake put your hand up
okay just checking how many of you have
bosses all right some of you how many of
you have bosses that like to invent
vaporware that you have to implement you
know why I'm about to do this
presentation that's that's what this is
all about my boss at the Center for his
studies in Cambridge is a really really
clever guy but he doesn't do the cybers
and he likes to talk about a PT's and so
he does stuff like ask me how many
people are in energetic bear you know
how much money did they spend on this
type of ransomware and I'm like how the
hell would I know that right
but then an idea started to form this
idea of logistical budget so basically
that's what this entire presentation is
about Bruce is going to tell you a
little bit about the implementation
because I didn't have the time to
implement stuff so I used my speaker
fees at other conferences to get Bruce
to do this work for me
partly because Bruce taught me to
program and he's a better programmer so
I'll let him introduce some of these
okay so as Erin said we want to be able
to present stuff to non-technical people
so they know which threats to
concentrate on and what to concentrate
stuff on which direct threat actors can
do the most damage and save around
somewhere they can get over you from
reading literature but what if we can
actually generate visualization directly
from risk data in an easy-to-understand
data to
to see how threatened through structure
is changing so has anyone heard the the
the term logistical burden before no
yeah sort of okay so I was working at
the Center for risk studies and they do
all kinds of risks they do environmental
risk they do you know labor unrest they
do interbank lending risk whatever it
was great I got to hang out with all
these amazing people and so I was
working with these counterterrorism
professionals and they came to me
we were talking about adversarial risk
like how do you quantify adversaries
that change and adapt and think and do
stuff right and they introduced me to
the concept of logistical burden so you
go to a site let's say like a ship or a
building and you you take some special
forces people and you ask them to ask me
how many people were to take to storm
this building or to drive a truck bomb
here how much money would it cost how
big would the bomb need to be these
these kinds of questions right and they
estimate the size of a threat that would
be required for a particular target
right now I didn't particularly want to
do that I realized we could do this the
other way around
and that's what the rest of this
presentation is going to be about it's
if we take indicators and we assume that
they have a cost in money manpower time
then we can start to get a sense of the
logistical budget of different apt
actors right wanted to quickly
prototypes and visualizations so we used
Primus to grab data from Aaron's Mizpah
server and pickle to cache it locally so
that we can very quickly iterate through
stuff so what we want to do is scan
through miss events and attributes and
filter based on galaxies and date ranges
and then accumulate score for the
entities that we found we used portly
initially for heat maps because it's
really easy to output data that plotly
understands and later on a new plot for
a bit more flick
but it has some drawbacks I think so
first we generated heat maps for a
threat back to activity and then
generated scorecards which are
comparable with each other for threat
actors but also ransomware because it
was a really easy extension one things I
struggled with not having Aaron's
background in threat intelligence and
this was domain knowledge so after
grabbing the data from the server we
quickly were a Python script to dump the
fields and to count frequencies and so
on that made it very easy to get a
better understanding of what data is in
Aaron Smith server and how we should be
writing scoring functions so this is a
kind of stuff go back so this speaks
very much to your point under s about
you know when people first encounter a
Mis server they don't know what the
fields are they don't know what the data
looks like and it's fine if you're like
a front-end user using the GUI but you
sometimes need to dig around and inside
the mists server to figure out what what
you've got and an interesting point here
is we wanted to start with with things
that had attribution that had threat
actors attributed to the events which is
not my favorite thing to work on like
attribution is essentially a political
act and I find it very complicated so
the first thing we wanted to know is
what percentage of of events inside my
mr. server were attributed to a
particular threat actor and it was like
8% I imagine most of you it's fairly
similar so one of the things we can talk
about later is how we might increase
that in the future and some of the work
you were doing would work really well
with that right so so I could talk a bit
about the scoring functions we really
want to discuss the scoring functions of
the community
the kind of first stab they can be
approved a great deal currently we're
looking at context-free analysis so
we're looking at an event and events
attributes with no correlation of other
data within less me saying a few things
about this as well so there's scoring
idea is essentially how do you translate
observables or iOS ease into one of or
all three of money manpower time and
it's not as easy as it my team right
like what does an ipv4 address worth to
an attacker if an attacker switches for
one ipv4 address to another what would
you say the cost is in money come on be
interactive I'm really exhausted
anybody fairly low right it's not it's
not super hard so I wanted to put an
actual number on that and I went digging
around in ipv4 auctions and you can
basically buy a new ipv4 address for
four bucks so there's a number I can put
on it right and we all agree that's not
the right number but what I'm trying to
get here is that we can put a sort of
constant of scores on like how long
kilobyte of binary takes to write and
some of you could go out and do further
research on that which is what we want
to talk about here but for now the point
is that everybody shares the same number
so when I was in Center for risk studies
there was a brilliant counter terrorism
risk professional he's written a couple
books on the subject Gordon whoo and and
he said to me all risks should be
comparable or all risks are comparable
or should be and I found that really
frustrating because like you know what
we do is special it's different it's
it's not like other risks but if you've
really progressed in the risk world then
you can be compared to fire risk or you
can be compared to pandemic risk or you
can be compared to kidnapping ransom of
piracy or whatever so that's the point
here is by putting some of these numbers
on here all of these different apts and
all the different ransomware families
can be compared even if we know those
numbers aren't exactly right the
constant is wrong for all of them and we
can at least compare them so Bruce will
show you more about how he achieved that
okay so the scoring functions are kept
separate from the mechanics so you don't
have to be an expert
and how the mechanics work to be able to
write scoring functions and as I said
before the dump of the attribute data is
really useful for writing them that's
almost impossible to read even for
myself so but it's basically just a
really simple piece of Python that takes
in event and the corresponding
attributes and scan through and
accumulates based on the attribute data
and then returns the score if you have a
URL it's got this much time to manage or
this much money if you've got an IP
address it's worth this you get the idea
if you've got a binary and it's of this
size then you have some idea of like how
much time the thread actor put into it
so that's all there and that could so
the scorecards that I mentioned before
looks something like this so we're
trying to estimate the organization size
and the amount that they're spending in
for on infrastructure the estimated time
investment and this is we're going to
compare Dharma and want to cry and you
can see we're giving some fuzziness to
the to the actual results but if we look
at one a cry much bigger organization
size spend and time investment so these
should be noted these are log graphs of
the tics or a little bit disingenuous
but they're for different score cards
and that's important because some threat
actors operate it at like an insane
scale so like you look at the number of
URLs involved in a sofa C campaign and
it's just extreme so you have to do some
of these things on a log scale right and
the score across the bottom for those of
you who can't read all of these it's
estimated organizational size at the top
that's the other one infrastructure
spend so the amount of money is the red
one time is the blue one and the last
one that's in black is basically the
aggregation of those three different
scores right so if we click back and
forth between these two you just get the
idea that
Dharma probably spent less money
manpower and time than wanna cry and
that's all we really wanted to do with
this but of course you don't have to do
this just for ransomware you can do it
for other things too right so we also do
this for the threat actors so we hit
here we have energetic bear and equation
group and then we have heat maps that we
generated for the threat actors so this
is taking threat actor events 15 bins of
30 days and then ranking them based on
their aggregate score so we can get some
nice idea of bright points and
corresponding dates we can also do the
same for weekly plots 15 bins this is
the event but scaled based on their
threat levels so the high gets a
significantly higher score than a medium
or low and we didn't want to put like
200 of these heat maps in here but we
can do them not just for events we can
also do them for binaries or for
networks or for files or for whatever
and then we worked on a sort of scoring
function that took all of those into
account and made one now it's worth
pointing out here that the time bin
across the bottom is detection time and
we all know that dwell time can be
really high so I don't take the time
line of this entirely seriously but I do
take the heat map to be of interest so
what I'm trying to say there is that you
know this little white spot here for
sofa C might have actually occurred a
time bin before or before that in terms
of when the attack occurred so this is
detection time but it still it still
lets us know that there was a lot more
indicators in that time period that we
could use for something so yeah this is
an idea of the code that Bruce is
written and we've made open source on
github
we have other ideas of how we can
visualize like perhaps you would do
treemap sort of structure where the the
files will be on one side and like you
know the the network indicators would be
on the other or we can do heat maps for
ransomware we've got a lot of ideas
about how to visualize this data but we
probably need a little bit of help and
then we want to talk a lot about scoring
functions so if you know that there's
academic work estimating the amount of
time that went into a binary based on
how many kilobytes it is or how much
network infrastructure costs for
attackers or so on I'm also giving a
talk tomorrow about ransomware well
you'll see a little bit more about where
some of this came from and some of that
work is replicated there in terms of how
much an incident costs by comparison to
how much attackers made in ransoms maybe
everything that you said oh can we use
unattributed mess data in our
visualizations how does the community
feel about this how do you feel like
sended
I mean so this wasn't super expensive
like Bruce works really hard and he's
got a new company but like like I said
this is my speaker fees for a couple
months right and I'm really glad about
that but we do think it could go a lot
further so if you're interested or you
have time we don't necessarily need
money we also just need people to
contribute so you know we could just
give it to you guys and you can do
something with it if you want that's
fine too but I'm also interested in the
reaction from the community like is this
total BS because it's based on money
manpower and time and you don't like the
scoring function or do you actually
think this is useful would you sit
around comparing apt groups and
ransomware no I mean from I think from
our perspective it looks really
interesting and maybe something that
could be interesting as well as
especially once you're refining your
scoring for the different types after a
while would you be interested for
example in feeding the data back into
the frittata galaxies because this would
I think would be very valuable for the
community out there to get the cigarette
and then for further developments we can
we can issue we should talk about this
yeah that's the other thing so we have
we basically have a bigger research
ongoing for the exploration of
indicators and we're looking at
different components and different
things that we can take into account so
which we could work together on that as
well
so yes I think there's another question
back here
yeah
exactly so I'll repeat the question for
the cameras as I'm supposed to even
though I'm tired I remember the rules of
Cooper
so essentially software development
houses have that data already and that's
the sort of thing that we should be
incorporating so once they've written a
piece of software you could look at it
and work backwards and say how many
people did you have on this project for
how long and how much did it cost and so
on now the costing would be the one I
would question in terms of timing
that's probably all very accurate when
you can player malware and you compare
standard software but in terms of money
it might not be the same pay structure
and the underground economy right people
might be coding for a share of the brand
some of our profits or you know they
might be stealing other people's code
before they get started there's a lot of
details in there but but I absolutely
take your point the traditional software
development studies are useful to this
and we didn't dig that deep into this
because we just wanted the proof of
concept where we could show you the
visualization first and then we could
deep dive on each of those numbers
especially if we can get you interested
to help us with that so great idea next
question
okay
yes so the the comment is essentially
about the associativity of tags inside
Misbah vents
yeah of course I mean the more that the
correlation engines run underneath the
more that we will have possible to
visualize right especially if you're
enriching events where like you happen
to know this domain and this domain are
linked by Whois data and then it grows
right
yes I mean we know that we have a naming
convention problem for ransomware and a
PT's because it's essentially marketing
reports that we get most of this
information from which is yes in fact I
would counter this entire conversation
with the fact that you can run our code
on your missing instance so if your
confidence in your misclassifications is
better then you can do the visualization
on your data that's why we wrote it this
way so yeah any other questions or
comments yeah yeah I mean you did some
lightweight analysis in that sort of
area but not not in a scientifically
rigorous way and it's absolutely
something we'd like to do it's just we
wanted to prove the concept with the
visualization and then talk to people
about how to do that so if you're
interested we'd love your help and I
think I have to wrap up for the next
speakers that's it from us
[Applause]