[Music]
so
welcome everybody to the last talk for
today so my name is still matthias
this talk is also tlp white and after
the talk we will have surely a lot of
time since it's the last talk for today
for a q a session so if you have any
questions please line up at the
microphones
as usual in the first app there is a
possibility to
to
rate the talk so
if you enjoy the talk please give the
speakers feedback afterwards if you have
any devices mobile phones whatever that
can make noises please be so kind and
turn them off so and uh
it's now an easy introduction for me
because the two speakers have the same
name so these are both pebbles they are
working with the the polished cert and
the thing they are gonna present you
today a central system for metal malware
analysis it's an open source platform
it's called mwdb and now give them a
warm round of applause and enjoy the
talk
uh thanks for introducing us um
and thanks uh for all of you to coming
to our talk it's uh it's been a long day
uh
for all of us
um
yeah we we came here to talk about uh
our open source tools uh open source is
uh of a key in in what we are going to
present because
we really try to
put uh majority of the code uh
out there on github for our others to
use
um
so
i will start with a quick uh
introduction of uh you know
our and where we came from uh
why why we started building uh the kind
of tools in in in this particular way
also
um so a very quick introduction of the
team uh cert pl is a national uh
national level team as the name suggests
for poland
we have a pretty long history
which we'll not cover in this talk
we are also part of a
research institute called nask
that does multiple things it's also
registry for dot pl
uh
but that is also the kind of the risk
kind of
a research
environment that that our team is
located in that
has some impact on how we
approach things
um
what is important uh for for uh for what
we do that uh you could say that we are
in a threat intelligence business and
have been
well for for many many years
um being a national national level team
means that uh to some degree we need to
uh take care of
various threats voice incidents that can
affect
either regular citizens uh small
enterprise or or big critical companies
in poland
and
well nowadays as
as as we know
fishing in terms of
quantity of incidents uh is probably the
dominating the the landscape
but malware was always you know
important uh threat uh important uh
vector in in in attacks
a tool used by threat actors uh and
still uh like if we look at our
statistics is the second
uh biggest uh
biggest source of of incidents in in our
constituency
and
so what we want to do uh is to
make sure that whenever there's a new
malware campaign in poland
uh we are able to detect this as soon as
possible
ideally
before actually victims get infected
then we want to
whenever possible to warn uh victims uh
or it might be affected institution that
is targeted especially for for financial
institutions
or uh the the victims that were
were infected uh and then
if we can mitigate uh those campaigns uh
either by uh taking down the
infrastructure uh supplying iocs for for
companies to block
things on their endpoints and so on
and
if we
take a look at a bit of history how how
our
our tooling our automation developed
um in
the early days uh
we well we always had some out level of
automation but it was mostly um case by
case
for particular malware families we had
some uh truckers or whatever
that
was
target well suited for that particular
family
and when we had some other incidents
other campaign we
and it was
suitable for automation we developed
some other scripts typically
um
however that
changed or or that the environment we
operate in changed a bit
in
let's say around
early
2010s
so
the decade ago
give or take it i'm not being very
precise here uh
where there is a rise of
a commodity malware especially bankers
in our case where you could see that the
malware actually can get used by people
with a little technical knowledge the
criminals are just buying uh
builders on the market they can just
supply
configuration that they provide and they
don't have they don't need to write
malware they just buy it off of the
shelf
and that means that we got uh campaigns
that that with with the same uh family
with
different configurations
uh so we get a lot of samples uh but we
then we could actually the automation
made even more more different for us in
terms of our ability to analyze those
campaigns
um
so
in let's say around around mid 2010s
we had our first kind of generic
approach where uh where there was
actually uh our tools become
capable of analyzing multiple kinds of
malware and in some consistent uh manner
that was a beginning of some common
framework for automation on our team
a couple of years later we
we kind of our maturity uh as a team
increased so we we are able to track
multiple botnets like almost real time
by uh by by looking at that at the
command control infrastructure
and
recently in recent years we we kind of
put quite a significant effort into
releasing many of those tools as like a
proper supported open source projects
that
you can also use
and
if we look at the what the malware lab
is
it's uh
in how we define it it has three three
three main uh roles one is collection of
of data of samples of
technical threat intelligence related to
malware iocs and so on
and then it should provide a way to
integrate
family specific analysis tools so
specific tools that can analyze specific
molar family
that we can plug into
some common framework and ability to
share this information some export this
information and share it with the with
our partners for of our constituents
and uh
we are going to talk about mostly open
source projects you can find them on our
github uh
and let's start with mwdb core which is
the
core of all the things
so uh hello everyone
okay so we start uh to uh
make a brief
introduction to the projects it will be
built on on the canvas of
these three points that were shown
before
uh collecting analyzing and sharing the
data
so the first thing is
we need to
collect somewhere all of these data we
want to analyze and that's why we made
mwb co
so the nldb core is kind of
the central component of our
laboratory design
it is meant to be repository for
organizing all the artifacts that
we are working on as a malware analyst
and as we have a central point where we
are collecting all of the data we can
also use that as an interface for
example interface to to a whole analyzes
pipeline because all data are aggregated
in single point
we can use that use that to to
explore
for some
[Music]
correlations and so
uh on
that
you you can
look at that
component as
like for a database
[Music]
and but it's not it's not it should not
be
considered as a
general uh malware threat information
uh sharing system
the common question at this point is how
is it
alike or different from misp actually
it's a completely different kind of
thing because it's more focused on the
malware artifacts and this kind of meant
to
be used by malware analysts
so
the main
data model design is strongly focused on
the on these artifacts and they are
divided into three types
files configurations and blobs
and the first thing uh that are
pretty common are files
uh but uh files are meant to
be
the whole chain
of binary data that can be used
by the analyst
so from from the
infected archives
to documents with macros droppers
to the exact
the dxe files with the with the with the
malware
uh finally uh we can also put some some
memory dumps from our sandbox
and uh
the second thing are configurations as
part we said before
malware is
highly reused
it's
bought by by the criminals
in
a kind of generic form
so it needs to be parametrized and we
can parametrize the model operations in
two ways
the first way is static
where the
data
crucial for the operation are put in
directly in the binary like for example
encryption keys
cnc addresses
uh botnet version
api version and so on
all of these data that are
somewhere contained in the binary can
can be extracted into a single structure
that describes the
[Music]
the malware pretty uniquely as it was
configured by the by the criminal by the
actor
other type of configuration is dynamic
configuration
well
it isn't kind of
configuration it's distributable form
it's kind of representation of uh
the data that are coming from the
command and control so for example if we
have some uh
p2p
botnet uh
it can be list of peers
uh there can be commands web injects uh
but what is what is important uh if it's
contained in the
configuration type
uh it means that we are able uh to
uh
label this data
to see their semantics and to put
them in in structured form that can be
useful for further automation and
usually the structure is determined
uh by the author of our
analysis script
that is
extracting this this
configurations so it highly depends on
the malware family there
isn't any standardization
so configurations are
kind of uh
end goal for for malware analysts
because they contain also some
consumable ioc information
so they can be used for different
purposes
and usually these configurations are
coming from
these memory dumps because
malware is usually protected by some
other another layer by
parker protector
and
we have some hybrid approach
firstly we are
doing
analysis in sandbox
and our sandbox is tuned to
damp regions that may contain the
unpacked code the unprotected parts that
may contain the
parts of configuration and parts of code
that are determining the semantics of
this
this data
and if we are able to to dub the
unprotected code
then we are shipping all of these dumps
to
something called configuration extractor
which is a bunch of scripts
designed to
reap the configuration from supported
currently supported malware families
the next thing are blobs
blobs are
text data
which
are nice to have in repository but we
are not so good in
parsing them in making structures out of
it
so for example
if we have some
decrypted strings for example that are
protected by the malware somehow so
there is the caption routine we can
as an early analysis we can make a
script that
reaps all of these decrypted
uh
protected by malware strings
into the text file
and then
we can use
some other approach that will choose the
appropriate strings for for
other operations
the same for web injects the format for
some botnets isn't very stable
and actually we have tried tried to to
pass them at some point but uh
it seems that very often uh they are
much more more useful in its wrong form
so when we are passing something
there is a possibility that we are
losing some information because for
example we don't expect
that information in in the
exact place
and how we are getting the dynamic
information well the static
configuration is kind of
parameters
for bot execution so if we are able to
uh somehow
implement the protocols and the
communication we can use these
parameters to automate automatically
gather the
gathering the
dynamic information and that's something
called amtrak that was described in
some
previous articles and presentations
and all of these things can be connected
to each other so
if we are lucky enough we can have some
nice graph representation a few
architectural graph
so we are able to
to find some connections like for
example samples that are connected to
the configuration are fetched
dynamically and for example they can be
plugins modules or updates that can also
provide another configuration and then
and so on
so we can have a whole
malware infection chain
within our repository
and
also we have lots of
different metadata like tags or
attributes
that are describing the the source of
the samples or or some
text metadata that i related with with
with
that artifact for example we have some
we can have some relations to other
systems
like our instant tracker
well that was about collecting
the second thing is about analyzing
so
analyzing things
was usually done
in set pl via a set of loosely tangles
connected scripts
that were
used for various purposes
and
that network
came to be very complicated at some
point so we
won't be able to to quickly attach
another script because
uh we
needed to analyze the whole
network of connections before we attach
something new
so we also decided to make a framework
for that
called carton
and
the design is
based on
something that we need to do very often
as mobile analysts which is pareto rule
so if you are writing some processing
script uh the the main effect the the
most important effect that is gathering
the data from some artifacts is pretty
easy to
to achieve
because you need to have
do some
malware analysis and
make a script
well
but there are much uh
more challenging things like integrating
that script so it can work on the on the
live feed on the live data
uh
to make some appropriate logging so we
can
debug
the workings of that script after some
time
if it still works for for current
samples
proper error handling so
it doesn't crash silently
and and so on
and
we decided to make
we we started to notice that these
things are pretty common across these
scripts so there is
a
similar code base that is handling all
of that stuff that is meant to integrate
our script with other services
so we decided to make
a small framework
that will provide
all of these common things out of the
box
and that
that is highly
[Music]
visible in the design so
shortly carton is
meant to be a micro micro service based
framework
for
making dynamic
malware analysis pipelines
and
the main design goal
was to be able to quickly deploy new
scripts to the system
[Music]
we tried to
use
some
ready-made approaches
but unfortunately
they
didn't really
know how to say that
[Music]
they were not solution
solution that was able to work in that
plug-and-play manner
so we decided to make something simple
based on radis
that provides the the cues the tasks
used
and a minion for common
file storage that are needed
for the time of analysis
and
what i meant
what i mean by by
talking about microservices
so microservice
is
meant to
do the one job
and do it right
so we are splitting the responsibilities
across
various scripts
that have
well-defined inputs and outputs
so the single script is
attaching to the queue by defining
what kind of data
it wants to receive
and then it has some well-defined output
what kind of data it emits so other
scripts scripts can also
adapt to that workflow and and
the autos know what data to expect
so for example we have
the most
basic kind of script is classifier that
everything it does is the classification
of file type because it isn't that
obvious very often for example for
scripts if we use the file command we
see only ascii text with long lines and
so on
so for example in that service we have
some holistics
that is looking for common keywords for
script languages and this choosing the
appropriate type
the second thing is extractor
that also tries to
match the common passwords for encrypted
archives like infected because
it's usually the easy way
for distributing malware across analysts
in in some corporate environment so we
need to have these files encrypted to
not be spotted by some antivirus
solutions and so on
and that extractor
just extracts the archive and
is producing the tasks
with with the files contained in it
and as we have
some
actual malicious file for example
document that comes from that classifier
stage
uh based on that
that document can be
uh
consumed by some specialized scripts
that are for example
uh
designed for extracting
macros from these word documents
or for analyzing the exact macro
for specific draw for specific campaigns
that are
using a specific dropper
for example for emotots they were very
common documents and we have
tried to
collect all of the distribution urls
actually from these markers and as they
were very similar to each other we
decided to make specialized scripts
especially for that
for that purpose
and uh
what is really important in the
microservice
frameworks is the inspection of the
queues and and the whole network of
services
so carton comes with a small dashboard
which
allows to administrate
the system and and make an inspection
about
supported filters by the services
i think we will try we will try to
implement the output definitions as well
at some point but currently they are
determined
kind of dynamically
based on the his history of tasks that
were emitted by service
but it's work in progress i guess
so okay
that's about carton and finally we are
getting to the last point which is
showing
because
all of these systems were made
mainly for our own purpose
it was a long story of
rewriting
designing and then
making it able to deal with newer
campaigns
and at one point we decided to make it a
kind of open source
so
we decided to
publish the frameworks the tools without
the
modules
to make people able to
[Music]
design a similar malware lab using our
tools
but
because
not all of you
need to have own malware lab and
most of people in id security are just
using some some services for that that
are already made
we decided to share our
central system as well under the mwsb
set pl portal
which is
um
i forgot the what
in general you can use our own
systems our own analysis capabilities
via that interface
so you can access
samples configurations
and
lots of outputs from our analysis via
this service
so there are some statistics uh
also we added at mwdb we are trying to
make a small community of malware
researchers
we are trying to do that from 2018
and we have over 1 000 accounts
of course all of you are also open for
the registration
and we have extractors for 130
families
but
most of them are not working because
they were made at some point for some
excellent version and
then
not all of them are maintained because
the maintenance of these scripts depends
on the current threats in poland
so
[Music]
we are trying to make some cooperation
in that manner but but it's also some
work in progress
the whole repository isn't
very big it is in the scale of various
total but it's pretty okay
we have some nice integrations like uh
malware basel integrated with us
and is pushing all of the samples
submitted to malwarebaser to nwb3pl as
well because we are the effect
intelligence vendor for it
so malwarebuzzer also is using our
service for malware classification
and yeah
about the community
we have a small slack
channel
and we're also trying to
use that service to share some artifacts
with community via twitter or other
channels
okay so that's about the
uh
main products
yeah so this uh what was already
available uh
and uh kind of what can you use the mwdb
srtpl is what you can use
as
without setting anything up or you can
set up your own instance of the
of the whole pipeline
and now a bit
what is coming up uh that would be also
it will also demonstrate uh the
extendability
uh
so we are
what i'll be talking about is using
a plug-in functionality in mwdb which
allows you to
extend the
well the interface and the
operations of the
well
of the whole system
so
this in this case uh we we are uh
looking
we're doing some research into
uh
malware similarity analysis
uh it's not a new topic however it is
some direction we would like to take uh
to extend the existing functionality in
mwdb because currently we are mostly
using yara rules
and
some python codes to to extract uh
interesting parts from
from from the
memory dumps usually
um so by adding a bit of the automation
for for comparing the
similarity between samples we can
we can try to
provide classification
for families even in situations when
there are new variants that are not
necessarily
[Music]
matching our our existing uh urls for
example
uh and also to
when we have new samples the similarity
allows to to pivot uh this is like the
the idea is not new however this is a a
new feature that is that has been
already developed for uh
for for mwdb
um
and um
so very very quick overview of what what
we have already in place uh
we we are uh
using um a combination of dynamic and
static analysis uh so
dynamic analysis is what was already
mentioned
uh using sandbox to obtain memory dumps
uh
this those memory dams are then analyzed
statically
this is there is a tool api scout by by
daniel ploffman
that extracts
potentially interesting
uh
windows apis uh which form a feature
vector that is then uh used for for for
comparison
and for for classification if there is
uh if there is a similar uh
malware so this is like a multi-stage
process that is implemented through
uh for carton modules actually in the
back end
and
the the plug-in functionality in in and
of the db allows
us to extend the interface uh by
providing here it just provides
additional uh tab in the results uh
where we can visualize uh the api
vectors so the the the color color for
picture is the api vector or the api api
qr uh uh
with that with the bits that are
highlighted corresponding to the
used apis in a particular memory dump
and
provide uh like a
pretty easy easy to read overview of
which family is
similar to what
which degree
uh the text might be a bit small here
but we have like 100 percent
similarity much with one of the existing
uh families in these examples of course
that uh sometimes it works sometimes it
does not as these methods uh do there is
no uh
with this fuzzy matching there is no
exception it's not fuzzy mushing but
this is this this kind of uh
approaches uh based on similarity that
they are not guaranteed to provide
meaningful results in in all cases
um
but that's not all of it there is also
an upcoming
extension to that where we are not
looking at
um the api as a static feature but we
are actually
doing disassembly of malware samples
and especially unpacked malware samples
thanks to the sandbox functionality that
we have built in
and we are able to determine what is the
reuse of on a code level looking at the
functions disassembled functions and
normalizing them to a canonical form
which is then compared across our uh
entire data set
and that is we already have some proof
of concept integration with mwdb for for
that system
and
i expect that we'll be able to
to to provide this uh in the inter mwd
cert pl interface
fairly soon
in the back end it looks like this so
there we have
uh like very very rough interface to for
for if someone's get into details but
actually in nwdb it will be
much more user friendly
for also for you to to interact with
that
um
so this is uh kind of what is on our
near-term roadmap uh
the api vectors are already kind of
running just not visible to the uh
general yes they need some polishing
before
we can make it public yeah so
still it's a better testing period
but yes that's kind of the
one of the directions we are taking into
kind of
building additional systems for for
malware analysis and then integrating
results of those systems in nwdb and i
think it kind of
demonstrates that it's kind of really
flexible tool
and you can
adjust it to whatever
requirements you might have
um
so actually that is
all we wanted to to show
so we will end with some pointers to
where you can find more information or
if you if you're interested
um
obviously
official documentation
quite a lot of effort was put into that
yeah that's for example there's some
user guides that will guide you from the
through the
main
points and concepts
so it's nice to
look at i guess
there is uh
their training materials actually with
with tasks where you can go through
uh various parts of nwdb also uh create
a carton module and so on so it's a uh
we
it's an evolution of a training that we
uh did last year at first the virtual
retro virtual training with
mwdb carton and so on uh these materials
updated their updated versions are
available uh for for you
also
uh
there is automation library that likes a
wrapper around the the apis and also
come online interface uh for for
interaction
uh with with nwdb
um and what we did not have time to talk
about today uh a maldek
probably it's it's kind of um swiss
knife for malware researchers that are
writing scripts for for configuration
instruction python
in addition it it has
a
configuration extraction framework
so it is able to run modules that are
compatible with maldoc
formats
so important part of our
code base
that's all we had prepared
for this
like high level a bit a bit high level
talk
uh we get some grunts for for part of
this work
uh if you want to register for uh mwdb
certified that there's the registration
is open
there's just a vetting procedure
we i know we have a backlog of those
applications so if if you applied you
didn't and if you did not get a response
please ping us
we'll make sure that that
that will get pushed through
if you if you have any questions
we'll be here
a whole week actually
but if there are any questions now we'll
be happy to answer them
thank you thank you
[Applause]