uh
taking up uh the challenge of teaching
you about yara in uh 30 minutes
or less actually
um
so it's a short workshop and actually
the idea was to share about
[Music]
how how you can learn yara
exactly
you have the floor thank you very much
so hello everyone my name is dominic
kargetsiova and i would like to
share with you some of my knowledge and
experience with yara
few words about me i am a researcher at
avast
and i'm also a phd student
at
bernard university of technology in brno
my research includes formal models
languages and their use in security
i'm also
interested in pattern matching
and also sometimes i'm working with
blockchain technology
during my projects at the university i
also worked and i'm or i'm working with
essa european space agency and czech
police
what to expect from this talk
we will shortly talk about what exactly
is
even though i believe most of you
is well aware of it
but
later on i will talk more about
some specific and mostly
practical like details that could be
really useful for you
if you are using cara
in your company
and your business
i would also
like to share with you some information
about my changes in era
that i proposed and also changes
that were done by my colleagues at avast
so let's talk about yara
you probably know that tierra is
amazing tool
for
mostly matching malware
based on statical and dynamical analysis
it was created by company virus total
you i think heard about them by the way
next talk is about verse total i'm
looking forward to it so don't skip it i
recommend it to stay here
uh
and it's widely used uh it's even the
company was bought by google which also
means something for me
uh and you can
meet the era basically everywhere
more specifically about uh yeah it's the
tool but also language
where the basic syntax you need is
so-called rule
rule has name you can imagine it as a
name of the function
or something like that and it's usually
it's contains usually three parts meta
information which is like description uh
name of outros etc
strings this is like for the static
analysis
meaning we want to find these strings in
files we can
declare text string as you can see the
first case
we are searching cmd dot exam
or you can define regular expressions or
hexadecimal strings
i will talk about
the option behind the string ask say
white no case a little bit later
then you have part condition that has to
be
evaluated as true
to
found a match in a file
so this is like boolean expression
basically i have here i want to match
both
text string and regular expression and i
also calling some function from modules
here has several modules that you can
use
like
for example for google module so you are
passing google
reports
or in this case i'm using module math
where i calculating entropy of file size
from giving offset
and you can also access specific offset
of the file
and you can change check the value there
this is just simple like rule as example
um you can see it's
called a bootcamp malware so nothing
real so
you can try to use it but it will
probably
it won't probably find anything useful
but
this is actually a great example
to my talk today
because if we use this rule on some data
i selected random
directory in in my system
that has around six gigabytes of data
with around 40 000 files
we found out we have a little bit of
problem
why firstly uh yara will generate free
warnings
if you are one of my students you can
say hahaha okay warnings that's not
problem right if it's not error that's
who cares
but in practice their system for example
by virus total that limited the use of
rules with warnings so you actually want
to avoid them
and also
they are usually
quite meaningful they can be like
persistent but they
tell you that your rule could be a
little bit better
in this case it's saying that
a rule is matching
too many things
so it's basically saying maybe you are
searching
for basically everything are you sure
it's
what are you like looking for
maybe check it out
also
uh there's like more specific warning
we will talk about that a little bit
later
and the final one which was also like
the motivation for the speech and my
research in general
uh the
rule
may slow down scanning and this is
problem because
uh when you need to scan
um basically data
get the answer very quickly
this could be really painful
and as you can see from the last uh last
line
yes
only
like in uh
just like i really want like for example
it's like only six gigabytes took
45 minutes to we scan by era which is
bad like this is really bad we don't
want to use these rules in practice
because
otherwise we cannot detect basically
anything so what we can do with it
which is the very topic of this talk
yara performance
to understand what's uh went wrong with
the rule i will quickly summarize
something but don't worry i will not go
into too much detail because i think
it's better to read some information
like this i will share the presentation
with you don't worry and i also have a
paper that you can read early larger
explanation and long explanation about
basically everything i will talk about
here
but shortly
yara does not search
like whole strength advance
yara will select substring of statical
part of the strings
based on
some heuristics
these
substrings or atoms are 0 to 4
characters
and
basically it will search the file for
potential matches so in the end it has
like okay there are four places or ten
places where the match can be
potentially found and then later it will
check if the match is really there or
not
what does it mean
you
should think about it while defining
your strengths because if you choose
two general strengths that mean
that yara will slow down because it
needs
deeply examinate every byte of your file
for example
i have some
examples here the first first regret
expression is
kinda okay there are there is one atom
wavelength of three characters which is
kind of nice
also in binaries and stuff it's probably
quite unique so this can work as well
the second one is also okay we have even
four characters
be aware that uh the one or two prefixes
will be checked later during the
matching process
uh the third one is excellent
demonstrating
we have four characters but
be careful about the zeros in general in
rules because as you can imagine
there's lots of files that contains
large section sections of
of zero bytes
and
in one version of the area it was even
possible to search for
uh zero bytes in atoms we actually from
from my company proposed a change during
this because the slowing down was really
really significant
the
next example is a little bit worse
because uh liera in first step will work
only with
basically two symbols like c and d
which uh can
took some time to check all files
and if you can see the trend you
basically want to specif be specific as
possible the last one is like the worst
idea because firstly
zero rank atom is selected which
basically mean
that every byte of the file is checked
more deeply if
the regular expression is matching the
part of the string which is very slow
and also it's very generic like you can
see
okay there has to be some alpha number
uh symbol then we can have basically
everything and lastly there is one
number
basically what are you searching with
this thing you should always think that
it if it's too generic
you maybe don't want to search for it at
all
we will see
also i talked about
or i mentioned modificators before
you can use it for both
text strings and regular expressions
there are some basic options asking
there is basically like the natural
regime for yara to work with
uh there is support for like byte option
which is
it's
something like utf-16 but not really
because the first byte is actually just
zero
and the second byte is from ascii so be
aware of it it's not the true utf-16 as
you can
or you could expect
or you can use both
but also be aware that
uh you should know if
it's really the case if and if you
really need it
you don't don't use it and your rule
will be
just just a little bit faster
okay sensitive modificators it's a big
debate in the air community
in moocs cases
using no case option
it's quite okay but if you have a big
rule set that mean
you are scanning with multiple rules at
the same time it can
it can have
effect on on your speed what you can do
instead and which is actually a little
bit like faster is to specific just
uh for example one first letter that you
want to match both
upper and lower case and be dead you can
get actually better result just quick
note don't forget to escape the dot
i think this is very like common mistake
it's easily forgotten right and then you
will match
a different thing so
be careful about it in regular
expression you need to escape dot to
match actually dot and not everything
everything else
again be specific as possible this is
like the main message of this talk uh so
be be careful about that uh
of course it's very hard i know i work
with a malware analyst and they will
come to me and say hey i really need to
specify this more in more general way
but yara is telling me no please don't
use it so what's the solution
social is basically just try to find the
middle ground
when your rules is working like you need
it but also the era is like happy and
does not generate warnings for example
you can specify uh the the range of how
many symbols you want to match
the star the path symbol that is
matching like zero or one two basically
infinite number of letters is not useful
for errors so be a little bit more
specific
what can be surprising for you
is that yara really likes fixed prefixes
what i mean that if you are matching
something and you know the start of the
string or regular expression
use it
because
yara will then
found
more
the largest or longest match possible
and is much quicker than the other way
around so you have suffix the end for
example here i have something
and tom
that's not the ideal for the the yara
because
then it will find every possible match
because
the prefix is not fixed basically so in
our case uh with regular expression for
example we want to match some some file
path so if we know that the path start
with let us see
with disk c
then we can use it and uh yara will be
very happy for us
too many matches this is a very
troublesome problem
we are dealing with it at about
basically every day
it means that your your rule is too
too generic and some files can trigger
like over one million
matches
on it which is the cause of this warning
previously it was even error so you
couldn't actually like really use it at
all
now
it's only warning but still the result
can be like invalid because
uh yeah cannot store all the matches
in their in their system
what we can do with this there is no one
simple solution for this problem
unfortunately
but i have collected some hints that
could help you what is behind this
problem first of all the quantifiers
like
dot star dot plus dot
star even like question mark
is basically matching everything so
uh don't use it
until
when it's not really necessary
because sierra really can
cause a problem and even with
combination with
unfixed prefix
you can easily get over one million
matches in one file
then
whatever doesn't like as well is
quantifiers without upper bound so say
okay i want to match at least for
example 14 10 on whatever
characters but i don't mind too much
more this is also a problem because here
i will try to find the longest match so
it can even match the whole thing like
whole file which is not probably what
you want
large
two large arranges are also problem just
keep it minimal what you really need to
match basically
uh it has
i'm sorry hexadecimal strings you can
use so-called gems jumps so you can skip
a large part of files
this is useful but again be careful how
you use it because if you skip for
example almost whole file then it should
be probably used in different way and
you can for example use the function
that check specific uh specific like
offset of the file
white cars characters like dot and etc
also
uh can
really
uh increase the time necessary for
scanning
so maybe even consider to split your
strings into two or maybe even more
version
so you are just
matching the parts of it and the things
between are just left behind
alternation era unfortunately doesn't
work very well with alternations
so
even though the readability is important
of course
maybe try to split if you have
multiple alternations in one string
split them into more
definition you can find out that the
speed actually
increase and the rule works better that
way
and also even though i was saying before
be careful with options like white no
case etc there are options that could be
actually useful for example food world
that is
like
we are saying that has to be
white space around the match which is
also useful for yara because it will
actually
it will eliminate some cases of
potential matches
conditions conditions as i said before
are there like boolean expression and
the main part of the rule when we are
calling uh some other functions and we
are working with previously defined
strings
what is not the the best i think the
evolution evaluation of static part of
rules
is done before the conditions
so if for example you have like you want
to eliminate the input and said okay i'm
carrying only about files
where the file size is less than for
example 100
it won't help you because the for
example very expensive frequencies or
complicated hexadecimal strings will be
evaluated first and then
here i will search oh okay this file
actually has like 10 billions gigabytes
oops sorry
so
this is problem we were also talking
with auto sofia about it it's basically
work in progress maybe in the future
it will be rotated a bit differently but
please for now keep it in mind that this
is a
thing that will not influence your your
rules what you can do instead uh use
short circuit evaluation there is
already in error
and basically
it means that the functions and other
stuff besides the static parts
you can change the order and actually
improve your speed
in previous version of the rule i was
calculated entropy first and then i
checked the first like
bytes of the file
entropy as you probably know is quite
expensive operation so if i switch the
the condition and the order of this
the speed will also increase because
it's very simple to check first bytes
versus to calculate entropy of the file
size
what is also nice is the last point here
uh
jara has integer loop optimization
which means that if you are using for
off or for any
first case
that breaks
the condition
in in case for all its first like
false
expression in case for for any it's true
expression the loop will stop it writing
so this is basically work done
for you by era
now we have a little bit modify a rule
uh here
we are not longer using no case but we
are specific that we
have regular expression just with one
class
where c can be both upper and lower case
uh we
also play it a little bit with regular
expression again this is not like
general solution
but if you try for example on your data
set or your examples
to find
some
more specific strength that still works
for you
the better and i changed the order in
condition so the the bytes
at the beginning of file are checked
first and the result
we
can now scan
all the data just in three seconds so
i'm not sure if you remember firstly we
it took uh 45 minutes and now we only
need three seconds which is i think
quite good right
additional tips i will be very quick
because i'm running on the time
uh
one i think i would say it's really
great uh yara has now option
skip larger which is it's said that
files on over some size will be skipped
entirely which also can drastically
improve your scanning so check it out i
have some more resources here of course
github pages documentation maybe you
know
florian wrote we worked together a
little bit uh
on guidelines how to improve your rules
so check it out and also if you are
using era
i would strongly recommend to
use key base
channel yara friends where you can
discuss possible improvements into
future versions and generally ask for
help
in some specific cases
just very quickly
our changes because i
speak a little bit longer than i
expected
motivation
we again
work with warning slowing down scanning
it's really really everywhere and we
wanted to
basically
match bitcoin addresses which are yes
regular expression but it's very
specific format so we try to find a way
how to work with it
i propose the solution that changes
basically every step
of the static evaluation
i would recommend check my paper that i
wrote with my colleagues uh just some
highlights the bitcoin addresses in our
version are 10 times faster than in
upstream version and scanning with no
case option is about
27 percent faster
i'm not going into details
i don't want to give you headache but
you can read the paper i'm saying and
there's more uh avast is very active in
your community so we are posting bug
fixes our new like additional functions
and etc so you can actually watch our
twitter and you can see like many many
many changes and in the future we are
planning
more
other resources my paper and also i have
to request in upstream i promise i will
update it soon because it started like
tons of
match conflicts and etc
i hoped i will
done it before this conference but like
you probably know like to the list is
too long
and etc so i will update it soon and
for conclusion you can follow me on
twitter and link it
in next like uh
years or months
dealing now with more like behavioral
analysis
under my teeth generation of yara rules
and i'm also working with google modules
so you can
look forward to it
and if you have any question please
ask them now or later for example during
the lunch
thank you very much for
your time
thank you
thank you dominica
any questions any year anna's yes
thank you for the presentation um i'm
also a yara contributor so from time to
time great for you
there for us everyone
slide 14 i had a question
if you can go back
if you
take the condition on second line
to check the first byte if you
push it up before the strings evaluation
does it change anything
no unfortunately no this is actually
like the the line here
uh the strings are checked first and
then here i will search okay what the
condition actually is
okay thank you i wasn't sure okay
there's a yara test before the gala
tonight i was looking at the net module
recently and uh wondering quite whether
evaluating some of the um
parameters you can extract with that is
slow
or whether it's uh um
well do you think about it
yeah yeah i understand they're actually
the thing about modules they are very
useful but as you said in most cases
they are very slow uh my colleagues are
working on the on that module and we are
trying to improve it uh
my specifically i'm working on cuckoo
module which is also quite slow
so in the future
i i cannot promise you but
hopefully it will be better but yes but
also
i would ask if you have some problem
with yara and you want to improve some
some stuff
a contribute that's the best option or
at least be asked for it because the
main entrance are very active but also
very busy people and if you will push it
like hey i really need this they will
make time for it and maybe in next
version it will be solved so please be
active it's very helpful for everyone
thank you another question
up there yeah
uh i have a question about the strings
which are evaluated first
is this a scheduled improvement to
evaluate the things in the order of the
condition
i mean
the person writing the yara rule
would know the
the expected uh things that would be
slow or fast and make the condition in
the right exactly yeah yeah we are
actually discussing this with virus
total uh to be honest they have some
reason why they implemented it this way
they need uh some specific features in
various total like
their platform
where they're scanning like
unbelievably large amounts of data but
for us uh
is not always the best solution so we
are trying to find the middle ground and
yes this is uh one topic we are
actually discussing with them so maybe
in the future there will be some option
for that
okay a last question over there
morning jogging
hi thanks for the talk a quick question
did you evaluate the
performance of the new modifiers such as
base64 or xor modifiers for the strings
yeah we are working with them as well
i just didn't want to complicate the
presentation even more
i would say in generally
be careful with them because they were
also discussing
how many things you can match with them
and
the
main object of these new features
obviously
he also was quite worried that
two general
would actually really slow down error so
yes
use them if necessary but be careful
about them because they will generate
many like
options that ya is searching for and
this is again this can be really really
slow
thank you
okay thank you very much
thank you have a nice day
thank you