all right good afternoon ladies and
gentlemen distinguished guests I want to
thank the NATO CCDC OE for inviting me
here to speak today with these excellent
guests in this technology session this
events icon is a spectacular event where
we blend technology policy and many of
the other aspects that are relevant to
the cyberspace domain the works you'll
see here today capture technical feats
that demonstrate the potential of open
source intelligence
I'm from the army cyber Institute in the
United States of America and I'm proud
to be joined here on this stage by these
three speakers mr. Roland Meyer a PhD
student in the Department of Electrical
Engineering and information technology
at ETH Zurich dr. Martin Stroheim er a
postdoctoral researcher in the
department of computer science at the
University of Oxford and mr. Andre
melegari the chief marketing and
innovation officer of cypher gate a
proud sponsor of this event thank you
please join me in a warm welcome and
round of applause for our speakers in
this session you'll hear each of these
speakers present a piece of work
relevant to the potential of the open
source intelligence process following
each of the remarks they'll be available
for questions well it's still their
materials still fresh in your mind so
I'd ask you to please think through
their presentations how you can
challenge them and endeavor to have a
discussion on this important topic
let me now set the stage our world is
awash with data with the growing
connectivity of devices and increasing
digitization there's an unprecedented
level of publicly available information
the president of Estonia put it
perfectly this morning in her remarks
that perhaps we can acknowledge a new
natural law as sure as the law of
gravity that when she said if it is on
the internet it is visible this simple
powerful statement if it is on the
Internet is visible it is visible
we can begin to see the potential of
open source intelligence I'll go one
step further and claim that open source
intelligence is the process of taking
this publicly visible information
analyzing it and then generating
insights analysis and in value it's
important to highlight that the raw data
alone is not sufficient it's only when
it's been processed and analyzed that it
can generate intelligence I'll go one
step further and say that intelligence
alone intelligence just for the sake of
intelligence is insufficient
intelligence is only truly valuable when
it is actionable there's a saying in the
military the intelligence drives
operations and I think therein lies
where we can see the great potential of
open-source intelligence if what we seek
is to maximize the effects that we have
both positively and minimize the risks
inherent to this domain data alone is
insufficient intelligence alone is
insufficient we must strive to apply
open source intelligence to truly
deliver those effects and enable
operations whatever your operations may
be with this in mind please join me in a
warm welcome for mr. Rollem Mir to
present his work on feed rank
methodology for ranking cyber threat
intelligence feeds
thank you very much hooray for and thank
you very much cyclone for having me
today I'm it's great to present our work
on feed rank a temper assist method for
the ranking of cyber threat intelligence
feeds this is joint worth work together
with Cornelia Sharon and Laura from
baver from ETH Zurich as well as David
googleman from elección analytics and
Vance our lenders from armistice but now
before talking about cyber threat
intelligence feeds and how you as a
network defender can pick the right feds
sets of feet to deploy in your network I
want to start with another example and
this is ranking websites so as you all
know ranking website is a very important
topic a particular when it comes to
search engines where they need
to have a matching or a ranking given a
query which is the most relevant website
for this query now a very naive strategy
to do this would be to just take the
website that has the highest overlap
between the search query and the
contents of the website however this is
not going to work because if I was the
operator of this triangle website and I
offered trips to four cities
I would not only describe these four
cities but I would just add as many
keywords as I can think of such that if
you search for any city or if you also
search for videos about cats you will
find text on my website even though it's
not at all about these topics so the
underlying problem there is that there
is not necessarily a relation between
the content of a web site and the r11
relevance for the website for these
topics and the solution that this widely
used today is called PageRank
a famous algorithm that is used by
Google and this does not this algorithm
does not rank websites based on their
content but based on their relationships
with other websites and in particular
based on which other websites provide
links to up to my website for example so
the intuition is that if a lot of
high-quality websites contain link
that's point to my website then my
website is maybe also of considerable
quality but now let's move from ranking
websites to cyber threat intelligence
feeds these feeds are basically lists of
IP addresses that are somehow associated
with malicious behavior for different
categories for example that there can be
a feed listing IP addresses of servers
that were used to send spam of servers
that host phishing websites or of
servers that distribute malware of some
other kind or whatever malicious
activity there are feeds that cover them
there is a large ecosystem of both
commercial and open such feeds and as a
network defender they are very important
source of information as you would like
for example to block all traffic towards
IP addresses that appear
phishing feet or you will block email
addresses arriving from servers that are
known to send spam however because there
is such a big the ecosystem of these
feet it's also difficult to pick the
right set of fees that are deployed in a
network because if you pick the wrong
ones or too many or too few you will
block either too many IP addresses not
enough IP addresses or maybe the wrong
IP addresses and all of this has
consequences for the security of the
network or for the usability in the
sense that you either don't block
malicious websites or you block websites
that are not actually malicious so we
were looking at how we can evaluate such
cyber threat intelligence feeds and the
problem there is similar to the ones
with websites that as a feed provider
you can list arbitrary addresses or
arbitrary IP addresses or domains on
these feeds and it's very hard to tell
given such a list of IP addresses what
the quality of this feed is you can also
copy entries from other fees and combine
them in your own feet and the idea the
underlying idea behind our approach was
that we look at correlations between
feeds and we look at IP addresses that
appear on multiple feeds and on the
sequence in which they appear on
multiple feeds with the idea that if an
IP address is malicious because it
appears on many of these cyber threat
intelligence feeds then we say the first
feed that was listing it is probably a
good one because it was first and it was
an IPS it was later confirmed by many
other feeds
so in summary with our feed rank
algorithm we wanted to design a
mechanism that allows us to identify
high quality feeds that is customizable
for different purposes of network
defenders that does not require any
other information besides the contents
of the feed that scales to large
ecosystem or to a large number of such
feeds and that is robust against feed
providers that try to cheat with the
ranking like in the example before with
the website similar to adding many
to a website you could just add many IP
addresses to a website to answer to a
feat and this feat will appear very big
and maybe some users could think that
this is a good fit because of this
speaking of good feeds we argue that a
high quality feed satisfies three
properties
the first one is completeness that it
contains all IP addresses that are
malicious at this time the second
property is accuracy that it does not
contain IP addresses that are not
malicious and the third one is speed in
a sense that if a device or not
addressed becomes malicious it should be
listed on the cyber threat intelligence
feed as soon as possible the underlying
architecture behind feed rank is shown
here so we first start with collecting
input data then we perform two kinds of
analysis one for the for correlations
between feeds another one for the
contribution of each individual feed and
based on these two analyzes we can
compute a final ranking of all the
considered feeds and I will now go into
into each of these steps in a bit more
details first regarding the input there
is not much to say there all we need is
at least two snapshots of the cyber
threat intelligence feeds that you want
to consider in your network and the two
snapshots being taken at two different
points in time while the time between
the two snapshots should be chosen such
that defeats have the chance to update
in between so in our case we collect
many snapshots with with one hour in
between them then second we go a bit
more into the details of the correlation
analysis here the basis for this
analysis is a graph where each feed is
represented by a node and edges between
this node represent correlations between
feeds I will explain this with a small
example cause in during feed a and B
here a has the entries 1 2 3 4 P has
entries 5 6 7 at time 0 and now later
maybe one hour later at time one we have
feed a still with entries one two three
and four
and feet beat now also has entry three
and four listed so if we look at time
one both feet agreed at entry three and
four are malicious but we see that a was
earlier in reporting these two entries
so we would say that feed B confirms
these two entries from fides of heat P
confirms that these two entries are
malicious and we represent this in the
correlation graph by adding an edge from
V to a with the weight corresponding to
the percentage of entries that were
confirmed so in this case it's two out
of four entries of a so it's 50% the
second analyzes that we do is for the
contribution of each individual feed
here we start by taking the union of all
IP addresses that were reported by one
of the feeds and then we compute which
percentage of them was reported by was
reported first by any of the considered
feed so the results could be for example
that we end up with 50% of the IP
addresses appeared first on feed a and
some smaller percentages on feed B C and
D based on these two rankings one can
then compute a final score for each feed
either by just computing a vited average
between the two between the scores of
the correlation graph on which we are on
a weighted version of the PageRank
algorithm and on the contribution score
or one can compute combine these two
metrics in a more sophisticated way so
coming back to the three properties that
I mentioned in the beginning
completeness accuracy and speed we see
that completeness is represented in the
contribution analysis while accuracy and
speed are represented in the correlation
graph where the accuracy is represented
in the edge weights which say how many
entries of one feet are confirmed by
another feed and the speed is
represented in the correlation graph by
the edge directions which point to the
feet that was falster in listing the
entries I also said that we want to
design feed rank in a way that this
robust against malicious feast
try to improve their ranking so the
scenario is that I for example have my
own feed I have a list of IP addresses
now what can I do to improve my score in
this scheme I can either add entries I
can remove entries or I can combine this
and replace entries however all these
strategies will not be successful
because first if I add entries I somehow
need a way to select them if I pick them
at random they will not be confirmed by
any other feed so they will not help me
if I copy these entries from other feeds
then I will always be slower than the
other feeds so it doesn't help me either
the second one for removing entries I
can be lucky and remove the good entries
so the IP addresses that are confirmed
by other feeds in this case I try
improve my score but then I thought I'm
not really cheating because I'm really
improving the quality of my feet and if
I'm unlucky and I remove entries
I removed good entries that were
confirmed by other feeds then my score
obviously decreases and finally
replacing entries is just a combination
of the above two strategies in the last
part I want to give some insights in the
evaluation that we did unfortunately I
don't have time to describe everything
here but I want to briefly describe the
data set that we used and then show the
results of the contribution another
analysis of this data set as well as the
impact of malicious feeds in both in
with feed rank and with some individual
more easy to compute or ranking schemes
if you are interested in the other
evaluations please have a look at the
paper or talk to us at the first four
authors of the paper are all are all
here so please
ping us in the paper we also show the
two case studies very precisely describe
how network defenders could use feed
rank to select fees for different
purposes so the data set that we use for
the evaluation consists of 27 freely
available IP address feeds of various
size between only 20 entries and 50,000
entries
amongst others they are from these
providers that are shown here all of the
providers provide more than one feet and
in addition to these real feet we also
analyze the impact of dishonest feet
sofie's the try to cheat with the
ranking and these fees we generated
ourselves the first one we call it a
random feed which has 50,000 IP
addresses that were chosen at random and
the second one is called copy feed which
copies the best the entries from the
best two real feeds and distributes them
as its own entries so for the
contribution we can see that feed rank
allows to pick the best feeds with
respect to contribution so in our data
set by picking the best fees according
to our ranking the five best fees
contribute 80% of all IP addresses of
course this depends on the data set that
you use but it's also interesting to see
that only the size of a feed is not a
good not a good metric to to pick fees
because of large overlap between some of
the feeds and this you can see with feed
rank the other experiment that I want to
show is to show the temporal resistance
of feed rank compared to individual
metrics
these individual metrics that we
consider where the feed size or the
number of n the number of entries listed
and the update rate in terms of how many
IP addresses are added and removed over
a certain time interval and we see that
for these individual metrics we have the
good or the malicious feeds scoring very
well while for feed rank we have the
real feed scoring well and the malicious
feed store scoring at very bad positions
with this I'm come to the end of my talk
I hope I could show you how feed rank
works that it allows to identify high
quality feeds for different purposes of
network defenders and it does work
solely based on the contents of feeds in
at least two different snapshots and
does not require any actual validation
of the entries it scales to the large
ecosystem of cyberthreat feeds and it is
robust against malicious feed providers
so thank you very much for your
attention and I'm happy to take some
questions now
[Applause]
any questions I've one wrong when you
look at these feeds how did you choose
which feeds to analyze is it better to
have all the feeds you can possibly get
for our analysis we picked just as many
features as possible that you as many
open feeds as possible that you could
get if you later use feed ranked the
idea would be more that you pick the
feeds that you actually consider so if
you are if you want to select feeds for
a spam filter you would analyze only
feeds that focus on on spam IP addresses
for example so it depends on on the gold
what you want you what you want to
achieve with feed rack it will just feed
rank will allow you to select the best
feeds out of those that you consider as
an input another related question is you
demonstrated with feed rig on IP
blacklists
very structured data how would you apply
this technique to other forms of threat
intelligence beyond IP addresses well as
long as you have a list of entries so if
this is IP addresses or domains or other
distinct entries the technology does not
really change so we compare if the
entries appear on one feed and later on
other feeds so if this is an IP address
or domain name for example this has not
really an impact there any other
questions yep you sir in the back
I'm wondering if you compared your model
to other feed scoring models for the one
particular I'm thinking about is Alex
pintos thread IQ where he looks at
novelty overlap population but there are
few other ones out there so far not as
far as I know so I'm not unfortunate not
familiar with this exact model but this
would be future work choo choo consider
that as well yes thank you
wonderful let's again give Roland great
thanks for his presentation thank you
and so while Roland presented a building
block for how we can choose which data
we want to clue include in our open
source intelligence analysis Martin will
now present his work on a another domain
looking at air traffic controller data
and showing how once you have selected
which data to use you can build to
determine facts from that hello and
welcome to my talk on extracting open
source intelligence data from aviation
communication my name is marcin
Strohmeyer and this is joint work with
my colleagues messiest myths and even
Martina witch from the University of
Oxford
Danny Moser from ETH Zurich Matthias
Schaffer from Kaiserslautern and Winston
lenders from amethyst and some of these
people are also in the audience so you
can go find us afterwards so let's have
a talk look at what we talked about
today first I will quickly introduce the
enabling technologies that allow us to
gather widespread open source
intelligence from aircraft communication
after that I'll explain the methods that
we use in particular for a study before
presenting the results in particular the
scope and the scale of the problem and
some case studies which should
illustrate the real-world impact this
can actually have finally we'll have a
look at potential countermeasures that I
used currently and see whether they're
effective in any way so on data
collection there's two different types
of data that we collect the first one is
structure
tighter the most important one is on the
movement of aircraft so this is simply
telling us what aircraft is going where
at what point on related we need to know
who is using these aircraft so we're
also collecting the aircraft metadata
importantly this helps us to find out
what is interesting to us is an aircraft
sensitive is it used by government
institutions and military beside that it
also allows gives us information about
the country when aircraft is form but
also aircraft type for example the
second type is also some unstructured
data that we can get from a traffic
communication mostly what I'm talking
about as free text messages of course
there can be all different kinds of
interesting content in these the two
main technologies that enable collection
of this data are called
acres and NTSB and in short they're both
technologies that are broadcast by the
aircraft to the ground or anyone who
listens and you can easily receive this
with a USB stick software-defined radio
that is can be as cheap as ten dollars
and you can receive all of these
communication it's unencrypted from as
far away as 500 kilometers and even
further if satellite communication links
are being used so there are two ways of
actually getting this data the first one
we could deploy our own receivers as
we've seen is cheap and easily doable
but this would strictly fall into the
realm of signal intelligence
communications intelligence so that's
not the way we pursued it here the
second option is to obtain the same data
collected the same way by existing
institutions organizations websites so
this is actually open source
intelligence the most well-known of
these such institution is probably
flightradar24 large commercial website
collecting this data we however for
study got the data from the open sky
network which is a nonprofit
organization based in Switzerland which
collects air traffic data specifically
to help researchers study various things
not only security privacy but anything
you can do with a traffic data as you
can see here the focus at the time in
particular was on Europe and the US but
this has been widening widening steadily
and you can get communication from all
over the world now not consistently but
it's a lot this illustrated by our first
data set on government aircraft so what
we did is collect the metadata on all
government aircraft that we can find
from various online sources not going
into the specifics here you can find how
the sources and how we did that in the
paper but suffice to say that we found
113 governments and the aircraft in this
data and indeed a large majority 94% we
could actually see during the year 2017
in the open sky data the majority is
from Europe as you can see 8,000 of the
14,000 flights but we've seen flights
from all corners of the world this is a
visualization of the data set just
showing flights from governments to
Europe different regions as you can see
of course there are hotspots in London
Brussels Paris as would be expected but
also Switzerland or nice the second data
that we examined for this study was the
military we looked at 6,000 aircraft
over the course of one month you can see
the origin of these aircraft here of
course a majority of these comes from
the u.s. reflecting surely the global
size of the Air Forces but also the
coverage area of open sky the next
biggest countries were all in Europe
Germany UK France but again you can see
we've had data from countries from all
corners so to illustrate the scale of
what can actually be the problem here as
we looked first at multilateral meetings
so a meeting was defined as
a government or a craft from three or
more governments being in the same
airport at the same time so you could
expect that there's at least some reason
they're there of course on average there
might be some noise in there but and in
general we expect that there's a reason
that they end up at the same place at
the same time one illustrative examples
of these 116 four meetings that we found
during 2017 is this example from the end
of January in Zurich where of course all
the participants are most likely going
to the World Economic Forum in Davos and
Zurich is the closest airport so as you
can see here there's 12 aircraft landed
on the runway in Zurich and this is
basically the point where we lost
contact to them so this is how closely
you can track these aircraft
besides meetings you can also or besides
multilateral meetings you can also look
at my lateral meetings which can give
away relationships between countries
over time this is a table showing the
meetings that we have the relationship
that we have seen most often in our data
and of course not all are necessarily
very unexpected for example Joni and yes
having aircraft and the same place is
probably not very surprising they're
closely related Allah is saying for
France and Morocco of course
historically sorry Arabian France might
be more interesting but then again we
are not necessarily with this study
looking always into the underlying
reasons why things happen or just why
really which is something that I think
Andrea will be trying to address and in
his talk afterwards another example
would be looking at where aircraft
actually going our time so these are the
top foreign destinations of nine
different governments just an example
again not everything is very unexpected
at least on aggregate people are going
to
Brussels of course very often in Europe
but also to the neighbors not too
surprising but the point is that we can
illustrate all of this very in very
detailed fashion for example we can look
more closely at what the aircraft of the
government in Dubai has done in 2017
seen the landings over time with the
clear hot spots in the UK in particular
in London so of course there are some
consequences to these intelligence
gatherings and that's something that we
need to illustrate the most striking
example is probably set up by two
investigative journalists in in Geneva
they set up a Twitter bot called
dictator alert and what it does is it
uses the same technologies that we use
here and looks into all incoming
aircraft into Geneva Airport it then
compares that with a list of
authoritarian governments and their
aircraft and automatically tweets out
whenever the incoming are going and of
course this has attracted some it some
attention over time and even had some
real-world impact as illustrated here
where the president of the vice
president of Equatorial Guinea actually
had some assets seized in Geneva partly
because we tailed engines gathered
through aircraft tracking through these
spots because these spots as we know
actually watched by international
organizations and also NGOs looking for
for reasons for a corruption for example
and use this for the intelligence
gathering we can also like if as we see
this can be a problem so you would think
there's some awareness of this problem
and people would try to hide the
aircraft from these intelligence
gathering undertakings and they actually
do as we can see in different ways one
example is this military aircraft which
switches off their positional
broadcasts every time they start and
land so it's not not something that is
possible or really allowed for
commercial aircraft but military
aircraft have the capability of doing
this and it is very clear that they do
this so people cannot at least very
easily track where they're going where
they come from a second and the most
notable example of this kind of interest
in privacy that people who use their own
aircraft have is this one where on
flight radar and many other commercial
websites aircraft are effectively
blocked from you and not shown and this
is you two lists provided by the
aviation authorities such as the
American FAA and if you're on these
lists you're either not shown and all or
s in this case the aircraft is shown and
where it's going but it's anonymized so
we do not know who's actually using this
aircraft and indeed in our data sets we
find a lot of evidence of this happening
so people clearly care about it of all
the government's more than 60% had their
aircraft blocked from you and these
commercial websites of the military it
was even 85 percent of all aircraft
beyond this you can see in the
comparison here the US or the Americas
overall have the highest number of
blocked aircraft with over nineteen
eighty percent respectively but it's
fairly high across the board however
from a security researchers point of
view we argue that is mostly security
through obscurity because as this case
very clearly shows during the
presidential campaign early 2016 Donald
Trump's aircraft personal aircraft was
taken off the commercial websites and
put on the on the Block lists however
while it's indeed not visible on these
websites anymore
you can still gather your own data it's
all out there it's not a problem
actually finding it
and one of the other system that I
mentioned that actually undermining this
privacy is the a car system which has
often been duped text messaging for
aircraft in our data sets seven point
seven percent of the military aircraft
actually used this system and even
thirty percent of the government again
this is entirely unencrypted and neatly
it uses the satellite data link a lot
and that means that we cannot only
receive it from up to five hundred
kilometres but pretty much or within
range anywhere on earth the majority of
the users of this system actually give
away their position using it and even
the indication in advance where they're
going so of course then blocking you
your aircraft on the website is not
really helpful beside this we have some
unstructured data coming from the system
free text communication one example I
can illustrate here this is a special
air mission conducted by Air Force two
in December 2016 and the information
sent here on the configuration is
presumably not only sensitive but we
also get more information about the
cargo including some code words - just
great ghosts and white elephant then we
don't really know what they mean but
they surely mean something to someone
more examples includes information about
the fuel loaded the crew enlisted on the
aircraft and even specific information
on the passengers from the number of
people in a number of senior staff
twelve in this case - how many people
are in the stateroom on this particular
mission if we take all of this
information together we can see you for
this specific flight some information
that like get us the whole picture so we
know that the aircraft took off from
Saudi Arabia around 5:00 in the morning
a couple of minutes later we get the
flight plan received in Central Europe
an hour later we get the text message
that we've just seen before later on at
Point 2 here it enters the then
reception range of open sky where we can
track it with different technologies at
some point it leaves it again
and finally in lands in Island so if
these options do not really work what
can we do well one way would be to use
commercial transport and in the head of
state have been known to actually do
this presumably not for privacy reasons
because we do not know any kind of
sophisticated technology or analysis
measures when you do that it's after a
short time this will be all over Twitter
and all over the mainstream media's in
this case for example so to conclude
modern technology and now enables
basically anyone to collect aviation
data which can be turned into open
source intelligence easily is trivial to
track pretty much any aircraft out there
we also usually get information on the
owners of these aircraft and we've shown
that there are definitely some privacy
consequences to this type of open source
intelligence collection there are some
approaches to mitigate this but while
they're popular they're certainly not
very useful at the moment that I
conclude my talk there is a demo on this
link here if you're interested we'll
also tweet it but yeah feel free to play
around but this is for me thank you
we'll open it up to questions
sure thank you
you said at the very beginning that you
get some information from the deck the
text messages in the open source way of
course which one they are how do you
have the the value or the first ability
of that kind of message the value or the
trust and Trust ability sorry yeah what
is the the value in terms of trust on on
trust I mean there's a good question
because of course since it's all
unencrypted people could also use this
for deception and in this point of
course let's say if an example from a
military aircraft they could send this
on purpose
absolutely we don't know whether the
content is true it's not it's not
impossible since the system is entirely
unencrypted unauthenticated you can and
principle do what you want with it on
the satellite communication this is at
least a little bit more difficult
technically than the direct line of
sight communication
the message come directly from the
aircraft so this is why a wireless link
occurs in this case for the messages for
the text messages it's called a question
sir
question over here I'm curious from a
logistics and securing logistics and
supply chains have you applied this open
source type model and thinking to rail
as well as commercial shipping and
freight not yet we know people are doing
that so in particular for shipping I've
seen lots of commercial entities there's
a very similar system to edges be used
for shipping is called AIS and people
are absolutely following this to see
what's happening on the oceans in
combination with I think a lot of
satellite photography to see where where
ships are but we haven't done this for
any other domains yet no one question
for you or as you look at this some
people might be surprised that all this
information is publicly visible and
available you mentioned that they're
kind of the methodology of obscurity is
not a fantastic solution do you think
there are technical solutions that are
feasible
are there regulatory where do you see a
way of making progress here yeah I mean
the technical solution is probably what
we have been working on for years of
informing people not only about the
privacy consequence but the actual
security consequences that are very real
you can introduce new technologies it's
we know how to secure wireless
communication it's not a fundamental
problem it is mostly the case that
aviation and the legacy legacy systems
used in aviation other critical
infrastructures it takes a very very
long time to develop implement actually
deploy them globally in such a system so
this is something that needs to be done
but it will not actually cover
of the problem even if it's done now for
the next 20 years I would say so on a
regular Terry basis I mean I think there
has been a topic of debate today already
I I think some there are some countries
like the UK who actually have a law
against collecting wireless data that is
not addressed to you specifically
however how do you a prevent this or
even prove this and unless you tell
anyone so I mean it's not encrypted so I
there's potentially some space here to
regulate like all these aggregators but
they may also just move to a different
country of course so this will be very
very difficult
speaking of these aggregators what are
the incentives for individuals to
participate in this you know these
networks that are you know kind of by
individuals choosing to join and
contribute to open sky enthusiasm
it is obviously plane spotting just like
transporting husband
it's a surprisingly large copy like many
people are doing this and it's become
much easier than ever so much of the
metadata is collected by people you can
collect this also on on Twitter let's
say people are going to Zurich during
the World Economic Forum looking at all
the machines coming in may be I'm sure
people were here at Tulane Airport ice I
saw several governments when when I
landed right so I'm sure there were
people there they collect the call signs
the registrations and this is posted and
yes people are very happy to have their
own receivers for all this kind of data
installed and share it with people and
well they see the value of this kind of
crowdsourcing because if they feed it
they also get the data from everyone
else so they love to do that yeah any
other questions so I give Martin a hand
please
so what Martin just demonstrated was how
you can take a specific source of open
source intelligence and really fully
develop it to come to surprising bits of
information that might not otherwise be
readily apparent particularly when
combined with other sources he
specifically mentioned this move from
structured data to unstructured data and
how the combination of those two sources
aundrea now will provide us a
demonstration that really pulls all
these aspects together and shows some of
the power of open source intelligence as
well as particularly in this
unstructured data realm thank you thank
you for your presence
I promise live demo so just few slide in
order to introduce the topics and then
go through immediately to live demo so
last year I attended this conference and
Robert Khalil one of the most brilliant
director of the intelligence USA
intelligence community said that now we
have one data and in five years we will
have 1 million data so the growth of the
data will be from 1 to 1 million but the
most interesting thing is that it needs
a lot of analysts and I wondering if
Robert Cardillo needs 8 million new
analysts how many analysts must be
involved in open source intelligence
social media intelligence communication
intelligence and so on and for this
reason Cardillo is changing the NGA
strategy and he decided to invest in a
AAA technology so the goal is to
implement artificial intelligence
augmentation and automation the goals
are very obvious first of all improve
productivity
second deal with complexity and
lastly try to involve the analyst in
what the human being can leverage in the
best way most of the knowledge we deal
with every day is what we call
instructor data so we are talking about
email documents web pages article tweets
and so on 80% of the knowledge is
instructor and dealing with the
structured data is oh very stressful if
you have two pages with twenty nine
sentences almost 800 words manually you
need 30 minutes to extract to tag all
the information you can find here for
example topics and it is relationships
and so on what you can extract this is a
partial list of the type of entities you
can find in a document in a text
generally speaking if you can extract
topics entities domain entities
relationship write print facts and so on
what we can do is to consider that the
artificial intelligent technology can
help us but please forget to obtain 100%
of quality okay the best-in-class
machine learning technology is able to
give you a precision of nineteen ninety
five percent but is really a very good
support because instead of thirty
minutes in just few seconds milliseconds
you can get a very interesting result
let's start with the live demo so is
something by the way you can you can't
ask because the website of this first
live demo is open so WWE intelligence
API dot-com this is a BBC website we
have a lot of good example of instructor
text let's try to consider one of the
instructor test this is a very famous
news and let's try to understand how we
can use the machine learning technology
in order to process all the information
you can find here so i copy the URL then
i go to the live demo and i past year
the URL and then the system the machine
learning technology is processing the
documents so this is a very quick
summary of what we can extract first of
all on the Left we can extract
categories and emotions emotions
sentiment is not enough we can try to
summarize the document with different
type of metadata is something that you
can see here we can automatically build
the entity relationship maps and we can
J locate on a map the different
Geographic elements we found in the text
let's try to understand more first of
all we love to tag the document
extracting different type of semantic
metadata and the summary what you can
get is a summary where you can find the
most free relevant sentences
that are present in the document but the
same time here you have a lot of other
semantic metadata it is something that
we call collocation concept lemma and so
on so for example who cry Prime Minister
ball Audrey Grossman
it is a colocation credit card is a good
example of a collocation you have credit
you have card but credit card together
as a specific meaning the most
interested thing is that the system can
understand the topic mentioned in the
document an ear on the left you have the
different categories that the system is
tracted from the document you have many
taxonomies on the left the first one
contains 800 different categories the
second one crime taxonomy is for the law
enforcement agency about 40 categories
and then you have terrorizes taxonomy
and probably if we scroll down we have
also geography taxon let's try to
understand what what the machine
learning is able to do here for example
we see a and space acident
take a look at the text here you don't
find the key or Ariens pass acident you
find crashed tu-154 okay it is a
semantic categorization if we click on
act of terror if we click on explosive
if we click on on the side so it's not
important to say the document talked
about on the side but it is important
for the analyst to go immediately to the
specific portion of the document where
you find the specific atomic part of the
information that is related to the
specific topics so we can categorize
with a good quality we can
extract many entities take a look the
capability is not simply to extract
proper name but also he okay so we can
deal with a lot of capabilities that you
can see here and this is something that
is very useful what is possible is to
extract fact so we can say that's a
reason act of terror where the Kremlin
is involved one sentence one sentence
and I mentioned the emotion why why
boredom here take a look become routine
we extract eighty different motion it's
not so simple because the starting
emotion is really really very stressful
activity to implement in an algorithmic
this is the result we can we can
obviously build automatically or the
connection that we found in the document
correlating the entities with other
entities with the other relation
obviously we normalize sad take a look
here we have a specific relation where
we say where there is sad sad means say
communicative this is part of what we
can do so let's try to add the
possibility to ingest this capability
inside a platform where you have
ingested a lot of data specifically for
this goal we have ingested a lot of data
coming from an open source database
flight Explorer
and we put everything inside the
platform so in this platform you have
the data from flight tracks and you have
a lot of and structure data with this
dashboard you can immediately analyze
the content of the unstructured data so
for example here we see military asset
and we found that there are some flight
racks that are related to military
aircraft and now here we see that we
have a part of the aircraft that change
the route and now we have data here and
we can say okay I'm interested in
analyzing this area and the system now
is filtering Wow we have Israel Air
Force here big cloud I click on Israeli
force I can try to understand more and I
can click on a specific track and now I
have all the databases where the Israeli
Air Force is present since we selected
it so now we know that we have Israel
Air Force in two areas
let's try to maximize it and it is
pretty normal that Israeli forces here
by I don't understand why Israeli air
forces present over Greece and this is
the problem we know what Israel force
we'd know where over Greece we know 1 6
April but
we don't know why the capability to
combine two different data structure
data and the structured data could help
us
now I'm asking please investigate inside
the instructor document if there is
something correlated to Israel I force I
have a lot of data you can see tweet you
can see a lot of data we have 150
results I don't want to read all these
results but you can see that the system
is telling me oh there are a lot of
result are correlated to armed force the
topics we have extracted with machine
learning at the same time the system is
showing me that there are some
geographic information here I'm
interested in Greece two documents the
second one explained that the Israel
force was there for a combined activity
with other eye force and as you can see
here the document is associated to all
the semantic metadata we have extracted
before and it's pretty interesting that
here if we scroll down you see that
there are part of the document there
where you can find the geographic
categorization here you don't find early
speed you find something that is
correlated with we close the loop the
machine learning technology works pretty
good combining a structure data with the
structured data could tell you why and
support support you not replace you in
the best way thank you
are there any questions for Andreea sure
so I've learned about the machine
learning you mentioned what type of
machine learning algorithms are you
using or these supervised or
unsupervised what are you using for your
ground truth data to train on a it's
interesting question when you apply
machine learning technology for some
specific situation you can you can base
everything on the algorithm analyzing
text languages is not exactly the same
of analyzing other type of information
so our approach is to combine supervised
and unsupervised learning in this way
when the machine learning algorithm
doesn't work you can adapt it this is
our strategy and this is probably the
best way to get quality over ninety
ninety-five percent yes over there sir
can you apply this capability on a large
you know huge terabyte data set that is
stored on a storage area network and how
long would it take to to parse that
large of a data set to index it or
however it computes it yeah another good
question first part generally our
customer our government so the data set
are big huge billions and billions of
data that you have to manage in a
real-time imaging all the data from
communication intelligence or now also
from open source intelligence if you
consider for example social media feeds
so we can apply this technology to a lot
of data generally the time you need to
analyze a document it is strictly
correlated to what metadata you want to
attract okay what happens is that the
customers generally scale or is on tally
okay using the cloud is the private
cloud is pretty common if you have a lot
of data in a specific situation
automatically new virtual machine start
and you can process many document the
same time did you have a question you
showed us quite a specific kind of
unstructured data which is English text
would that work for a different kind of
a structured data let's say just another
language that has a different structure
than English example German or Italian
with like directly you're just small
training or would that need a different
set up of the entire system no we we
have developed the technology for 14
different languages developing a new
language needs time but we have the 14
languages what is interesting is that
with the approach that is a mix of
supervised and unsupervised learning
what we can do is customize also the
specific language for example German for
dealing with a specific Jerell
expression you know in the cyber there
are communities that speak different
language
so what you can do with the approach of
supervised machine learning is to adapt
the machine learning results to us to
the dialects that you can find inside a
lot of communities the most interesting
community has a lot of dialects
no question Andra do you see individuals
taking you mentioned using not just open
source intelligence but also then
private data the benefit of fusing these
two sources how does that work in your
experience yeah generally all the
customers try to fuse different type of
intelligence typed different types of
data the main concern is having in the
same infrastructure private and open
source data so generally they have two
systems one works outside the firewall
and then they exploit the data and the
result of the open source intelligence
now comes inside the firewall and then
you fuse with the internal data but this
happens frequently I think we now sit
down we can have a little bit of a
discussion about open source
intelligence amongst the three of us one
theme that's come up is that open source
intelligence while there's a vast amount
of data and it was mentioned we may not
always know the ground truth or the
veracity of this data you know and how
that may impact the conclusions we can
draw in your specific works how have you
addressed this potential noise in the
data or the lack of a known good ground
truth so in in our case we we basically
avoided looking at or validating the
data as such it was one of the key
design choices to only look at this with
the relationships between feeds or the
trust of so there can be relations if
what I've shown that IP addresses appear
in multiple feeds in not at the same
time it could be either because one feed
is copying from another feed which which
can be or it can be that the two feeds
come
to the conclusion that these IP
addresses are malicious from independent
sources but both of these means that the
FIR defeated was first in listing these
IP addresses is somehow it was faster so
it's somehow a bit better
nitrogen others were independent came in
coming to the same conclusion or they
were copying from these feet so they
implicitly trust this one but we didn't
actually have to validate the
correctness of of the contents itself
this is an interesting insight that
maybe you you can indeed sidestep some
of this question by relying on you know
that crowd-sourced nature yes or do you
have any thoughts can in your experience
well you could read our psyche on paper
from last year which actually covers
covers that topic very precisely
homework no the the crowdsource aspect
is exactly what we also exploit so on
the one hand if you have all of these
enthusiasts that I mentioned before it
is possible that some of these people
might poison the data people could
actually like enter these large
crowdsource networks and send data that
is not accurate right however if we have
a lot of receivers in the same area then
of course we can across validate these
things on a network level on like
relatively straightforward level but we
can also use physical layer features
such as the time the signals need from
the position where an aircraft claims to
be to the ground we can usually find
that an aircraft is not in the ground if
someone sends these signals from the
from the ground right it's very far away
and this is actually something that is
very very hard to achieve since you
cannot send faster than a speed of light
so that's very difficult however of
course on a network level if your threat
model is a government military actor
then it can be very very difficult we
know that capabilities are extremely
large so they could subvert the network
as a whole if they really really wanted
to but
at the moment I would say that at least
reasonably unlikely simply because it's
not high-profile enough fantastic answer
again knee you can ability to rely on
constraints in the data to you know pull
out some may be true through some
invariance you want to enforce you need
further thoughts data validation is a
central point if hold the process we are
involved in obviously what what I have
demonstrated is exactly the exploitation
evaluation activity that you can find in
all the intelligence cycle activities
and probably the most common use of what
we produce as all technologists here
presenting largely automated solutions
their solutions that are providing new
technology to tackle this data where do
you see the integration of kind of
humans of analysts into these processes
for your specific fields in order to
take the either raw data and the tools
that you've provided to generate
insights real Liberty could you the I
stressed the concept because now there
are some IT companies that describe
application that can replace the analyst
fake news fake news the analyst is
necessary the the challenge is to him to
implement to use the analysts in the
best way
so the analyst must refine the data but
the technology must support the analyst
in the best way
analyzing this type of data and
structured it is quite impossible
without the human being I think it's not
possible and you know analyzing text
with different nuances is so difficult
so the the analyst will be present also
in the future inside the intelligence
agency not eight millions like Carrillo
said but probably some millions will be
part of the
is community so hardening here we'll
still have jobs in spite of all the
automation you're breaking don't worry
Martin I absolutely agree I mean I think
whatever mention is that we are the
authors of this study at least we
computer scientists all of us are very
technical so we do develop the methods
and try surely to highlight problems and
we know security we know privacy however
we're not in the business of actually
analyzing the data to this extent this
is something we are definitely
interested in we like applications it is
just not our core business that we
usually do we're happy to collaborate
with people on this but experts let's
say International Relations for example
but this is not we do ourselves but as
you see clearly yes we would need humans
to to go further with this because we
don't have the expertise and I don't
think you can just do this with
automated automatic approaches and then
rolling I mean obviously these threat
intelligence feeds you identify they're
going to somebody correct yes exactly so
I think in our case the human is
involved by design of the algorithm we
just we provide a tool to evaluate feats
but somebody needs to pick the fees that
are analyzed so if you pick phishing
feeds and you and you will feature and
we will tell you maybe the best ones
amongst them but then if you use them in
your spam filter it will not work so
it's still you need to choose the right
input data set and then we will give you
this course and you need to combine them
in a way that makes sense for for your
environment and for your purpose there
any questions for our speakers today in
that case I just like to conclude by
thanking you for coming to our session
like to highlight what we've observed
today is the identify identification of
how you can identify which open source
data which raw data you can build your
open source intelligence on you can then
discover insights from that data and
then really get to a place of where
you're determining
why in order to inform decisions so you
can either drive operations or achieve
effects and so thank you and let's again
give our speakers
[Applause]