1 00:00:02,730 --> 00:00:07,839 okay my name is Aaron Leverett this is 2 00:00:06,970 --> 00:00:10,900 Bruce stunning 3 00:00:07,839 --> 00:00:13,960 we are CEOs of companies that we just 4 00:00:10,900 --> 00:00:15,639 made up that's sort of true but 5 00:00:13,960 --> 00:00:18,279 genuinely he's a start-up I'm a startup 6 00:00:15,639 --> 00:00:20,650 that's how it is so it's really late in 7 00:00:18,279 --> 00:00:22,660 the day I was traveling for 24 hours 24 8 00:00:20,650 --> 00:00:24,580 hours ago I was in Cleveland so I'm kind 9 00:00:22,660 --> 00:00:26,859 of exhausted and a little bit you know 10 00:00:24,580 --> 00:00:28,720 fuzzy so if you give me lots of energy 11 00:00:26,859 --> 00:00:31,330 I'll give you lots of energy and Bruce 12 00:00:28,720 --> 00:00:33,750 and I'll have some fun so how many of 13 00:00:31,330 --> 00:00:35,829 you are still awake put your hand up 14 00:00:33,750 --> 00:00:39,399 okay just checking how many of you have 15 00:00:35,829 --> 00:00:41,500 bosses all right some of you how many of 16 00:00:39,399 --> 00:00:46,300 you have bosses that like to invent 17 00:00:41,500 --> 00:00:47,680 vaporware that you have to implement you 18 00:00:46,300 --> 00:00:49,209 know why I'm about to do this 19 00:00:47,680 --> 00:00:51,700 presentation that's that's what this is 20 00:00:49,210 --> 00:00:53,079 all about my boss at the Center for his 21 00:00:51,700 --> 00:00:55,649 studies in Cambridge is a really really 22 00:00:53,079 --> 00:00:58,780 clever guy but he doesn't do the cybers 23 00:00:55,649 --> 00:01:00,579 and he likes to talk about a PT's and so 24 00:00:58,780 --> 00:01:03,280 he does stuff like ask me how many 25 00:01:00,579 --> 00:01:04,869 people are in energetic bear you know 26 00:01:03,280 --> 00:01:06,790 how much money did they spend on this 27 00:01:04,869 --> 00:01:09,040 type of ransomware and I'm like how the 28 00:01:06,790 --> 00:01:11,110 hell would I know that right 29 00:01:09,040 --> 00:01:13,869 but then an idea started to form this 30 00:01:11,110 --> 00:01:15,340 idea of logistical budget so basically 31 00:01:13,869 --> 00:01:16,780 that's what this entire presentation is 32 00:01:15,340 --> 00:01:17,860 about Bruce is going to tell you a 33 00:01:16,780 --> 00:01:21,729 little bit about the implementation 34 00:01:17,860 --> 00:01:23,950 because I didn't have the time to 35 00:01:21,729 --> 00:01:25,659 implement stuff so I used my speaker 36 00:01:23,950 --> 00:01:27,189 fees at other conferences to get Bruce 37 00:01:25,659 --> 00:01:28,720 to do this work for me 38 00:01:27,189 --> 00:01:30,220 partly because Bruce taught me to 39 00:01:28,720 --> 00:01:32,880 program and he's a better programmer so 40 00:01:30,220 --> 00:01:35,770 I'll let him introduce some of these 41 00:01:32,880 --> 00:01:40,199 okay so as Erin said we want to be able 42 00:01:35,770 --> 00:01:42,820 to present stuff to non-technical people 43 00:01:40,200 --> 00:01:46,140 so they know which threats to 44 00:01:42,820 --> 00:01:50,258 concentrate on and what to concentrate 45 00:01:46,140 --> 00:01:52,509 stuff on which direct threat actors can 46 00:01:50,259 --> 00:01:59,500 do the most damage and save around 47 00:01:52,509 --> 00:02:03,909 somewhere they can get over you from 48 00:01:59,500 --> 00:02:06,670 reading literature but what if we can 49 00:02:03,909 --> 00:02:11,790 actually generate visualization directly 50 00:02:06,670 --> 00:02:11,790 from risk data in an easy-to-understand 51 00:02:13,890 --> 00:02:18,250 data to 52 00:02:15,430 --> 00:02:29,980 to see how threatened through structure 53 00:02:18,250 --> 00:02:33,430 is changing so has anyone heard the the 54 00:02:29,980 --> 00:02:37,840 the term logistical burden before no 55 00:02:33,430 --> 00:02:38,889 yeah sort of okay so I was working at 56 00:02:37,840 --> 00:02:40,180 the Center for risk studies and they do 57 00:02:38,889 --> 00:02:44,079 all kinds of risks they do environmental 58 00:02:40,180 --> 00:02:46,150 risk they do you know labor unrest they 59 00:02:44,079 --> 00:02:47,260 do interbank lending risk whatever it 60 00:02:46,150 --> 00:02:48,400 was great I got to hang out with all 61 00:02:47,260 --> 00:02:49,328 these amazing people and so I was 62 00:02:48,400 --> 00:02:51,099 working with these counterterrorism 63 00:02:49,329 --> 00:02:52,689 professionals and they came to me 64 00:02:51,099 --> 00:02:54,970 we were talking about adversarial risk 65 00:02:52,689 --> 00:02:57,010 like how do you quantify adversaries 66 00:02:54,970 --> 00:02:59,290 that change and adapt and think and do 67 00:02:57,010 --> 00:03:01,480 stuff right and they introduced me to 68 00:02:59,290 --> 00:03:04,060 the concept of logistical burden so you 69 00:03:01,480 --> 00:03:06,790 go to a site let's say like a ship or a 70 00:03:04,060 --> 00:03:08,620 building and you you take some special 71 00:03:06,790 --> 00:03:10,450 forces people and you ask them to ask me 72 00:03:08,620 --> 00:03:12,040 how many people were to take to storm 73 00:03:10,450 --> 00:03:13,929 this building or to drive a truck bomb 74 00:03:12,040 --> 00:03:15,189 here how much money would it cost how 75 00:03:13,930 --> 00:03:16,659 big would the bomb need to be these 76 00:03:15,189 --> 00:03:18,310 these kinds of questions right and they 77 00:03:16,659 --> 00:03:21,220 estimate the size of a threat that would 78 00:03:18,310 --> 00:03:23,260 be required for a particular target 79 00:03:21,220 --> 00:03:24,879 right now I didn't particularly want to 80 00:03:23,260 --> 00:03:25,599 do that I realized we could do this the 81 00:03:24,879 --> 00:03:26,530 other way around 82 00:03:25,599 --> 00:03:27,940 and that's what the rest of this 83 00:03:26,530 --> 00:03:30,819 presentation is going to be about it's 84 00:03:27,940 --> 00:03:32,919 if we take indicators and we assume that 85 00:03:30,819 --> 00:03:34,869 they have a cost in money manpower time 86 00:03:32,919 --> 00:03:37,090 then we can start to get a sense of the 87 00:03:34,870 --> 00:03:42,280 logistical budget of different apt 88 00:03:37,090 --> 00:03:45,819 actors right wanted to quickly 89 00:03:42,280 --> 00:03:49,689 prototypes and visualizations so we used 90 00:03:45,819 --> 00:03:53,858 Primus to grab data from Aaron's Mizpah 91 00:03:49,689 --> 00:03:56,198 server and pickle to cache it locally so 92 00:03:53,859 --> 00:03:59,799 that we can very quickly iterate through 93 00:03:56,199 --> 00:04:02,549 stuff so what we want to do is scan 94 00:03:59,799 --> 00:04:05,829 through miss events and attributes and 95 00:04:02,549 --> 00:04:09,909 filter based on galaxies and date ranges 96 00:04:05,829 --> 00:04:14,319 and then accumulate score for the 97 00:04:09,909 --> 00:04:18,009 entities that we found we used portly 98 00:04:14,319 --> 00:04:21,728 initially for heat maps because it's 99 00:04:18,009 --> 00:04:26,159 really easy to output data that plotly 100 00:04:21,728 --> 00:04:28,330 understands and later on a new plot for 101 00:04:26,159 --> 00:04:42,729 a bit more flick 102 00:04:28,330 --> 00:04:44,258 but it has some drawbacks I think so 103 00:04:42,729 --> 00:04:47,909 first we generated heat maps for a 104 00:04:44,259 --> 00:04:50,110 threat back to activity and then 105 00:04:47,909 --> 00:04:53,469 generated scorecards which are 106 00:04:50,110 --> 00:04:56,020 comparable with each other for threat 107 00:04:53,470 --> 00:05:00,849 actors but also ransomware because it 108 00:04:56,020 --> 00:05:02,530 was a really easy extension one things I 109 00:05:00,849 --> 00:05:04,630 struggled with not having Aaron's 110 00:05:02,530 --> 00:05:10,960 background in threat intelligence and 111 00:05:04,630 --> 00:05:15,240 this was domain knowledge so after 112 00:05:10,960 --> 00:05:19,000 grabbing the data from the server we 113 00:05:15,240 --> 00:05:21,610 quickly were a Python script to dump the 114 00:05:19,000 --> 00:05:25,449 fields and to count frequencies and so 115 00:05:21,610 --> 00:05:27,789 on that made it very easy to get a 116 00:05:25,449 --> 00:05:30,610 better understanding of what data is in 117 00:05:27,789 --> 00:05:33,750 Aaron Smith server and how we should be 118 00:05:30,610 --> 00:05:39,909 writing scoring functions so this is a 119 00:05:33,750 --> 00:05:41,560 kind of stuff go back so this speaks 120 00:05:39,909 --> 00:05:43,300 very much to your point under s about 121 00:05:41,560 --> 00:05:45,370 you know when people first encounter a 122 00:05:43,300 --> 00:05:46,539 Mis server they don't know what the 123 00:05:45,370 --> 00:05:47,830 fields are they don't know what the data 124 00:05:46,539 --> 00:05:49,870 looks like and it's fine if you're like 125 00:05:47,830 --> 00:05:51,520 a front-end user using the GUI but you 126 00:05:49,870 --> 00:05:53,949 sometimes need to dig around and inside 127 00:05:51,520 --> 00:05:56,109 the mists server to figure out what what 128 00:05:53,949 --> 00:05:59,289 you've got and an interesting point here 129 00:05:56,110 --> 00:06:01,120 is we wanted to start with with things 130 00:05:59,289 --> 00:06:04,240 that had attribution that had threat 131 00:06:01,120 --> 00:06:05,949 actors attributed to the events which is 132 00:06:04,240 --> 00:06:07,330 not my favorite thing to work on like 133 00:06:05,949 --> 00:06:09,490 attribution is essentially a political 134 00:06:07,330 --> 00:06:10,690 act and I find it very complicated so 135 00:06:09,490 --> 00:06:14,440 the first thing we wanted to know is 136 00:06:10,690 --> 00:06:16,300 what percentage of of events inside my 137 00:06:14,440 --> 00:06:17,740 mr. server were attributed to a 138 00:06:16,300 --> 00:06:20,020 particular threat actor and it was like 139 00:06:17,740 --> 00:06:21,550 8% I imagine most of you it's fairly 140 00:06:20,020 --> 00:06:23,500 similar so one of the things we can talk 141 00:06:21,550 --> 00:06:25,180 about later is how we might increase 142 00:06:23,500 --> 00:06:26,770 that in the future and some of the work 143 00:06:25,180 --> 00:06:34,330 you were doing would work really well 144 00:06:26,770 --> 00:06:37,389 with that right so so I could talk a bit 145 00:06:34,330 --> 00:06:39,520 about the scoring functions we really 146 00:06:37,389 --> 00:06:40,140 want to discuss the scoring functions of 147 00:06:39,520 --> 00:06:46,260 the community 148 00:06:40,140 --> 00:06:50,190 the kind of first stab they can be 149 00:06:46,260 --> 00:06:53,400 approved a great deal currently we're 150 00:06:50,190 --> 00:06:56,219 looking at context-free analysis so 151 00:06:53,400 --> 00:06:59,520 we're looking at an event and events 152 00:06:56,220 --> 00:07:05,550 attributes with no correlation of other 153 00:06:59,520 --> 00:07:07,169 data within less me saying a few things 154 00:07:05,550 --> 00:07:09,230 about this as well so there's scoring 155 00:07:07,170 --> 00:07:12,810 idea is essentially how do you translate 156 00:07:09,230 --> 00:07:14,820 observables or iOS ease into one of or 157 00:07:12,810 --> 00:07:17,400 all three of money manpower time and 158 00:07:14,820 --> 00:07:19,560 it's not as easy as it my team right 159 00:07:17,400 --> 00:07:21,539 like what does an ipv4 address worth to 160 00:07:19,560 --> 00:07:23,820 an attacker if an attacker switches for 161 00:07:21,540 --> 00:07:26,880 one ipv4 address to another what would 162 00:07:23,820 --> 00:07:29,330 you say the cost is in money come on be 163 00:07:26,880 --> 00:07:33,240 interactive I'm really exhausted 164 00:07:29,330 --> 00:07:35,849 anybody fairly low right it's not it's 165 00:07:33,240 --> 00:07:37,110 not super hard so I wanted to put an 166 00:07:35,850 --> 00:07:39,180 actual number on that and I went digging 167 00:07:37,110 --> 00:07:41,070 around in ipv4 auctions and you can 168 00:07:39,180 --> 00:07:43,290 basically buy a new ipv4 address for 169 00:07:41,070 --> 00:07:45,060 four bucks so there's a number I can put 170 00:07:43,290 --> 00:07:46,770 on it right and we all agree that's not 171 00:07:45,060 --> 00:07:49,470 the right number but what I'm trying to 172 00:07:46,770 --> 00:07:51,530 get here is that we can put a sort of 173 00:07:49,470 --> 00:07:53,970 constant of scores on like how long 174 00:07:51,530 --> 00:07:55,590 kilobyte of binary takes to write and 175 00:07:53,970 --> 00:07:57,060 some of you could go out and do further 176 00:07:55,590 --> 00:07:59,190 research on that which is what we want 177 00:07:57,060 --> 00:08:00,660 to talk about here but for now the point 178 00:07:59,190 --> 00:08:02,400 is that everybody shares the same number 179 00:08:00,660 --> 00:08:04,050 so when I was in Center for risk studies 180 00:08:02,400 --> 00:08:05,400 there was a brilliant counter terrorism 181 00:08:04,050 --> 00:08:08,190 risk professional he's written a couple 182 00:08:05,400 --> 00:08:10,169 books on the subject Gordon whoo and and 183 00:08:08,190 --> 00:08:13,469 he said to me all risks should be 184 00:08:10,170 --> 00:08:15,090 comparable or all risks are comparable 185 00:08:13,470 --> 00:08:16,980 or should be and I found that really 186 00:08:15,090 --> 00:08:19,049 frustrating because like you know what 187 00:08:16,980 --> 00:08:21,690 we do is special it's different it's 188 00:08:19,050 --> 00:08:23,700 it's not like other risks but if you've 189 00:08:21,690 --> 00:08:25,380 really progressed in the risk world then 190 00:08:23,700 --> 00:08:27,030 you can be compared to fire risk or you 191 00:08:25,380 --> 00:08:28,710 can be compared to pandemic risk or you 192 00:08:27,030 --> 00:08:30,479 can be compared to kidnapping ransom of 193 00:08:28,710 --> 00:08:31,739 piracy or whatever so that's the point 194 00:08:30,480 --> 00:08:34,320 here is by putting some of these numbers 195 00:08:31,740 --> 00:08:35,909 on here all of these different apts and 196 00:08:34,320 --> 00:08:37,770 all the different ransomware families 197 00:08:35,909 --> 00:08:39,240 can be compared even if we know those 198 00:08:37,770 --> 00:08:40,799 numbers aren't exactly right the 199 00:08:39,240 --> 00:08:42,570 constant is wrong for all of them and we 200 00:08:40,799 --> 00:08:46,160 can at least compare them so Bruce will 201 00:08:42,570 --> 00:08:48,930 show you more about how he achieved that 202 00:08:46,160 --> 00:08:50,459 okay so the scoring functions are kept 203 00:08:48,930 --> 00:08:52,319 separate from the mechanics so you don't 204 00:08:50,460 --> 00:08:53,820 have to be an expert 205 00:08:52,320 --> 00:08:56,670 and how the mechanics work to be able to 206 00:08:53,820 --> 00:08:58,470 write scoring functions and as I said 207 00:08:56,670 --> 00:09:01,050 before the dump of the attribute data is 208 00:08:58,470 --> 00:09:03,270 really useful for writing them that's 209 00:09:01,050 --> 00:09:05,760 almost impossible to read even for 210 00:09:03,270 --> 00:09:08,010 myself so but it's basically just a 211 00:09:05,760 --> 00:09:11,910 really simple piece of Python that takes 212 00:09:08,010 --> 00:09:14,569 in event and the corresponding 213 00:09:11,910 --> 00:09:18,150 attributes and scan through and 214 00:09:14,570 --> 00:09:25,350 accumulates based on the attribute data 215 00:09:18,150 --> 00:09:28,140 and then returns the score if you have a 216 00:09:25,350 --> 00:09:29,730 URL it's got this much time to manage or 217 00:09:28,140 --> 00:09:31,620 this much money if you've got an IP 218 00:09:29,730 --> 00:09:33,510 address it's worth this you get the idea 219 00:09:31,620 --> 00:09:35,520 if you've got a binary and it's of this 220 00:09:33,510 --> 00:09:37,050 size then you have some idea of like how 221 00:09:35,520 --> 00:09:40,680 much time the thread actor put into it 222 00:09:37,050 --> 00:09:43,130 so that's all there and that could so 223 00:09:40,680 --> 00:09:45,180 the scorecards that I mentioned before 224 00:09:43,130 --> 00:09:47,340 looks something like this so we're 225 00:09:45,180 --> 00:09:51,180 trying to estimate the organization size 226 00:09:47,340 --> 00:09:56,700 and the amount that they're spending in 227 00:09:51,180 --> 00:10:00,000 for on infrastructure the estimated time 228 00:09:56,700 --> 00:10:05,460 investment and this is we're going to 229 00:10:00,000 --> 00:10:07,980 compare Dharma and want to cry and you 230 00:10:05,460 --> 00:10:13,560 can see we're giving some fuzziness to 231 00:10:07,980 --> 00:10:17,490 the to the actual results but if we look 232 00:10:13,560 --> 00:10:24,270 at one a cry much bigger organization 233 00:10:17,490 --> 00:10:26,850 size spend and time investment so these 234 00:10:24,270 --> 00:10:29,579 should be noted these are log graphs of 235 00:10:26,850 --> 00:10:35,460 the tics or a little bit disingenuous 236 00:10:29,580 --> 00:10:37,560 but they're for different score cards 237 00:10:35,460 --> 00:10:39,180 and that's important because some threat 238 00:10:37,560 --> 00:10:40,770 actors operate it at like an insane 239 00:10:39,180 --> 00:10:43,050 scale so like you look at the number of 240 00:10:40,770 --> 00:10:45,030 URLs involved in a sofa C campaign and 241 00:10:43,050 --> 00:10:46,530 it's just extreme so you have to do some 242 00:10:45,030 --> 00:10:48,449 of these things on a log scale right and 243 00:10:46,530 --> 00:10:49,589 the score across the bottom for those of 244 00:10:48,450 --> 00:10:51,810 you who can't read all of these it's 245 00:10:49,590 --> 00:10:53,760 estimated organizational size at the top 246 00:10:51,810 --> 00:10:55,709 that's the other one infrastructure 247 00:10:53,760 --> 00:10:58,200 spend so the amount of money is the red 248 00:10:55,710 --> 00:11:00,150 one time is the blue one and the last 249 00:10:58,200 --> 00:11:01,620 one that's in black is basically the 250 00:11:00,150 --> 00:11:03,449 aggregation of those three different 251 00:11:01,620 --> 00:11:05,010 scores right so if we click back and 252 00:11:03,450 --> 00:11:06,089 forth between these two you just get the 253 00:11:05,010 --> 00:11:07,379 idea that 254 00:11:06,089 --> 00:11:09,660 Dharma probably spent less money 255 00:11:07,379 --> 00:11:10,800 manpower and time than wanna cry and 256 00:11:09,660 --> 00:11:12,809 that's all we really wanted to do with 257 00:11:10,800 --> 00:11:14,128 this but of course you don't have to do 258 00:11:12,809 --> 00:11:16,740 this just for ransomware you can do it 259 00:11:14,129 --> 00:11:19,050 for other things too right so we also do 260 00:11:16,740 --> 00:11:23,220 this for the threat actors so we hit 261 00:11:19,050 --> 00:11:31,109 here we have energetic bear and equation 262 00:11:23,220 --> 00:11:35,249 group and then we have heat maps that we 263 00:11:31,110 --> 00:11:40,499 generated for the threat actors so this 264 00:11:35,249 --> 00:11:45,689 is taking threat actor events 15 bins of 265 00:11:40,499 --> 00:11:49,949 30 days and then ranking them based on 266 00:11:45,689 --> 00:11:55,399 their aggregate score so we can get some 267 00:11:49,949 --> 00:12:01,229 nice idea of bright points and 268 00:11:55,399 --> 00:12:10,639 corresponding dates we can also do the 269 00:12:01,230 --> 00:12:16,459 same for weekly plots 15 bins this is 270 00:12:10,639 --> 00:12:19,399 the event but scaled based on their 271 00:12:16,459 --> 00:12:22,258 threat levels so the high gets a 272 00:12:19,399 --> 00:12:25,800 significantly higher score than a medium 273 00:12:22,259 --> 00:12:27,839 or low and we didn't want to put like 274 00:12:25,800 --> 00:12:29,279 200 of these heat maps in here but we 275 00:12:27,839 --> 00:12:31,019 can do them not just for events we can 276 00:12:29,279 --> 00:12:33,059 also do them for binaries or for 277 00:12:31,019 --> 00:12:35,610 networks or for files or for whatever 278 00:12:33,059 --> 00:12:37,019 and then we worked on a sort of scoring 279 00:12:35,610 --> 00:12:38,939 function that took all of those into 280 00:12:37,019 --> 00:12:41,249 account and made one now it's worth 281 00:12:38,939 --> 00:12:43,439 pointing out here that the time bin 282 00:12:41,249 --> 00:12:44,670 across the bottom is detection time and 283 00:12:43,439 --> 00:12:46,980 we all know that dwell time can be 284 00:12:44,670 --> 00:12:49,529 really high so I don't take the time 285 00:12:46,980 --> 00:12:52,139 line of this entirely seriously but I do 286 00:12:49,529 --> 00:12:53,699 take the heat map to be of interest so 287 00:12:52,139 --> 00:12:54,899 what I'm trying to say there is that you 288 00:12:53,699 --> 00:12:56,998 know this little white spot here for 289 00:12:54,899 --> 00:13:00,209 sofa C might have actually occurred a 290 00:12:56,999 --> 00:13:01,920 time bin before or before that in terms 291 00:13:00,209 --> 00:13:03,719 of when the attack occurred so this is 292 00:13:01,920 --> 00:13:05,579 detection time but it still it still 293 00:13:03,720 --> 00:13:08,100 lets us know that there was a lot more 294 00:13:05,579 --> 00:13:10,258 indicators in that time period that we 295 00:13:08,100 --> 00:13:12,029 could use for something so yeah this is 296 00:13:10,259 --> 00:13:14,370 an idea of the code that Bruce is 297 00:13:12,029 --> 00:13:16,049 written and we've made open source on 298 00:13:14,370 --> 00:13:17,730 github 299 00:13:16,049 --> 00:13:19,860 we have other ideas of how we can 300 00:13:17,730 --> 00:13:22,439 visualize like perhaps you would do 301 00:13:19,860 --> 00:13:24,929 treemap sort of structure where the the 302 00:13:22,439 --> 00:13:27,449 files will be on one side and like you 303 00:13:24,929 --> 00:13:30,238 know the the network indicators would be 304 00:13:27,449 --> 00:13:32,248 on the other or we can do heat maps for 305 00:13:30,239 --> 00:13:34,439 ransomware we've got a lot of ideas 306 00:13:32,249 --> 00:13:36,749 about how to visualize this data but we 307 00:13:34,439 --> 00:13:38,519 probably need a little bit of help and 308 00:13:36,749 --> 00:13:39,629 then we want to talk a lot about scoring 309 00:13:38,519 --> 00:13:42,269 functions so if you know that there's 310 00:13:39,629 --> 00:13:43,920 academic work estimating the amount of 311 00:13:42,269 --> 00:13:46,319 time that went into a binary based on 312 00:13:43,920 --> 00:13:48,479 how many kilobytes it is or how much 313 00:13:46,319 --> 00:13:50,998 network infrastructure costs for 314 00:13:48,480 --> 00:13:52,799 attackers or so on I'm also giving a 315 00:13:50,999 --> 00:13:54,059 talk tomorrow about ransomware well 316 00:13:52,799 --> 00:13:55,889 you'll see a little bit more about where 317 00:13:54,059 --> 00:13:59,100 some of this came from and some of that 318 00:13:55,889 --> 00:14:01,889 work is replicated there in terms of how 319 00:13:59,100 --> 00:14:11,489 much an incident costs by comparison to 320 00:14:01,889 --> 00:14:15,470 how much attackers made in ransoms maybe 321 00:14:11,489 --> 00:14:18,860 everything that you said oh can we use 322 00:14:15,470 --> 00:14:25,709 unattributed mess data in our 323 00:14:18,860 --> 00:14:29,389 visualizations how does the community 324 00:14:25,709 --> 00:14:32,388 feel about this how do you feel like 325 00:14:29,389 --> 00:14:32,389 sended 326 00:14:35,880 --> 00:14:44,370 I mean so this wasn't super expensive 327 00:14:42,480 --> 00:14:46,080 like Bruce works really hard and he's 328 00:14:44,370 --> 00:14:48,150 got a new company but like like I said 329 00:14:46,080 --> 00:14:51,030 this is my speaker fees for a couple 330 00:14:48,150 --> 00:14:52,560 months right and I'm really glad about 331 00:14:51,030 --> 00:14:54,480 that but we do think it could go a lot 332 00:14:52,560 --> 00:14:56,099 further so if you're interested or you 333 00:14:54,480 --> 00:14:57,450 have time we don't necessarily need 334 00:14:56,100 --> 00:15:00,510 money we also just need people to 335 00:14:57,450 --> 00:15:02,190 contribute so you know we could just 336 00:15:00,510 --> 00:15:03,180 give it to you guys and you can do 337 00:15:02,190 --> 00:15:05,100 something with it if you want that's 338 00:15:03,180 --> 00:15:07,439 fine too but I'm also interested in the 339 00:15:05,100 --> 00:15:09,360 reaction from the community like is this 340 00:15:07,440 --> 00:15:10,920 total BS because it's based on money 341 00:15:09,360 --> 00:15:12,120 manpower and time and you don't like the 342 00:15:10,920 --> 00:15:13,439 scoring function or do you actually 343 00:15:12,120 --> 00:15:15,750 think this is useful would you sit 344 00:15:13,440 --> 00:15:21,510 around comparing apt groups and 345 00:15:15,750 --> 00:15:22,680 ransomware no I mean from I think from 346 00:15:21,510 --> 00:15:23,850 our perspective it looks really 347 00:15:22,680 --> 00:15:25,020 interesting and maybe something that 348 00:15:23,850 --> 00:15:27,990 could be interesting as well as 349 00:15:25,020 --> 00:15:30,480 especially once you're refining your 350 00:15:27,990 --> 00:15:31,710 scoring for the different types after a 351 00:15:30,480 --> 00:15:33,300 while would you be interested for 352 00:15:31,710 --> 00:15:35,700 example in feeding the data back into 353 00:15:33,300 --> 00:15:38,790 the frittata galaxies because this would 354 00:15:35,700 --> 00:15:41,100 I think would be very valuable for the 355 00:15:38,790 --> 00:15:42,839 community out there to get the cigarette 356 00:15:41,100 --> 00:15:44,750 and then for further developments we can 357 00:15:42,840 --> 00:15:46,560 we can issue we should talk about this 358 00:15:44,750 --> 00:15:48,000 yeah that's the other thing so we have 359 00:15:46,560 --> 00:15:50,280 we basically have a bigger research 360 00:15:48,000 --> 00:15:52,380 ongoing for the exploration of 361 00:15:50,280 --> 00:15:53,819 indicators and we're looking at 362 00:15:52,380 --> 00:15:55,680 different components and different 363 00:15:53,820 --> 00:15:57,030 things that we can take into account so 364 00:15:55,680 --> 00:15:57,420 which we could work together on that as 365 00:15:57,030 --> 00:16:03,089 well 366 00:15:57,420 --> 00:16:05,719 so yes I think there's another question 367 00:16:03,090 --> 00:16:05,720 back here 368 00:16:33,420 --> 00:16:36,020 yeah 369 00:16:36,399 --> 00:16:41,209 exactly so I'll repeat the question for 370 00:16:39,140 --> 00:16:42,890 the cameras as I'm supposed to even 371 00:16:41,209 --> 00:16:43,880 though I'm tired I remember the rules of 372 00:16:42,890 --> 00:16:46,339 Cooper 373 00:16:43,880 --> 00:16:48,560 so essentially software development 374 00:16:46,339 --> 00:16:49,670 houses have that data already and that's 375 00:16:48,560 --> 00:16:51,018 the sort of thing that we should be 376 00:16:49,670 --> 00:16:52,310 incorporating so once they've written a 377 00:16:51,019 --> 00:16:54,019 piece of software you could look at it 378 00:16:52,310 --> 00:16:55,640 and work backwards and say how many 379 00:16:54,019 --> 00:16:58,820 people did you have on this project for 380 00:16:55,640 --> 00:17:01,399 how long and how much did it cost and so 381 00:16:58,820 --> 00:17:03,410 on now the costing would be the one I 382 00:17:01,399 --> 00:17:05,750 would question in terms of timing 383 00:17:03,410 --> 00:17:07,399 that's probably all very accurate when 384 00:17:05,750 --> 00:17:10,510 you can player malware and you compare 385 00:17:07,400 --> 00:17:12,920 standard software but in terms of money 386 00:17:10,510 --> 00:17:15,260 it might not be the same pay structure 387 00:17:12,920 --> 00:17:17,360 and the underground economy right people 388 00:17:15,260 --> 00:17:19,910 might be coding for a share of the brand 389 00:17:17,359 --> 00:17:21,079 some of our profits or you know they 390 00:17:19,910 --> 00:17:22,939 might be stealing other people's code 391 00:17:21,079 --> 00:17:24,649 before they get started there's a lot of 392 00:17:22,939 --> 00:17:26,240 details in there but but I absolutely 393 00:17:24,650 --> 00:17:30,650 take your point the traditional software 394 00:17:26,240 --> 00:17:32,600 development studies are useful to this 395 00:17:30,650 --> 00:17:33,650 and we didn't dig that deep into this 396 00:17:32,600 --> 00:17:34,730 because we just wanted the proof of 397 00:17:33,650 --> 00:17:36,800 concept where we could show you the 398 00:17:34,730 --> 00:17:38,230 visualization first and then we could 399 00:17:36,800 --> 00:17:40,370 deep dive on each of those numbers 400 00:17:38,230 --> 00:17:43,610 especially if we can get you interested 401 00:17:40,370 --> 00:17:45,879 to help us with that so great idea next 402 00:17:43,610 --> 00:17:45,879 question 403 00:17:52,790 --> 00:17:55,690 okay 404 00:18:18,270 --> 00:18:37,900 yes so the the comment is essentially 405 00:18:35,440 --> 00:18:40,890 about the associativity of tags inside 406 00:18:37,900 --> 00:18:40,890 Misbah vents 407 00:19:17,430 --> 00:19:21,490 yeah of course I mean the more that the 408 00:19:19,840 --> 00:19:24,040 correlation engines run underneath the 409 00:19:21,490 --> 00:19:25,540 more that we will have possible to 410 00:19:24,040 --> 00:19:27,760 visualize right especially if you're 411 00:19:25,540 --> 00:19:29,800 enriching events where like you happen 412 00:19:27,760 --> 00:19:32,320 to know this domain and this domain are 413 00:19:29,800 --> 00:19:34,649 linked by Whois data and then it grows 414 00:19:32,320 --> 00:19:34,649 right 415 00:20:06,590 --> 00:20:12,060 yes I mean we know that we have a naming 416 00:20:09,180 --> 00:20:14,850 convention problem for ransomware and a 417 00:20:12,060 --> 00:20:16,020 PT's because it's essentially marketing 418 00:20:14,850 --> 00:20:21,899 reports that we get most of this 419 00:20:16,020 --> 00:20:23,190 information from which is yes in fact I 420 00:20:21,900 --> 00:20:25,380 would counter this entire conversation 421 00:20:23,190 --> 00:20:27,480 with the fact that you can run our code 422 00:20:25,380 --> 00:20:29,850 on your missing instance so if your 423 00:20:27,480 --> 00:20:32,490 confidence in your misclassifications is 424 00:20:29,850 --> 00:20:35,159 better then you can do the visualization 425 00:20:32,490 --> 00:20:42,420 on your data that's why we wrote it this 426 00:20:35,160 --> 00:21:06,510 way so yeah any other questions or 427 00:20:42,420 --> 00:21:09,600 comments yeah yeah I mean you did some 428 00:21:06,510 --> 00:21:14,370 lightweight analysis in that sort of 429 00:21:09,600 --> 00:21:16,560 area but not not in a scientifically 430 00:21:14,370 --> 00:21:19,110 rigorous way and it's absolutely 431 00:21:16,560 --> 00:21:20,820 something we'd like to do it's just we 432 00:21:19,110 --> 00:21:22,590 wanted to prove the concept with the 433 00:21:20,820 --> 00:21:23,820 visualization and then talk to people 434 00:21:22,590 --> 00:21:26,220 about how to do that so if you're 435 00:21:23,820 --> 00:21:28,050 interested we'd love your help and I 436 00:21:26,220 --> 00:21:32,830 think I have to wrap up for the next 437 00:21:28,050 --> 00:21:38,389 speakers that's it from us 438 00:21:32,830 --> 00:21:38,389 [Applause]