Okay, so welcome back.
We now have our closing keynotes.
I am absolutely delighted and honoured to be joined by Brian Hurd.
He was going to do his closing keynotes on strategic perspectives and mostly true war stories.
So the floor is yours, Brian.
Close us out.
And of course, thank you for doing this as well.
We really appreciate it.
No, Stu, Kevin, and all the members of the founding parents of this organization, thank you.
I really enjoyed a lot of the other speakers on the tracks today.
And I hope to hold my own in such company of great intel analysts.
So I'll get right into it.
I know everybody's had a very long day, both here in the U.S.
and across Europe, attending the conference.
I really am honoured to be kind of the capstone.
My name is Brian Hurd.
I currently work at Straus Free Burger Company.
I'll tell you about in all of 15 seconds later.
And I've been in the community since around 1993 when I founded the U.S.
Navy Cyber Counterintelligence Program.
A lot of people have the gratuitous splash side for their resume.
I decided to be nice to all of you and do mine in logos instead of a text for an end of day.
I had the honor and privilege to be the founding father of the program for Cyber Counterespionage NCIS, Microsoft Cybercrime Center's first chief of intel and the chief operations of the entire U.S.
watch listing system.
I've analyzed, unfortunately, long nights, weekends and weird places, about every type of case you can.
From dead bodies and dots on a map to cyber breaches for going on 30 years now.
And I'll talk about some of the older cases and I'll talk about some of the ones from last month.
And it'll be more about the craft of analysis than it is about the findings of the case.
Because it was amazing to see today practitioners exchanging that kind of information and ideas.
Before we get into that, I had a community service alert that the management asked me to put out about dihydrogen monoxide.
There's been a lot of exposure to it, even with the pandemic.
It causes deaths all around the world, is a major component of acid rain, and usually people die by inhaling it.
Dihydrogen monoxide's chemical makeup looks like this.
It's commonly referred to as H2O and is commonly referred to as water.
So to start this discussion for Intel analysts, our job is to make it less complex.
Our job is to communicate.
Our job is to take massive amounts of RDP connections and emails and attachments and lines of code or dates and times for a pattern of life, and I'll talk about that in a bit, and distill it down for decision makers.
But let me be really clear, in my whole career, it is never dumbing it down.
You might distill it, but it really is leveling it up, whether that's intelligence or information technology or whatever.
I've briefed secretaries of defense on many occasions, and in making it so that they can make a strategic decision is actually harder than just presenting my results to fellow analysts.
You have to think a lot of those levels, and hopefully I'll talk about some of that today.
So when they were talking about this, they asked me for a couple of quotes from dead guys, because you can't have a keynote without dead guy quotes.
So my favorite, Lord Kelvin, if you cannot measure it, you cannot improve it.
And when you cannot measure it, your knowledge is of a meager and unsatisfactory kind.
Also, Julius Caesar, I came, I saw, visualized, and conquered.
And then Colin Powell, experts often possess more data than judgment.
Colin's not dead, but he does owe me a dollar, so I'll include him in here.
The other thing they asked me to do was ask the US intel agencies what I would tell an international audience of commercial government, law enforcement, and other intel analysts about who's the threat.
The State Department says there are no enemies, only emerging allies.
The CIA knows who the enemy is, but telling you it would endanger the source.
NSA knows who it is, but you're not cleared.
Director of National Intelligence says they're in charge.
Marines, doesn't matter, mess with the best, die like the rest.
And the US FBI thinks it's the CIA.
But who is it really?
It is these people we've been talking about, and Kevin and the team have been leading the fight against for years and years and decades.
And I've worked in many of the exploitation cases at NCIS and across the world as well.
There are truly evil people in the world.
They steal money, they steal lives, they steal dreams, and they steal the very existence of others without a blink.
Cybercrime has now become a service.
This is not just a small group or kids in hoodies.
It is an international ecosystem of evil in the trillions of dollars.
It is not small groups, kids in basements, things like that.
We are dealing with almost another continent of capabilities and marketplaces.
Malware, when I worked at Microsoft Cybercrime Center, we did a lot of the takedowns like the TrickBot takedown, the Citadel takedown, and others.
These were two of the visualizations of all the infected computers that we had from our takedown operations calling back into us.
On the left, you can see the different color codes of the botnets and the spikes for the number of systems.
The spikes coming out of Beijing could stop satellites were they real.
And in the image on the right, there was financial malware that had infected all of those hotspots you see across Western Europe.
But here's something to think about visualization as I go forward.
This one visualization shows something that no spreadsheet ever could.
If you consider this heat map to be the electrical grid of that region, it should continue all the way to Moscow, but it doesn't.
The heat map, the infection pattern of a bot on computers, seems to know where a border is that it doesn't know anything about.
It was encoded not to attack those with the Cyrillic keyboards on their computers.
And therefore, not to come home and, you know, crap where you eat.
So these are the kind of things that as I go forward and talk about some of the operations in our craft, visualization is not what you do when you're done analyzing.
Visualization is a great part of what you do to get to the answer.
So another visualization, there's a stack of $200 million.
Now you have to cut it in half.
So you have to visualize cutting that in half.
$100 million was taken by the Lazarus Group probably out of Bangladeshi banks.
Now, for any of us on the side of good and right, and I hope all of the viewers are, even we take a pause when one operation nets $100 million.
But even looking at our adversary, if they were then to turn around and go, can I do another one tomorrow, boss?
I'm pretty sure the Approval Chain and Risk Committee will allow them to do so because they can net $100 million or more.
These attacks on our banking system and ransomwares and other things I'll talk about are stealing just tons of money every single day from hardworking people in every single one of our country.
So there's not just ecosystems, there's loosely affiliated groups, anonymous.
Hector Monsignor, the gentleman pictured there, is a friend and I've worked with him on several occasions.
In fact, I was on a corporate board with him.
But that was after he had been a member of the group and then the FBI invited him to be helpful.
And for those who might harsh on him a little bit, when faced with 100 years in prison, almost everybody will ask, and what's the other option?
So there are a lot of decentralized groups, and they aren't groups.
They don't have a registry, they don't have membership cards, people may join for one operation only.
And this causes us a lot of problems.
This transactional nature that crime can become, not only in the cyber world, but in the real world of human trafficking or other things.
It's easier to have these transactions.
It's easier not to be a known entity and get things done.
And that's a struggle for us on the analytic side.
So the other problem we all face is nation state tools are in the hands of mere mortals or groups.
Based on the doxing of the shadow brokers and other things, national level tools that have caused some of the worst problems we've had with WannaCry and Alpetya and all that, came out of the place of spy versus spy, where they had better controls.
And we now see them on the open market and some of these capabilities hitting a lot of companies.
This is not just a cyber issue.
For those who like to see a real world impact, besides one of my favorites is the attack on the BTC pipeline, not my favorite because of those of our friends viewing today in Turkey, Teşekkür ederim.
I'm not a fan that it happened, but an attack using cyber capabilities and near person cyber capabilities on the pipeline to disable its alarms, to super pressurize the pipe and have it explode, with a different group hacking in and deleting the videos,
is a pretty interesting set of capabilities for physical and cyber.
A lot of other examples come out of the Ukrainian arena.
And what does that mean for us as Intel analysts?
We all know this picture, the surface web, what most humans think is the entirety of the internet, the deep web, which most of us get a lot of information on, and the dark web where we go to get the rest of it.
This layers of capability mean there's a lot of noise in open source intelligence on the surface web, a lot of opinion and we'll talk a little bit about that later.
And a lot of rich information, if you know where to find it on the deep web, that may be in places like, and I'll talk about the congressional records at thomas.loc.gov and the US government and pros and cons of it.
And then of course, as many of you are probably already on an onion browser today or using Tor, thanks to the US Navy for developing it, go Navy, there is a dark web capability where everything pretty much looks like Amazon, except it's drugs and guns and DDoS attacks.
So I've spent way too much of my life tracking people through all three of these layers.
And it poses each of them pose a different analytical problem for us.
So for those of you that like a little history, what I'm going to do is two parts.
For those of you that don't like history, it might be fun, trust me.
I'm going to talk about the early days.
Back in the early days in the mid and early 1990s before the web existed when we started, we used chisels and rocks and things like that.
But one of the first books, if you've never read it, was The Cuckoo's Egg by Cliff Stoll, an astronomer who caught basically East German hackers under the employee of the KGB stealing intellectual property from the United States government because he was kind of bored and had to rectify two accounting systems.
It was an amazing case.
I've met and talked to Cliff a bunch of times.
His work on the case is great.
He's an interesting character.
I really love him for that part.
And him, myself, Jim Christie, Kevin Mandia, and a bunch of others were working cases in the early 90s.
This was a great look at what even a non-investigator could do when given the time and some of the things he did about alerting when somebody was in the system are tried and true today.
Another thing about for open source analysis.
Many of you may have heard way back when that in the late 1998 timeframe, the Navy and Army shut down their web pages because they were worried about OPSEC.
Here's the reason why.
One night we were doing a strike on some Bin Laden facilities in Yemen and decided I was going to do an open source intelligence exercise for the United States Navy.
I was still running the cyber counterintelligence program at the time.
So I took my father, who's an accountant, and gave him a web browser and said, go look stuff up.
So he started on the CNN.com page and then found that they had links to the Navy pages, links to the list of ships.
Each ship had a homepage.
Each homepage had this commander's bio, including the name sometimes of his wife and children.
The one jump I gave my father was to go to whitepages.com, which still exists, by the way, I'm told here in Seattle.
And then was able to look up base maps and using the pictures from the pages on the website of the Navy ship, could walk to the park and meet any of those kids and know what their dog looks like to know who the commanding officer's family is.
And then also at that time we had problems with the ship schedules, a classified bit of information, sometimes being shared by family members who really didn't understand that.
I gave this brief to all the secretaries of the services and the Secretary of Defense.
The resulting output of that was the Army shut down its web pages for a month or two, the Navy refactored a lot of bits.
And a year later, the San Diego Union Tribune did the same analytic and published it in the newspaper, where they went out and looked at open source information to then talk about the most powerful thumb in the Navy, and the name of the gentleman on the ship who had fired the missiles.
That's not a great place to be because this was long, long ago and bin Laden and them were still targeting us pretty harshly.
These kind of things are the early days of open source, and they haven't changed much at all.
Our craft continues to evolve, but the core analytic capabilities remain.
So, one of the other things I was talking about the deep web.
In the mid 90s, the congressional record of the United States Congress was digitized to make it more available to the public.
When it was, somebody didn't realize that when general officers were promoted to flag rank, when you get your first star as a one star general or one star admiral.
Their social security number was put in the congressional record to uniquely identify them.
Yes, all of you are laughing.
Yes, it was a remarkably bad idea.
And nobody cared because you had to go there, really, and check out the book for that year.
It was a very inefficient system back in the old days.
However, once it was digitized, one group just talking about free speech made a list of all 4400 of the socials available, including Colin Powell and others, and published it on the internet to prove freedom of speech.
A group of identity thieves then use that list to create bank accounts all over the world.
I helped lead the case with the Secret Service and CIS and others.
And here was the, the issue was they had advanced algorithms, even back in the late 90s, using machine learning to figure out where there was fraud.
You know what they didn't do?
Use the list as a whitelist and just flag any new account created with those socials.
It didn't exist back in those days.
Identity theft protection and locking your credit wasn't really a thing either.
We had to work with all the credit bureaus, Navy, Federal and Equifax, TransUnion, USAA, all these were military members, to create capabilities that are now taken for granted.
But we had to stop the considerable amount of pain these people were facing.
I'll switch now to the other side of, again, an older case.
Elian Gonzalez was a Cuban child that was here in the US.
I'm not taking a side in where he should have been or those kind of things.
It was an immigration discussion.
Because of the things I'd done in the Navy, the INS asked me to monitor the operation on the internet and let them know if there were any concerns.
So the night the operation went down, we were in a watch center and I was doing the thing we all now do in the cyber world.
And I was not focused on Elian himself or the case.
I was focused on the agents.
The agent you see in that picture where you can see his face pointing toward you and all that, you could take that back before facial recognition.
Again, this was the olden days.
You could figure out probably the year he graduated from the ATFs and INS schools and find a graduation picture with his name.
Then you can go to white pages and get his phone address.
Same for the female agent.
They did.
In fact, we found that it was able to be done in 15 minutes.
Warned them, and then after the operation was over that night, moved the agents out of their homes about three hours before hostile surveillance was stalking them.
So looking not only at your skills as an investigator, but also as a protector and other components of your organization is a way not only to maximize what our craft does, it maximizes your role in leadership and understanding the various facets of what our skill,
what our community can do to increase mission capability and protect lives.
And it all comes down to what all of us know as the analyst bathtub.
So, the analytic effort on the y-axis and time and activities on the x-axis, find the intel loop, the ODA loop, observe, orient, decide, act, whatever you want to call this for the progression along the bottom.
You collect data, you manipulate data, you get some draft findings, and you do your presentation.
The problem is, we all spend all of our time on the left hand side and right hand side of that, calling it the analyst bathtub.
Very little time to analyze because you spent so much time trying to get the data.
Get it in the right format, transform XML to HTT or to tab delimited or vice versa, depending on which tools you're using.
Normalize the terminology, normalize the date times.
Oh my god, I hate that.
And all these other types of things.
So what happens at that point is effort is stolen from analysis by what precedes it and what follows it.
The dream is to invert the analyst bathtub and spend much less time collecting, manipulating, ETLing, or transforming, as you kids say, data.
ETL is extract, transform, and load.
It's an old database term.
Making that data, at that time, more time on target for you to be able to think and manipulate data and for those turns to be quicker.
There's also a thing called analytic momentum about how quickly you can ask a question in your analytic and how quickly you answer.
If it takes 12 hours for the computer to crunch it out, you kind of lost your momentum a little bit.
So there's a struggle there that we have had and still have in some cases.
And that's the analytic gap is what is the difference between the time it took you to collect data and present it and how much time you actually had to manipulate and analyze it.
Another blast from the past is called Moonlight Maze, an investigation.
Oh, sorry.
The investigation from 20 years ago, 1998.
I'm not really going to talk about the investigation, but I will talk about the technologies, is that we use technologies from the Pacific Northwest National Labs called Inspire and Starlight.
The thing on the left is 10,000 documents and the mountaintops are the topics on 200 dimensions of n-space about what's in the data set.
So here's the difference.
Every time there's exfiltrated data and you're dealing with it, everybody wants to know what's in it.
And then somebody, usually probably a junior kid goes, well, I queried missile and it said it 90 times.
I said, 90 out of what?
He goes, what do you mean 90 out of what?
I'm like, well, how many times did nuclear weapon come up?
Well, I don't know.
Yeah, exactly.
So we're randomly picking keywords and we're randomly getting numbers out of a data set.
That thing took all 10,000 documents, sorted them, including foreign languages, and then arrayed them into both that mountain view and the associated galaxies view.
The data told me what it was.
I wasn't inserting analytic bias by picking query terms.
Similarly, the visualizations on the right, mockups of the ones we used, was where we were looking for after an adversary got in.
What documents did they hit?
What machines and locations did they hit?
And where they might go next.
These advanced visualizations, again, the analyst bathtub was not inverted.
I spent a lot of time manipulating data.
And when we did that analytic, I wrote the nation's report for Moonlight Maze.
And then a year later, went back with better technology and reassessed our report and questioned all of our findings.
As an analyst, write up the counter opinion.
Document what you're leaving out.
Be willing to give the alternate opinion a place in your document like it's a Supreme Court finding.
Here's my pros and here's what we think happened.
Here's the appendix where if the following things change, we're probably wrong.
Footnote, footnote, footnote.
If you're not willing to footnote your sources, then peer review is much more difficult.
And then you're playing a game of you have better access than your fellow analysts, not you have better analysis.
And what I'll do is move on.
This type of visualization brought us insights we would have never seen in spreadsheets.
We could have never written out on whiteboards.
We could have never seen by looking at Excel for time gaps and jumps.
Make sure you're picking tools that make your life easier on analysis, as well as presentation.
We'll talk about that in a little bit.
One example.
This is a chart of the CIA World Factbook life expectancy around the world.
Green and big is good, red and small is bad.
You can immediately see where the life expectancy is more 28 years old, as opposed to 82 years old.
You can also see over by Florida, a small red dot, which is the Dominican Republic back in this slide from a long time ago.
That showed there was an outlier that would have jumped out that you wouldn't have seen if you were paying attention to the sub-Saharan Africa and region.
These types of things in analysis is a difference between displaying data and visual analytics than briefing somebody from a PowerPoint.
Similarly, for those of you that have complex migrations, this was the migration of a search engine across an entire enterprise.
And there were about 40 data sets, each represented by a different, as I call them, lucky charm shape on this visualization.
And this is how the overall migration would go.
The search findings on the left are what the rank of the top 25 documents would be, the search finding on the right.
So if your document was number one in the search result before the migration, it should be 1-1, 2-2, whatever.
So there's a smattering.
So everybody said, oh, it looks great.
Until, of course, you look at each individual data set, where many of them line up like you expect, except that one on the right.
That data set is going to have a problem.
Result number two would turn into result number 25.
Now, it was an XML data set.
It was not apt for the search engine indexing, so we had to manipulate it in a different way.
But it was a critical data set to the enterprise that would have been overlooked had we not segmented our analysis and be able to walk through each of them individually.
A pointer is not always an answer.
And sometimes, a lack of a dot is more important than one there.
And these are the kind of things that, as I went through different investigations and used different analytic techniques, as an example, this is a bunch of investigations around malware attacks on the left.
And believe me, these are blurred, so you can't read them.
That's not the point.
Arraying the data so you can see the activities of adversaries with time across the bottom and either the machine names or the types of activity across the y-axis.
Arrays data so you can see the patterns that just jump out at you.
But a lot of us, every single one of you, spends a lot of time keeping that data in your head when the systems can be helping us a lot more with it.
A similar example, an entire ransomware case start to finish.
Down bottom left, those yellow crosses are the phishing attacks.
Just above it is the disabling of the AV and the account creation of the pink stars.
The lateral movement and data access as the red and orange pentagons and circles move across.
And the black events is the launch of the malware and the deletion and encryption of the enterprise.
Now, arraying this allowed us to build this case and make sure we had the data and events for our master timeline.
It also was the very thing we briefed not only our internal analytic team, but the client, the client's lawyer, the insurance company, and many others, which gave them an ability to understand the case in about three or four minutes and ask questions about each location,
machine or person involved.
Similarly, and Kevin, I know you have a great love for this too, pattern of life.
This one was an insider case with 40 suspects.
The line across the top are press and enterprise events that I was looking for.
The red accesses are people accessing the document that was compromised.
And then other activities of creating documents in those little dashes.
So this pattern of life analytic allowed us to move people up and down the suspicion list based on the timing of their access to certain documents or their activities in Slack before or after a news story broke around the incident itself.
So it brought data from the real world, data from the systems, data from communications, accesses, and other things, all onto one chart for analysis.
This is what we use to ask and answer questions, and then ultimately present to the client.
I would be willing to share this out to those who want it, but if you want one recommendation for me for a book to read, it is an odd one.
Why people believe weird things.
And the reason I love this book and have bought it a lot of times for, geez, over 200 analysts that have worked for me, is chapter three, the 25 ways analysis goes wrong.
And this is the list that I still use to this day, besides many deception matrices and other things I've learned in the Intel community.
When I'm writing a national level product, and it's going to go up, I run through these 25 different ways analysis goes wrong.
That my theory influenced what I looked at.
That my observations change the behavior of what I'm measuring.
That the equipment I used constructs the results.
In Moonlight Maze, the equipment we had didn't bring in Unix documents without a Microsoft Office extension.
So all of the text from the system administrators and things talking about things in Moonlight Maze, that wasn't a .doc, a .txt file, wasn't brought in to that map in SPIRE.
We brought it in a year later and then reassessed our damage assessments and findings about what the adversary had acquired and why.
There's a bunch of other things in here, potentially applicable to for those of you looking at today's news.
Bold statements don't make claims true, no matter which political party makes them.
I'm agnostic.
Hearsay does not equal correctness.
Burden of proof is on the person that's making an outrageous claim, not me to disprove it.
Just because something's unexplained doesn't mean aliens did it.
So Michael Shermer, great author, I'm a huge fan of his, is a skeptic.
And he applies this theory to UFO believers, Holocaust denial, and a bunch of other stuff.
So it's great to look at these types of things as analytic biases.
And a couple of our tracks today talked about analytic bias and other things related to our profession.
It is humbling to find yourself falling into these traps, to ignore a source that doesn't fit your mold, and later find out you probably shouldn't have.
And I've had that occur many times in my career.
Or overlook one little data set that has that one date time stamped event that makes that pattern of life analytic actually make sense, or removes a person from being a suspect.
And letting go of a suspect, an espionage suspect, a terrorism suspect, because you've proven them innocent, is an important thing to embrace and document.
And a struggle.
So as I said before, it is tempting for us to wrap ourselves in the technology, in the terminology of threats, vulnerabilities, exploits, IP addresses, tradecraft of our adversaries.
But we should be talking about the mission.
We should be talking about business strategy.
What should a decision maker do with the output of your analysis?
Will it stand up in court?
Will the board approve money because of it?
Or are we ready to impact somebody's civil liberties because you're confident enough in your analytic, proving they are the guilty party, or at least guilty enough to move to the next phase of enforcement?
These are the things that many of you, if you're junior in the field, you'll be doing it soon.
If you're not related to military or law enforcement, you'll be doing this for your companies.
Intel analysis is a universal craft.
And it has fared me well, and I do hope that each of you gets in different ways to have just an amazing career.
I couldn't have picked a better career to be at in terms of the great things I get to do and the amazing capabilities that I've been able to tinker with, so to speak.
And one final thing, and Kevin, I promised you I'd put this in.
Follow the money.
Somebody at the end of the day wants to buy a boat, diamonds, a DACA, a fur coat, a $500,000 car.
Great presentations today from our speakers about how to go through Venmo.
A bunch of you probably fixed your Venmo privacy settings today, I'm sure.
Absolutely, track payments where possible.
And for those of you that feel Bitcoin completely precludes that, there are two things.
One, not soon if ransomware keeps hurting so many people, I'm sure somebody on the government side is going to declare it a clear and present danger, and they're going to put the kind of analytics that can even burn through the anonymity of Bitcoin.
And two, the money has to come out somewhere.
Somebody wants to buy that boat.
Somebody's making money and somebody wants that money in a physical form.
So even if there's a couple hop, skips and dead drops we can't get through, we can still get to them.
It takes years.
It takes patience.
But it works.
And it takes a lot of analysis.
So, finishing with plenty of time for Kevin to pepper me with stump the chump questions or stew.
My final call is literally, you're the mission.
This conference and conferences like it, where we trade, how to do our jobs we talk about the impact we can have as it as individuals and a team, and the support for great groups like the NCPTF, and, and the, and the sponsors that helped us today.
And for Stu and his team, and especially my hero in the AV, Andrew, and others for putting all this together.
This was an amazingly well run conference, and one of the better missions of a group, I've seen gathered recently.
So thank you for your attention throughout the tracks today.
Thank you for those of you that stuck around for the old guy to be cantankerous and complain about the old days and when we used to use pen and paper.
I look forward to chatting with you for a little bit, or maybe letting some of you over in the UK get some sleep.
Over to you Stu and Kevin.
Excellent.
Yeah, I don't know if Kevin's still on the call I think he might be dropped off slightly, but I'll certainly do it.
I'll certainly ask some questions.
Just first of all, before I do that, can I say thank you.
That was an amazing, amazing talk.
I really enjoyed it.
And especially the case studies as well.
So, I have a question for you actually before I kick off with everyone else because I know there's lots of questions.
What's your favorite story?
That you can talk about from, you know, all the stuff that you've done, either amusing story or case study that you haven't mentioned that you can talk about and share with everyone.
Okay, so, and the pause is just for me to change the date, country and adversary.
I was at an international counterintelligence conference in Italy.
And as easy as you might be like, okay, but wait a second, all the spy chasers got together and we had a conference.
So I'm literally in the counter espionage department of the United States, sitting across the table drinking bottles of wine with the gentleman in Russia whose whole job is to make my job completely ineffective.
And my whole job is to make his job ineffective.
It was, it was the closest thing to a James Bond meeting I think I could get.
Besides the support I did with the people that do operations, but I'm not the guy doing the meetings.
Sometimes.
The sitting across the table.
It was this conference for all the evil people in the same thing, but let me, let me say this the thing I loved about that meeting and that it was like five day meeting.
It was professional.
You will never see that type of meeting encounter drugs or counterterrorism, you all won't get together and talk about the rules of the game.
The funniest thing about it though I learned was humor really doesn't translate between a lot of languages, and we were trying to tell jokes, and what was some of the jokes were funny, some weren't.
But the translations were even funnier in the attempt to explain why something was humorous.
So sometimes every once in a while the Russians or the, or the, and it was then Czechoslovakia, the Czech guy wouldn't get something, and we'd say, well don't let the bear chase you and for some reason that was funny because that was one of the odd translations the Russian did.
So it was, it was just the oddest meeting I've ever been to in my entire life.
And it was fun, and I had professional respect for the person that sat across the table for me, because he had my job, he just picked a different flag.
And that's, as you look at Intel analysts across borders and across the world, on the good, on the side of good.
Those people on the evil side?
Yeah, F those guys.
But on the professional Intel analysts, professional journalists, professional other ones, they're members of our craft, they might be just have picked the wrong jersey to root for.
So I don't know if it's funny to anybody else but I just, that always just stuck in my mind is one of the oddest weeks of my life.
That sounds amazing.
Thank you.
I love that.
I've got some questions now so let me go, let me go through some questions.
So, okay, there was a question around recommendations for courses on visual analytics.
You know, oh, yes, hold up one sec.
I got the guy stepping off screen for a sec.
Okay.
Edward Tufte.
So, Edward Tufte is one of the world's best courses on, it's like a one day course, it's only a couple hundred bucks in the US.
I'm sure there's a whole website.
He has four different books on visual analytics.
I am a fan of Tufte, not because of what he visualized but this is historical visualizations from across the ages.
And in fact, just right here behind me is my favorite visualization on the planet, which is Napoleon's march to Russia and back.
And it shows all on one visualization, the number of troops, the temperature where the rivers were, the directions, and the fact that, you know, 20x went out and 1x came back.
5% of the troops returned to Paris.
But that visualization has so many attributes of size, shape, color, direction on one screen.
Like those things, in fact, I'll see if I can...
And that's why you'll see me doing visualizations like this.
Color means something, size means something, shape means something, and which line it's on vertically means the date and horizontally means the entity, the person or computer.
This is why I'm such a huge fan of selected annotation to let the user then become a participant in the visualization.
My career in all of these things, and especially going from Tufti to running tools that literally are jet engines of capability from the Department of Energy's National Labs, I was honored to run Spire and Starlight for the first time in an espionage case.
It was not easy.
But I would start with Tufti, and then pick your own.
Nate Silver's really great.
Many other visionaries.
I would also look at mind mapping books.
Mind mapping is doodling.
For those of you who doodle, hey, you were ahead of us.
We just thought you were just scribbling in the margins, but it actually is a mnemonic and you people were right, I'm sorry.
Those types of things will get your mind thinking in visuals, and then you will turn that and take tools like TIBCO Spotfire, Tableau, ClickTech, whatever, and you will apply them.
The one thing I'll say is I like TIBCO Spotfire.
I use it all the time.
I've used it in running whole intelligence agencies, knowledge management for all intelligence agencies.
And it's a tool, and I'll go back down to one of these other visualizations.
In this visualization, one of these things is not like the other.
The three visualizations at the left, right, and bottom right are about the case.
The visualization at the bottom left is how many hours each of my investigators are dedicating that day.
These spikes here are in the 17 hours a day range.
We were working pretty hard on a case.
I used the same tool for investigations to see which of my team was about to burn out.
So if you're good at visualization, there's a little bit of a U.S.
bias.
There's a thing called Frank's Red Hot Sauce.
They say I pour that shit on everything.
I do that with visualization.
I use this to look at the billables and hours of our team.
I use this to look at the case.
I use this to look at how much billable hours, expenses, and other things we need to tell the client about.
With the very same tool, I use to hunt evil.
And if you have a tool that only does one or the other, you might want to look at moving those spreadsheets around a little bit and consolidating and really tricking out one tool that does both.
So I'm a huge... I hope that over-answered your question probably way more than you ever cared to know.
But yeah, Edward Tufte, T-U-F-T-E, and then of course Nate Silver and the others, and then mind mapping books.
Mind mapping.
Thanks.
That's really good.
Thanks, Brian.
And in a sort of follow-up to that as well, there's another question as well.
So on a global level, how do you move from raw collected data from several sources to those fancy visualizations?
A horrific amount of data transformation.
And here's the thing.
So I took the job and ran all of the identities for the U.S.
watch listing system.
Now for those of you that are going, what's that?
It's the no-fly list and the selectee list and the border list.
I was in charge of every single identity record for it.
I was hired after the underwear bomber slipped through because of what we call a sharded identity.
So my job was to take that entire system and fix it and make sure Osama bin Laden had just one record, as opposed to the 26 entries he had when I took over.
And I worked for 10 years doing that.
The things you're going to run into after you get the format between XML tab delimited, whatever, whatever, then in one of your data sets, it says England.
And in another one, it says the United Kingdom or Republic of the Congo, Congo, comma, Republic of.
So you're going to spend a lot of time doing a bunch of join tables to clean up the tags so that all of the stuff for Congo gets on one timeline.
All of the dates and times for the cell phones, for the communications, for the images and other things are all in one time zone.
I would usually put all your data in UTC and then visualize it in something else if you need to bring it to Eastern Standard or Pacific.
I would always try and unify, especially time and location data.
And then like you see on these charts and again they're not really for reading but it says RDP shell open, RDP shell close, file delete, file create.
Yeah, those tags didn't come out of the computer so we had to do a little bit of manipulation to get a consistent term for the things coming out of the logs.
Sometimes by grepping.
I used to be an expert at grep, sed, and awk.
But those skills have atrophied as I've been more and more executive.
But I did these visualizations.
These aren't from my team.
So I still get to pick up a wrench once in a while and it's a lot of fun.
Excellent.
Thank you.
And I guess there's another question that's been asked.
So what's your opinion on how you deal with a problem like Russia?
In general?
I guess from a cyber perspective, I don't know, from the way that crime is, you know, is, is operated and such and so forth.
Understood.
I'll take the question on a couple levels.
I was, geez, I'm old.
I was born in 1971.
Many of you don't know when that was.
That's no way to look at your father.
So Russia has always been a competitor.
And much of my career, that's been a constant.
I was born in the, you know, ragged edge of the cold, or in the middle of the Cold War, I guess I could say.
Actually, 71.
Yeah.
And 1989 was when the wall fell.
So Cold War was the start of my career, I was already at the US Naval Academy.
And when I came out, while the wall had fallen, the KGB had not really hung up its hat.
So working Moonlight Maze in the mid 90s, it was, it was Russian oriented intelligence.
But again, I view the competitors in our craft, especially counter espionage, counter intelligence, to have honor and integrity to the degree that we do in the field.
So, you know, again, I admire patriots in their own countries.
And I do understand they're vying for the health and security of their families.
And we picked a different jersey.
So it's like the NFL, except I'm on the Seahawks, go Seattle, and somebody else picked the 49ers.
Why, I have no idea.
So, or, you know, for those of you out there, Liverpool and Manchester.
I think I got that right.
But you know, I can be corrected.
The rest of it, in the world today, for those of you doing open source Intel analysis, news outlets used to be a primary source.
That is no longer the case.
Can't trust any of them.
Can't trust their fact checkers, can't trust the bias in much of the media around the world.
And it hurts my soul a little bit to see how bad it's getting.
And I don't care what side you're on.
It doesn't matter to me.
It's all spin.
So as you look at being an analyst related to global events, 9-11, the Boston bombing, I got a Lifetime Achievement Award for running the Boston bombing.
I don't care about the politics of the Boston bombing.
I cared about finding the suspects and getting them watch listed.
It's my job to keep terrorists off planes.
It's not my job to determine, you know, whether their cause is right or wrong.
Now that's not blind fealty.
It's not blind loyalty.
It's a focus on the analyst's role in a job.
As a forensics person, I've been at NCIS, I'm now at Strauss-Friberg, one of the world's greatest forensics companies.
We were hired by Facebook for Cambridge Analytica.
I led that case for a while and did all of its case management and things like that.
My job was to figure out what happened.
My job wasn't to have an opinion of whether it should or shouldn't have.
And as a forensics analyst or an intel analyst, when you show up at the scene of a car accident, you will describe the ages, genders, and physical condition of the victims in the car.
It is an unfortunate incident.
People probably died.
But that's not your role.
Your role is to describe it right and tell the world what happened.
Now, if you're doing that for a case of any type, you bring truth.
Other people bring opinion.
That's really powerful.
I love that.
That's really, really good.
I'm a little preachy today.
I apologize.
It's been a long time since I've been in front of so many great intel analysts.
I miss this crew.
And I apologize if I'm a little like, I've got to get my messages out to the next generation.
But preparing for this briefing and dusting off some of my old stories reminded me of when I was 21 and founded an entire program for the U.S.
Navy because one or two people saw what I was doing and understood the truth of it.
And you can have that global impact.
I've made so many world-changing analytic programs.
There were smarter people than me.
I will repeat that every single time.
Apparently, they were on vacation and I got the job.
But I did everything in that job to change the world in watch listing and cyber espionage.
For EDS, I founded their program.
For Microsoft, I helped found theirs.
Wouldn't change a thing, except maybe I'd be smarter.
Well, I probably disagree, but I think you've done amazing things.
Thanks.
I'll send the 20 bucks for that later, but go ahead.
I'll take it.
I'll take that later.
That's fine.
I'm conscious of time, so I want to make sure you get these questions.
There are so many questions now.
There's always the way, right?
So, Rag asks, what's your opinion of the use of AI and behavior analytics to not only interpret, but also to predict the future motions of a threat actor?
Great, great question.
I spent three years at a machine learning AI company with Richard Clark from the U.S.
White House and Mudge Zatko, the father of Cult of the Dead Cow and Loft.
So, Mudge and I are great friends.
We basically made an AI tool that tracked adversary behavior through a system.
So, a couple things about machine learning and AI.
One, those terms are over abused.
Two, you got to know what type of machine learning you're talking about.
And if you're just talking about linear regression, then, you know, it's pretty good at predicting the past, maybe the present, but not so much the future, because it's assumed the threat actor will keep doing the same thing.
And if somebody says, oh, no, it's AI.
Again, what kind of AI?
Deep learning, random forests?
You don't have to be an expert in what those are.
But if you can't go, if you don't understand what happened from the spreadsheet of data to the thing the computer is telling you, that's a problem.
You don't know what happened to the data in the sausage maker.
So there's a, here's the traceability in an ML AI system, you need to be as an analyst able to trace back a recommendation or finding back to the set of raw data that made it.
Now what happens between that is the algorithms will have coefficients on certain variables that are related to that process.
You don't have to program those coefficients, but you better be able to explain which five or however many, for me it was usually five to 10 gave me enough, but depends on how complex the system is, why this thing's being recommended, the top three to five reasons at a minimum,
and what the coefficients were, and why they're that way.
What gets you a 26 suspicion score?
What does that mean?
What 10 documents led to the suspicion score?
Where did it come from?
Why does the computer think Brian's evil?
They do, believe me, I've had this problem.
But you as an analyst, if you're going to put, as we say, bullets in bad guys or warheads on foreheads, you better know why the system spit out a name.
And the algorithms, a lot of times, they don't have explainability.
Even the guy that programmed it doesn't know why it gave it that score, but he's not the one that's going to brief the Secretary of Defense tomorrow.
So I put it back to you as analysts, you have to understand, not every bit and byte, but you need to know what analytic process deleted your data, summarized it, made it across table, and how you can drill back to figure out in a human explainable and defendable and legally defendable way,
what you're telling somebody to do with their life.
All right, that's brilliant.
I think we've got one more question.
Okay, the question actually is, right, okay, it's final question from Cyber Viking, there you go.
Does Brian have a training course?
Would be a beyond amazing to learn from him.
One, I know you added that second part, but Cyber Viking, thank you if you actually wrote that.
No, I've trained my own teams.
A lot of times, I have not hung a shingle.
Right now, I work for Strauss-Friedberg, which is owned by Aon, the global insurance company.
I run massive breaches.
I'm running a bunch of cases now.
We just ran one where a hacker for hire pwned a company.
And within five days, I have pictures of the hacker's home address.
So that's going to be fun next week.
So I'm running mega breaches all over the world, as well as doing that.
But thank you for the compliment.
There are a lot of great trainers.
Kevin and his team do amazing capabilities as well.
But drop me a line on LinkedIn.
Again, if you want to know a book or two or, you know, exchange a thought or two, just let me know.
Awesome.
So I think we're out of time.
But I think a lot of people would still like to learn from you.
So I'm going to be really selfish and ask the last question.
I run a podcast.
The Many Hats Club runs a podcast.
Would you want to come on that at some point in the future?
Putting you on the spot in front of everyone?
Oh, absolutely.
Challenge accepted.
I'll message you at some point in the future.
Another way to call out my manhood on the stage.
But yeah, yeah, yeah, I'm in.
I'll just say Kevin made me do it, so it's fine, right?
But of course, we blame it all on him.
And my final thank you to the audience for not only all the hours you spent today, but I know you didn't have them because you spent them all week long, too, in whatever job you're in.
So thanks for time to allow me to pontificate and share some old war stories.
Now, seriously, I know everyone's just said this amazing talk, and you can't read the chat, but everyone say it's an amazing talk.
And I will mirror that.
And I've been blown away by this.
So what an amazing way to close out the talks today.
So thank you so much for being such an awesome keynote and all the things you've done.
Literally, even when I spoke to you in our talk before this conference, when you started reading out your background, I was just like, wow.
Just amazing.
So, yeah, thank you for taking time to share your experience with everyone here today.
Seriously, amazing.
No, and thank you.
Talk soon.
Thank you, Brian.
And before we go, just to let everyone know, the conference isn't quite over yet.
We do have YT Cracker coming to do an after-party set.
He's waiting in the wings.
I've just got to send him a stream key.
So everyone can come and hang out, listen to music.
And like I said, if anything, like his last set he did for us, if he's any as good as that, then you're all in for a treat.
So I think we're going to hold fire for five minutes or so.
So just hang on here and we'll be ready to go.