Alright, we're gonna get started here now.
Soon as the guy with the ponytail finishes his conversation, talking with his hands, with his back to me, hey, there we go.
Yeah, oh.
Wasn't you, it was the other guy with the ponytail talking.
So don't worry about it.
So, my name's Andrew Hay.
Uh, this is bootstrapping a security research project.
This talk actually became a talk, uh, back when I was managing researchers, who would finish a project and then come to me and say, hey, what should I research now?
Like, well, you're the researcher, what do you want to research?
Like, oh, I could do this, or I could do that, or... And finally, it just, it, there needed to be a process.
So, hopefully, if anyone here is a researcher, or is an aspiring researcher, uh, is anyone here an academic researcher?
Yes?
Really?
Get out.
No, so this is not a very academic process, this is just a, here's what works, uh, with a little bit of academia rubbed on for style, I'll say.
I will preface this by saying that I did foosball my way out of college, so this is in no way academic.
I'm a hell of a foosball player, though.
And also, before we get started, uh, I don't know if Ian's still in the room or not?
No?
So he really referenced that pyramid of pain, that, it, it's really near and dear to his heart, that's actually the Turkish bathhouse he worked in one summer, so he keeps bringing it up, it's, he really likes that place.
So I am the CISO at Data Gravity, uh, if anyone here has been a CISO at a startup, it means you do absolutely everything, when needed, and at any hour of the day.
Uh, I used to be the Director of Research at OpenDNS, now Cisco, uh, I was the Chief Evangelist and Director of Research at Cloud Passage, still Cloud Passage, uh, was at 451 Research as an Industry Analyst, and... Yeah, they are still 451.
Uh, I worked at a university in Western Canada, yes, I'm Canadian, come at me.
Uh, you know, just cause we invented basketball doesn't mean the Canadians really care about basketball.
Side note, Canadians invented basketball, so, there you go.
Uh, I worked at a bank in Bermuda, uh, it sounds a lot cooler than it actually was, Dave can attest to that, cause he also is a Canadian who lived and worked in Bermuda.
Uh, I also was, I think, employee 34 at Q1 Labs, which is now IBM, um, no, I'm not rich as a result.
I also wrote, uh, three or four books, which, judging by my financial statements that come in, none of you have read, um, I blog infrequently and I spend a lot of time on planes, and as you can tell, I am built for coach.
Very happy to be in coach.
So, these are the concepts that we're gonna talk about.
I'm gonna do a very quick overview.
I'm not gonna, you know, drone on, like, Webster's defines research as blah.
Uh, in fact, I think I hid that slide.
So, I have managed researchers, I have done research, uh, I still do research, I am a self-taught data scientist.
I should have put that in quotes, too.
Uh, and R, Ruby, and Python hacker, and not in the sense of, hey, I'm really awesome, I'm gonna hack this thing, and it means that my code is so bad that it's a complete hack job.
I would hire interns and I would just make them do code review of my code.
It was awesome.
If, if you have the opportunity, I, I very much encourage that.
They're really one and the same.
Uh, I did have someone come up to me at RSA and say, hey, you're the Hello Barbie guy.
Which is not really what you want to be known as.
Um, so choose your research projects very carefully.
Cause you don't want to be like, oh, hey, you're the, you know, the adult toy hacker.
Yeah.
That's probably not gonna get you in Forbes.
So let's talk a little bit about bootstrapping.
Oh, see, I thought I, Webster's defines research as.
I thought I got rid of this.
So when I talk about research, I'm not talking about the classical, academic, sit in a basement for ten years until something's done kind of research, because nobody's really got time for that.
We have lives.
Uh, I like money.
I like to make money.
And I like to spend money even more.
So I don't have time to sit in a dusty closet day in day out.
Uh, so that's what traditional research is.
What I'm going to talk about now is bootstrapping research.
So how to get to the end point faster.
I, I toyed with the idea of saying like, you know, agile research.
I'm like, nope, I'd have to punch myself every time in the face if I said that.
Has anyone here heard of pastor's quadrant before?
The academic researcher guy in the back, maybe?
Seen it in a textbook?
I'm sure.
Yeah.
Expert.
All right.
So this is really used to define applied versus basic research.
Uh, I'm not a big fan of the term basic research.
It's more knowledge based research.
So we'll do that.
Knowledge and utility.
So for pure basic research, Niels Bohr, Josh Corman.
I worked for Josh.
Really nice guy.
Lots of big ideas, but it's not applied research.
It's, I have this idea.
Let's put it out there and someone else can run with it and make something of, here's my baby.
Go put some lipstick on it and make it beautiful.
That is probably the worst analogy I've ever used.
Now, pure applied research.
This is the Thomas Edison kind of guy, you know, inventing a light bulb because, Hey, we need a light bulb.
Uh, and I put deviant on here too, because all of his research is around how to get into locks or how to, how to really skirt the locks protections.
It's not about building a better lock.
He's not in the business of creating research around him manufacturing a lock.
So in the use inspired basic research top right, that's where we're getting the past doers, the car hackers.
And this is where I think everyone wants to be from a security research or where people should be from a security research point of view, because you're creating big ideas that someone else can run with later on, but you're also producing fact-based research that is verifiable and reproducible.
Does anyone been following the DBIR debacle on Twitter?
The, well, the one.
So I found this picture and this is really, um, what a, what a data scientist looks like.
Uh, there is some overlap with what a researcher and what a security researcher looks like.
Uh, I would say that me personally, uh, math and statistics, not my forte programming and database.
Nope.
Uh, domain knowledge and soft skills.
Yeah, sure.
Communication and visualization.
Yeah.
A woman.
Nope.
So glasses.
Yeah.
Uh, I do have a tablet, so, and a smartwatch.
No hair.
And a purse.
Yeah.
So I, you know, there's, you don't have to have all of these things to be a data scientist or to be a researcher, but this gives you an idea and I will give you guys the slides later, um, so you can actually read this.
You know, unless you're, you have a scientific background and manage a lot of data on a daily basis, machine learning and statistical modeling, probably not that important to you.
But if you are curious, event and action driven in that, Hey, someone released Hello Barbie and kids are going to play with this.
Uh, I wonder how bad that's going to be for my child's privacy.
I'd say that's event driven.
Uh, time sensitivity.
I found out that someone else is researching this exact same thing.
I have to do it faster because if I release after them, it's just me to research.
And then, uh, you need to want to do good or bad as the case may be.
Researchers are not always good or bad.
It's a great big gray area.
Um, yeah, I'm not judging anyone, but you're probably not going to see a lot of bad researcher, bad researchers that are publishing reproducible research and showing you how they made an exploit, uh, unless you pay them.
So now when you're deciding on what to research or if you even want to undertake a research project, I've defined three buckets here.
So if you do net new research, you have full control over the experiment.
You get to decide how you're going to research this thing.
And I'll just keep using thing because it could be an IOT device, could be a piece of software.
It could be the ripening of cheese, whatever.
That's probably previously done research.
Um, you also have the potential to be first to market.
You could stake your claim on being the researcher for X.
Yeah.
Hello, Barbie.
It is very time consuming because you're starting net new.
Uh, and there is that time sensitivity of, Hey, if a whole bunch of other people are researching this, what if they go to market first?
What's my research going to look like?
Or if I publish my research, is it going to be in direct conflict with what they or the direct opposite of what they published?
If you're continuing research, a little bit more control.
Um, especially if you're using your own findings from a previous experiment, you're familiar with the data, you know how to present it.
Um, you don't have a lot of wiggle room though, in redefining the experiment or the question, you can create another question based on, uh, preexisting data to try and answer something else.
But, uh, you know, you're, you're not breaking anything really net new.
Most of the time it's either rehashing current findings or finding a new, you know, subtlety in your data and then challenging someone else's research.
This is not calling someone out on Twitter.
That is not a validated or a valid research methodology.
One second here.
So if you challenge someone else's research, which by all means is very important, it helps if you have access to the exact same dataset that they have used.
Otherwise you're going to get different results.
So if someone publishes data and their tools they use to create their analysis and their findings, you can go and look at the code.
You can go and look at the data, the smoothing, the sampling that they used and maybe come up with different results.
Maybe try bucketing the data to see what pops out and, and what could shed some light on some additional context.
Or you could look at their code and say, Hey, this is just like random data that's being created.
Or you're mashing together two disparate points of data, DBIR, that have no direct correlation or relationship to each other, DBIR.
And one of the big problems here is that it could be very time consuming, especially if the data is not presented in a consumable way, or you're just unfamiliar with the data.
So you're gonna have to ask a lot of questions, you're gonna have to make a lot of inferences.
Hopefully the other researcher will help you understand the data.
If you tell them it's because you don't believe what they're saying, they're probably not going to help you that much.
So hopefully they're following proper procedures and documenting everything, so that you don't have to reach out to them to challenge.
So who has ever sat back or at the end of a conference saying, you know what, I want to research something ?
Anyone?
That was, that was the most reluctant hand raising.
Yeah, whatever.
So if you want to research something , absolutely anything, you should ask yourself these three questions in sequence.
So what is the question you're trying to answer?
For the Hello Barbie, for an example, I wanted to know what privacy issues existed or would be surfaced from allowing children to use a doll that uploads data to the internet and monitors pretty much everything they say when you push the little button.
Do you have the data to answer the question?
I will admit to spending about three hours shopping for Hello Barbie on Amazon one day at work.
I'm kind of glad nobody questioned me on it because my proxy logs would have been ridiculous.
It's like, I'm just trying to find the best price for a Hello Barbie.
Do you have kids?
No, I don't have kids.
Yeah, yeah.
Free candy.
Still trying to black out the windows, but whatever.
So if I have the data or the means to generate the data, if I could answer the question that I posed in number one, could I use the answer?
Does it matter?
Could I personally use the answer?
Yes.
Could others use the answer?
If you're buying a Barbie for your kid around Christmas time and you want to know that there's huge security issues, which there were, and privacy issues, which there still are, would I want to know about that before buying the product?
And the answer is, I would hope, yes.
Otherwise, you don't like your kid very much.
So there's a bunch of different types of questions that you can ask of data or of an experiment.
I'm not going to go through all of them because it's a lot of reading and it's Sunday and it's early and I can just come up with excuses for days on end.
I will say that when you're doing security research, the majority of your questions are going to be either exploratory or inferential.
So exploratory, I have a bunch of data, what can I find?
So that's almost like sifting for gold.
What nuggets are going to fall out of this data?
And then inferential, you're asking a question of, I wonder if, you know, I believe that Hello Barbie is insecure because it doesn't, because the end server that the doll communicates with is still vulnerable to 10-year-old attacks.
Which it was.
Then if you take a look at this, the epicycle of data analysis, I stole this from Roger Peng, who's a data scientist at someplace.
Thank you.
Johns Hopkins.
So when you're looking at your data, you have to set the expectations.
What am I, what do I want to find?
What am I looking for?
And then, you know, you start collecting the data.
So for Hello Barbie, for example, you need the doll, you need to be able to capture network traffic, you need to understand maybe the components that make up the doll, the software, the mobile app, any other software that gets installed on an endpoint,
how the flow of data and even the, like, the authorization and follow-up emails, you need to understand and what the process is of using the doll.
And you're not going to find that in the documentation, because it's complicated.
People ask questions if you document it.
So you have to figure it out yourself.
And then revise your expectations.
So let's say my hypothesis originally was all of the data is transmitted in clear text up to the Hello Barbie servers.
If I look at the traffic and I realize that it's all SSL encapsulated, then that hypothesis no longer stands.
So I have to revise.
So I know that the data is being uploaded or transmitted to the server in an SSL encapsulated session.
Then I can ask a follow-up question.
Is the server susceptible to known vulnerabilities or known attacks?
And then you just keep going.
One, two, three.
Until you're happy with the question.
So these are the things I believe you need to get started.
Time.
If anyone here has a significant other, my advice is to pick a television show that you hate but they love, and there, bang, your research time.
My research time is Dancing with the Stars.
Because I cannot stand that show.
But my wife loves it.
So that gives me two hours a week to do something.
Using that time in the past, I have learned Ruby.
I have learned R.
I have learned Python.
I researched Barbie.
I did a whole bunch of stuff.
It's great.
Thank you, Dancing with the Stars.
So, yeah, just find something that you hate.
But if you don't have time, then maybe researching something right now isn't a good idea.
Because it is time consuming.
You also have to be very curious.
You know, just researching something for the sake of getting a presentation at a conference or getting an article in a major trade publication, that might be good.
That might motivate you.
But it's not going to inspire you to really keep at it.
So if you're curious about the question that you're answering or you're posing of the data, you're going to have far more personal buy-in and you're going to love your research a lot more.
When you get the results, you're going to be extremely happy or sad, depending on how the results come, but you'll have a vested interest in this research other than I'm going to publish a blog post.
You can publish blog posts all day long.
Doesn't mean someone's going to read it.
A question.
You obviously need a question .
You can't just say I'm going to research coffee.
What are you going to research about it?
Coffee.
Doesn't work that well .
Ability to execute is a big one.
This can be a learned trait.
So if you want to, if you want to dissect or reverse engineer a piece of Android code or an Android app, but you have no idea, you don't own an Android device, you have no programming experience, you don't even know the language that it's written in, that's going to be a big barrier for you.
So what I advise is if you don't have the ability to execute, but you have the time curiosity and a question, and maybe even the asset device or whatever you want to research, you could probably find someone to help you execute on that.
And then having the asset device, if it's an IoT device, it's very easy to test yourself.
If it's software, generally you can download it.
I will go over in the next couple slides how to obtain equipment.
So you can buy.
You can go to an electronic store.
You can buy software.
You can go buy toys.
You can borrow them from work or from a friend or from a neighbor.
I was going to put a slide here.
You can steal it, but that's kind of implied under borrow.
You can rent.
I don't know, do you guys have rent a center here or something similar where you can rent like a TV and, you know, try and impress a significant other that you're richer than you actually are by having all this stuff in your apartment or house.
You can get like a big screen smart TV from rent a center for like 54 bucks a week.
So 54 bucks, you got a toy to play with for a week.
That's not bad.
Or you can download software, download the VM.
If it's a SaaS or PaaS application, you usually will get a trial license.
I will... I should have prefaced this by saying, you know, you may be violating some of the terms and conditions of the software and hardware licensing, but as a Canadian, those are your rules.
They don't apply to me.
I've got a passport .
I can flee the country.
I know what side... I know what side of the wall I'm going to be on when I have to start building it.
So these are some of my favorite places to buy things or to... sorry, to rent things.
Costco has a fantastic return policy.
Full refund on everything.
The only caveats there are around Super Bowl Sunday, for obvious reasons.
Don't research a big screen TV over Super Bowl weekend.
Items that must be returned within 90 days.
Televisions , projectors, computers, cameras, camcorders, tablets.
I don't know who buys iPods anymore or MP3 players.
Cell phones.
Again, that's probably related more to the pay-as-you-go cell phones.
Target... the target branded items, you can return within a full year for a refund or exchange, which is kind of cool.
Unopen new... or in...
unopen a new condition 90 days.
That doesn't really do us much good.
Non-contract cell phones, computers, cameras, and gaming systems within 30 days.
Music, movies, video games, software that have been opened.
You can't return them, but you can just keep exchanging them until they tell you you can't exchange it anymore.
So if you want to research software, that's not a bad model.
Fries.
I don't think... fries isn't on the east coast, is it?
No.
Is there a... yeah.
And California.
And Vegas.
This is... so next time you go to Vegas, check out fries.
It's pretty awesome.
It's like an electronics geek's Shangri-La.
But it also has a lot of toys.
My wife let me go shopping there for research one day.
I was gone for like three hours.
It was awesome.
I came home with like $300 in kids' toys.
Again, nobody called the cops.
It wasn't loaded into a windowless van or anything, so I was good.
But you can return pretty much everything.
Best Buy.
They're going to have a lot of the new stuff.
So yeah.
Get stuff from Best Buy and return it.
At some point, you're probably going to end up on a watch list.
So this is where you get friends to go borrow stuff on your behalf.
And even if you start rotating storage, you'll get on a list.
Let's be honest.
We're probably all on lists already.
So maybe not that big a concern.
So this section, I titled it core research ideas.
It's more core research methodologies and processes to really get started on a research project.
I'm not giving you ideas to go research.
That's up to you.
I'm not your mother.
So when I'm talking about a physical device of some sort, this is usually what, this is my pyramid of pain.
Where I find at the bottom is the easiest.
So I come from a very strong networking background.
I know offsets in headers and packets and what certain protocols look like compared to another protocol.
So I find network level is very easy.
Also, you don't have to rip anything apart.
You plug something in, TCP dump, analyze the results.
Software, again, I'm not a developer.
But I code a lot.
Poorly.
So software knowledge, it can be learned.
You can dump something in IDA, poke around, you can figure out what things are happening based on your level of expertise and knowledge.
I generally get someone to help out with the software side.
Platform, this is really operating system and whether it's hosted in cloud infrastructure.
So if it's a SaaS product, infrastructure as a service , platform as a service, whatever, where the software or where the device is being hosted.
So platform could also apply to the physical device, but I split that into hardware and circuitry.
So I did... so from a hardware perspective, this is taking the device, opening it up, taking a look at the chips, taking a look at the boards, getting serial numbers, getting names, finding documentation associated with that.
That's going to help you with your research.
And that way you're not researching in a black box.
Obviously, if you start ripping these devices apart, you're probably going to avoid the warranty and it's going to make it very hard to return.
You could probably epoxy those things back together and be okay.
A couple times.
But that could be an issue.
And then circuitry, you know, if you're plugging, if you, you're trying to figure out the JTAGs or you're taking digital multimeter and oscilloscope, that's what I'm talking about circuitry level.
That is, you know, I was very good at that in my first year of college before foosball.
After foosball, not so good.
Circuitry and knowing the resistors and the path of data as it flows through a board.
That's a specialization that not all of us have, myself included.
If I had to start doing soldering, it would, you know, I'd end up soldering my thumb to my forehead in no time.
So your mileage will definitely vary on this based on your current skill set.
And the level of difficulty will definitely vary.
So some of the example questions based on the research level.
So from a network perspective, where's the device calling out to?
Is it encrypted?
How frequently does it call Korea?
Is it the good Korea or the bad Korea that it's calling to?
You know, does the software have any flaws?
Why, why, why, why is it still susceptible to freak or heart bleed?
Um, is it sending anything in plain text?
Is anything configured within the software plain text?
Uh, Hello Barbie had a whole bunch of keys that were the same on every single... So in the app, the app itself on the mobile device, um, there was a certificate that was the same on every single installation.
Because security is hard.
Let's be honest.
So is the way it's hosted make it vulnerable?
Uh, if data is being sent to cloud infrastructure, typically a SAS platform, um, the answer is most likely yes, it's vulnerable to something.
You will have to carefully poke and prod at this.
Again, Canadian, not a lawyer.
Your laws, not mine.
Uh, I say that until I end up in Gitmo.
But, you know, is the way it's being presented or interfaced with vulnerable?
Everything I've tested has had vulnerabilities in one shape or form.
In fact, one, uh, a friend of mine who works for a company that does IOT devices, during one of my big research reports when I was at OpenDNS, he's like, yeah, you found a whole bunch of vulnerabilities that our due-diligence team that used this other product didn't find.
And, uh, now I'm gonna go yell at them.
Can you send me your validated findings?
Like, here you go.
Give them shit.
So what are we getting by directly interfacing with the hardware?
One thing that not a lot of people think of is what happens if they're using the wrong resistors on the circuitry?
Or cheap components that have been known to fail?
These things, like IOT devices, for example, are meant to be cheap and disposable.
What happens if you rip this apart and you realize that there's recalled hardware or recalled components that blow up in the face of a child?
You probably don't want that in that toy.
So that's a finding in and of itself.
So the research team in an ideal world, all of your research will have a three person team at a minimum.
And that's an ideal world.
I have never had this happen.
And if you have people that have hardware, network, and software expertise, then you can divvy up the work.
So you're not spent trying to boil the ocean yourself.
You can just kind of divvy up the tasks and then come together at the end and publish your research.
Oftentimes, you'll get one person that has knowledge that spans multiple areas.
This seems to be more likely in what I've run into, where you have network and software expertise, but no hardware expertise, or you have hardware and network, but no software expertise.
And this isn't... this doesn't mean that your research has to stop.
This just means that you have to change the way you're going to research this.
So you're not going to research absolutely everything.
If you're missing a particular type of expertise, you can try and reach out to someone and see if they want to help.
When we were doing the Hello Barbie stuff, you know, I was the network guy.
Andrew Blach was the software and mobile app guy.
And then we had another individual who was helping with the hardware.
And then when we told him we wanted to publish the research, he said, well, I'm just kind of doing this for fun, and I don't want my name associated with this.
My day job won't like that.
So Andrew and I just said, you know what?
We don't need the hardware.
We will publish our findings from a network and software perspective.
And then if someone else wants to take the hardware side that actually knows hardware hacking, have at it.
Maybe we'll revise our research with their findings later on.
Which we probably won't do, because that's a lot of work.
But, you know, we've put research out there that others can use.
And, uh, yeah.
It's actually helped people, which is good.
So from a sample device workflow, you'll often find that as you're researching something, you'll be like, oh, crap, I forgot to get this.
So if you sit down and say, okay, I need software.
I need to plug into all the different pieces of the software chain.
So that is software that you may download from the website.
Firmware.
The mobile app that gets installed across various platforms.
So I'm pretty sure you don't have to worry about Blackberry anymore.
So you can probably cross that off the list.
But if you're testing a mobile app designed for an IoT device, or just a mobile app in general, you're probably going to want to test that on Android and on iOS to see the different versions.
Because the Android app store is far more lenient in their upload and verification process when compared to Apple.
So there could be differences where vulnerabilities have entered through the Android side, but not the Apple side.
And you should test both.
If you have the means to test both.
The device itself, if there's any other hardware or any other software associated with the device, then you're going to want to take a look at that, too.
And, again, this is if you have the software expertise.
From a networking perspective, it is very easy to test anything networking.
Except when you're trying to find a 10100 hub.
You have any idea how hard it is to find a dumb hub these days?
How many do you need?
I'm good.
You could probably make a fortune off your aftermarket.
Yeah, they're always in, like, people's basements next to their Commodore 64 box.
So the reason I say a hub, it used to be that hubs were cheaper than desktop switches.
Now you can't even find a desktop hub anymore.
And if you do, it's a smart hub, which means it's a switch.
Thanks, guys.
Thanks, marketing.
So if you can't get a hub, get a switch, but make sure it has a span port.
And you need the span port so you can watch the traffic as it flies by.
One thing I didn't put on here that I should have, and I will in future revisions, is you can also get a tap.
So you can make a tap, as well, if you have the hardware capabilities.
But one thing to know is when you're using a network tap is you're usually getting one direction of the traffic.
So if your results look kind of funny, you're only seeing send but not receive traffic, then you probably need a regenerative tap or two taps in the path.
And then two routers.
Very easy to find routers.
Because what you're going to want to do for any IoT device, it's most likely going to be wireless, but, you know, if it has a physical connection, you're going to want to test that, as well.
Connect it to your internal lab router.
Then when it... your external router is going to be connected to the internet and you're going to watch the traffic as it flows.
As you're doing your analysis, make sure that you're saving the files.
And I recommend putting them in a directory with at least date stamps so that you can say, yeah, I can point back to this.
On this day, this is what I saw.
So a good example of the networking side of things is... does anyone here have a Samsung Smart TV at home?
Is it connected to the internet?
Dave, is yours connected to the internet?
It's all right.
It's the Canadian internet.
It's dial-up.
You're safe.
So the Samsung Smart TVs, when I was joking earlier about something phoning home to Korea, Samsung is a Korean company.
It has legacy code that still beacons back to a server instance in South Korea, even though they have replaced that software with two other acquisitions.
They just have never removed the software, but it still beacons out like four or five times every minute.
You're not going to find that in the documentation.
That's something you're going to have to find via research.
So then the device itself, you're going to connect it, and then the bidirectional communications.
You're going to want to see not only what it's calling out to for command and control, but also what it's calling out to for analytics, because these companies love their... they love them some analytics.
So you're going to see all sorts of communications to non-branded or non-associated manufacturers' websites for tracking purposes.
And again, legacy code that was maybe never removed.
Now software, again, never overlook where the data is being written to.
A lot of applications that are run off of desktops are going to be storing in a SQL database.
Nine times out of ten, it's probably going to be unencrypted, or it's just going to be obfuscated.
APIs of both the device and the platform you're connecting to, you could probably brute force those or at least get some additional intel around how those devices or how the APIs are being used.
And there may be vulnerabilities.
There may be something that you can do with those APIs that was not expected.
And then just looking for bugs in the code.
Whether it be hosted or the actual installable code base for the device or something else.
Testing the UI.
I'm fairly certain in this day and age, you know, don't quote me on this, but a lot of fuzzing and a lot of rudimentary web app sec testing on the portals for these devices occur.
I don't know if they're automated.
Some of these companies are getting pretty sophisticated in that they're hiring people to actually test it as we would like web apps to be tested.
Some, you know, if you're getting these crappy $30 webcams from Southeast Asia, they're probably not going to have as much rigorous testing as something with a major North American presence.
So those, I call those ones the fun ones.
Now, hardware.
Device firmware, mobile app device.
This, again, becomes a little complicated if you don't know how to deal with hardware.
Now, I'm not a hardware expert.
I would like to get more into this, but it's just not my area of expertise right now.
So lab equipment and software.
You might want to take a picture of this one.
You're going to get the slides anyway, but whatever.
Hub tapper switch with a span port, two Wi-Fi routers, RJ45 cables, packet capture software of some sort for networking.
Software, get a good debugger that you know how to use.
Fuzzers are going to have some value, maybe, but, uh, I, I like to think that these are run through fuzzers pretty frequently.
Uh, and then any vulnerability port scanners, because you're going to want to document what is open to the world.
You know, does an IOT camera really need to have Telnet enabled?
The answer's always no, but it may, it may be enabled.
Or R protocols.
Oh, I love finding devices with R protocols enabled.
That's awesome.
For the hardware, I'm not going to go into this too much, because again, I'm not really a hardware guy, but if you look at this presentation at the very bottom, this is one of Mudge's presentations from Black Hat, uh, 2014, I believe?
Yeah, 2014.
And he lists all of the tools that he uses for hardware hacking, including like the standard oscilloscope, uh, digital multimeters, but then he gets into software, um, touches briefly on software-defined radio, that part's going to be cool when you start looking at remote devices like car door openers or garage door openers and anything where there's a,
like an RF signal.
So knowing when to stop.
This is really a big question, is when do I stop?
And I picked four reasons that you can and should stop.
So sometimes there's an artificial or organic, organically applied time constraint.
And that time constraint could be marketing.
Does anyone here work for a vendor where they've been held to a marketing timeline?
Dave, I know you're not putting your hand up, but you are.
And Ian, if he's still in here.
Sometimes marketing has a campaign to run.
They do not want to hear that the research will be done when the research is going to be done.
Uh, so in managing researchers that were academics and, you know, I I'd say are still academics, they have this concept because it came out of academia, that the research will be completed when they're happy with the results.
It could be a month , could be three years.
Well, you know what researchers that are hired to work at vendors, they kind of need a quicker turnaround time because they're not hired just to, you don't throw money at researchers just to have them sit there and think really hard.
You need to turn that thinking and the discoveries into something tangible and that tangible asset is why they were hired.
So these artificial and organic, uh, time constraints help put them in a box to say, what can you research in this period of time?
Because we want to have a go to market strategy that follows on the end.
And that's not something that you're going to hear in academia other than, hey, I've, I have a paper that needs to be written and submitted to this conference or this journal by this date, that will speed them up a little bit.
Diminished relevancy.
Sometimes someone will release research and you start reading it and like, oh, this is way better than what I was doing, or this has far more relevance or the way it was research is just better than mine.
So I'm going to bow out and go do something else.
It happens.
It happens a lot.
Success.
This is what we want, right?
We've successfully answered our questions.
We don't have to look at the data anymore.
We're happy with the results.
Failure.
Don't look at failure as a bad thing.
A negative result is still a scientific result.
Think of what happened to the, to the people who tested preparation A through G.
So what does success look like?
Again, you need to define what your expectation of success is.
And that could, I always say that success is something new was created.
That's not always good enough because you could create some new knowledge based on your research that isn't applicable to anyone but you.
That, you know, that could be good, that could be bad.
That's not going to be earth shattering.
You're not going to change the world.
You're influencing decisions or policies.
You're, you're pushing the way the world works today into a new direction.
You've generated a report presentation application script that can prove your results.
I say that's part of success as well.
And then yeah, is, is it learned that the data can't answer the question?
Again, this comes to the failure.
Some might say it's success, like, oh, well, the data I have wasn't enough to prove my hypothesis.
I'm going to go back at a later time and run through the experiment again, but adding these data points.
So failure, again, not every project is a success.
I know we were all told that we're very special snowflakes and everything we do is fantastic.
If you, if you see research that just comes out and it's pristine and absolutely perfect, Twitter is going to tell you you're wrong.
This is why you need to validate your research and make it reproducible.
So some negative outcomes, maybe your findings completely disregard the evidence.
Maybe it doesn't really add value in one side, one direction or the other.
So it's kind of flip floppy.
And then it prevents or there's uncertainty.
I don't trust my data or I don't trust the source of my data or the results I created, because I don't know how to use Python.
So I think this is what I have.
So I'd say that's failure.
We're on the homestretch here.
And I'll go over that.
So when you're presenting your research, there's a number of mediums that you can present your research in.
Blogs, definitely the easiest.
You're trying to pepper it out to a whole bunch of people in one fell swoop.
It's a very effective way of sharing research.
But it's not all that engaging .
You should probably do a series of posts to have people come back.
You know, no one wants to sit there and read a 6000 word blog post.
Someone once told me that your blog posts should be only as long as it has two sittings on the toilet.
Any longer, like two sittings maximum.
One sitting ideal.
Anything longer, they're not going to read it or their legs are going to fall asleep and they'll probably never read anything from you again.
So if you split it up into manageable chunks, then granted the bathroom might get used more, but your research is going to get digested more.
But articles, if you're publishing your findings in an academic journal or trade rag, this again, wide readership, not really getting a good way of fostering any interaction as a result, unless you put contact information and again, they'll just blast you on Twitter .
One thing you can do is if you find something that you think is really, really awesome and different, go to one of the big reporters and say, Hey, do you want this as an exclusive?
Because if you give them an exclusive, they love excuse exclusives, by the way, you give them an exclusive, you get the web equivalent of above the fold in a paper.
So it will get pushed out through their marketing channels, their sales channels, it really helps with the broadcast of your research and syndication, academic papers, you know, I'm not an academic, I probably will never write an academic paper, because I don't need to.
But it is required for some conference submissions.
Be prepared.
This is it's almost like a mini dissertation.
This is something that will be heavily scrutinized by your peers in that industry.
So you have to be very careful about how you prepare an academic paper.
And then white papers, marketing loves white papers.
Because it's, it's something that could be left behind at a customer.
It could be something you could send to the media so that they can glean information from it.
It's you have to be careful when you're writing a white paper, though.
And this is you could host it on your own website.
Don't put it behind a paywall.
Everyone hates paywalls, or registration wall.
But know your audience.
If you're publishing a white paper that you want reporters to consume, don't put all the technical details, put the high level bite-sized chunks, because they want sound bites, they want to be able to copy and paste your findings, the high level results,
because it makes it much easier for them to create a story and move on to the next thing they're doing.
Technical people , however, want the technical details, for obvious reasons , they're technical people.
Charts, if you have never used raw, I highly recommend it.
It's a free web app that you can just upload data to, or copy CSV or Excel spreadsheet to, and you have all of these really sexy visualizations.
So it's... if you've ever seen, like, a whole bunch of D3 charts and thought, wow, that's great if I could put my data in there, but I don't know JavaScript or don't care to know JavaScript, this will generate a lot of those non-Pi chart type of charts for you.
So alluvial diagrams, scatter plot charts, all that.
Really cool.
If you want to interact with the data, Tableau Public, it's free, plus it gives you a place to publish your data, and you can just point people to the link and they can fiddle and futz with your data.
Open Graffiti , this was written by one of my guys at OpenDNS, so this is a 3D immersive graph visualization application, and you can use it with Oculus Rift, so there, you've got justification for work to buy an Oculus Rift.
And then graphics.
Pictures, very important.
Pictures are loved by the press.
Very easy to copy and paste it into their report.
Infographics, I'm always on the fence on infographics.
They...
you just see, as an analyst, I saw so many of them, and you just kind of become numb.
But if they're created in such a way that adds value, they're pretty good.
But, you know, it depends who has final say on the structure of the infographic.
If the last point is by my product, it's not a good... not a good infographic.
Conference talks, kind of like this one.
Very great way... so I love presenting research at conferences, because you have the audience who could either be with you, against you, or indifferent.
So you have to be prepared for all of those cases.
So they may challenge you right there on the spot.
They may challenge you after the fact.
They may agree with you and say you're the... you know, your mom was right, you are a special snowflake.
And, you know, it could be very stressful, because you don't know what's coming.
If you know that there are going to be certain people in the audience that frequently challenge research, make sure you know the kind of questions they ask ahead of time, so that you can be prepared.
Press interviews, again, know your audience, use the four letter S-B-A-C, sound bites, and charts.
They love sound bites, they love charts, because it's less work for them to do.
And they will admit, you know, you're not insulting them by giving them sound bites and charts, you're helping them move one story off their plate in favor of the 10 other stories that they have to work on that day.
Right, Dave?
If you use code to produce your final results, you should publish it.
So if you use code, if you generate code, publish the code.
If you have the ability to publish the data, that's even better.
So if you get... if someone comes at you and says, I think your findings are complete bull, which will happen, DBR.
And you can say, well, here, here's the code I use to generate my findings and my charts.
And here is the actual data set that I used for my findings.
Prove me wrong.
If the answer is, thanks, I will prove you wrong.
Awesome.
If the answer is, well, I don't have time to do this.
I just know you're wrong.
It's like, well, you know what?
I did my part.
I shared the data with you.
Someone else can analyze it and come back and challenge me.
I'm being ethical.
I am being forward in presenting you with the data and the tools and the methodology.
There's only so much I can do.
You know, you can lead a horse to water.
Revision control, for God's sakes, don't keep it on your laptop.
Laptops die.
Laptops get coffee on them.
One of the researchers that I worked with, he would keep everything on his laptop and then his laptop would die.
And he would have no backups and he would have to start over again.
And it got to a fight every quarter.
Like, have you uploaded data to Git this quarter?
I'll do it now.
Come on.
It couldn't be easier.
Really couldn't be easier than using Git.
So if you publish code, people can reproduce, challenge your findings.
They may even find something interesting in your data that you may have missed.
That's great.
Thank them for it.
What was that?
Something about meth?
If they take my 30 lines of meth?
Oh, yeah.
That's my code.
Exactly.
That's my code.
Interns.
Keep this in mind, is that when you are writing any code or even doing analysis or publishing your findings, you are doing so so that you can collaborate with yourself in the future.
I can't imagine... there's been so many times where I've looked back at code where it's, like, nested functions within functions.
Like, this seemed like a really good idea at the time, but I have no idea what this is doing right now.
I know, I'll check my... oh, no, no comments.
All right.
Maybe I'll just write a new function.
So when you're defending your research, expect to have your findings challenged.
If they're not challenged, that's not saying you did a bad job, but it might say that people don't care about your findings.
Maybe it's not applicable to them.
You may get shit disturbers who just want to see the world burn.
Everyone knows some of those.
You'll get shitheads who just, you know, they are who they are.
They got picked on in school or whatever.
And then you'll just get grandstanders who want to point and say, you're wrong because you didn't do this my way, or I don't like your parent organization, so you're a fraud.
And then friends, colleagues, and peers, they might not like what you're doing either, or they may not understand your findings, and they want to hear you defend your research.
Cool.
I'll do that all day long.
So always argue with your data.
Don't just go off book and say, you know, you're just a bitter jerk.
That's why you're attacking my data.
Go and say, well, you know, my findings were based on these tools, this data.
Here are the visualizations.
Here's my process.
Go to town.
Prove me wrong.
And again, publish your code.
You can also ask for alternative findings.
That's a great conversation killer.
It's like, hey, maybe you can take a look at my data and find something I missed.
Oh, well, I don't know Python, or I don't know R, or I don't know, you know, regression models.
I can help you with that if you want.
No, that's a lot of time.
I don't want to do that anymore.
Some people are just jealous.
You know, there are a lot of people that challenge other people's research because they're jealous of them.
It happens.
So I won't go through this because we're running out of time here.
But these are questions that you should ask yourself when you're doing any sort of research, exercise, or engagement.
And, you know, if you answer yes, or if you answer no to any of these, you should probably take a moment and think of why you should be doing this.
Especially the, was any part of this done by hand?
If so, are those parts precisely documented?
Emphasis on precisely.
So just to summarize, so anyone can research anything.
Absolutely anything.
You just have to know whether you have the time, the knowledge, the desire to do so.
I encourage everyone to pick a device up that maybe they bought at Best Buy or they got for a birthday or Kwanzaa or whatever, and test it.
See what it's doing .
You'll learn a lot about the way the device acts and interacts with your network and the internet as a whole.
If you establish your own process, it'll make subsequent research activities much easier to follow.
So you're not reinventing the wheel in the process every single time you do it.
Just like an incident response activity.
If you can fall back on documented process, you know what boxes you have to check to go from A to Z.
And, you know, again, some people are just want to watch the world burn.
Nobody, some people do not believe your initial findings and will challenge you.
Always argue with your data.
Fight with your data.
So I, you know, I sat down.
This is excellent courseware to go through during Dancing with the Stars.
This, the first one is a very high level of, you know, what data science is and how to manage a data science team.
This data science course, this is going to teach you how to use R whether you want to use R or not.
Doesn't matter to me.
You can do, use a lot of that information in pandas, in Python, if you want the same results.
But this will help you start thinking about how to work with data, how to clean data, how to get data.
And then Code Academy.
Who doesn't love Code Academy?
It's awesome.
No?
Someone said no?
They don't love it?
I don't care.
So I, I've read and purchased every single one of these books.
I love this one.
I used to keep this on my desk when the researchers would come up and ask me a question.
And I just said, what page is it on in this book?
I really recommend if you're doing any sort of network analysis to get the TCP IP illustrated book, if you do anything networking related, this should be on your bookshelf.
Because that way you don't have to remember offsets and headers and all that crap.
It's been around forever.
One kind of unorthodox one is the open source intelligence techniques, the bottom right hand corner.
That's a great book to show you how to do kind of non non scientific research on publicly accessible data that you can then in turn feedback in to your research project and enrich.
And then a plug for Bob and Jay's Jay and not so silent Bob's book, data driven security.
How to present your data in a way that doesn't look like garbage.
Avoid pie charts unless you're trying to infuriate someone.
This will show you what is valuable and what is not valuable from a data perspective.
And with that, I will leave the negative 30 seconds open for questions.
Any questions?
I'll be around all day too.
Thanks.