Awesome.
Hello, everyone.
We're gonna scare ourselves to death today.
And it's exciting.
This is a crazy topic.
It's obviously very topical.
Anyone who's read the news in recent days, weeks, months, you know what's coming.
And there's a whole bunch of stuff.
So, I want to kind of start out with me first.
So, my name is Josh.
I've been doing this open source thing for like literally decades at this point.
That's a picture of my Mastodon profile and InfoSet Exchange.
If you don't use Mastodon, you should because it's awesome.
It's where all the security nerds hang out.
When this XZ fiasco went down, Mastodon was on fire.
And there was so much going on.
I also host a couple podcasts.
One's called the Open Source Security Podcast.
And that's the title says it all.
But the second one, I do for CypherCon right here.
It's called the Hacker History Podcast.
And if you haven't listened to it, it's a super fun show.
But more importantly, is if you want to tell your story, come and find me and let's talk.
Everyone has an amazing story to tell.
And it's really fun telling it.
And it's really fun listening to it.
Every guest I have had on the show has said, no one wants to hear my story.
It is a lie every time.
Everyone's story is amazing.
So if the first thing that popped into your head was, oh, no one wants to hear my story, they definitely do come find me and tell it.
It's so much fun.
Trust me.
All right, we're going to get moving right, right away.
Okay.
All of the data, well, most of the data you see today, there's going to be kind of two big groups of data we're going to look at.
I am a data nerd.
I used to work for a company called Elastic.
They do Elastic Search.
And I took an enormous amount of data and I shoveled it into Elastic Search.
And then we look at it for fun things.
Most of the open source data comes from this site called Ecosystems.
A friend of mine named Andrew Nesbitt runs it.
It's free.
They take sponsorships.
If anyone's looking for some really good open source data, it's where it's at by far.
And it's absolutely amazing.
To give you an idea of what I did, he, so my funny story is, Andrew hosts the site through, I think it's CloudFlare, it backs it.
And you know, all the usual, no one can overwhelm my site stuff.
It's one day I said to him, like, hey, if I start hammering your API, am I good?
Do I need to do anything special?
He said, oh, I don't know.
It'll be fine.
And a couple of days later, he's like, can you stop what you're doing?
Because you're hammering my site.
I was like, all right, fine.
Give me, can I get a dump of your data?
And he gave me a dump.
It's about 300 gigs of Postgres dump that I then extracted and shoveled into Elasticsearch.
Like, it's a lot of data.
I was actually surprised how well Postgres handled it.
I didn't think it could do it that well.
We're talking, I think it's a couple hundred million records.
Like, it's quite a bit.
But anyway.
So, the first thing we're gonna do is understand what is the shape of it.
And I made a word cloud of the various repos that contain the data we're gonna look at.
So, NPM is the biggest because NPM is the biggest.
And I've got some better data on this.
We'll kind of go over later.
But then this is all of them.
This is everything that's being tracked by ecosystem.
So, it's quite a bit.
It's all the big repositories, as you can imagine.
And one of the things that ecosystems does that makes it so easy to work with the data is they normalize everything.
Because if you try to, for example, query NPM's API, which I used to do, and it returns some data.
And it's kind of a mess because it's like 20 years of data.
And then you look at Docker, it's completely different.
And so, it's a big headache.
So, anyway, this is the actual graph to give you an idea of what kind of sizes.
Can you see my mouse?
You can.
Aha.
Okay.
So, the orange is NPM.
Unsurprisingly, right?
Biggest by far.
And then Docker is the purple here.
And then everything is scattered in between.
And we'll kind of go over what they all are.
And I've got many other pictures.
But to give you an idea of what we're looking at, this is the packages published in each repository.
So, what I mean by that is you look at the latest date a package was published.
Okay?
So, anything published very recently is obviously way over here on the end.
This cuts off in March.
Early March.
So, we're looking at about a month old data right now.
But anyway.
And this goes all the way back about 10 years way over here.
I just did 10 years of data.
Because once you get past that, it's zero functionally when we're looking at the data.
So, anyway.
Okay.
Now, how big is big?
Okay?
8.1 million packages are in this data set.
I don't mean like 8.1 million versions.
We'll talk about versions on the next slide.
8.1 million open source packages.
This isn't everything.
I know there's GitHub repositories.
There's so much extra data here.
But 8.1 million is the number we're dealing with for open source packages.
And every package has multiple versions.
We'll explain that in a minute as well.
8.1 million is a huge number.
Okay?
And I'm gonna break that down into a number we can maybe sort of understand as humans, but still probably not.
But this is like there's an old saying about when we kind of look at data as humans, there's one, two, three, and more than three.
And that's what we can understand.
And this is a hell of a lot more than three.
But here's where things get really wild.
Now, every open source project has multiple versions.
Actually, most open source projects don't have multiple versions.
I'll show that in a minute.
But you get the idea.
If we take every version of every project that is tracked in ecosystems, it's 93 million is the top of that graph.
And you can see it's a beautiful graph.
I mean, in terms of like the data is beautiful crowd, that is a gorgeous graph.
I did not doctor that.
That is actually the graph.
But you can see what the growth looks like.
It's maybe tapering off at the top.
I don't know if that's just because we have an incomplete bucket or if open source is slowing down.
I don't know.
But it is gigantic.
It is hard to understate how big this is.
And I'm showing you this.
I have no answers, by the way.
At the end of this talk, you're not going to walk out and be like, I know what we need to do.
Nobody knows what we need to do.
No one understands this problem, though, either, which is why I'm doing this.
This is really hard to fathom.
And I'll explain why in just a minute.
Okay.
So I talked about every package has versions.
Right?
This is the graph of how many versions every open source project has.
And the first time I looked at this, I was like, what the hell is this graph?
35,000.
There is a project in NPM that has 25... it's actually more than this now.
Because they literally publish a version like multiple times a day.
This is like an NPM test package.
I mean, they define it as what?
A test package to use during smoke tests.
But this is like the NPM QA does this.
25,000 versions of one package.
It's marvelous.
And so that amused me.
So anyway, here's a real graph that we can actually understand.
And this one I think I did to about 100... I can't tell.
It's very small.
About 100 versions.
Because most things don't have 100 versions.
But you can see there is a gigantic spike.
The spike is at one.
Right?
Most open source packages have one version.
I don't know what that means.
Is it good?
Is it bad?
I can't say.
But that is what we're looking at.
Now, remember, we're talking about tens of millions of packages here.
Right?
So even though a bunch of them only have one version, there's still a ton of stuff that have multiple versions.
Okay?
Now... okay.
What does it look like?
This is the graph we saw way at the beginning that I had broken apart into different pieces.
Right?
So what if we split it up and we kind of overlay the ecosystems on top of each other?
Green is NPM.
Right?
It has that weird spike.
I have no idea what that spike is.
It's probably a spam attack of some sort is my suspicion.
One of the things ecosystems does is they don't delete packages.
They'll tell you it's been deleted, but they leave it in their dataset, which is nice because then we can do things like see a giant spike for no apparent reason.
Then you've got Docker underneath.
And then it's Go and Python are in this as well.
And it becomes zero once you get down lower.
You can't deal with the numbers.
Right?
Because they don't show up.
This is one the challenges when we're working with a dataset like this.
It's gigantic and it's either NPM or zero is what you end up with.
And this is one of our challenges.
Right?
When we think about how big open source is, how do we look at these in a way that makes sense?
And sometimes you have to split things apart where you might say NPM is special and we're going to treat that different.
Right?
So 4 million packages.
Now, I remember I have a story I'm going to tell.
So I work with the Open SSF, which is the Open Source Security Foundation.
This is an organization that's part of the Linux Foundation.
And they're tasked with securing open source, which is, of course, a gigantic and ridiculous task.
And it's unattainable by all means.
And there's a conversation that was started there some time ago where some of the Open SSF leadership was like, hey, we should figure out, like, what is the correct number of maintainers, right, that we should have in an open source project to say it's healthy and good.
And then, of course, there was a lot of discussion and hemming and hawing and they landed on like five or six or something I think is the number they were talking about.
And I showed up one day and I was like, hey, most open source projects have one maintainer.
And I'll show you the graph of that in just a minute.
But they were like, oh, well, what does that mean?
Well, actually, I shouldn't say that.
I said I have all this NPM data.
Specifically, I only had NPM data at that time.
I was like, if we look at NPM, literally almost like 90% of the packages are one maintainer.
And then one of them says, hey, that's just NPM, right?
Python is totally different.
NPM is weird.
Python doesn't have this problem.
So I found the Python data.
I ran the data.
And the Python data is literally the same.
I'm like, the graphs are like you can't tell them apart.
And then they just kept talking about five or six maintainers being the right number.
And this is one of our challenges, right?
When we're dealing with all of this open source data with the giganticness of this problem, it is really hard to talk about things in a way that makes sense.
So one of the other inevitable conversations that comes up when we're like at the open SSF and talking about things is people say, we should audit all of the open source packages, right?
And you think, okay, I mean, that's a big job.
Let's think about this and decide what it means.
And now so let's say we take 4 million packages, right?
We know there's more than that.
We know it's closer to 8 million.
But let's say there's 4 million in NPM.
And I'm gonna pick on NPM just because they're the biggest, but they also have the best data.
So it's easy to pick on them for both of those reasons.
Don't think less of NPM necessarily, I'm going to pick on them a lot.
We get the idea, right?
So let's say we audit a thousand packages a day.
At 4 million, that's 11 years.
What does 11 years ago look like?
What was 11 years ago?
2000 and what?
14?
13?
I don't remember what happened in 20... I don't remember anything before the pandemic, but even if there wasn't a pandemic, right?
But here's the thing, even if we wanted to do this, even if we wanted to try to audit all this stuff and vet it and understand what it means, there's 8,000 new releases every day right now and it keeps growing.
So it wasn't 8,000 new releases a couple weeks ago, it was less.
And so this is the thing that is mind-blowing about all of this data.
What can we do when we have a system growing faster than we can even imagine?
Now, I don't know how many of you do security audits.
Auditing any application is generally a multi-month engagement.
This isn't like one person can run one audit and have it done in a day.
Maybe some of the very, very, very tiny open-source projects you could think about that, but the vast majority, absolutely not.
And the dependencies are madness, right?
For anyone who was at B-Sides Milwaukee a couple days ago, I gave a talk where I showed what happens in a dependency graph of NPM, where I started with, I think, 11 packages that I depended on for my project.
And when I did NPM install, I ended up with 162.
But this is the nature of open-source.
This is every ecosystem, right?
NPM is easy to pick on.
Every single project works like this.
NPM is not special.
Okay, I kind of hinted at this, but how many people are maintaining open-source?
So this graph goes up to in the neighborhood of 500.
I know you can't read any of this because they're all so freaking big, but you know what that value is way over on the left.
That's not the axis.
That's the actual graph, right?
So we're going to zoom in a little bit.
Zero to 50.
Again, we're going to focus on NPM because NPM tracks this maintainer data.
So you can see, literally, almost every NPM package, this is one.
This is one maintainer.
Some of them have zero.
I don't know why that is.
I think it's an error in the data.
When you're working with data on this level, sometimes weird things happen, and that's just life.
You deal with it, right?
We don't obsess over it.
But to give you an idea, there are 3.6 million packages in my NPM dataset.
3.1 million of those have one maintainer.
What does that mean?
Does that mean they're secure?
Are they insecure?
How many maintainers did XZ have?
Well, it had two a couple weeks ago, but now it only has one again.
So this is where we stand, right?
This is how open source looks like.
There are countless projects that are one person.
And it's easy to say this is broken and wrong and it's insecure and the world is going to end because we only have one maintainer for all of these projects.
Some of the maintainers are overworked.
Some of them love being the only maintainer on their project and they don't want your help.
They don't want you to come and save their day.
It's really complicated and it's really weird.
We talk about open source as this one thing.
Open source is 3.6 million things.
And I guess if you really want to cook your noodle, I've got a friend of mine who every time I say that most of NPM is one maintainer, he says actually it's less than one.
And then I tell him to shut up because no one cares.
But what it really is is how many of these packages one person has three or four packages.
Right?
More than one, for sure, by all means.
And so, this is it's so complicated.
Okay.
I'm going to keep going.
All right.
Now, I mentioned when the folks at the open SSF said, oh, Python's different.
Right?
The other thing that comes up is they say, okay.
Okay.
So, most of them are one.
Right?
3.1 million packages.
But of the .5 million, that's the important stuff.
Right?
That's the stuff that has extra maintainers.
You, of course, know where this is going.
But right.
So, we can say, okay.
So, let's look at the so, here's another crazy thing.
95% of NPM has less than 2,000 downloads per month.
Which is kind of mind blowing as well.
So, let's look at things that have 100,000 downloads or more.
Because some of this NPM, 1.4 billion, that's how many downloads.
There are NPM projects with literally billions of downloads per month.
Not total.
Per month.
And some of them have 25,000 versions.
But again, NPM is weird.
Okay.
So, let's take 100,000 downloads.
Hmm.
That kind of looks like the other graph.
All right.
A million.
Right?
A million.
Everything with a million is different.
Yep.
Pretty much.
It all looks like this.
Every, no matter how you slice and dice this data, I have tried so hard to not get that graph.
That's all you get.
It doesn't matter how you look at it.
That's the graph.
That is the reality of open source.
Our universe, like the foundation of society, is built on that line right there.
Think about that.
It's mind blowing.
Okay.
What about a billion?
Right?
The things that have more than a billion?
Yep.
We've got the top one is eight maintainers.
And at the bottom is one.
You can see there's one, one, four, twos, a four, a five, and an eight.
Those are a billion downloads a month.
And for everyone, I wrote them down.
Has flag, ANSI styles.
This is very NPM feeling, right?
Has flag, which is I think you're giving command line options.
ANSI styles, color.
Supports color.
You got to know if your terminal can print those colors for ANSI styles, yeah?
Chalk, which again prints things to the terminal.
Debug, I don't know what that does.
Semver and TSLib, which is TypeScript.
But yeah, like it's mind blowing.
Literally every NPM package probably includes these things.
And the biggest one is eight people maintaining it.
Yeah, yeah, TSLib.
Right, right.
Okay.
I want to also talk about who these people are.
Because one of the other things that comes up, anytime we talk about the size of open source and who's doing what, there's inevitably going to be someone who talks about, oh, a bunch of these developers, they've got their day jobs to work on the software.
And you've got people who are being paid by Google or they're getting grants from the government or whatever.
So there's a company called Tidelift.
It's a bunch of red hat nerds I used to work with started this company.
It's a cool company.
Their motto is pay the maintainers, which doesn't solve all the problems either, but I'm not going to talk about paying maintainers today.
But basically they went out and interviewed a ton of open source maintainers and they basically said, why do you do this?
What's the status of your employment and the projects you work on?
60% of the open source maintainers they talk to do not get paid to work on the open source they work on.
That's a pretty big number.
And even then of the others, 13% are only paid to work on it sometimes.
You've got 22% make most of their, oh, I'm sorry, 13% make most of their income, 22% make very little, right?
And this is one of those things we like to think about.
Oh, there's all these companies paying all these developers to write all this software.
This is even true of the Linux kernel.
In fact, I was talking to the kernel maintainer, I'm going to talk about him in a little while about CVEs.
But he also said that when I was talking to him, he was on my podcast actually.
And he mentioned, well done, thank you.
But he was talking about how an enormous number of Linux kernel developers are not like paid by Google or Microsoft or Red Hat or any of those people.
It's just like random people doing random work in the kernel, which is really cool.
But also it like, what does that mean?
We don't know.
All right.
I want to go back to the downloads graph, because why is the giant spike over on the left?
So we're going to zoom in and enhance a few times here.
And what we get is most of the packages in NPM, again, we're focusing on NPM because they have the best data.
Most of the packages in NPM have like four downloads per month.
That's zero, right?
When we're talking about some of this stuff having like literally billions of downloads, most of them have four.
Now, you can also make the argument that most open source isn't used.
And that's not unreasonable at all.
I think a lot of this open source is not used.
It exists forever.
This is one of the challenges we see when we look at open source repositories, is if I add a package today and I never update the package, in 10 years, it's still going to be there.
Because who's going to take it down?
Stuff doesn't get removed.
This is the graph here where we look at when was every package last released?
And this isn't just NPM.
This is everything.
Okay.
But when was the last release for every package?
Green is more than a year old.
Blue is less than a year old.
2.3 million things are blue and 5.7 million are green.
So, basically, what?
Like two-thirds, a little more than half, are things that just exist in these repositories more than a year old.
What does that mean?
Right?
Is it abandoned?
Is it done?
Are the developers alive?
We have no idea.
Some of the people, some of the green is because the people are literally not with us anymore.
Some of the green is because people didn't care.
My goodness knows, how many of us have 100 GitHub repos that we haven't committed to in more than a year?
I don't even want to look at that statistic for me because my GitHub is riddled with half-finished projects and crap I started forever and a day ago.
Most of it I don't publish to NPM.
But you get the idea.
These are like bananas numbers.
I don't know what they mean.
I don't have good answers.
I also want to talk about CVE.
CVE has been in the news because NVD kind of collapsed about a month ago, maybe two.
And we have the XZ fiasco we've heard of.
I mean, it's a security conference, obviously.
It's part of what we're talking about here.
So this is the graph of CVEs.
And I know if anyone watched HD Moore's keynote, it was like an hour and a half ago, give or take, he had a similar graph.
He actually had the same numbers.
And he talks about how big CVE is getting.
And this is a similar graph we see, kind of the growth of open source.
We've got the growth of vulnerabilities.
Every graph we look at of our industry, it's all going up and to the right at an alarming rate.
And it's crushing tons of teams, especially security teams.
Last year, the line over here, there we are.
That's like 30,000 CVEs in 2023.
And 2024 is looking like it's going to be even more.
I'll show this graph first.
If we graph the CVEs by year, so this is 2021.
And you're talking about, oh, it's like just over 4,000, almost 5,000 CVEs.
And if we look at 2024, which is up here, we're at over 9,000 already.
Oh, man, that would have been, that's a Dragon Ball joke, I didn't know.
Anyway, you get the idea, right?
We're growing vulnerabilities at 20% per year.
And so this is why NVD collapsed.
Because NVD can't keep up, right?
If we look at what happened to NVD, so for anyone who doesn't know, the National Vulnerability Database, we've got CVE IDs, which come from an organization called MITRE.
And then NVD, the National Vulnerability Database, is part of NIST in the United States.
They're completely unrelated.
They don't work together.
They sort of like each other.
So NVD would take CVEs from MITRE, they would apply severity, and they would apply version and package information to all of those IDs, right?
They did this enrichment of the data.
And one day they just stopped.
Like, they didn't tell anyone they stopped even.
They put a little notice on their website after they had stopped, saying, we're going to experience a slowdown in NVD data, and we're going to make a consortium to fix it.
And then this happened.
And for anyone who... it's funny, like, we all complained about NVD, and we hated it until it was gone.
And then we're like, oh, no, NVD is gone.
What are we going to do?
And the answer is, we don't know.
But it's definitely a problem for a lot of organizations.
Anyone who does FedRAMP, for example, that's like the federal government's, like, we're going to verify these cloud services are A-OK and all that jazz.
FedRAMP specifically says you have to use NVD severities.
There are no NVD severities.
Mission accomplished.
We all pass, right?
Actually, it's funny, because there's a whole bunch of federal organizations that were like, NVD is gone.
What do I do?
We're like, you're the freaking government.
Like, you tell me.
And it's funny, because then they're like, I don't know what to do.
And we're like, we don't either.
Go find someone to talk to.
And we don't have a good answer, right?
NVD says they're going to use a consortium.
But the reality is, like, when you have this sort of growth, we don't know how to solve these problems.
We can talk about, oh, we should use open source to fix it.
I mean, I do think there's probably some open source ideas we can leverage here.
I mean, but the reality is, expecting, like, one underfunded government organization inside of NIST to be able to handle this kind of growth?
Absolutely not.
I mean, do you think their budget from 2021 has been doubled into 2024?
No, they cut NIST's budget, actually, right?
They have less money today than they did in 2021 to enrich all of these ideas.
And the other thing I love is they're behind now.
Do you have a question?
Are you raising your hand?
Okay.
All right.
You can also interrupt me.
But they're falling behind.
So, now, they can't keep up, and they've fallen behind, right?
This feels like that Simpsons episode where they put Bart in the remedial classes and he moves slower and he doesn't know how he's ever going to catch up.
Same idea.
We don't know what to do with this.
We have no idea.
I also want to talk about the Linux kernel.
This is where the talk of Greg KH comes in, right?
So, recently, the Linux kernel became what's called a CNA, a CVE numbering authority.
That means the Linux kernel has the ability to coin their own CVE IDs and push them into the CVE repository, okay?
I mean, that seems like a decent idea.
Why don't we do that?
The Linux kernel's only, what, the most widely used piece of software on the planet, probably.
I didn't know this, actually.
When I interviewed Greg, Linux on Android devices, there are 4 billion Android devices that run Linux.
He even said, like, Android devices, 4 billion, everything else is zero, right, when you look at the numbers.
And it's wild, because it runs on rockets, it runs on milk machines, like, it's everywhere.
What do we do?
And, again, so the Linux attitude towards vulnerabilities is we're going to assign a CVE ID to anything we think is maybe a security vulnerability, right?
And people are freaking out.
The blue on the graph is how many CVE IDs used to be assigned for the Linux kernel.
This is just 90 days of data, right?
So about half right there is where the cutoff is.
Blue is what used to be kernel IDs.
Green is what the Linux kernel is now assigning themselves.
And you can see the number is, like, dramatically higher.
It's wild.
People are freaking out about this, because, like, also interpreting kernel CVE data, it's not like you just read it and know what it means.
Like, this is complicated stuff.
And if we look at the vulnerability data, like, overall, this is all CVE data being published.
Green is the kernel.
Like, it's noticeably larger.
So now, here's the thing we got to think about, right?
Open source is growing at an alarming rate, and we're also missing a bunch of vulnerabilities.
I'll talk about that in a second.
You've got CVEs growing at an alarming rate, and now the Linux kernel accounts for 15% of all CVE IDs being coined.
So instead of growing at 20% per year, we're probably going to grow at about 30% per year, thanks to the Linux kernel.
So that means in about three years, not four, we're going to double the number of IDs in the CVE data set.
I mean, you got to remember, like, we're looking at here, 30,000, it's going to be 60,000 in a couple years.
How many of us can keep up with 30,000 CVE IDs?
No, you can't.
Okay.
Right?
None of us can raise our hand.
We all get it.
So this is, like, this is where we are as an industry.
We are all being crushed by this volume of open source, this volume of vulnerabilities, and we have no freaking clue what to do with it.
Anyone who's trying to do, like, AppSec in your company, how many times do you think you have to upgrade something to a certain version?
And there are two versions past where you even were when you looked a couple days ago.
This is, like, just unsustainable, maddening growth.
Yes.
You have a question ?
Okay.
I'll explain.
Okay.
So the question was, kind of, the Linux kernel can assign CVEs, like , who's responsible for doing that?
Is it part of MITRE?
Who is it?
So the way this CVE project works , and this is kind of, this is all very confusing, I'm not making this up.
The Department of Homeland Security pays an organization called MITRE.
MITRE is a giant defense contractor.
They're a four billion dollar defense contractor.
They do anything you can imagine for the federal government.
It's an amazing organization.
You've probably heard of things like the MITRE attack framework, for example.
Same people.
Okay.
The Department of Homeland Security pays MITRE to run the CVE program.
They have recently created, like, they call it CVE.org, and they're claiming it's a community-run program.
We'll see.
I'm not super confident in that community angle there.
MITRE's still driving the bus.
But part of that is there are these things called CNAs, CVE Numbering Authorities.
Anyone can be a CNA as long as you're willing to do the paperwork and jump through their hoops.
So, like, Red Hat, Microsoft, Oracle, all the big companies are CNAs.
Okay.
When you become a CNA, you are responsible for, like, your sphere of software.
So, like, in the Red Hat instance, they're responsible for Red Hat software and some open source.
Microsoft is responsible for Microsoft software.
Like, obviously this makes sense, right?
The Linux kernel is now a CNA.
So, they, like, the Linux kernel is responsible for the Linux kernel now.
And one of the ways this works is MITRE has a very strict definition of what a vulnerability is.
And we'll say the Linux kernel is pushing it to the very edge of what they're filing.
But it is fundamentally, it's Greg KH and his team are deciding what is green.
They get the last say.
Now, you can go to Greg KH and say, Greg, I don't think that's a security vulnerability.
And there are a couple of things they've withdrawn after people have looked at them.
But generally speaking, Greg's answer is, well, we think it is.
And in the CVE ecosystem, it is.
Like, that's just the way it goes.
You can go to MITRE and beg and plead, maybe.
But generally speaking, they're going to just point at Linux and be like, go talk to those folks.
There's nothing we can do.
It's wild.
It's completely wild.
And then NVD takes that data and they do their thing.
But they're not part of any of that whole Rube Goldberg contraption.
It's wild.
It's completely wild.
And it's gigantic.
Okay.
All right.
We're all being crushed, right?
So, here's where the talk, I guess... are we doomed?
I don't think we're doomed, but it doesn't look good.
I don't have any answers.
I wish I did.
But I don't.
But here's the thing.
So, understanding these sorts of problems is really, really hard to do.
We don't understand how big open source is because we're humans.
And humans are really bad at big numbers, right?
I said way at the beginning, one, two, three, more than three.
That's what we're dealing with here.
We don't know what 93 million packages looks like.
Because we can't fathom it.
Now, back when I mentioned about vulnerabilities, right?
30,000 vulnerabilities per year.
Do you think 30,000 vulnerabilities per year is correct when there are 93 million packages floating around out there in the universe?
Yeah.
Now, think about that.
What do you think the number should be?
Probably north of a million is my suspicion.
And how many of us can handle 30,000 CVEs?
None of us.
So, what do we do when we have a million CVEs?
We don't know.
These are the problems.
This is the thing.
This is the problem with exponential growth.
Exponential growth sneaks up on you, right?
By the time you realize it's exponential growth, it's too late.
You can't do anything about it.
And we're there.
We're there with vulnerabilities.
We're there with open source.
Most people don't even know we're there with open source, right?
Like the whole XZ debacle that just happened.
How many other places is that happening?
When you have tens of millions of open source packages, it's not one.
But we don't know what it is either.
It could be 10.
It could be 100.
It could be a million.
We have no idea.
And it's terrifying and crazy.
And we don't know what to do with it.
Now, here's kind of the way we need to think about this.
I guess I do sort of have a call to action in this talk.
Almost.
There's a concept in game theory called finite and infinite games, okay?
When we think of finite games, this is like baseball or basketball, where you have rules.
You play the game.
Someone wins the game, in theory.
Maybe you tie.
You get the idea, right?
We got March Madness right now, okay?
March Madness, great example.
Someone has to win, right?
There is no tying in that game.
They have very well-defined rules.
We know who the players are.
We know what's going to happen.
When you turn on a basketball game, you mostly understand what's going to go down on your TV, right?
Or if you go or whatever.
How many of you have heard of Kelvin ball?
Yeah, a bunch of hands.
So, there used to be a comic long ago that died in its prime, we'll say.
And it's become a bit of legend with this old geezer.
It's called Kelvin and Hobbes.
Yeah, so good, right?
Ironically, I bet if it hadn't been like plucked from us, it would have died a horrible death and we would dislike it.
But because it was taken from us too early, we love it.
Anyway, there's this game they played in Kelvin and Hobbes called Kelvin ball.
And Kelvin ball has no rules, right?
You make them up as you go.
What is it?
You can't use the same rule twice.
This is like this whole list of rules.
Kelvin ball is an infinite game, right?
Where you have known players, there's unknown players, the rules can change at any time.
The objective of an infinite game is to keep playing.
The Cold War is often used as an example of this, where you had the United States and the USSR engaged in this like nonsensical game of each trying to not murder the other, sort of.
Anyone who's my age grew up in this.
It was wild and ridiculous.
But this is an example of like, there's no end, right?
You can't play to win because there's no way out.
This is open source, right?
There's the post office is another good example.
If I delivered every piece of mail at the post office today, tomorrow there's more mail, right?
This is the same thing.
If we audited every single open source package for security vulnerabilities today, there's 9,000 new ones tomorrow.
And that's how this works, right?
We can't think about securing infrastructure or open source or anything as a, oh, I just have to do this thing and then we're done.
That's not how it works.
That's how it's never going to work.
It's hard to think about this stuff.
It blows the mind.
I look at these graphs every day.
I think about them every day and there's days it's just like terrifying and I don't know what to do.
Okay.
There's also just kind of what do we do about open source and what does it mean?
This is how open source works.
There's going to be people who tell you open source is broken or we need to fix open source or we need to secure open source.
This is open source.
There's nothing wrong with it.
This is how it works.
If you don't like it, I I'd say don't use it, but you don't have a choice anymore.
Like it won.
The Atlantic Council has a really nice white paper about all this where they define open source and kind of two nice buckets.
One is like infrastructure.
You can imagine things like we have roads, we have power lines, we have, you know, water that runs underground and comes to our houses, that sort of stuff.
Right.
This is public infrastructure.
Polite society needs infrastructure to survive.
We need open source to survive now.
Right.
Open source has become public infrastructure.
But how do we what do we what does that mean?
How do we pay for it?
Who's taking care of it?
Right.
We pay taxes to the government and the government makes sure we have roads.
Who is making sure open source functions?
Nobody.
Right.
We don't know.
And and they also break it down in the context of water.
They use like the water system, right.
The water table, like lakes and rivers and all of this.
Many of you don't know this, but back in like the 50s and the 60s, there were rivers that lit on fire in the United States.
Like we had a horrible pollution problem.
The water was terrible.
You couldn't drink water in many places.
I grew up in a city that you still can't eat fish out of the river.
Right.
This is this is up by Green Bay because it was so polluted.
I mean, it's way cleaner than it used to be.
But you still they're like, don't eat the fish.
Right.
That's the rule.
And I get it because whatever is in the water.
So when we think about kind of open source in that way, you can have something like water.
And if you pollute it, you ruin it for everyone.
Or we can work together and try to do things like keep it clean and take care of it.
And then everyone can use it.
And so like, I don't know how this ends.
I don't even know how this starts.
But I can tell you, there's a lot of people thinking about this now.
You've got like people in the federal government in the U.S.
at CISA that are having lots of discussions about like what's going on in open source.
How can we work with open source?
How do we pay for some of this stuff?
They get a lot wrong because they're still like obsessed with, oh, we need to fix open source.
Like, no, no, you don't.
Figure it out.
There's a group in Europe called the Sovereign Tech Fund that's doing a lot of work into how do we fund some of this?
What does it look like if we just start paying people to work on this stuff, like give them grants with no expectations?
We don't know exactly what that looks like yet.
There's companies like Tidelift doing work on this.
You've got a lot of big companies are giving various grants and they're hiring people to work on some of this stuff.
We don't have any answers.
I wanted to give this talk today just because like this is so freaking huge.
Yes, you have a question.
Okay.
So the gentleman in the back asked about the White House released a document talking about, I think it was CISA technically published it.
And they were did like a request for comment on like memory safe languages and software.
It wasn't specifically geared towards open source, but they did.
They actually did have some comments about securing open source in that.
Cause I remember I replied to that RFC and I actually, a bunch of the graphs in here are in my response to the RFC I sent them.
But so the question is kind of, so like the White House is, we'll say making these requests.
They're, they're providing this guidance.
They're saying like, stop using C, right?
Which I don't think anyone who's ever used C would disagree with.
And you like, you like C?
Well done.
That's all right.
You're allowed to like C.
That's fair.
He said after 25 years of C, he doesn't want to learn Rust.
And I get that like for sure.
For sure.
Rust is a cool language though.
But however, all right.
We're not, we're not going to start this fight right now.
We'll go, we're going to go have a fight outside after this talk is done.
Yeah.
We'll like, we'll go to town, but I love C too.
I'm not going to lie.
I've like, I've been using it forever, but so, so here's the thing, right?
You've got the White House.
They also did an executive order maybe two years ago where like, for example, they said you need SBOMs with all your software.
And they talked about, you know, some secure development activities and, and they released the secure software development framework and just released a, an attestation form.
You're supposed to fill out and send to them about your secure development practices.
And it is useful that the federal government is paying attention to this stuff, but these are all fundamentally unfunded mandates, right?
If the government says you should do something, but they don't give you money to do it, it almost never happens.
Like there will be organizations that fill out the SSDF attestation form.
There will be organizations that say, we're going to stop using C in some of our projects.
You might see medical device manufacturers that say like, we're going to do some of these things, but fundamentally, if there's no money behind it, it has no teeth.
And that's just the nature of the government.
I mean, it is what it is until, I mean, this is kind of like, if you look at pollution, pollution is one of my favorite, right?
Or when you have volunteer adherence to some environmental regulation, nobody does that, you know, nobody.
And I think this is similar with software until we have federal agencies saying, if you don't do this, we're going to fine you, or we're just not going to give you any money.
That changes the game.
But on the other side of things, one of my favorites is when the federal government says, all right, we need you to do all of this secure development work.
You're like, great, we need to charge you 20% more.
They're like, no, not like that, right?
Then they're like, on second thought, maybe we don't have to do all this stuff.
You know, can you do some of it?
You're like, 10%.
Never mind, it's fine.
And that's just kind of how the government works, right?
I am hopeful that we will see something with teeth at some point in the near future.
But the other part of this is, even if they try to give it teeth, it's going to take a long time.
The example I use is, so we all know about railroads, right?
Everyone's familiar with trains.
We all love trains because they're awesome.
Now, about 100 years ago, the way they would stop trains is you had people running on top of the train, like twisting those wheels to turn the brakes on.
And you can imagine with people running, literally running back and forth on top of the train, they fell off all the time.
You fall off, you're dead.
Like, that's it.
That's how it worked.
So tons of people died.
They invent air brakes, right?
Perfect solution.
You put these things on the train, you've got air hoses turning the brakes on.
Great.
It took 80 years from the invention of the air brake until the government finally said, all right, you have to start using these.
Like, stop letting people fall off your train.
80 years.
How old is C?
50, maybe?
Like, I mean, but think about that.
Like, our industry can't fathom the speed at which government moves.
And they can't fathom the speed at which our industry moves.
They don't understand that graph, right?
When you show that graph to government people, they're like, I don't know what this is.
I don't know what it means, right?
Because that's only 10 years.
What is a graph?
What is that graph going to look like in 20 years or a hundred years?
A hundred years of growth in open source?
No way, right?
I mean, something has to change.
I don't know what it is yet.
Anyway.
Okay.
I've been rambling on for a really long time.
Are there any other questions anyone has?
All right.
I saw the gentleman in back.
I saw you first.
I'm going to call on you.
Okay.
So the question was basically, for anyone who watched H.D.
Moore's keynote, he talked about prioritization when he talked about vulnerabilities.
And then you asked, like, so if I'm using open source, we need to obviously prioritize all of these security vulnerabilities.
And whose job is that, basically?
Who's responsible?
So when I worked at Red Hat, we had a saying that was, if we ship it, we support it.
And we took on the burden of supporting everything we shipped.
Sometimes it was going to open source projects and helping them.
Sometimes it was just taking their fixes.
Sometimes it was doing literally all of the work.
Because open source project could be gone or dead or something.
And we don't have a good way to do any of this today.
I mean, you know, there's that graph, like, this one, where we talk about, like, what's more than a year old, right?
If I'm running something in the green and it has a security vulnerability, what happens?
I can go ask.
Maybe the developer's dead.
What do I do then?
And we don't have good answers for any of this.
But I think fundamentally, if you're an organization selling software or services to another organization, and they say, oh, you have this vulnerability in your software, and you're like, it's fine.
The developer died.
You're going to be like, what?
That's not an answer.
So I think at the end of the day, yeah.
You're on the hook for it.
And so what does that mean?
No one knows.
You, sir.
Okay.
So the question is about pinning dependencies.
And this is where you either say, I'm going to literally just run, like, NPM upgrade every morning.
I'm going to pull in everything, and we're going to YOLO it.
Or I'm going to say, okay, I've installed a bunch of dependencies.
I'm going to now tell my package manager, don't upgrade any of this stuff.
Keep it there.
And I'll upgrade it as I need to.
And there's kind of two camps.
It's always amusing to me, because I love arguing with everyone on every side of this, because there's no good answers.
But you run into problems of, if I don't upgrade things, I start falling behind, right?
We end up with, like, the NVD graph, where, like, I start falling behind, and eventually I'm so far behind, it becomes really hard to catch back up, right?
Or I'm bleeding edge.
But if I'm bleeding edge, now I've just pulled in, like, the latest version of XZ, and now I have a frickin' backdoor in my environment.
And there is no good guidance on this.
This is one of those things that we need someone to do the research to understand, what should we be doing?
Everything you read about how to, whether you should or shouldn't pin dependencies and what you should or shouldn't do, is literally made up crap.
Every single piece of that advice.
It's what I like to call things a smart person thought was a good idea.
We have zero science behind any of the advice.
I would love to see, a university or someone, do a multi-year research project where you look at, if we upgrade very fast, what happens?
If we don't upgrade at all, what happens?
If we upgrade at this cadence, what happens?
What is the burden like for developers?
How does it work?
And the other thing you get all the time is, let's say I don't upgrade for a month, and then I run NPM upgrade, now everything explodes because they just changed all the APIs.
You know what the developers do?
They're like, we didn't need to upgrade that anyway.
So you just keep it forever.
And then five years later, they've quit and moved on.
And the new guy comes in and they're like, why are we running a 70-year-old version of React?
People like, shut up.
You've got real work to do.
Don't worry about it.
And yeah, it's a tough one.
Again, there are literally no answers.
I truly hope someone does a research at some point in the near future to figure all this crap out.
Because we have to figure it out.
We don't have a choice.
This is running society.
We depend on this.
This is keeping people alive.
All right.
One more.
Yes.
Yeah.
Okay.
So that's perfect.
This is the perfect answer to end this.
We're nearly out of time anyway.
So he basically said, if there are no good answers, why don't we just let everyone do their own thing?
Okay.
And that is what we're doing today.
Now, you can say there's nothing wrong with that.
And maybe there isn't where every organization gets to decide how fast are we going to patch?
What are we going to update?
What aren't we going to update?
And look, society functions, right?
It's easy to show these graphs and be like, doom, everything's terrible.
But the reality is, I mean, how many... I logged into my bank this morning to pay some bills.
It works, right?
Nobody has come and murdered me or kidnapped me because I logged into the bank from a conference or whatever.
Society functions.
So what we're doing, it's working.
But I also have a suspicion we can do better, right?
Where you look at what we've accomplished with open source, with all these people working together, building upon the works of others.
And I do have a suspicion, if we don't do something, we're going to get crushed at some point.
Because I don't know about everyone else, but from where I stand, my job and everything I do, it truly feels like there's a crushing weight coming, right?
When we look at vulnerabilities, we look at open source dependencies, trying to keep track of all this stuff, trying to stay ahead of it.
It feels different than it used to.
And maybe it's just me.
I don't know.
But I think that is the question.
Like, what do we do?
Maybe nothing.
Nothing is an answer, right?
Every problem on the planet, nothing is usually what happens.
If we look at humans, humans are really good at doing nothing.
It's by far our best skill.
So I don't know, maybe it's nothing.
Maybe nothing changes.
And maybe I'm just some old kook showing graphs and complaining.
And in 10 years, they'll be like, all right, grandpa, it's time to go.
But I don't know.
We're going to find out.
That's the thing.
We're going to find out, right?
I'm hopeful that things work out for the best.
And I think they will.
Because I think there's a lot of smart people talking about this.
And I think there's a lot of interesting ideas out there.
I also recognize that this one's big.
Like, it's a really big problem.
All right.
I'm going to probably get the hook here in a minute.
So I'm going to go stand outside.
If anyone wants to come yell at me or tell me I'm dumb or my data is wrong.
Or I'll put a link, I'll drop a link in the Discord to my slides.
So if anyone wants graphs, they're super fun.
Come talk to me.
I love talking about this stuff.
Like, I won't shut up.
So thank you, everyone.
It was fun.