so please keep in mind we'll have a
short q a session after the talk and we
have microphones lined up there and
there so if a question please go to the
microphone
there is also a survey in the app so do
the speakers a favor and fill out the
survey in the official first app after
the talk and if you have any devices on
you that somehow make noises please be
so kind and turn them off now
and now for the first talk of the
afternoon here is uh crop he's the
director of security communications at
intel product assurance and security and
he's also active in the first pset
services framework and the lead of the
open source security foundation
and open source is also a big topic of
his talk he has a very cool presentation
design he has a lot of cats so everyone
please give him a wall welcome and the
stage is yours thank you natalie
hey everybody
i'm krobe
uh i do stuff we'll see a little slide
about me in a minute
uh while we're
thinking about this presentation this
topic of open source i wanted to pull
the room a little bit
so who here in the room raise your hand
if you represent a piecert a product
security team
okay lower your hands who here
represents a corporate or computer
security team a c-cert
okay academic
other
so
pretty even mix of c-certs and p-certs
i am here representing the open source
ecosystem the maintainer persona
we'll talk about a couple different
personas here in this room piecerts
would be typically a supplier persona
and sea certs you all get to be probably
the consumer persona because you
probably aren't doing the work in
upstream but you definitely benefit
through products and services
i've had the great fortune to do open
source security for under a decade
pretty close
and i've done a bunch of nonsense you
can see up here
while making this presentation
i was think trying to find a theme i was
going to hand illustrate each slide and
i said man that's a lot of work
and so i'm i had a crappy powerpoint
presentation deck
and then my cat walked in and started
yelling at me demanding to be fed
and that really inspired me because and
ultimately made the topic open source
doesn't care about you in general the
cat doesn't care too much about you
unless it needs fed or some of its other
needs need attended but you absolutely
should care about it
we're going to talk about kind of
understanding your open source problem
we're going to talk about how the open
source ecosystem really works and how
there's some differences with closed
source like if you get software from a
commercial vendor that's typically
going to be closed source software and
they function a little differently
we're going to talk about the open
source threat model and then we're going
to talk about some solutions because i'm
not just going to complain about stuff i
actually i'm going to help you solve the
problem
so in the room
how many people here know that they use
open source at work
i see some hands not raised
you're
you're fooling yourselves all right
how many people think you know all the
open source you use
right answer
the answer is not everybody knows
everything they use
who here gets an s-bomb from their
suppliers
wicked
zero people alan friedman is out in the
hallway crying right now
so obviously if you don't have an s bomb
you can't use it to try together
uh vulnerability information so you i'm
not able to solve that problem at this
point without having the s-bomb
so let's talk about open source
uh
i worked at red hat for a substantial
part of my career
and i always thought about the
engineering teams as kind of hippie
cowboys
they're off
on their own out on the range doing
their own thing
a little strange but generally they mean
well and when you think about developers
developers have a lot of different
motivations there's a lot of different
ways to look and you know
perceive
the open source developer
so we'll talk about there are many
different unique goals of open source
projects
not there is no monolithic the open
source everybody functions a little
differently
some people get into open source and
this is the key reason most people get
into open source is they want need to
solve a problem and they solve that
problem through creative software but
also you have people doing academic
projects at universities they'll get
their mass you know bachelor's master's
doctorate then software engineering so
they'll have projects on open source
many people do open source because they
want to be build and be part of a
community
it's amazing the
the sense of purpose and well-being you
get in an active nurturing community
a lot of people do open source because
they want to have fun or they're
teaching themselves some new skills
some people want recognition from their
peers i wrote the absolute greatest
encryption algorithm ever look at me i'm
so amazing
other people feel that they because they
use open source they feel that they
should contribute back to that open
source
some others just generally agree with
the altruistic mission of free and open
source software software that's created
for the public good
but then you have some people that
contribute to that because it's going to
help their career i'm going to get a
promotion because i have you know x many
projects in github and so many stars and
whatnot
and then the very last category of why
people can
contribute to open source is some people
get paid by their company to do this
work
and that is the reality there's a lot of
different complex motivations some
people may have
one or two of these motivations some
could have all
but the open source is not very cut and
dry not very black and white
but one thing if you take nothing else
away from this talk aside from cats are
great
is a substantial amount of commercial
software today uses and is based off of
open source
uh depending on the day of the week and
the study that you read between 80 and
upwards of 91
of all commercial software has
major components that are free and open
source
and
uh
this xkcd
cartoon very humorously illustrates
humorous uh the fact that there's a lot
of stuff in your environment and then
there's this one little project that
hasn't been touched since 2003 that you
probably don't know about
so you know my first controversial
statement here you can read it on the
screen
but by using open source software you
are knowingly benefiting from someone
else's work and not paying for it and
that's okay open source again is
generally created for the common good or
to solve a developer's personal problem
but the developer's personal problem and
this altruistic nature
does not service your needs you did not
pay for this there is no contract
between you and the developer in fact if
you read most almost all open source
licenses they state as is
so there is no obligation from the
developer to do anything to their
downstream community
but you know your business
and that's that's fine that's how the
ecosystem works
but you need to understand that these
development these upstream goals don't
necessarily align with your business
objectives and the fact that
they have a release schedule and you
have a different release schedule and
they're out of sync it's not really the
upstream maintainers problem that's your
problem to try to understand and under
make compensating controls to deal with
that
so what does open source do for you i
have a couple things here to tell you
what open source
doesn't do for you
first and foremost open source never
gave you a support contract
upstream maintainers aren't in the
business of supporting your business or
your software
you may pay a commercial vendor for open
source support and that's cool there's a
contract there you have some rights and
they have some obligations there but
upstream does not have that obligation
open source doesn't care that you have
dependency conflicts or you have a
critical system you can't take down to
patch or you can't update to the next
version because of dependency conflicts
that's not there they are there to
create their software and move along
solve their problems and you are able to
benefit from that
open source doesn't care that you gave
your customers a 10-year support
contract
generally
open source is focused on n minus one
versions so they will support the latest
version and the version right behind
that that came out right before it and
that's it that's your expectation from
open source most open source projects
they don't do this long-term support
there is some other commercial entity
that if you need this they can step in
and try to fill that gap for you or you
need to take that responsibility on
yourself if you need to support a really
long life product
open source also doesn't care that you
can't upgrade the latest version because
that is almost always the fix in open
source upgrade to the next version
please
because i'm not paid to support this i'm
going to support the latest thing so
again it's up to you to figure out how
you're going to protect yourself and
your customers your constituents
and finally open source doesn't know and
open source doesn't care how you cobbled
together your solution in your
environment on your equipment with all
this other software
they don't care they're not there to
support that that is on you your system
integrator or your vendors that are
supporting you so don't ask open source
to help you troubleshoot your amazon
cloud instance of nonsense because they
they don't do that that's up to you to
figure out how to do that
so how open source really works
typically a developer has a a problem
they're trying to solve so they'll
lovingly
develop this awesome thing then they'll
share it with the world you know again
sometimes just for recognition you know
look at me i did this amazing thing or
sometimes it's i want to help a
community or they're just pushing it out
there because they don't they don't want
recognition but they want to generally
help the world
then you know that developer after
they've shared this pride and joy with
the universe they kind of wander off and
do other stuff but then other people
discover this amazing gift that open
source person gave the world they go
this is great
this is this solves my problem except i
need blah blah blah and blah i need all
these additional functions or these
additional features or i need it to
behave differently or i need the screen
to be green not blue
and
the great way how open source should
work is as you find these
personal deficiencies with that software
you're supposed to contribute and give
back
hey
you submit a pr or you open an issue
saying i found this problem could be
could you change the behavior to do from
this to that
or better yet best of all here's a patch
to fix this bug i saw on your product
your your software could you please
integrate that into the main branch
that's how open source is supposed to
work
this is how many suppliers treat open
source they see this beautiful project
out there on the internet they don't
have to pay anything they incorporate it
into their products and this is what you
get
so i'll let you
admire that while i take a sip of water
many commercial suppliers do not
participate in the communities of
software that they use
and they all
depending rarely
contribute patches back or any kind of
resources back to help that community
there's a lot of different ways you can
engage with communities to support them
beyond
money
so unfortunately being that open source
is out there and it's beautiful and
delightful
it also can become interesting to
attackers
the all of our source code is fully
publicly viewable which is awesome so
you can take that software
look at it understand exactly what it
does
so can the bad guys unfortunately
it is typically most open source
communities those that aren't like
single contributors tend to be globally
distributed where patches are welcome no
matter what part of the world is no
matter what your
sexual orientation is what nation you
live in we will take patches from
anywhere and there are generally some
rules vetting getting things into the
main branch but open source development
is geographically distributed they are
rarely a
small group of people in one spot that
does this so that's a nice
avenue that attackers could potentially
infiltrate that community because there
isn't these tight controls and tight
circle of trust between these people
because many of the transactions that
open source are done anonymously
uh there are different economic
incentives and feedback loops um so when
you're thinking about enterprise
software attackers
generally want to get in there because
they want to
install ransomware and they want to
threaten you or they want to siphon off
corporate secrets
within open source the motivations for
the attackers are can be very different
and also can be a lot broader think of
things like the solarwinds attack where
a compromised package was able to help
unfortunately get integrated into many
supply chains and that the scope of a
potential attack if you're doing kind of
a poisoning the well type thing
is potentially broader than just
attacking
large bank or government office
and it draws different types of threat
actors
what is almost universally applicable is
that open source projects have a lack of
have a scarcity of resources they might
have limited
development environments they might have
limited peer developers they might not
have access to expensive corporate tools
again because a lot of them don't get
directly paid to do that thing
so that
brings in some problems if you're
looking at software from a more
traditional commercial enterprise they
typically do have you know very well
managed developer environments they have
things like sdlc or they have scanning
and other protections whereas an open
source because it's all out in the
public they don't necessarily have all
these some of them do some some
organizations some projects do sdl very
well and have very tightly controlled ci
cd infrastructure
others don't
and unfortunately and that that kind of
draws into the very last point there's a
real lack of consistency you can't look
at one open source project and directly
compare it to another because there's a
lot of different factors they could have
different skilled developers on each
project they could have different access
to testing infrastructure or tooling
and that is you know the unfortunate
reality of open source is that not
everybody's equal not everyone has
access to all these amazing platforms
and amazing tools
so
how security vulnerabilities get
addressed in different communities and
it really depends on the type of
community you're looking at
so if you have
so if you're a supplier you're
potentially downloading thousands of
open source packages if you're a
consumer maybe you are
a build-it shop maybe you're a bank and
you write your own um
online banking platform so you will
actively download and participate in
open source
but there's a lot of different types of
communities within open source the first
model and this is unfortunately
depending on some language ecosystems
a
peer of mine josh bressers is doing
analysis of maven and npm and the go
communities and the overwhelming reality
in those communities is that this single
maintainer
there's one maintainer that does that
software and that's it and these are
millions of packages and when you have
an individual contributor style solution
you're looking at one to two people this
tends to be a passion project
uh you're going to have little to no
process because these folks are doing it
for fun or they have to try to so they
have a quick fix and they throw it out
there to share
there's likely
no formal or secure means of sharing
security related information normally
they might have an email address
they're not going to have access to a
secure jira or bug tracking system
they're not going to use private gitlab
github issues
there's definitely no slas on the review
it's kind of buyer beware you download
the software there's probably not going
to be a lot of expectation on how you
can get support
and
they have minimal if any documented
devsecops practices
if you're lucky you get a man page or a
readme file that kind of describes
things and that's probably as good as
you're going to get in some of these
smaller projects and this is a large
percentage of open source software
that's used today
and especially in things like maven and
go and some of the newer more modern
languages
in the middle stands kind of project or
community focused efforts so
this particular piece of software has
been able to capture the interest of
enough other developers that you have a
group of people doing this you could
have um
half dozen a dozen 20 or so folks
actively contributing to this project
they will have basic processes because
when you're trying to integrate commits
from different developers you just have
to have that process otherwise you get
collisions and everyone's stomping on
each other
they probably will have some type of
page that lists their contacts or like
what the rules of that community is how
to get a hold of them
and they pro they definitely will have
some type of bug review process so as
defects or security vulnerabilities
reported they'll have a way to triage
that
uh they'll probably have
that guy
or you know sally our devops gal she
manages the ci system they'll have
typically a handful of people that are
focused on some type of
formalized devsec ops practices
and uh they'll definitely will have
documentation because once you have more
than two people you really need to have
this documentation
and it helps so they will share that
with their broader ecosystem
you still should accept best effort
coordination expect best effort
coordination
and they might have cvd practices with
some key either suppliers maybe they
depend on something upstream so they've
made an arrangement to get
pre-notification for example or they may
understand that we have these four or
five downstream projects that depend on
us so we'll notify them but this is not
generally formalized or mature in a lot
of ways
and then the last type of open source
project you can have is like a
foundation level project these are
things like kernel
apache security kubernetes openstack
these large very mature programs with
these types of open source projects
you're going to have a dedicated team of
security people that typically oversee
that project there's going to be
oftentimes many different related
projects i can't remember
the kubernetes team told me last week
how many projects they had underneath
their umbrella and it was in the
hundreds of they're kind of managed
under this umbrella so that's very big
requires a lot of coordination
high maturity style project there
so with that you're going to have
well-developed documentation in process
you're going to have
well-communicated and well-documented
communication channels this is how this
is our security policy this is how we
want to intake issues these are slas
people can expect for a response and
generally kind of what our patching
methodology is these larger communities
will have all this documented
you'll have dedicated people performing
compose build deployment and security
functions within that project they're
going to have documented release
timelines and they very typically will
have a well thought out cvd process and
will be well integrated with other
communities they depend on or supply
so again there is no one the open source
you'll see any mixture of this so no one
kind of puts a sign up saying you know
we're foundation level or we're an ic
style project
generically oh i have a laser
oh it's a green laser too nice um
generically open source projects will
follow a very simple
ingestion triage and disposition process
some type of somehow some way a security
flaw is discovered somebody reports it
they're going to evaluate it
they're going to do something and it's
generally fixed or won't fix if they fix
it great the patch is thrown out in the
world and the developer moves on with
their life
this whole second half of the screen
second two thirds this is your
downstream suppliers
and what they have to do
to support that change upstream
and the worst situation
is when you get into a won't fix
scenario with upstream and your
commercial supplier has some type of
long life support plan in place for you
so that becomes an interesting little
exercise in negotiation
upstream
not upstream
so kind of comparing the two when you're
looking at closed versus open source
typically closed source will support is
supported by some corporate entity or
entities and
updates will typically only come for
that one supplier versus open source
where it's supported by the community
update could come from anywhere anyone
that has the interest and ability to
push a patch
potentially can help contribute to fix a
solution
for closed source they're typically
governed by stakeholders regulations
and they have customer demands so
they're going to be reactive to
customers so if you are supplied open
source software by a commercial software
vendor you have ability to influence
their roadmap influence how then
when and why they fix things
versus open source uh there generally is
little or no governance and they they do
open the open source does not have
regulatory oversight they are not
beholden to pci
um gdpr yes um but the generally the
typical regulation things are not
compelling to open source because again
they're not a commercial entity and
generally as i mentioned up here with
number one a lot of times
when you're dealing with people in open
source you don't even get their real
names we use hacker handles or
pseudonyms
very typical in this community so if you
are an organization that
some
nations like to state we will we only
like software created in x or y region
from people from a or b country
you don't have that luxury within open
source just because of that the open
global nature of it
looking at closed source
they generally have a process around
security reports
and communication is not always done in
public
oftentimes with commercial software
you'll see embargoes where there'll be
some type of period where that supplier
needs to get all their ducks in a row
and release an update
versus
for better or worse open source
almost always the defect tracking
process is open
it's a very mature very large project
that has a private
disclosure process
and
almost
to a fault all open source developers
are when the fix is ready it goes out
the door i'm pushing it immediately
because they're focused on their
downstream in this way they want to get
that fix out to their consumers whereas
commercial entities have a lot of other
uh constraints on them like you know
shareholder value and they can't
necessarily stop everything to release a
patch
this afternoon
when you're looking at
closed source generally those releases
are tied back to some predetermined
release schedule you have a quarterly
release cadence you have patch tuesday
whereas open source is again you know
screw it when it's done it's done it's
going out the door great good luck we
hope you all are happy with the software
around and then five typically with a
commercial vendor they'll try to
orchestrate updates especially if
there's any kind of interdependencies
between vendors
and they're
looking at the open source counterparts
there typically isn't a lot of that
understanding of
who ingests them and who we need to
notify so you're it's going to be much
less coordinated 10 minutes no way
oh man
all right here is a non-exhaustive list
of threats that could happen within an
open source ecosystem
very exciting
because on the internet no one knows
you're a cat
i'll tell you the story behind that in
the future uh so
when you're thinking about how a lot of
commercial suppliers support open source
it's not really great they don't really
do a ton of effort to help improve that
maintainers
ability to execute and deliver software
more quickly
so how can suppliers work better with
open source first and foremost
it should be the obligation of every
commercial supplier to understand every
piece of software that is delivered in
an offering they supply to someone that
pays money for it
that should be the first thing that any
well-intended company should do is
understanding what i'm ingesting
where i got it from what the rules of
that community are if i need to request
a change
you also need to understand how issues
are worked upstream
some issue some i said some communities
are we only have a public defect tracker
so if a researcher submits a security
vulnerability the world can see it which
as a defender you know the clock starts
ticking right there
others are more minimal more mature more
understanding of their downstores their
downstream ecosystem so they will have
practices that allow them to take some
issues in privately kind of triage it
beforehand before working on a release
suppliers need to understand
how patches are released by that project
is it pushed out on github is it out on
some random aws file system they
understand how that community works
where they're putting their software and
how they can get notified of it
a lot of times advisories are not
something that upstream communities do
so the
downstream consumers like suppliers need
to understand how they get notified and
how they need to notify their consumers
so let's talk really quickly about this
thing
who here has heard of log4j
so my perspective
the upstream apache team the security
team while not perfect
did an admirable job they did they
handled this vulnerability to the best
of their ability under the constraints
they were given and they
actually executed pretty well
so on december 10th there was a cve
released
very bad oh cvss 10. ooh scary so the
team within uh
the embargo broke
and they actually
the information got leaked by a vendor
that scraped github commits so somebody
started tweeting there's gonna be a big
apache event
so
six hours from that embargo leak the
apache community had the bolton out had
patches ready and it notified their main
downstreams
unfortunately
the first fix wasn't complete which is
uh
can be common especially when you're
working under pressure and the fact that
things went about three or four days
early it was an incomplete fix
so
just four days later the apache team
released a second fix that closed up
that little hole that wasn't addressed
and you'll note
that what wasn't addressed was
non-default configurations so someone
altered the behavior of the software the
developer intended to make it work a
different way and but they generously
released a fix for that
then a couple days later oh there was
another issue so while they were
investigating or mediating this they
released they found more issues as part
of this response and they actually at
the end of the day for this particular
problem there were the two the main
issue the
incomplete fix and then two secondary
cves that were discovered
so to think about this
the reason that they leaked and they
went early is because they had evidence
of active exploit in the wild so they
did for the best interest of their
community released early and
unfortunately it was confusing not
everybody got the message in time
but they were still able to get
everything out within six hours which i
don't know many organizations have that
ability to spit a patch out that quickly
the security team published it within
hours yay
first patch wasn't sufficient
boo
uh orgs working on testing and rolling
out the initial uh integration and
testing of the patches had some rework
to do that's an unavoidable
uh after effect
if you are working with open source
software again the goal is to move
quickly as they can
so they move quickly and sometimes it
requires some rework downstream
and then what was really fun is some
vendors were supplying log4j version one
which was publicly deprecated five years
ago
and then they sent demand letters to the
log4j maintainers demanding them what
are they going to do to fix this
software that they had publicly said
five years ago we're not doing anything
with anymore
not cool
um
so the hope hopefully if you got nothing
else from log4j i'm gonna do questions
here in a minute if you take nothing
else from this you need to ask questions
of your vendors you need to understand
what software they're supplying you and
some of the practices of how things are
done
so i will my slide should be available
after the fact i'll make sure that they
post it up there but i have some
suggestions and first and foremost the
best way you can fix this problem is by
contributing to software that you use
and contributing could be
having engineers participate in the
project fix or bug or two offer to help
do documentation help them with
advisories uh donate tooling so that
they can do this testing that downstream
consumers want
um
all right and there is somebody doing
something about this so the other group
i work a lot with is the open source
security
software foundation
say that fast five times um in response
to the executive order released by the
biden administration last fall the open
source uh
secure software foundation has released
a 10-point plan a mobilization plan on
how the foundation a group of volunteers
will be working on fixing some of these
problems
like
education risk assessment digital
signatures of software getting software
rewritten into memory safe languages
working on sharing data doing code
audits better scanning tools
i get to create an open source cert
yay me
and then you know s-bombs and improve
supply chain security so the industry is
working on this and we need your help if
you are interested in any of these
topics don't hesitate to reach out to
the foundation show up to a working
group meeting it's fully publicly
available everyone's welcome and then as
we're developing uh solutions like
academic scholarships and
s-bombs and pcert you'll be able to
participate and help shape those
solutions so by engaging open source the
right way
it can be pretty cool
so i think we have like three minutes
for questions
so what questions might have i inspired
the group
to have
did you learn a little
so if you have any questions please line
up about microphones and at first give
chris a wrong applause for his awesome
talk
hello uh i'm dog i'm not a programmer so
might be a stupid question
is there any system by which
one entity can in a way sign or approve
a piece of code
published by b entity
so that i can look at the code and i can
see oh entity a which i trust has
approved that code
so there is a project that the
foundation supports called sig store
that is all around
digital
identity of developers and code signing
and so you can get that pedigree in
providence and it's all kind of a
blockchain style repository so that once
that software is signed you can always
check the hash with the main repository
to be able to validate that so yes there
are tools that are coming and that are
um actually they just announced that
kubernetes is moving over to using sig
store so yes there are ways you can do
that
thanks you're welcome
is there a
a listing of open source projects in
danger
should you say
that are focusable undersupported or
where
where there would be a need for it which
are widely used
and where there would be actually a need
for
more support
on that
and what is the your opinions on on the
best ways of providing
financial support in into the general
open source security ecosystem is it
through that foundation or are there
other
projects on this area so to your first
question there is no
list of
known at-risk projects but there is the
converse
we partnered the linux foundation
partnered with harvard university and
they did the harvard census ii so they
ranked the top
117
most used most critical projects and
that's what openssf is focusing in on to
try to go in and do
one-on-one developer code audits with
these people trying to get them access
to tools and whatnot so there isn't an
at-risk list but there is a list of we
know that these pieces of software are
used a lot and we're going to focus in
on it and then we're also focused on the
next 10 000 projects where we have a
project called alpha that's directly
engaging the top 1 115 and then omega is
trying to address the 10 000 below with
what we learn on how to engage those
projects kind of filtering those
practices back to the larger ecosystem
and openssf is one foundation that you
potentially could partner with and you
know again contributing doesn't
necessarily need to be money
but it could be that helps us get
contractors or helps us buy tools or get
moving infrastructure from
i worked with a guy that had a
array of raspberry pies underneath his
desk in the netherlands we were able to
help get him funding to be able to
convert that into an aws kubernetes
cluster so there are a lot of different
ways so open ssf is one foundation but
there's a lot of great foundations the
cncf uh the confidential compute
consortium
a lot of different ways but if you have
specific questions don't hesitate to
come ask me afterwards
good we have time for one more question
did you like the cats
all right well thank you i appreciate it