I work at indoor Labs we are a supply
chain security startup
um I am also an open source contributor
and a maintainer I contribute a few open
source supply chain security projects at
the moment uh I am one of the
maintainers of scorecard with us talk
about Brian
all right I'm Brian Russell I'm on
Google's open source security team I'm a
product manager there and I work a lot
with the scorecards project
so today
we wanted to just start with the problem
that scorecards is really setting out to
try to help solve
and if we look at this diagram that
comes courtesy of xkcd
what we start to see here
is a representation of of what modern
digital infrastructure looks like
if you peel back the veneer of of almost
anyone's you know end product you start
to see
a real hodgepodge of different shapes
and sizes of the dependencies that that
an organization has taken on over the
years or taken on just by adding one
dependency and getting that cascading
set of dependencies behind it
and you know we we don't have a lot of
insight into exactly how strong or or
weak some of these dependencies are and
how well maintained they are uh you know
there's a there's a lot of different
concerns that could come out of this but
when it comes specifically to security
you know we we don't have a lot of
insight and we'd like to have some uh we
want to know on top of how big and small
these These dependencies are we actually
want to know are these safe or not
because even the smallest one uh that we
have is a concern to us
so imagine you know if we could start
looking at this and start getting a
little bit of extra information about
some of these dependencies imagine if we
had numbers like these that gave us a
sense of this dependency is in a really
good State its maintainers care a lot
about security
they are really trying their best to
make sure they follow just good
principles and good development
practices versus here's some projects
that that actually aren't doing that at
all and then you know here's some
projects that maybe they're trying but
they've still got a little ways to go
just having that sort of insight starts
opening up the world to us in terms of
what decisions we want to make what we
should do with these dependencies how we
can take on
um kind of different different pieces
so enter in openssf scorecards we are
seeking to add that extra information to
all of these open source projects and
dependencies that that are being taken
on this is an example of what what one
looks like some of you are likely
familiar with it and some of you might
not be I'll spend it just a little bit
of time explaining what we're looking at
what we see here is an overall score and
we have a number of different checks
that scorecard is looking at
to determine whether or not a project is
being maintained in a way that's
considered secure
so without going through each and every
check line by line I did just want to
take a couple of examples and take a
look at what these really mean so if we
look at the first one you know we have a
check for binary artifacts that is you
know is this project storing binaries
and therefore obfuscated source code in
the project or can we see all of the
source code that's that's in a
particular project
with each check if you look at the table
that's on the project page and I hope
you all visit it after this talk we go
through each and every check
that we provide in scorecards and
in there you'll see you know what it is
a little bit more information about it a
risk level in terms of is this a serious
risk if if a project is not adhering to
it
and then the amount of access that's
required which I'll get to in just a
second in terms of how a scorecard
gathering this information it needs
certain levels of access to determine it
and then just any other notes that that
might be beneficial to know when it
comes to those checks
uh so that brings me to well how is how
is the scorecard generated it's not
people manually filling out surveys and
trying to score themselves this is all
done programmatically we're using
github's apis uh to basically bring that
data in and to score it
and
we can do that in a couple of different
ways right now we have a GitHub action
that if you're an individual project
that is interested in maintaining a
scorecard and using that to help inform
security decisions that you're trying to
make
you can run that as just part of every
other GitHub action check that you would
run
there's also a way to just run this as a
command line tool you can run it locally
it will still end up calling those
GitHub apis to gather that data so
there's not manual assertions that are
feeding into it it's all done
programmatically
so just to give folks a sense of where
this project is today
um
it lives within the open ssf and we've
had over a thousand installs of just the
GitHub action
we're seeing an increased number of
people that are watching the project
working at starring it our goal right
now is to continue to put those Trends
on the on the positive
and just keep the momentum going that
we've started to see
we have had some some mentions and some
adoption also at the organizational
level
um Eclipse has been using this as a tool
to start looking at how different
projects can
make some security improvements uh Envoy
did some piloting and has been giving
feedback working with the openssf to
improve scorecards
and sonotype included
scorecards in its annual state of
software supply chain report and I'll
dig into what they said in just a second
now since this is a cncf event we
thought we would mention how the cncf is
using scorecards right now and thanks to
Chris for for giving us just a quick
quote on how things are being used today
you know scorecards is being used as a
tool
to look at different cncf projects to
help improve that security posture and
it's being used in at least one
competition as just an indicator of of
good security practices
so I encourage you to read sonotypes
annual state of software supply chain
report in it there's a specific section
looking at what are good indicators of a
Project's likelihood of being vulnerable
over time
when scorecards was included in this
study it was shown to have
good potential in terms of indicating
whether or not a project if following
these best practices was going to incur
some sort of vulnerabilities over time
sonotype went further
and broke down scorecards into a check
by check basis which you can see there's
some some correlation between certain
checks provide better indicators than
others
but again I encourage you just to take a
look at the report it goes into a lot
more detail than we have time to today
so we'll jump into a demo in just a
second but
I wanted to give a sense of kind of
what's existing what's new in terms of
major features for scorecard
we originally uh
launched and then added a GitHub action
that does exactly what I mentioned
before it's checking projects scores
really on a regular basis as many times
as you've set it up to trigger for my
GitHub action perspective
and we've also created basically a
weekly scanned Cron job of gathering
scores for the top million projects that
was originally bootstrapped on some
infrastructure that is undergoing some
improvements at the moment so some of
those scores are stale as of December
but we're working on launching an
improvement that will basically get
those updated
any scores that are running the GitHub
action are also pushing their data into
that same repo so those are all up to
date so if you're using scorecard
as an action uh you can you can be
assured that the API that we're about to
talk about is is reflecting the most
up-to-date data
so without further Ado I'll hand this
over to Naveen to talk about what the
API looks like
thanks Brian
um
why did we build scorecards API
um like Brian mentioned
um we we've been running scorecard for a
million repositories for a little more
than a year and all of this data is
available free for everyone using
bigquery and we spoke to a few customers
and not everybody wanted to use bigquery
they wanted a standard HTTP endpoint
which they can use to get this data
so that was the motivation to use the
build the scorecard API
standard HTTP API with rest endpoint you
can hit the API at API dot security
scorecards.dev
and we also made sure that the API is
predictable like you can see here
because it is a cncf conference we
tagged in kubernetes kubernetes so you
can take the detail and and replace that
with whatever project you want as long
as the data is available like like Brian
mentioned if you have a million 1.2
million repository that we scan and you
should be able to get that
and second and another critical thing we
don't have any tokens you don't need any
tokens with this API this is served as a
public good instance for people to come
and get this data
um another another thing
we are not throttling any of the API
requests so please don't DDOS this
uh
like I said um this API has a get and a
post
um but I'm going to talk about get at
the moment
um this will be if you go hit this API
for whatever project is it will be it'll
give out a Json document similar to what
Brian showed as a scores being in a
table format but I'll give it a Json
document so anybody should be able to
utilize this
we launched this API in last fall we are
seeing about 85 000 to 100 000 requests
every day we see this trend only going
up so that's the little bit of update
people are adopting
that goes on to the next one who's using
the API
to be honest we don't know who's using
the API uh if any of you are using the
API please come and update or read me uh
there was one person who came and asked
us who was using uh like the API weight
limit and we got to know from that uh
looking at that person that person is
from a university in in Germany who's
trying to use the API
okay
all of this is great but how can how can
this API be useful what can it do
what you're seeing right now is
um the code review for code review data
over a span of year of top thousand
critical projects
how do we come up with top thousand
um we didn't come up with that there's
another open openss project called
criticality score and it and that
project's primary motive is to take
I could be wrong 100 000 projects and
decide
and give it a score between zero to one
and the algorithm for that was
um was based on by Rob Pike's algorithm
and we used that data
we want to shout out to them that the
data is available free so anybody can go
pull the data
um so we use that data
use scorecards API to say now pull the
data and show us how the top thousand
clinical projects are doing looks like
most of them are doing well
there are few places that require need
some help
um another check that we said is
are they are they able to do binary
artifacts how many of them have binary
artifacts within the top thousand
critical projects looks like most of
them don't have probably TNT but of them
have
um
why are we doing this
this is essentially to go back and help
the critical projects and tell them how
they can fix and there are a lot of
people from
within the community of helping doing
this
also
if we can use for the open source
projects
all of us can use for our own Enterprise
Solutions any other things that we're
doing this API is available for that
okay
enough with all dark show me the code
okay
um so I decided it's not nice if we
don't show what the code is all about so
here's like I mentioned
here is the piece of code this is
written go this is a sample code
that I was shared with where this code
is hosted this code what you're seeing
right now is this is written go I'm
familiar with go scorecard has written
go so but you should be able to use any
modern programming language to go query
this data so what you're seeing here is
this is the meat of the code that goes
and gets that hits that API like I
promised does not have any tokens any of
that goes and hits that API gets a data
and unmarshalls to a struct that's the
that's the code here
and if I go on to the top over here I
take the Thousand critical projects I
use some go magic to concretely go hit
the API and it should be able to get the
data I'm not going to run this I'm not
I'm a little scared of the demo Gods I
it might not work so I'm not going to
run this moment but that's it
uh the code is available in this
repository it's my personal repository
uh it's not part of up it's not an
opennesses of anything anybody should be
able to code for clone and use this
let's talk about the usefulness
yeah so we we thought we would just kind
of act out a scenario here uh in which
Naveen is going to be a contributor to
an open source project and I am a
maintainer and I care quite a bit about
security so uh I'm gonna have a little
chat
Brian I want to submit a PR with the new
divinity
so I think that's a great idea uh but
I'm really concerned about if this
dependency is healthy can you talk about
that
would score god be helpful Brian
uh
uh I'm not sure what that is
have you checked out the scorecard's
repo
well uh actually now that I have this is
really great in fact I think I'm going
to use this API for checking all my
future dependencies
in fact I think I'm going to go further
I'm going to use this in my CI CD
workflow as well
it's just so easy to use
and why doesn't everyone use this at
this point
so these are these are the kinds of
questions hopefully conversations that
we'll uh we'll be seeing a scorecard
so
My Demo uh
is going to be equally unambitious I am
going to just walk through a quick run
through of how easy it is to add open
ssf scorecard badges so here's an
example of what you can expect to see
it's a pretty standard looking badge on
a GitHub readme when it comes to
actually adding this in
kind of narrate What's Happening Here
you can just go to the openssf
scorecards GitHub page there's a
specific section in it that talks about
how to add Badges and if you go to that
section there's just a snippet of code
that you can grab you can copy it you
can go into your readme of a project
that you have
you can paste it in
and then you just have to tweak it a
little bit take the name of the project
that you want the badge to represent put
it there
and then
once you have those changes made it's a
pretty standard change to to the readme
just commit it announce that you're
adding a scorecard badge and as soon as
it's pushed you should see the badge
generated right away the one caveat here
is that you do need to have data behind
that badge in order to populate it so if
you're already a large project and
you're one of those 1.2 million that
we're scanning on a regular basis you
can expect to see that badge populate
itself if you're not or you just want to
make sure to guarantee it you can
install that GitHub action and at that
point you'll have that data guaranteed
to be in the repo itself that we're
pulling from
and it will populate that way
so we don't need to see that again
but uh Naveen if you want to talk about
just how to get involved yeah
um we have a bi-weekly you should be
able to hit the
the QR code we have a bi-weekly meeting
uh please come and hang out we are
looking for contributors or if you're
even using the project please come and
let us know how we can improve
um
we are all active in the slack and the
issues we constantly are all the
maintainers are pretty active in helping
grow this project
so I think at this point we are way
ahead of schedule so we we have plenty
of time for questions
um if people want to just ask uh we can
just kind of try to ask me anything
right now
yes
um
so that's a fantastic question uh and
actually one that's I would say talked
about a fair amount in the project
itself
honestly both of those cases I see is
very viable so
if you're in that case of trying to take
on a new dependency we think it's useful
information to have so that you can make
a more informed decision about is this
really a dependency that I want to take
in you know or is this something that
that maybe needs a little bit more work
or maybe has a few follow-up questions
before you'd want to take it on
if you are the maintainer of a project
or you're looking to contribute to one I
think it's a great starting point to see
what are some of the key security
changes that could be made that would be
really impactful for a project so we've
seen that in a few different places too
where
anyone who's looking at trying to just
improve how they're doing software
supply chain security they'll generate a
scorecard as a starting point and then
say what do we need to do to bump up
this score or celebrate that they're
doing a really good job already and
hopefully put up a badge
we're getting a 10 on that and the whole
score is hard scorecard does not have a
10 so just giving some heads up
Olympic style scoring
any other questions
yes
but are you talking of some sort to have
access
yes
um that's a configurable you don't you
don't say anything you can say please
don't publish results and the land
publish anything
so
um second thing if we find issues We're
Not Gonna
the moment your score is bad we're not
going to put it out open for everybody
to come and attack you
we're going to write to your security
dashboard so you can go as a maintainer
or administrator of the project to go
address them so we take that we take
that we take care of that just wanted to
add that to that
so yeah should be flexible and fit
whatever kind of data you do or don't
want to share
so great question at this point it's
still something that we're working on
we we do want scorecard to be flexible
in that it can run not just on GitHub
projects but you know any open source
project or dependency that someone would
you know either like to be making
improvements on or be considering
whether or not to pull into their own
project
there are some things in the work uh
there's get lab work going on yes
um and to add to that there are some
take example if you have a good boss
tree uh on gitlab or somewhere else
you can run scorecard but it'll not be
able to get all the checks it will be
able to get you a contrive amount of
checks with score only pertain to that
so it'll be able to say hey are you
pinning your dependencies
um do you have this XYZ do you have
binary artifacts we don't because it's
irrespective whether it's in GitHub or
gitlab or anywhere it is so there are
something that it will run so to answer
your question on that we should be able
to run them but not get comprehensive
results like what you get and get up
right now
you're welcome
very good question we have all of this
data available in our bigquery it's
available free so you should be able to
go pull this data to plot out to say
repository was here
are they going up or coming down
up okay to answer your question we ran
we run weekly scans scanning
1.2 million repositories with the amount
of tokens GitHub tokens we have takes us
roughly about four or five days
um to do that so we ran if I'm not wrong
last year 52 times so you should be able
to take
whatever the current window is yes
you should be able to take that and plot
it
absolutely yeah right so um
on that front we are having plans on
release working on a feature Within
if you're going to add a new dependency
what if your pull request gets a
scorecard result on that to say
and you decide I want of these 17 checks
I want four checks I just want these
four checks and I don't want it to go
below eight below seven whatever number
you want to configure
so be it it's on our works we are
working on that
so to answer your question on that so
that you don't have to fly blind when
you're taking new dependency
yeah and I think you know obviously the
more granular we can get the better off
we'll be you start bootstrapping the
infrastructure to get that initial set
of scores I think
you know generally not much changes on a
day by day so much as a week by week but
but
we made a decision a year ago somebody
goes in one
to talk about these things
yeah yeah you can at least get that to
that date range yeah
yeah that's a great use case
yes
um so so what are you trying to say is
hey from the cve I want to go back and
look at that underscore
okay
right now great great feedback but at
the moment we don't have that
there are
other projects which are working trying
to integrate scorecard into that
um but we can take offline talk about
that
yeah essentially you'd need some
third-party service to do that lookup
right now that's that's another open
source project which is trying to build
that
give me the questions
okay going once going twice
all right maybe extended coffee break
thank you
[Applause]