welcome to
introducing onefuzz i'm justin campbell
i'm with the microsoft windows
security team here with mike walker uh
the leaders
lead responsible for the leader of the
group responsible for building one fuzz
uh today we're gonna introduce uh one
fuzz itself
talk about how windows is leveraging it
to handle the
security of the scale of windows and
also concurrently be releasing it open
source
so uh spoilers we're going to kind of
jump to the end of the presentation at
the beginning
and describe the vision that we had at
the beginning of the
project which is that developers could
introduce a change
to their code base be that windows or
anything else at microsoft
wait for the build nightly validation
come in the next morning to an
automatically delivered message
with all the information uh necessary
to reproduce and fix a bug that had been
automatically detected
reproduced deduplicated and with uh
all the technical details that we could
possibly gather automatically
and the goal was to get a developer up
and running in one day
onboarded into the windows cr system uh
to keep this
to make it as simple as possible for
developers to take advantage of the
technology that we had built
and broaden adoption as widely as
possible so
dealing with security at the scale of
windows
is a tough job uh there are roughly six
million
source files uh mostly cc plus but
a variety of languages 100 prs a day
that comes out to roughly 19 a minute
thousands of developers all across the
globe committing to windows
every day and 440 official branches
and an official branch is one that
receives a nightly build and nightly
test validation
and acts as an excellent choke point for
security teams like mine
to perform validation steps including
static analysis dynamic analysis like
fuzzing
and so on and so the way we tend to
think about finding vulnerabilities
and software flaws in native code is
part of this
vulnerability discovery funnel and at
the top of the funnel the most broadly
applicable
uh and cheapest to employ techniques
like static analysis
and at the bottom are things like red
team and so when we think of static
analysis that's automated code analysis
uh microsoft has world class static
analysis built into the compiler and in
additional tool chains
and we try to do those things at pr time
and as early in the developer loop as
possible
and we're now experimenting with uh an
ide
uh inner loop experience there as well
we then also like to think about
scalable fuzzing how do we
have developers perform automatic
dynamic testing
of their code roughly the same time they
would run their unit tests when things
are expected to work
we perform automated attack surface
analysis so this is churn
where you have changed code on something
that's already on the attack surface
as well as when you're adding new attack
servers and the best time for us to do
that is run
build and integration time there are
also other techniques dynamic sanitizers
that we'll be talking a little bit about
today when we do flight testing and
flight testing is
a microsoft term for when we deliver
early copies to
early adopters and enthusiasts as part
of the windows insider program
and then there's pen testing which is
predominantly the activity that teams
like mine do which is
validating the implementation matches
the design
and also looking for possible
implementation flaws
be they architectural or code flaws
and then finally all the way to the
right are red team and so that's a true
black box start from binaries reverse
engineer find a flaw write an exploit
and use that to help teams and
stakeholders understand any actualized
risk
and so the things we're going to be
focusing on today are going to be
scalable fuzzing
and dynamic sanitizer technology that
enables scalable fuzzing
for developers so let's also think about
the scale
of issues that come into windows so a
cve
if you're not familiar with it is
essentially just a unique identifier for
vulnerabilities in published software so
flaws
in any piece of software that were never
shipped to customers won't receive a cd
and so this chart is showing the number
of
issues reported either internally or
externally
in one of a shipped windows product and
you can see
over the years there's been an
increasing amount of attention both
internally with investment
and externally from external researchers
with strong incentives to find issues
and so 2020 is looking pretty good if we
naively annualize that
we're looking to be flat or a slight
uptick from 2019
roughly 80 percent of these issues are
common fuzzing
of course there is some bias in this
data
as i mentioned microsoft already
leverages world class static analysis to
find issues
early in the development cycle and so uh
that tends to filter things out earlier
in the pipeline and so then they never
make it to a deliberate product and thus
never receive a cve
there's also external finders of course
don't have source
don't have private symbols and so
they're limited in the tools and
technologies that they can use
to find vulnerabilities before they
report them back to us so windows has a
pretty generous
bounty program which pays external
finders and there are a variety of
incentives in the security industry
to be the ones to find issues and report
back to microsoft and that leads to some
recognition and pr as well
so all those incentives really lead to a
massive concentration of effort on
finding vulnerabilities in these shift
products
so if we think about the history of
fuzzing we can go all the way back to
the 50s
and i'll give you just a moment to
digest this anecdote here
but what it basically says is that even
in the days of punch cards the earliest
developers had figured out that there's
a real benefit to taking inputs for your
program that were syntactically valid
they were well-formed punch cards but
semantically invalid they were utterly
they were literally garbage there were
inputs that were never intended to be
processed by the program that you're
feeding them to us inputs
and so maybe they were the ones who used
the program code maybe they were
mistakes from earlier formats etc
but ultimately when introducing those
inputs into their programs
they shook out undesirable behavior and
it was the earliest form
of dynamic testing they didn't
call it fuzzing then but they had
effectively stumbled upon it
and then fast forward to 2016. so this
is a picture from the smithsonian
where out of a machine that won a darker
contest
to automatically find flaws in software
uh reproduce them and patch them so 70
years
has led to a lot of advancements and
it's still a rapidly shifting field
and so you know we want to take you or
we want to invite you to reconsider any
assumptions you may already have about
how expensive it is for developers to
begin fuzzing the costs of those
and capabilities of fuzzers
and one of the major advancements
in modern fuzzing and perhaps in the
last 10 or 15 years even
is in coverage guided housing and so
let's
start by considering this toy program
that i have on the left here
and we have parse function with an array
of bytes
and a corresponding size and we can
assume that size
is a trustworthy value for the valid
length
of buff and then we have a couple simple
check
functions there we verify that there are
at least enough bytes for us to do some
basic parsing
and then some early guard checks the
first night must be x the second must be
y
and so on and then we can imagine after
those initial checks
we do our actual work and so perhaps we
allocate some memory we do some further
parsing
and then in a refactor or simple typo
we've introduced a double free and a
double free is an exceptionally powerful
vulnerability when it does occur it's
very likely that
when it does occur an attacker will be
able to leverage that for arbitrary code
execution
were they able to influence the state of
the program and that input
so let's then think about how we fuzz
this target
fuzzing is fundamentally executing the
program and monitoring for crashes or
other spurious behavior
and the way we start fuzzing is by with
coverage guided fuzzing at least
is by selecting an input from a corpus
and mutating it
and corpus is just a term of art for a
collection of input seeds
as i mentioned before it's those cement
uh syntactically valid semantically
invalid
or perhaps even semantically valid at
the point of uh
at the point where we're starting
fuzzing and so if this is
uh you know an image parser perhaps it's
a well-formed image or perhaps not
and then mutate it is uh just a
another again another term of art for
performing perturbations
on the input prior to introducing it to
the program
um and by that we can mean randomizing
certain values we can smash in magic by
values that we know are likely to cause
problems
we can invert the values of bits we can
transpose sections of the input
really the sky's the limit to the in
terms of the creativity of the
fuzzer author as far as what can be done
as part of mutations
and there's a huge body of work here in
academia and in the security industry
on what makes a more valuable mutation
and what types of mutations
are more effective on different types of
parsers and programs
so then we need to introduce that input
so we trigger parsing
by uh invoking the program
in such a way that the input that we
have mutated will reach our
target function on the left and so the
buff will then be
will then contain the value that we have
here's where the magic starts to happen
is we measure code covered
and with uh modern tools and techniques
uh
we there's a variety of ways we can do
this but we can for the moment assume
that we have compiler introduced
instrumentation
to determine which code was hit when a
new input was generated
and uh that can then lead to a variety
of valuable behaviors where
if we consider passing in uh the word
test as the value that buff points to
when it uh does that initial comparison
against x uh we can hint
to the runtime that is measuring and
executing and guiding this buzzing
that perhaps x is valuable
to the like as is worthwhile to consider
and introducing
somewhere in the input and then such
later
might actually be um might able
function on new coverage
once we know we have a value that gets
us past a check we save that input back
into the corpus
and it's now a candidate for further
mutation
and then we do this hundreds of
thousands of times per second because so
much of the work that we're doing is
fuzzing
uh is probabilistic we really need those
large sample counts
to uh get us the outcomes that we're
looking for
and so uh the first piece of feedback
that we always get when introducing this
concept to developers is
well that's great this is a toy function
um
my my program is infinitely complex it
has
all these edge cases that your thing
won't handle and
here are a set of sample inputs from a
blog
written by michael celeski who is the
primary author of afl a broadly adopted
fuzzer these are automatically generated
jpegs and so as
as the contents of this blog outline he
started with a single
null byte in his single input in his
test purpose
and coverage credit fuzzing in
combination with afl eventually
led to automatically generating fully
parsable and displayable jpegs
so something as complex and convoluted
as jpeg
um can be solved to this type of type of
process
it's very likely that the vast majority
of parsers are targetable by this
technique as well
and so coverage guided fuzzing will help
as i mentioned it will
help discover magic numbers and guard
checks
looking for immediate values but it
won't
calculate hashes it won't break crypto
and it doesn't understand it isn't
reasoning about the relationship between
an immediate
or uh where that goes in the input um
naive or relatively straightforward
coverage fuzzing instead relies on that
probabilistic approach of saying perhaps
this
immediate is valuable and perhaps we can
wait for it to land in the correct
position
um and in reality this actually gets us
very solid results that we'll talk a
little bit about later
so what is it like to actually harness a
target for
fuzzing with lip fuzzer uh so lip fuzzer
isn't it is a coverage guided
fuzzer written by google and introduced
as part of the lvm
project and is an incredibly valuable
shift in paradigm
where you can think of lube fuzzer as a
unit test framework with a fixed
contract
and its contract looks exactly like that
parse function that i showed earlier
there's an array of bytes with a
corresponding size and the corresponding
size can be assumed to be trustworthy
and your job as a test author
is to introduce that uh generated or
mutated input
into your program and so here's a real
world example where we're
fuzzing the x509 parser that's used in
windows
and we're taking the data and we're
specifying a number of configuration
parameters
which in this case is saying please
interpret my data as an x599 certificate
and then we're introducing it to the
target and then on line 17 through 19
there you can see we're doing some
cleanup so if this was successful enough
to allocate resources
uh be sure to clean those up so that we
can then uh continue and not
exhaust memory or exhaust other
undependent resources
and so this is it this is a 20 line test
target that's actually
in use in production today and in terms
of
building and what it looks like to
modify your build system
you can see what we've done here with an
edited sources
used with build.exe we've enabled
address sanitizer
and we've included the live fuzzer lid
that is responsible for kind of
implementing the driver and performing
the callback
that we've implemented here and so uh
this gets us a long way towards that one
day
of developer time goal to get started
fuzzing
if we look at another case study we
actually
also perform this technique on a variety
of parsers
in dhcp and so dhcp obviously has a
network protocol
and the windows implementation is native
code and it's actually a
variety of self-contained parsers and so
it's almost an ideal target
and with coverage credited fuzzing with
no seed corpus
like no human provided inputs uh with
200 000 iterations per second
in roughly 30 seconds we exhausted
uh or we reached 72 percent code
coverage and
uh roughly stabilized there that uh this
work actually then resulted in four
publicly documented vulnerabilities and
two of those were remote code execution
um and so uh the
uh the proof is in the footing as we say
like actually then
uh spending the less than a day
necessary to harness this target
start it running get this performance
and now there is a durable artifact that
can be run every time unit tests are run
and performs automatic regression
testing and also prevents future
vulnerabilities from
reaching customers
so let's take a slightly broader lens
and think about how we got to where we
are
with compiler inserted instrumentation
and fuzzing
the the way things the way most security
teams work today is we employ security
analysts
to target the production binaries so the
same binaries that ship to customers
and run on customer systems every day
and our goal
is to reach this target where developers
can write
their own unit test like binaries
that are born to fuss they have a single
purpose of performing fuzzing
and perhaps they share code in common
but not necessarily the final artifact
binaries
the problem comes in when we then act
when we then consider all the problems
that a security analyst has to solve
they have to think about how they're
going to gather coverage if we don't
have the source we're not able to modify
the source we're not able to recompile
the source
perhaps we're doing things like dynamic
binary instrumentation
using pin tool or dynamo rio maybe we're
using a platform specific debugger
or the platform's debugging apis to
perform that instrumentation
maybe we're using ph heap or electric
fence to catch uh out of bounds
overwrites and reads
etc and so there are all these uh 80
solutions that the security analyst is
expected to
be aware of piece together select the
appropriate combination thereof
write glue or write new tools
when existing tools don't exist and this
is unrealistic to expect developers to
also become subject matter experts in
the whole domain of
security and security testing in
addition to whatever they are already a
subject matter experience
and so instead we look to compile in
tools
and so a variety of these are again
tools built by pioneered by google
and now uh some of them available uh
as experimental features in the
microsoft sql software toolchain
and so for coverage we look at things
like sanitizer coverage for boundary
detection we look at
we look at things like address sanitizer
and other things we completely avoid
like having to introduce io by making
that a
a part of the contract
and really the goal here is to pivot
from the hacker bolt-on paradigm
to introducing developer tools we need
to reach developers where they're
already working
in tools they're already familiar with
and in paradigms in which they're
already familiar and operating
and through all of this then we can rely
on the security analysts to provide
fuzzing expertise
they can write new fuzzers they can
ensure that they reproduce correctly
they can assist with regressions and
automatic regression testing
and introduce and manage the corporate
responsible for
making widely available parsers fuzzable
and so once we've got self-contained uh
developer-created
fuzzing binaries like this we then need
to think about
how we fuzz them at scale and for that
i'm gonna hand it off
to mike walker who's gonna talk you walk
you through
more of one fuzz and how we perform
fuzzing at scale
excuse me thank you justin my name is
mike walker
i'm a senior director at microsoft
research
um thanks to cppcon for uh inviting us
to be here today
today's talk is about the open source
onefuzz framework that we're releasing
today
and that is a framework that was
developed through listening
uh listening to teams that develop
products around microsoft like justin's
team
and others uh to developers and fuzzing
authors at fuzz con the fuzzing
conference this year
to the customers of the microsoft risk
detection service that we've operated
for years
uh and both listening to security
engineering and developers
and the consensus story that we came up
with uh through all these conversations
was that everybody wants fuzzing and
everybody understands the reliability
and security upside
but everyone also agrees that fuzzing as
it stands and as it's been traditionally
done
is hard expensive to use and it's
expensive in terms of engineering
engineering work that could be better
spent developing new product features
so i want to dive deeper into the
revolution that justin was describing
through an example this is our
simple image resizing application that
we're going to use to talk about
fuzz testing and the compiler testing
revolution
um and the job of this service is to
resize images
its control flow is very simple it takes
untrusted inputs from the network
and it recognizes them as jpegs or pngs
and then it has to de-marshal those into
an
rpc state machine so jpegs and their
associated metadata go one way
pngs go the other inside that state
machine
is the work of converting jpegs or pngs
into an intermediary image format
and then resizing and then
re-serializing
in the user's requested format and
sending it back
now if i'm a security engineering team
like justin's
then there's a significant amount of
work to do to be able to test this
network service because it is a network
service it is stateful
that means that it has to be fast and
performant for the authorized users
they're going to use it in high volume
every day
and it also needs to be resistant to
black hackers who have vulnerability
research
capabilities so that means that we have
to be able to test it better than they
can
and my first problem is that i have to
develop a networked pseudo client that
tracks
every feature in this service to get
inputs
into a network service that is a
production binary i need to be able to
generate
high volume high speed inputs into the
service and then orchestrate
the correlation of the effect with the
input that's a significant amount of
work
the next thing that i have to be able to
do
is advance my slides
well um technical difficulties here we
are
um the second thing that i have to be
able to do is
instrument production binaries so
production binaries are designed for
speed
not instrumentation um and that means
that i have to bolt on coverage often
through a dynamic binary instrumentation
framework like dynamo rio
or intel's pin and i have to bolt on uh
crash detection either through parsing
cores or using a debugger
and then i have to be able to
differentiate real crashes and
investigate real crashes and
differentiate them from
instrumentation induced crashes
uh so the net result here is in our
example
uh security engineering has full-time
care and feeding of this
testing framework um they have to write
probably two clients one for the rpc
state machine
um one for the network service uh
they're going to be doing a lot of
uh investigation of each crash uh
there's
probably going to be per app
enlightenment using the dynamic binary
instrumentation framework
and reachability issues are going to
over test the recognizer
and under test the serialize and send
step
so what we're looking at here is a set
of homework not just homework for the
software under test
but homework for the testing apparatus
security engineering teams are tasked
with building high performance compute
root causing crashes
uh testing every single input for
reproducibility
um uniqueing crashes and figuring out
whether
all these crashes fall under one bug or
different bugs and then there's the
whole job of fuzzer selection corpora
updates
the rest so
let's take this legacy application and
rebuild it again with the compiler
fuzzing regimen that
justin was describing we're going to
take each component
and we're going to harness it with the
llvm test one input
function prototype
and for each component
we're going to have a unit test that is
cooked
into the build system and this unit test
is going to contain
the trifecta coverage sanitizers and
fuzzing technology
all built in and the goal here is
automated targets that you can build
once and compile forever and vanish into
the build system
kind of reducing that homework lift so
the compiler support that we've already
covered uh is half of the story
and the other half is cloud fuzzing the
job management to be able to do testing
in bulk with a
massive number of test cases that uh
justin mentioned
crash deduplication being able to check
uh and
and manage the uh massive number of
crashes you may get
making sure they reproduce making sure
they're unique handling the onboarding
of fuzzers
and and using telemetry to get
notifications
so let's let's try this out on the
command line
take a look at what it looks like we're
going to start again with that simple
test program
that is uh checking the input for x y
and z
but afterwards we're going to use the
next byte to select
a different critically important memory
corruption flaw and this particular
sample program
on its fourth byte will select from one
of eight
different harmful memory corruption uh
capabilities so
if we fuzz this thoroughly we should get
at least eight unique bugs
and uh fuzzing
um with modern coverage guided buzzers
is useful just on the command line
that's what it takes to build one of
these unit test binaries cooks into the
build system very easily
and that's how long it takes for a
modern coverage guided buzzer to
determine that xyz
is the correct uh the correct input for
this program
and it starts generating crashes and
it's completely feasible for a developer
to test this type of binary on their
local workstation
to harness one of these binaries into a
while loop start generating crashes and
start generating a series of faulting
inputs you can see the crash and then
it's hash
generated on the command line the first
one generated was
x y z 0. now as useful as this is
with real software this needs to scale
this needs to be a large process with
lots of compute behind it
and there's a lot of data orchestration
behind the scenes
so to solve the problem of how to take
these unit test binaries uh attach them
to multiple fuzzing technologies get
them into the cloud
and and handle all the data flows and
crash management
we developed one fuzz that's how far the
buzzer gets on a local workstation
we can use one fuzz to send it to the
cloud
so um i want to talk today about the one
fuzz framework
it is open source and it is self-hosted
so uh mit license this is here for you
to run uh
in your enterprise and to build and
remix
um and change and
it's also self-hosted that means it runs
in your subscription
so your jobs uh your there is no hosted
buzzing service here
all your jobs and your data live in your
subscription
and deployment of this framework does
not take
uh an azure expert uh the readme that's
linked on this slide
will allow you to if if you have access
to the azcli and access to azure
start this up in an afternoon and have a
fuzzing instance that can scale to a
company as large as microsoft
so i want to talk again about that
listening phase of listening to security
engineering teams listening to
developers
and talk about how the one fuzz
framework attempts to solve
the very different requirements of these
two different teams so as we talked to
security engineering teams around the
company and around the world
what we found was that we would hear
feature requests
but they weren't feature requests they
were complete blockers
security engineering teams often had
unique requirements where if they
couldn't meet that requirement they
would abandon the entire fuzzing
technology if they couldn't make the
change
those requirements went from windows and
other operating system dependent
products that needed to be able to take
daily builds of their operating system
and fuzz on that daily build so they
needed
complete custom operating system support
that was cross-platform we have this
problem on linux we have this problem on
windows
there were many teams that needed their
own custom crash analysis tasks
and they needed to be able to plug in
their own crash analyzer because there
was
rich metadata associated with their
program uh that needed custom analysis
we had many security engineering teams
that said we need to be able to onboard
our own fuzzers we have private fuzzers
that live in our subscription or our
enterprise
we need to be able to onboard them
easily if we can't do that we have to
use our own fuzzing rig
so what we developed was a python
software development kit
where it's not just that you can change
things because it's open source you can
change things because
everything in one fuzz is used and
composed through this sdk
so all of our fuzzers are onboarded
through the onefuzz sdk
and it's made up of a collection of
simple primitives
so jobs are collections of tasks tasks
perform
a primitive um within a fuzzing workflow
and every single task and every single
job is remixable and plugable so you can
add your own input generators
you can add your own coverage and you
can add your own analysis tasks
on the other side of the fence what we
heard from developers was we need
simple reliable recipes that just work
and plug into continuous integration
so onefuzz has a python cli or a cli
that's accessible as a downloadable pe
where you can send in very simple
recipes get
powerful defaults and fuzzing should
just work
so let's talk about how that line is
uh set up and navigated
component by component one fuzz had to
be a
scalable architecture we needed to be
able to handle software projects with
thousands of concurrent buzzer targets
thousands of cores
but it also had to work the same way if
you were fuzzing on one or two vms
so to do that we used scalable
primitives in azure
and uh the the end of the road here is
that
everything is structured data in a file
in an azure storage blob container so
seeds or files crashing inputs or files
executables or files crash reports or
files
and all of them live in the same by
default seven
storage containers that are set up on
every job your setup storage container
creates uh contains prereqs
inputs create contains both the inputs
you provide if you start out with a
corpus of valid inputs or inputs you
want to test and those you generate
through fuzzing
all your crashing inputs live in a
container your crash reports which are
kicked off automatically live in a
container and then the container that
you will
care about the most is the unique
reports container
where new entries in that unique report
container by default can kick off
messaging
uh to your continuous integration system
we also have
those bugs that failed to reproduce and
code coverage data
continuous integration on compile
fuzzing so
we talked a bit about how developers uh
want reliable recipes
uh that just work this is the commit the
default command line for getting
a fuzzy job launched in the cloud and
we're gonna uh
fuzz a component out of our sample image
processing application
start with a template take the re image
resizer project
jpeg functionality build number one
and the linux available pool
so virtual machines live in pools
they are hot spares so the pool is a
size that you specify and set up
and uh every task will pull from the
available pool that means that
in that phase in between local fuzzing
and continuous fuzzing where you're kind
of setting things up and working with
the framework
you get this very low latency observable
debuggable interaction with the
one fuzz framework build aware artifact
preservation means we keep
inputs over the life of a feature as
you're doing continuous integration
testing of the jpeg feature all the
inputs that were created by last fuzzing
run are
automatically ported to the next fuzzing
run and combined with your new
newly compiled fuzz target
and beyond these defaults there are
powerful options
so i mentioned the need to be able to do
custom operating systems
um that comes from a pool that you can
set up
and you can do everything from pulling
public images
out of azure vm image list off the azure
cli
anything in the url format works to
using
automation to take your daily build of
an operating system and create a custom
bootable image of that operating system
or if you're an operating system
dependent product
or you're doing something exotic like uh
fuzzing a hypervisor using nested
virtualization
you can create and test your own custom
operating system image and then do all
of your fuzzing on it
um
for scaling we wanted to make sure that
the same command line that you used to
do continuous fuzzing at very small
scale
could be turned into large-scale uh
fuzzing
with just one flag so scale is just a
number in one fuzz
if you want to fuzz with tens uh a few
cores
tens of course hundreds of course all
you have to do is change vm count
talking to security engineering teams
fuzzer choice was a really important
topic and the ability to onboard fuzzers
was a really important topic
so this is a screenshot from last week's
fuzz bench
um and you can clearly see that on when
testing sqlite 3
that you get breakout performance from
afl plus plus
more edges discovered more quickly so
it's clear to me
as a security engineer perhaps that
if i'm going to test my software i
should use afl plus plus
here's a screenshot from the same uh
fuzz bench
test from last week of libpng where
hong fuzz achieves breakout performance
and more edge coverage
security engineering teams love results
like this
uh they love learning about which
fuzzing technology should be adapted to
which software and tuning the fuzzy mix
um we also uh when we speak to
developers realize that developers
just want high quality reproducible
results they want the best testing
uh and analysis and they're not as
concerned about what's happening under
the hood
and we felt that the solution to this
was uh very recent
in in in terms of academic investigation
the um
ianfuzz paper ensemble fuzzing presented
at usenix last year
uh came up with the remarkable result
that when you allow these buzzers to
share
inputs throughout the entire job to work
as a team
uh lib buzzer afl fuzzers like radomsa
maybe a private fuzzer that you use or
fuzzers that are
domain specific to your target when
they're able to swap
inputs they can reach coverage and bugs
that no individual fuzzer in the team
can find on their own
so we decided that one fuzz would be
ensemble by default
and so when you onboard a fuzzer or use
the default fuzzers in the one files
framework they all swap inputs and work
as a team
and you don't need to worry about it as
a developer when you're using these
templates that just happens as a
property of the framework
and uh we wanted uh security engineering
teams to always be able to bring their
own fuzzers
so we have first class support for the
buzzer afl radonsa
but we also have an examples directory
linked on the slide
that shows how to onboard other fuzzers
d'amato which is
very specific to the browser domain hong
fuzz oss fuzz
these examples um
are are there in the directory
not just to be able to allow you to
onboard those fuzzers but to you
to work as a template for onboarding
buzzers that are unique to your project
even closed source buzzers that you may
keep only in your subscription or belong
to you
so i want to do some security
engineering
use the framework on the command line
briefly uh while also noting
that you don't need to do this in order
to use one files
but what i want to point out here is
that everything is programmatically
accessible in the framework
and all data is structured uh data is
json the jms
path query language and step one is how
all querying of the framework happens
and the idea here is that it should be
interactive it should be scriptable
and it should be able to interact with
your automation so in step one
i will search for a job with containing
the substring 8763
just enough of the uuid to be unique i
will pull the unique reports container
and i will pull one json structured
crash report out of that container
here's what it looks like on the command
line and we're going to start this
command line session where we left off
with a fuzz target we had just built on
the developer workstation
and we're going to launch a cloud
fuzzing job so there is that one line
template again from the one five cli
and this is what it takes to launch a
a cloud fuzzing job of any size
uh there are the seven uh containers i
mentioned
and it's off now the minute you launch a
fuzzing job you want to know how is it
doing so i just launched one fuzz status
top
this is fuzz top uh the development team
behind one fuzz loves this view
you can see everything in your entire
fuzzing instance in real time
at the top is the status of your pool
it's busy in free node counts
next are jobs so the job we just
launched uh that is
following the tiff functionality uh with
two crashes and nine inputs is
accompanied by longer running jobs
testing png and jpeg with more crashes
uh
based on when they were started and then
all the individual tasks that are being
kicked off by the jobs is at the bottom
now i'm going to dig straight into the
job i just launched on the command line
i'm going to say using just enough of
the handle
from that job 3fc just the substring
give me that container and list
everything inside it
and i want the unique reports
so those are my unique crash reports i
have two unique crashes so far
and if i pull that json file you'll see
again
uh that all data is structured um
everything is a file in a container and
the the ability to pull this uh and then
we'll run it through jq so we can
we can take a look at it uh in its
structured form uh
has sensible name fields that would you
can plug into your follow-on automation
uh maybe route different types of
sanitizer bugs
to different parts or
different notification pipelines the
entire call stack is there
uh the asan log anything that your
security engineering team wants to be
able to fire
as a introspection driven event
is programmatically accessible uh from
this cli
but we also understand that this is what
security engineering teams need but it
is not what developers want
uh when we talk to developers they
wanted to the fuzz testing process to
vanish into their build system
and to kick off automatically so
in the one fuzz samples directory linked
from this slide you'll see example yaml
files
uh that show how to launch fuzzing jobs
from an ado pipeline
and how to launch fuzzing jobs from
github actions
and uh we also have examples on how to
do
new unique crashes um and cr uh
have those create notifications in
microsoft teams or as
ado work items so
uh i'm gonna show an example from
microsoft teams
and like justin said when you work at
microsoft now you can get messaged over
your
chat application with a new unique
reproducible fuzzing induced crash in
pre-production code and along with that
notification
comes a direct link to the crashing
target the direct link to the input file
and a direct link to the crash report
with a complete call stack included
and uh it it really changes the
the uh the way that fuzzing is viewed
when
every single bug is reproducible
the screenshots here are from our dev
branch this is
a github issue automatically generated
by the one fuzz framework
uh talks about again the same fields
um uh crashing input in the in our
example program
um down in the github comments uh are
all the
reproducing cases so uh multiple
invocations of the same bug unique
as additional reports and then finally
uh the this part of the git hub issue
has
the same metadata as teams and i want to
call out one line
in particular which is the repro line
so this is a pasteable command
that allows for live programmatic
cross-platform debugging
of any bug found by the framework and uh
when we
launch this on the command line so i'll
copy it out of my
build system and paste it into a one
fuzz enable command line
a series of things happens whether it's
windows or linux we're gonna pull
a hot spare vm from the specified pool
we'll do the key exchange automatically
we'll set up an ssh session for you
and then on the target the known
faulting input
the target is deployed a debugger is
attached
and we'll halt on entry and because you
have the call stack this is a time when
you can set up
breakpoints and you can continue
and the program is going to parse with
sanitizers enabled
straight to the the crash condition
and we're using the same sample program
that we started off at the beginning of
the presentation with
um and
in this particular case you can see that
this is lip buzzer discovering xyz one
and it's even available at line
seven of the call stack that's the
faulting input
so continuous fuzzing should not be
about doing continuous homework it
should be
about spending a day setting up maybe a
fuzzing instance and some targets for
your program
and then after that the fuzzing program
should vanish into the build system
and it should be about notifications
that are
immediate and actionable uh that are
easy to understand and make your code
more reliable
help make you a better programmer
i want to take this slide um right
before we move to questions which
i've been i've been reading but uh
hoping that future slides would answer
but we'll get to them um to spend some
time
thanking uh the one fuzz team who spent
the last year building this framework
and getting it ready for global open
source release
our partners uh on product teams who
who've worked with
us uh used the framework driven its
development
uh justin windows jonathan normand at
edge
uh sphere sdl teams around windows
uh the visual studio team jim radigan
and the customers are our microsoft risk
detection service all of whose voices
are represented in the
the result we're releasing today we also
need to thank
google costia the llvm community for
pioneering this compiler fuzzing
revolution you'll see
their ideas and tools represented
throughout the open source fuzzing
community
and the attendees of cppcon and cppcon
for having us
so uh again we are open source
and self-hosted and we're going to be
using this continuously at microsoft
and pushing updates uh to github
um and we're releasing it globally
because we hope that uh
this is going to democratize and make
fuzzing easier for enterprises
who um for whom that security
engineering task has traditionally been
too expensive
so we hope that you fuzz with us on
windows and linux operating systems
using composable fuzzy workload
workflows that you can edit yourself
using built-in ensemble fuzzing by
default
programmatic triage that's that's
hookable and customizable and can use
your own
crash analysis automated result
deduplication
on-demand live debugging straight out of
your messaging system
um observable and debuggable with
detailed telemetry through app insights
which we don't have a slide on but it
but it's there and that uh app that app
insights telemetry
uh is something you can use to you to
create coverage
guided fuzzing programs and crash
reporting through microsoft teams
so at this point um i believe
justin can rejoin me
and we can take some questions from the
audience
so i'm gonna uh please bear with me uh
as i uh attempt to to repeat some
questions
um yes one fuzzy is open source on
github and we are happy to work with
developers
um and and we'll by the way be at the
microsoft booth for the rest of the con
uh to talk about people to
talk with people who are interested in
collaborating or contributing to the
framework
fuzzing can be used for testing non-i o
functions and there is a huge body of
work using fuzzers to discuss
uh to to discuss state machines
differential analysis of implementations
etc
um uh maybe falling onto that i know
that there is um
i believe the the project is crypt fuzz
but there is a a project that uses
fuzzing to discover
divergent implementation behaviors
between cryptographic primitives that
should always have the same outputs for
the same inputs that's maybe one
example of that to answer other
questions i'm going to need to move over
to the screen and scroll back
so maybe justin who's had more time to
prepare can pull one out of the stack
yeah let me let me go ahead and read one
question for you here
uh does throwing exceptions count as a
crash for purposes of
crash monitoring do you want to take
that one
uh yeah sure so so today in
uh technologies like lib fuzzer
structured exception handling is
uh we'll handle that correctly i believe
the public implementation of the buzzer
does have a gap
uh where c plus exceptions are concerned
that's something that
my team and the compiler team at
microsoft are working to close
one fuzz on the other hand that's
entirely uh configurable based on
how you've implemented your fuzzing
adapter and so
mike mentioned that it's composable and
you can bring your own fuzzer
and it's incredibly flexible in how what
behaviors you treat
as something worth reporting and whether
you treat that as a crash for purposes
of alerts and reports
uh second question here does the
framework work with any
programming language is it a black box
testing tool
mike you want to fill that one
it it can we say that fuzzing is good
for
any target that contains native code
so it sees c plus plus
but those are just the the primary
languages
uh that where where fuzzing is
traditionally used however
it's worth noting that even with
binaries that have
stronger guarantees of memory safety
anywhere there are potentially unsafe
components
at the end of a reachability path
fuzzing is an appropriate solution
so you see fuzzing in the rust community
as well
now i can read another one for you here
can we integrate with other ci
environments
uh for example team city
uh we haven't tested it but the spirit
of one fuzz
is that everything should be pluggable
and integration should be easy so
that uh onefuzz samples um directory
uh and those yaml files there are
recipes for how we've done it with ado
and github um but this is open source
and contributions are accepted so
uh our answer is try it let us know and
we're really interested to know whether
the sdk is adaptable enough to make that
happen
or what sdk changes we need to make
so that can happen
okay and next question can we extend
your framework with c-sharp
well it's open source um so i don't
think there's a reason you couldn't
um i will say that uh one fuzz
when we talk about the language choice
of the framework itself it is primarily
python um and
and there is also a rust component
and we've used rust uh exclusively for
the execution agent that happens
on the target virtual machine because of
rust's
well first uh amazing runtime properties
but also its ability to be able to emit
dependency free cross-platform binder
binaries without a lot of
platform-specific code
so it is already a multi-language
framework um
and i'm sure you could extend it in c
sharp
yeah and i'll add to that that um if you
wanted to write a fuzzer in c sharp
that's obviously
supported today with no additional work
and microsoft is actually working with
the.net and c-sharp teams on
using fuzzing to fuzz csr and the
dot-net runtime itself as well
okay i think that's our our total
backlog
i i believe we've covered the questions
queue
but um if you'd like to have um a
smaller discussion
after the talk um please come join us at
the microsoft booth
you