all right thank you so much yeah my name
is John Reese I'm gonna be talking to
you about our company's approach to
policy coming from a very roll heavy
focus security model again name is John
Reese work at plex the manufacturing
cloud get that a lot we're not the media
server we are actually a ERP system for
manufacturers enterprise resource
planning so we handle anything from the
shop floor all the way to the CEO so
thing shipping accounting their their HR
system anything they need to do to run
their business that's kind of our
wheelhouse I also love the go
programming language so if you love go
and then we're already best of friends
and we should hang out sometime so again
this is really gonna be how we have
approached security over the years the
company's been around since like 1995 or
so so little over 20 years and we've
gone through a lot of growing pains
going from different security models and
to into where we are now so to do that
I'm just going to kind of start where we
started from relating to roles and
security moving in to where we are today
and kind of where our mindsets are and
then where we really want to be and get
to in terms of security so for some
background in the beginning roles just
controlled everything within the system
so again we are a full solution for
manufacturers and we built everything
for them this includes again their their
HR system their therein out system
everything that they need to do they
would log into our system and they would
use us . this also means that early on
we as a company also used everything
within the system so maybe not the
greatest idea because we're a software
company not a manufacturing company but
that also means that everything that you
needed to do in that system was role
based if you needed to log in to to
check in if you needed to do any
action with the system through is gonna
be a role for that and honestly it
looked a lot more like permissions more
so than roles we had a lot of them and
even again we put our deployment system
in here we put everything you needed to
do in the system and so in a way for us
roles were way of assigning
responsibilities if if you were given a
role that was kind of your wheelhouse of
what of what you had to do so so for
example we had this idea of a module
champion you can think of a module as
just an area of the system right you
have an accounting module champion they
are responsible for the entire stack in
that area and I pulled the definition
from our internal wiki which again we
also built and it's essentially saying
it's an individual developer responsible
for that module and so this means if a
feature request came in that's on you if
something went down that's on you and so
this was a pretty good approach to start
but as we grew is just you can't handle
all that information but there's a
slight problem with that is that we also
had this idea of a standard member and
they were responsible for our language
standards you can think of this as where
the braces go in JavaScript or tabs or
spaces but they were also responsible
for deploying restricted source code and
we defined restricted source code as
really any code that we felt could
negatively impact the system in a large
way so for example the the menu system
that was a module and a menu is pretty
much going to be everywhere throughout
your system it's how your users navigate
so that would be a restricted
restriction module that a standard
member would have to deploy because they
know how to deploy restricted code but
the problem with this again we have this
very clear definition of roles is that
we had this situation where someone who
owns this system who knows everything
about it can actually deploy the change
so you have this guy on the standards
team who doesn't know anything about the
change itself
doesn't know anything about why the
feature came in but it's his
responsibility to deploy restricted code
because again he knows how to deploy a
restricted code rolls were often a
solution for us to prevent problems from
happening again so to put simply if you
have an employee who does a thing and
that thing didn't go right we would
generally create a thing do a role and
it seems a little extreme but that's
kind of how things evolved if we would
give employees a lot of privileges
because we trust them
and in the event that they didn't do
something right maybe it's their own
fault or not we would all get into a
room and would be like hey how can we
solve this problem how can we make sure
that this thing doesn't happen again and
you would we'd all be like boom that guy
like he's in that he is now the thing
doer it is your responsibility to make
sure that thing doesn't ever happen
again
and so this created a lot of problems
again as the company grew and grew and
grew where simple tasks would just
involve multiple multiple people so true
story we have a developer who who simply
just they asked the the network engineer
for a new domain this is the internal
domain it's gonna be with inside our
four walls he's just asking hey I need
to get this made we are using Azure so
in this case he needs to create a
resource group resource groups in Azure
are kind of where you put all the
resources it's responsible for for
billing and you can put some more
restrictions on it but because a
resource group isn't a networking
concern the network engineer cannot
create resource groups so he has to go
over to the system engineer and be like
okay yeah sure yep I'm gonna create one
for you comes back all set
realized hey this is a dev domain HTTP
like can I make a cert so he asked the
engineering managers who's responsible
for approving these types of requests
he's like yeah sure I guess cool domain
gets added and then this comes back to
me the developer and so
what to realize here is that you know
all these steps take time maybe you'll
get lucky and it's instant if they just
got nothing going on I could take a day
it could be a week if that individual in
that role just isn't there so a very
simple task me to put in perspective
this should have taken a couple minutes
to go with the log into Azure add some
resources and when we're good to go it
took like two weeks which is again it's
a very sad story and so so one thing but
I'd really like to to kind of highlight
especially to to newer devs or just when
people are complaining at the company or
even to everyone else right is like
these were probably the right choices at
the time I mean again the company's been
around for a really long time I have a
paycheck because of these decisions when
we are thriving because of these
decisions but you know there comes a
time where you actually do need to
change and need to evolve so when you
when you're working on like a legacy
system you're like how did they ever
make these decisions why would they ever
ever do this I mean there's a lot of
factors back in the day that create
these types of decisions so that's kind
of a a look into what we what we did
back then you know still a little bit
today but for the most part we have a
very different mindset so at the heart
of it it's automatic approvals through
policy so think of it as we had that
previous slide where we have to go
through every single individual step and
those are pretty much all approvals but
we do this because policy really
enforces what we actually care about and
this was really I feel a a green light a
a light bulb for us when we really
started looking into policy as to how to
secure our resources where you don't
really necessarily need an individual to
to stand these concerns up if you like a
virtual machine or a repository or any
any sort of thing that you might want to
request when you ask for those the
person in charge of that is just going
to make sure that the
the constraints are within reason you're
not asking for a ton of resources maybe
it's name correctly you know whatever
they care about in terms of like their
domain that's what they're looking for
and if you meet all the criteria they're
just probably gonna do it for you
so to kind of tackle this problem it all
starts with OPA for us open policy agent
if you're not familiar you can really
think of it as a yes/no engine for the
most part you give it a you give you
some data and then you have red go files
which are your policies and then given
those those to it will say can you do
this or can you not do this and so as an
example this is a really simple
regulatory to kind of show you how a
policy might look this is a kubernetes
manifest for for a namespace and this is
saying if input kind is namespace and
there does not exist a labels field
vanish it should it should deny and say
namespaces must have labels so one of
the really powerful things about about
reg o is that it doesn't necessarily
care about the the structure of the data
coming in that's something that you
define so input here is in blue because
that is a key word it's the input to reg
oh but you know kind is just something
that that we set because that is what is
in a community's manifest you have kind
operations but you also have metadata
labels but anything to the right can
really be anything and so in our case of
creating a namespace you can say here's
this manifest it's a namespace does it
have a label yes or no and so two to use
all these tools together we use
something called called comp test and
the the power of this tool for us when
we were introduced to it was the fact
that it supports just a wide arrange of
file format so to really kind of
showcase what can happen with files you
have yam all JSON I and I and docker
files for example and so we can take any
sort of structured data where we know
what the data is going to look like
throw it at com
contest has our policies in it and it
can tell us does this file violate our
policies so a really simple pipeline
could be that a developer is going to
submit a pull request and that pull
request is going to contain their
terraform it's gonna contain their
kubernetes resources and then when they
submit it contest itself will validate
the PR the key thing to note here and to
really highlight again maintain errs
define their policies because again
policies are really just a brain dump
for what the maintainer wants the
maintainer the person who might have had
that role is the person creating these
policies so as a developer I contribute
to your to your repository I must abide
by your policies and then if it
validates then you're just you're off to
the races because if you are within spec
everything is named correctly you have
all the right labels etc it's just going
to go off and do it in this case now a
lot of our a lot of our implementations
are using terraform so we can just say
hey this is valid terraform do a
terraform apply good to go
so again to really showcase some of the
other use cases that you have for this
that we do talked a lot about terraform
we have a means for if a developer wants
a repository and azure devops we don't
have an azure devops guy creating
repositories anymore we used to but now
that's all automated where the the
DevOps guys wrote a terraform source
module so they define how they want
their repositories to look and so when
you request a new one we just have to
make sure to say hey are you using the
Plex
source module for this for this
repository the the other use cases again
because it doesn't really matter what
the input is
we have kubernetes manifests that say
you have to have these labels and also
again docker files you can make sure
that it has a label for origin some
other examples could be the docker file
has to come from blessed images so if
you at your company have only images
that you allow your employees to use you
can make sure that like the from tag is
actually from
list of approved images super super
powerful if you're interested in
learning more about the tool itself this
talk just like blew my mind when it came
to came to policy aging and comp test
because we were working through this
just at this moment when I believe that
this talk came out where we were we were
thinking we knew we wanted to apply
policy especially at the at the
kubernetes level we knew that we wanted
to make sure that our policies are our
llamó files had specified labels on them
but we were really afraid of the fact
that how would we put this to the left
shifted to the left for developers
because we didn't really like the idea
of developers writing writing code
writing manifest and then they didn't
know it was actually going to violate
spec until it actually went to the
cluster and we were really afraid of
that but calm test again it can be run
on your local because it's just a binary
it can be running your CI CD pipelines
and then once it's deployed that's kind
of where the wheelhouse for for comp
test ends the other approach we are
taking is not only the creation of
resources but also the configuration of
those resources once they're already
created so this is actually pretty
pretty new for us we are doing
repository
policies through a policies that yamo
file where we can say in the root of
your repository these are the things
that I want set so you don't really need
any sort of entry 'extra permission
aside from being able to contribute to
your repository and once you make these
changes we have an azure function that
was basically monitoring for pushes to
policies yeah Mille it will notice say
hey did did anything change and if it
did validate against our ops team's set
policies even if it doesn't then you can
go ahead and you can modify that
repository so it's end-to-end
essentially of you can create this
repository and then once it's up and
running you can still make changes to it
you don't need to say hey yes we we
this repository but now we need to make
a change to it can you update this value
for me we can all do it through here
because again we used to have to say hey
we used to have two approvers but now we
want three and that's gonna be it no
suite so we use service now so yeah
service now ticket like update this
number from 2 to 3 that's just not a
good use of someone's time the the last
thing for for policy that we're really
starting to get into is is a tool called
gatekeeper and this is this essentially
picks up where comp tests stopped so
again we use com test for everything for
like local and CIC d and then gatekeeper
is responsible for enforcing those like
in the cluster themselves this is what
we were going to use for the actual
application of the policies and we
didn't know what to do with the the CI
CD part until contest but this is
allowing you to use the same policies
because both tools are built off OPA and
built off reco again reco doesn't really
care what the data is
OPA just reads it so you can shut any
kind of data and you can use the same
policies wherever because again policies
might change from an environment
environment and you can configure that
but for the most part if you say that
you don't want a late or you need a
label somewhere you shouldn't have to
rewrite that policy for for everything
you shouldnt have to deploy this to
gatekeeper and deploy this to comp test
you can have a central policy location
that has that defines all of your
policies in kind of to the point of like
it doesn't make sense for someone to do
this the the thing that really resonates
for me is that a bit another big push of
this again we want employees to innovate
and not just approve request right we
these individuals are incredibly smart
we we pay them well and we shouldn't
have to take away their time just to
approve request right there they're
working on highly complex things they
they see a list come in and in email
saying hey you need to approve this they
probably don't really want to they want
to be off doing this thing but if they
don't approve it it's
everybody down so with this type of
approach we can just make it so we
remove that load from them and they're
able to do what they actually want to do
I like to call I'll call this out too
because again it's not like rolls are
dead and Paul and policies everything
for for us it was really just a
different way of thinking about security
again early early days was we would
implement a feature and we would say
okay well this is new now we have to add
some role for that you know the the
thing doer this thing has to be
protected and who's gonna do this and
just everything was very locked down
from the get-go
whereas now we take very much the
approach of why can't we open this up
why can't we open this up to everybody
what are the things that we actually
care about and then take it from there
and again policy can use roles I mean
it's it could be very possible that you
have someone who is a kubernetes
operator and that's a role and you have
a policy saying that you must be in the
kubernetes operator role and you must
have have Eastfield on your gamal they
can work together so so for going
forward um then the easy bit of it is
that we just really want to continue
adopting policy we want to make sure
that that we are trying to remove manual
sign-off and just implement policy where
we can for me the the more interesting
part and where we're going to be
spending most of our time I believe
going forward is actually moving those
policies into the application level and
so what I what I mean by that is not
only do we have this role problem
internally here at our company but since
both systems are pretty much the same
that our customers also feel this like
they have a lot of roles that they have
to manage because our system is meant to
run their business and every business
has different policies different
security rules and so we give them the
ability to to define how their business
works but really only we have to do that
is to say hey like you can create roles
and then we have to check all those in
our systems so it
be really really cool if we can say
these are your policies you can kind of
define those how you want them to be and
maybe we could even set our own global
policies if you will of like this is how
typically manufacturers solve this
problem but if you want to make
modifications to it then you're more
than welcome to to write your own
policies and get them out of this this
very very roll heavy world so that's all
I had these are all be the links to the
tooling that I introduced OPA contest
gatekeeper and then Garris talk on
contest thank you so much I'm gonna be
here all week
we have time for questions now otherwise
we do okay all right
how do you how do you address like
emergence for example emergencies and
within the people needing to make like a
manufacturer hey something updated tons
is it like an Esso something you're
cutting out a lot really sorry oh I'm
sorry
how do you how do you identify how do
you adjust for like an emergency
something that wouldn't necessarily fit
like an SOP style policy okay how
quickly adjust for those those
emergencies changes so like if you need
to make a change there like violate
policy yes um well yeah I mean there's
obviously always going to be like we
overrides I mean the we get this is our
this is our typical CI CD pipeline the
individuals in these roles still have
full full access
again the so much of it again is the
maintainer z' defining their policies so
if you essentially if you were a
contributor someone who is trying to
contribute to the repository well then
just its policy full stop but you as a
maintainer you as someone who
as you know you can push the master you
will still have those permissions but
anything came up like you still had the
ability to violate policy there's
actually not a really cool thing about
about gatekeeper lis specifically is
that in even the previous talks and
continues compliance that if that ever
happens we are actually looking into
ways of running continuous compliance
and making sure that no one did just go
run cube cuddle add some things and
change configurations manually because
you want to be aware of those of those
changes but but yeah for the most part
yet they still have permission to do
that
how did you go about adding the new
requirement now that I regel itself new
learnings adding that low how did that
slowly out yeah so I mean for for us it
was I mean it was very slow a few people
it was definitely not an overnight thing
I mean these are these are approaches
that you can take at a team by team
level or even like a repository by
repository level so so for example the
the first thing that we ever did was
just a policy around our docker file
because we have a we have a policy that
docker files must have an Origin
repository gen label and so the owners
of that repository said okay I'm gonna
add comp tests into my CI CD pipeline
I'm gonna write a policy folder that
says that and we're good right and then
that just kind of naturally evolved and
grows throughout the system and then
even now I mean it's like we are as more
and more pot repositories are created we
are just leveraging this tool more and
more as we as we naturally grow there's
not like a hard department that thou
must do this we see it as a way to allow
you to not have to do this manually so
again it's not a company requirement but
if it's appealing to to the end user to
the maintainer which I would hope it
would be they get they can implement it
other questions got time for at least
one more yes this is member friends to
your repository policy can you elaborate
on how the you guys make decisions went
to update your repository based on
spirit policy so say you're expecting a
three but they put it too I've been in
situations where it's not simply be a
range of values how do you guys come to
that decision you decide how to update
that value so I mean in in that use case
that was definitely that's up to the
team from like approving from two to
three for example I think you were
saying yeah but in a that's a very
simple keep like ranges right
but they put it in two but how do you
know it was supposed to be a three if
there's range one through five how did
you guys go to that process sure I mean
so yeah I think the best way to put it
is that it's it's up to the team I mean
we can have you if you have a really
simple project that a team of response
before they can say hey this project we
don't really care about its internal
it's used in very small areas that's
just going to be one where as this is we
as a team we own this really critical
piece of software we think it should be
five or some number right the for us and
kind of the whole takeaway for this is
it allows teams to make it make those
choices and can figure out how they want
without bringing in the the actual owner
of the overall platform to look at me in
this case a sure so I mean I don't know
how like they would have made it that
decision from I know from my perspective
for on these types of projects that yeah
if we feel that something needs to we
say it's too otherwise it's just gonna
be one or three or whatever the team
decides to do alright thank you very
much big round of applause
[Applause]