I would like to thank everybody for
attending this talk there are a lot of
great opportunities out there and I'm
flattered that you're here I'll let you
know that there are a couple of bonus
slides at the end of this presentation
that I won't have time to cover
however this talk and it slides will be
available on the GDC vault and or you
can just reach out to me directly if
you're interested and also please don't
forget to fill out your evaluation forms
at the end I'm pretty sure I need the
feedback so now that you are here
whether intentional or you just realize
that you're in the wrong room you're
most likely wondering who the hell is
this guy well my name is Peter Dalton
and I am the technical director for Blue
Point Games as you can tell from the
gray my beard
I've been around for a little while I've
been working in the games industry for
over 18 years and I will never leave I
love my job I love the insanely hard
problems and I love the talented people
that I'm surrounded by over the eleven
plus titles that I have shipped my
specialties revolve around core engine
development streaming memory performance
I like to believe that it is my job to
basically make shit happen so Blue Point
games perhaps you've heard of us the
Masters of the remaster I have to thank
digital foundry for that title
essentially the digital foundry guys are
amazing their technical reviews are
spot-on and Blue Point is tired of me
ranting about how our frame rate and
frame pacing must be perfect or digital
foundry will call us out Blue Point has
made a name for itself by remastering
games many of which I'm hoping you've
heard of we've been blessed work with
titles that we absolutely love and from
my perspective it is insanely awesome to
that we get that I get to see exactly
how these games were made at Blue Point
there are two primary ideals that I try
to promote with number one being quality
is the foundation of success I believe
that it is absolutely true for our
industry if you want long-term sustained
success you must have quality to feed it
and number two every project that we
release must be better than our previous
release and hopefully you can see this
by looking at the titles at our past
projects the depth at which we remaster
projects has grown exponentially
starting with the God of War Collection
which involved texture cleanup to the
Uncharted collection which involved all
aspects of the game being modified in
fact for shadow we don't call it a
remaster we coin it a remake given the
complexity of the project and moving
forward to our next project we coin it a
rien visioning given that it goes well
beyond what we even thought was possible
in shadow now I want to take a minute to
show you a video from PSX 2017 showing a
comparison between the original shadow
and the ps4 remake to give you a better
sense of what we do
[Music]
[Applause]
[Music]
I feel like it is important that people
understand a couple of our overarching
goals that we take into each remastering
project number one we have to be true to
the design of the original game number
two we have to respect the decisions of
the original game team and number three
we must not lose the magic of the
original game and if you say to yourself
I can't find the magic in the original
then what the hell are you doing
and number four we have to acknowledge
that it is rarely a black-and-white
answer for what we do and as a result
there will be lots of internal debates
these debates are good and healthy and
most importantly they show passion to
help explain our thought process I like
to use an analogy as a kid growing up I
used to love watching cartoons
who didn't I'm sure if you think about
it you had your favorites certain shows
that stand out in your memories well in
my case I loved Thundercats and have
fond memories however if I go back and
watch them now or try to show them to my
kids
they aren't anything like I remember
what we're trying to do is bring those
memories back to life and enable others
to have similar experiences we're trying
to recreate the game the original game
developer would have released if they
had the technology that we have today so
let's take a technical look at the
development process that we use a blue
point when remastering a title besides
determining whether or not it makes
business sense to remaster a game there
are several key factors that must be
considered number one we absolutely must
have the access to the original east
package for shadow this was the final
package that was delivered to Sony for
distribution this should include all
packed
patches back-end servers basically we
need to be able to run the retail
project in-house and number two we
require all the final source code
required to rebuild the game's final
binaries now there are exceptions we can
work around miscellaneous systems for
example if
are missing an audio library chances are
we're going to replace the entire system
so no biggie however if we are missing
gameplay code it quickly becomes cost
prohibitive to reverse engineer those
libraries others examples include server
code if required as a bonus however we
can work around not having it tool
pipeline code is a bonus however we
really usually don't build our processes
around legacy tools and primarily only
use them for reference and finally any
source assets that were used to build
the original final distribution package
is a bonus but not mandatory so looking
at the development process that we go
through the goal for the first month is
to obtain all the required pieces and
rebuild the guy the game's binaries for
the original target platform we then
send this bill to our internal QA and
run a whole suite of parody tests I
should call out that it is critical to
the success of the project that we get
these first steps correct small errors
here will be magnified as we get into
full development leading to wasted time
and incorrect decisions being made here
we feel that taking time to ensure
correctness is preparing for success
moving forward once we have parody we
upgrade the original title to its latest
SDK to make life easier
then rerun all parody tests basically we
make incremental changes and verify
parody through all steps take shadow for
example very early on we decided that we
were going to use the remastered ps3
version as our base at that point QA R
an extensive parody test between the ps2
and the ps3 version combed old bug
databases search the web for user
feedback etc concurrently engineering
got the game up and running on the ps3
converted to the latest SDKs we
localized all Japanese code comments
rearranged code libraries to make us
happy and then reran all parody tests
and we fix the issues before moving
forward at blue point every developer at
their desk has full access to our target
platform which for Shadow was a ps4 and
the original game which in this case was
a ps3
kid programmers are expected to debug
and play the game on the ps3 dev kit to
determine original code intent and
purpose and to diagnose parity issues we
do the same thing for all other
departments art always has full access
to both platforms and are expected to be
familiar with both and in the case of
shadow queue a went the extra mile and
through the ps2 into the mix only after
parity is verified do we start porting
the code to the target platform porting
is usually a 1 to 2 man project that
takes approximately 2 to 3 weeks before
we have the code compiling during this
phase we are more interested in just
getting everything to compile on the new
platform not porting it basically we try
not to lie to ourselves and pretend that
we know how everything works and instead
we're using this time to familiarize
ourselves with code base we create power
defiance and disable whole system's
markup changes we have made to the
original code with the most important
goals being at number one to get the
code to compile and Link which is harder
than it sounds and number 2 make it
trivial to identify where we have
modified original code so
congratulations after about 2 months you
have a binary that you can lancha launch
on your target platform not only did you
get the game to compile you got it to
link which was a huge pain in the ass
but you can't help would want to play
the game so you hit run and boom
everything blows up in fact the
breakpoint that you put at main never
got hit you blew up during pre a minute
the call stack doesn't make any sense
and everything feels broken memory is
being requested you have 32 to 64 bit
compatibility issues plenty of code
assumptions that a pointer fits in a UN
32 and resource files are being
requested in fact where the hell is the
file system and you have Indian issues
everywhere and how do you deal with
memory map files that assume a 32-bit
pointer at this point you can't help but
wonder what you've gotten yourself into
and does
make sense to assign this task to
another engineer at this point I must
tell myself to stop being a little bitch
and just keep pushing forward so given
that the first step is just to get the
game to compile on the target platform
the next step is to get the game domain
and then to the main loop the main loop
will be pretty much commented out at
this point however our goal is to get
the game running displaying nothing but
a black screen getting to a black screen
as a critical milestone for us it is
really the point at which the entire
team of engineers can really start
piling onto the project and looking at
their respective domains it is the point
at which momentum momentum really starts
to pick up so I want to stop for a
second and I want to share a little bit
of bluepoint history with all the
projects that we've worked on before
Shadow of the Colossus we would work
within the realm of the original
codebase we would rewrite low-level
platform code to work on the new target
platform we would build a new rendering
system to handle the needs of the game
tools and processes would be adapted to
closely fit those of the original
development team the problem was that
with every new project it felt like we
were completely starting over code would
end up being very tied to a specific
title and that's not moved from project
project the team art engineering
including QA only did to rerun leap
relearn processes as we moved from game
to game basically it started to conflict
with our ideal that each project needs
to be better than the previous so for
Shadow of the Colossus we completely
revamped how we work we made a point of
building reusable technology and
processes that evolved from project to
project
rather than being recreated this
requires a deeper commitment to
processes and long-term planning rather
than just solely focusing on immediate
goals so we took our blue point engine
dusted it off and started to make a
major investment for reference the blue
point engine is a proprietary engine and
has been around for quite a while in
fact it has been licensed and used to
ship several titles how it was time for
it to evolve now getting back to the
goal of getting the game running we
achieve this goal by
urging the original code with the blue
point engine basically we structure the
code libraries so that the original game
is built on top of the blue point engine
this allows the blue point engine to
provide all platform centric systems
such as memory threading file i/o
rendering etc for us it is key to keep
in mind which original game systems we
want to port to the target platforms and
which ones will be replaced by blue
point systems basically we don't want to
waste time porting a system that will
just be replaced in the future with that
said I think it is important to make one
clarification okay when it comes to
gameplay systems AI character logic etc
we want to keep all of the original
systems intact well we will fully
replace core systems with gameplay we
take a much more surgical approach to
fixing and enhancing next I want to take
a minute to discuss blue points approach
to game assets well it is ideal that we
get the source assets from the original
game team and we do to varying levels of
success we do not depend upon them
rather we only use them for reference
the reason that we don't use source
assets is a starting point is that
they're often wrong or incomplete teams
are notorious for not checking
everything into source control often
local file changes or P for shelf
changes are used to build the final
package creating very difficult to find
parity issues this is exaggerated and
with dealing with patches as the
likelihood of a one-off local change
increases to eliminate this concern we
spend around two months extracting all
the data within the original
distribution package to BBE compatible
formats we also assume that before we
ship we will have the need to edit every
type of data not just key file formats
well this is time-consuming it has
several key benefits number one we know
that the data that we have is exactly
the data that's shipped and number two
we actually learn a lot about how the
game is constructed allowing us to make
smarter decisions moving forward
standing the content greatly helps us
understand the code enables us to make
decisions that work with the original
code as opposed to fighting it and
number three once we have extracted the
key file formats and converted them to
be PE formats such as models animations
skeletons textures collision data these
assets are immediately available within
our B PE toolset and finally because we
are taking full ownership of the data we
are no longer dependent upon the content
pipeline of the original team we don't
need their tools and we don't need to
follow unfamiliar practices it is
actually quite liberating that after you
the extraction is finished we follow
BluePoint processes rather than the
processes of a remote team that we don't
understand now to be completely
completely honest there are some game
specific file formats that we don't
always extract and will take shortcuts
we do however regenerate these files a
dressing like Indian as' and 64-bit
compatibility issues and we spend the
time to understand and document their
purpose for example in shadow we do not
fully convert all of the pathing data to
be PE formats because we figured that we
would never change this data this came
back to bite me in the ass and I'll tell
you more about it later
so let's take a look at the final
results that we achieved in shadow in
relationship to how we integrate two
distinct engines I like to call this the
dance okay without proper planning and
coordination chaos would take over
however with each engine assuming
specific responsibilities we can create
a harmony of technology what this
diagram is trying to illustrate is the
responsibilities of each engine the
errata the original shadow engine
maintained sole responsibility for the
majority of all gameplay if there was a
gameplay bug or a behavior that we
wanted to modify chances are that we
were going to modify the original shadow
codebase on the other side the blue
point engine handles all core system
responsibilities memory threading
platform services
along with anything visual the bluepoint
engine handles the management of the
world the static geometry in the world
the particle systems placed the lighting
etc it remains the responsibility of the
shadow engine to create all dynamic game
objects such as Agri wander and colossi
within the Blue Point engine in order to
build a complete scene which in turn
feeds the render take for example wander
the main character in the game the
original game has a concept an object
representing wander and also a link to a
BP engine representation of wander the
original game provides all the
simulation logic and pushes matrices and
the required state information over to
the blue point representation in turn
the blue point representation can add
additional functionality such as head
tracking and drives the render to ensure
that everything shows up in the final
scene so let's take a look at the game
and break down a scene starring Colossus
6so who is responsible for what shadow
is responsible for all dynamic
characters in the scene in this case
wander and the Colossus shadow is
responsible for AI behaviors when QA
would report behavioral problems we
addressed it within the original code
base shadow is responsible for collision
in retrospect this was perhaps a mistake
and should have been brought over into B
PE in order to make it easier to work
with and shadow is responsible for
building the animation blend trees and
building the final pose Shadow has these
responsibilities because the simulation
is dependent upon the final pose it
would have been better for us to use B
PE systems for animation however at the
time intertwining shadow and B PE
processes was very difficult the eye
state of the Colossus was determined by
shadow code past a B PE code where it
was managed used to determine the
correct eye color and drive dynamic
shader parameters it is the Shadow code
that handles the simulation of the bones
that hang around wanders waste these
were never lifted into B PE given that
the original coder did such a great job
so that is about where the original
shadow code
and the bpe engine takes over bpe
handles all rendering and scene
management the entire environment in
atmospherics that you see are all
handled by bpe
the physics simulation of wanders poncho
is handled by BPB bpe it is purely
visual and needed improvement in this
all aspects were removed from the shadow
code and moved into Blue Point bpe is
responsible for the dynamic rings
attached to the colossi in the original
the Rings are static and misaligned they
are now dynamic and finally bpe is
responsible for all audio and particles
within a scene one way to think about
our approach is that we are taking the
original game and overlaying visuals and
enhance gameplay
here's another scene with a path nodes
debug display enabled the blue lines
simply show available connections
between nodes while the purple line
shows the navigation path that a grow
your horse will take to get to you when
we started the game we figured that we
would not change the layout of the
environment enough that we would have to
modify collision this game this became
perhaps the biggest lie that we told
ourselves before shipping all collision
within the game was completely rebuilt
we also told ourselves that because we
were not significantly changing the
layout of the environments there would
be no need for us to modify path
information
this was also a lie as the environment
started to be finalized QA started
flagging issues where aggro could no
longer path to you because someone
placed a large tree on top of the path
node in certain areas wonder could not
find lizards because they were now
hidden under heels in other areas the
hawks and the birds in the game would
start flying through mountains that were
now in their pass after a bit of digging
it became obvious that it was not an
acceptable solution to require art to go
back and fix the geometry to match the
constraints of the original instead
within the last two months before
shipping we wrote a tool that allowed
for path nodes to be visualized and
edited directly in the game and
serialized back out before this the
pathing logic was the last piece of
untouched code and data well we didn't
necessarily set out to change every
single piece of data loaded by the game
it became an asset
so I guess the moral of the story is to
assume that if something can bite you in
the ass it will has a pretty safe mantra
for anyone in engineering so shado
released a year ago this February i
explicitly don't want this talk to feel
like a post-mortem but more of a sharing
of our approach and processes my goal is
to be completely transparent and share
what worked and where we fell short well
on shado we had a fairly sophisticated
worker job system we were not able to
retrofit enough code to take full
advantage of it we had this notion of
the shadow code completing its full
simulation before moving on to handling
bpe game objects the CPU frame was
broken up so that shadow took
approximately 70% of the game frame and
the blue point engine took about the
other 30% we could find areas to
optimize shadow code by Java fiying
however this basically just moved code
from one court to another and really
without really increasing the CPU
saturation as a result we do not we did
not get much parallelization within the
game code within the blue point engine
we represent a scene as a tree of game
objects where each game object contains
components that exposes functionality to
update a scene we would walk the game
object tree and update all components
the overhead of walking the tree quickly
became time time consuming to make
things worse we would actually walk the
game object tree a second time to build
a list of items to render this happened
during the sync point between the game
and the rendering threads creating a
major bottleneck well towards the end of
Shadows development we implemented a
couple of Hail Marys to address the
problem they were band-aids at best
moving forward to our next project we
knew that we needed to re-architect how
we coordinate the game loops between the
two engines basically was time to take
what we had learned and evolve the dance
the major breakthrough for us came when
we decided to stop treating the original
code base as special but rather is just
another think process that
happens at a specific point in the frame
you can think of the entire original
shadow game code as a component on a
game object exposing functionality and
fortunately this is easier said than
done to evolve to where we are today
beyond shadow we had to make the
following changes number one the first
thing that we did is we eliminated the
need to walk game objects to determine
what needs to be rendered rather we make
components responsible for adding and
removing persistent render items and we
support a command buffer style interface
for changing render item parameters to
keep everything thread safe number two
we no longer walk the game object tree
to update components rather components
register think requests delegates think
requests can be added or removed at any
point from any thread a single object
can submit as many think requests as as
desired based upon demand this makes it
very easy to create objects that only
need to think for one frame or under
very certain or under certain conditions
think requests can also be invoked at
any point if not explicitly invoked they
will be invoked when their bucket is
processed however it is trivial to be in
the middle of a think request start an
async job that you need to wait for and
invoke other think requests to ensure
that we never stall with these changes
and looking back at shadow now rather
than merging the shadow game loop with
the blue point game loop we would simply
have the shadow code register a think
request within the correct bucket
eliminating the distinction between the
shadow update and the bpe update when
the original shadow code needed to wait
for async jobs it would simply start
processing other think requests rather
than stalling this type of behavior is
proving to be critical on our next
project in fact this last week I spent a
day rearranging think request
dependencies to fix stalls saving
approximately 2 milliseconds within our
game loop I didn't optimize any code
rather I simply fix scheduling issues to
increase CPU saturation having the
flexibility to easily rearrange a frames
work to fill dead areas and remove
contention is proving to be awesome so
here's a
diagram showing our current running
layout it is much easier to use and a
lot more flexible at this point my only
regret is that we didn't implement this
system earlier so rather than looking at
shadow let's take a look at how we have
evolved and what's next
so this sorry I hope you didn't think
that I was going to expose our next
project I know this is a fight well
maybe it is a little bit to me well I'm
super excited about our next project and
the tech that we are building I'm pretty
sure BluePoint would make me walk home
all the way to Texas just to shoot me if
I screwed this up so next I want to talk
about what we have learned from past
projects and how it affects our systems
how we go about creating a harmony of
technology by doing a deep dive into our
memory system how it is designed with
flexibility and what steers those
decisions before I do so here's a quick
disclaimer all of the examples that I'm
going to share are based upon my
personal experiences and memories and my
examples are based on outdated
technology from the companies in
question and don't represent their
current technology or processes and
finally I apologize in advance if I
misrepresent anyone's work so memory who
needs it I'm pretty sure I do I've been
working or dealing with memory systems
since I started my career over this time
I have tried numerous approaches and
have shifted the way I view memory based
upon the games that I have shipped in
fact I wrote an article for game
programming gems years ago as I look
back I still embrace the ideas presented
but I'm a bit embarrassed that the
inbloom implementations were so
short-sighted
I actually got my first job in the games
industry working for beyond games
creating a Hot Wheels game the key to
landing this job was a PSP collision
system that I had written that require
less than half the memory footprint of
the current system basically cutting
memory requirements from 6 to 2 Meg's on
a ps2 if there's one lesson that I
learned then that is just as true today
it is that proper memory management is
critical to achieving performance and
shipping titles so let's take a look at
blue points hardest title to date
shipping titanfall on the Xbox 360 what
an amazing game respawn created and if
you buy me a drink I have a lot of
stories if you don't recall the original
titanfall was developed by respawn and
released on the PC and Xbox one Blue
Point released the Xbox 360 version
approximately two months later simple
summary if I ignore the performance
aspect it is not easy to ship a 5 gig
Xbox one game on an Xbox 360 with 512
megs so how did we do it a lot of hard
work or as we like to call it Blue Point
magic the original game used a fixed
sized bucket allocator for allocations
less than 32 bytes and dl malloc for
everything else where all memory
allocations were funneled into a single
allocator and dispatched from there
creating a choke point where thread
contention became a major issue I know
that a lot of people love DL Malek and
argue that it is sufficient well has a
lot of great features in practice on
titanfall we found that number one the
version of DL Malek we use did not have
any concept of virtual allocations and
thus a custom system was required number
two dl malloc was not ideal for large
page page aligned allocations DL malloc
works by placing tags at the beginning
of allocations if you need a texture
with 128 byte alignment the tag can
create memory waste and number 3 due to
DL malloc not being ideal for small
allocations virtual Alex or page size
allocations we need a DL malloc to
release unused memory back to other
systems in practice we found that DL
malloc held on to more memory than we
thought was ideal
the kita shipping titanfall from a
memory point of view was memory tracking
knowing exactly where all memory was at
all times
tracking fragmentation issues and
tracking memory by category
so that the peaks could be compared from
build to build to determine trends from
there we were able to optimize data
formats and nearly every system to cut
memory usage we reworked the memory
system - we used a fixed block size
allocator for small allocations less
than 512 bytes we use DL malloc for
medium-sized allocations and several
changes were made to DL malloc to get it
to aggressively release memory and we
used large large page based allocator
used primarily for textures and large
vertex buffers and we added a concept of
a single frame allocation where small
allocations would be placed within
unused DXT mipmap memory this memory was
only valid for a single frame and
coordinated with the texture streaming
system basically every byte was used by
the end we had spreadsheets backed by
more spreadsheets for every level
showing max memory usage how much memory
was available for textures streaming etc
well I'm still in shock that we pulled
it off looking back the whole system was
perhaps a bit more complicated than it
needed to be so uncharted what a great
project going from a ps3 target to a ps4
target how could there be any memory
considerations well there weren't any
real memory considerations provided we
kept streaming textures reasonable
however there are a couple of key things
that we learned number one all memory
was allocated at startup by the core
memory system from here all memory was
assigned to specific allocators there
was a hard coded table within the code
dictating how much memory allocator was
allotted the downside is that when an
allocator ran out of memory you would
play the russian roulette game to push
memory around until the problem went
away often over allocating there was no
virtual memory support and fragmentation
issues were evident there was code to
handle the shuffling of memory and
pointers one defragging however it came
across as air prone and touched numerous
systems the later Uncharted games
adopted a rule that memory could not be
allocated before hitting main I loved
this ideal given that it was
straightforward when to
create your memory allocator and at the
end of Maine you can easily check to
ensure there were no memory leaks and
finally uncharted introduced the idea of
a tagged heap allocator which is very
similar to a single frame alligator that
we implemented for titanfall just
evolved so what were some of the
philosophies that we felt were important
when designing the memory system for the
Blue Point engine number one we should
never run out of memory never expect no
from an allocation request in fact we
explicitly halt the game if this happens
number two we need smart allocation
schemes to allow for a wide variety of
allocation patterns basically encourage
small custom allocators rather than an
uber allocator number three allocators
should help to eliminate thread
contention and for ensure debug ability
across all allocators without a lot of
custom work to us debug ability in this
case means the ability to track and
categorize all memory allocations and
tools to help detect and diagnose common
memory issues with the most common being
memory stomps with these basic
philosophies in place we built a list of
our goals and I'm only going to touch on
a couple of the key points number one we
determined that all allocators should be
platform agnostic we didn't want to
write custom allocators for each
platform and allocator should deal with
memory patterns not the specifics of
where the memory came from number 2
allocators should be memory agnostic
basically every allocator should work
with both CPU or GPU memory well this
seems trivial it does have implications
during shadow our per frame dpu memory
allocators benefited greatly by removing
tags and markers that were being written
directly into the memory blocks for
memory tracking assume that memory will
be allocated at any point programmers
need power and flexibility to create
great systems number 4 we wanted to
eliminate the need for fixed sized
allocators basically allocators should
work with fixed memory blocks but also
should support growing and shrinking as
required we want to avoid over
allocating
and the memory russian roulette game to
get started let's look at how we ensure
all memory allocations are routed
through our memory systems we start by
overriding system memory routines which
is pretty straightforward and second we
create bpe macros to wrap all memory
requests we choose to use macros for a
couple of reasons number one they're
simple straightforward and require no
additional steps number to some sort of
redirection is required to support
malloc and free and virtual outlook will
also require custom wrappers to be
platform agnostic so it's nice to sort
of standardize everything and then
macros also allow us to provide
additional features such as location
tracking and passing additional
parameters directly to memory requests
such as alignment requirements and
requesting specific allocators
converting to using macros is not a big
undertaking I've done it about three
times in my career and each time it
takes about a week if you do not
redirect and control all allocations
within your code I highly recommend that
you start and using macros as the
cleanest method that I've found and even
if you forget to wrap a new or a delete
it doesn't matter it'll still get picked
up to overriding the system memory
routines and finally to ensure we
redirect all memory routines into our
system we simply control all memory on
consoles we allocate all physical memory
upon initialization as a result if we
miss an allocation due to a third-party
library there simply isn't any memory
for it to request talking about
third-party libraries whether audio
physics video playback or anything else
I think there are a couple of mandates
that I believe that we should all insist
upon number one being if a library does
not provide direct control for how it
gets its memory don't use it if you
can't control the threading behavior of
a library don't use it and if you can't
control how it loads files don't use it
there is one third-party library that I
would like to call out that we use
extensively the EA STL which is a
replacement for the standard containers
I hope that anyone that has been in game
development for any amount of time has
some at least an opinion on the debate
of whether or not to use standard
containers for us we don't however we
have found an awesome replacement that
provides a lot of really cool features
that while not always safe allow for
smarter code if you haven't checked out
the EA STL I strongly suggest you do so
let's take a look at a diagram
illustrating the flow of memory requests
as you can see it all starts with
overriding system allocation routines
which are redirected into our memory
coordinator from their memory requests
are funneled into appropriate allocators
these allocators manage their own memory
pools to service memory requests and
fallback to requesting memory pages from
the platform page allocator note that
the only system that is platform
specific is the platform page allocator
this helps to ensure allocators only
deal with memory usage patterns and
ensure that we get consistency across
all platforms so the MC contains some of
the hardest-hit code in the entire code
base its primary purpose is to redirect
memory requests to the appropriate
allocator all without creating thread
contention thus it must avoid locking
mechanisms we are able to achieve this
goal by ensuring that the MC is
basically stateless and that the few
stack based variables that it forwards
into the allocators are all stored using
thread local storage there's one caveat
to keep in mind our threaded job system
is constructed using fibers that we
switch in and out because we are using
thread local storage we need to prevent
fibers from picking up the wrong set of
variables one switched so within our job
system if we perform a contact switch
and switch out a fiber we also create a
copy of the memory local thread
variables then when the job system
switches back to the previously
suspended fiber we were we restored the
local thread variables that's keeping
everyone happy and consistent by using
thread local variables not only do we
allow memory requests to be redirected
to any specific allocator
it is also at the heart of how we
categorize memory in our engine every
single allocation is categorized even if
it is just categorized as a general
allocation our strategies once a
category starts to account for too much
memory we simply start splitting the
category and refining our tracking and
finally the MC gives us a great single
location to track all out of memory
issues eliminating the need to spread
that code throughout the entire code
base or throw out the allocators one tip
that you can see from the code snippet
is that we create a special function
that is never in line to handle out of
memory issues this has actually been
really great if QA ever encounters a
crash it is trivial to determine if it
was an out of memory issue by directly
looking at the extracted call stack no
need to look at the log or try to
interpret line numbers the allocator
or as I like to think about it the heart
of the system this snippet shows a
portion of the interface that all
allocators must provide the primary idea
here that I want to call your attention
to is the fact that we declare all
memory routines is protected basically
we want to prevent code from directly
bypassing the memory coordinator this
helps to ensure that all allocators
follow consistent conventions and are
properly registered with the memory
coordinator I mentioned earlier anytime
an allocator needs to grow its memory
pool to service a request it requests
new memory pages from our platform page
allocator for shadow on the ps4 we
determined that 64 K pages were the
ideal size we also aligned all requested
pages to 1 Meg boundaries we wanted
smaller pages to minimize waste and
fragmentation but also minimize the TLB
issues for us 64 K pages was the sweet
spot I also want to point out that when
coding allocators we strictly avoid the
use of mutexes while allocators need to
deal with threading issues we stick to
using atomic locks these are
significantly faster and the restriction
of not being reentrant is easy to work
around so
let's take a look at a few of the
different types of allocators that we
support The Dispatch allocator is the
default allocator that the majority of
all allocations get redirected to it is
a very simple redirector that basically
looks at the alignment and size
requirements of a request and forwards
them to the appropriate allocator we use
a bucket allocator for small allocations
we use a heap allocator which is similar
to DL malloc it uses an intrusive
red-black tree to store free blocks and
uses a heuristic of preferring recently
freed memory followed by best fit then
adds headers and post headers to the
allocations to provide tracking
information and to round it off we use a
page allocator for large memory requests
that fit nicely into our 64k pages we
also have several special use allocators
a simple ascii allocator that is only
used by tool code there is a virtual
page allocator to handle virtual memory
requests and there is a frame allocator
which is great for packing multiple
requests together to ensure memory
coherency and creating small blocks of
reusable memory and let's not forget
about the tagged frame allocator which
is awesome our tagged frame allocator is
based upon an allocator described by
christian Gerling in his 2015 GDC talk
the basic idea behind the tagged frame
allocator is that we are creating and
extremely fast allocators where
allocations are only valid for a short
period of time these are basically
temporary memory allocations that are
valid for n number of frames where a
frame is arbitrarily defined for Egypt
allocator the fact that we don't need to
track the memory - or call free to
release memory is a huge win not only is
the code requesting the memory easier to
write the tag frame allocator can take a
bunch of shortcuts knowing that
individual free calls do not need to be
supported it is also impossible to
create memory leaks with this allocator
as we discussed earlier one of the main
one of the primary goals behind
allocators is that they should help to
minimize
read contention this is achieved using
two strategies number one we use atomic
locks rather than mutexes and number two
by using thread-local storage several of
our allocators use the pattern
illustrated in the code snippet for the
tag frame allocator this allows it to
service memory requests for multiple
threads simultaneously without ever
locking if you've ever had to deal with
multi-threaded performance this should
jump out immediately as a huge win
within the blue point engine there are
more tagged frame allocators than any
other type of allocator each tag frame
allocator is built on top of a tagged
framed arena which allows all tagged
framed allocators to share a common
memory pool eliminating over allocation
issues the tag frame allocator order the
tag frame arena will also grow and
shrink upon demand thus all pool sizes
are dynamic if you don't have any
something similar to this in your code
base I would highly recommend that you
make it happen
and finally the platform page allocator
or the brains of the operation as we
have discussed this class is responsible
for abstracting away the details of how
64k pages are managed in practice we
restrict anyone from accessing this
class directly the only customers of
this class are the memory allocators on
windows this is a very simple class and
is built using virtual Alec the ps4
version is more complex due to platform
considerations at the heart of both
implementations is the concept of
virtual memory and mapping physical to
virtual addresses on-demand if you're
not familiar with the differences
between virtual and physical memory you
might want to investigate so looking at
the platform page allocator there's one
last problem that I would like to
discuss which is a problem in every non
uber memory system that I've worked with
given that we are avoiding an uber
allocator and embracing numerous
allocation patterns we need to be able
to take any random memory address and
determine which allocator it belongs to
this is required to properly direct free
calls
well bpe macros allow us to optionally
specify which allocator owns memory when
calling bpe free this is not ideal
instead we need to support the get
memory owner routine so how did we solve
this problem we through memory at it we
break up the problem by realizing that
we only need to determine which
allocator owns the memory page that
contains the memory address in question
within the platform page allocator we
use a 1 Meg block of memory to store a
direct lookup table which Maps one
terabyte of virtual memory to its owning
allocator if you have a better solution
I'd love to hear it so what do you get
when you put this all together hopefully
a game without performance issues and
runs within retail memory ok that isn't
going to happen but hopefully you have
the flexibility and tools necessary to
get there from a memory perspective
here's a screenshot showing our memory
statistics you can see the platform page
allocator at the top the top and each of
the various allocators below here's
another screenshot that shows memory
category tracking I want to call your
attention to the outline black boxes
that I've marked up most consoles have a
concept of development memory basically
the development kit has more memory than
the retail kits we make it very easy at
all times to see where memory usage is
in regards to a retail kit from the
screen shot you can see that the closest
that we came to running out of memory on
a retail kit was 130 Meg's or the
equivalent of the max memory of four ps2
games there's no way we could run out of
memory so to finish up I want to share
one additional feature in Shadow of the
Colossus is related to memory and made a
significant impact on development
texture streaming I'm sure many of you
have texture streaming solutions we use
GPU based feedback to determine what to
strain and specify a sixth streaming
texture memory budget foreshadow we
initially set this budget to one gig and
after some tweaking most of the game
looked great
some areas required a higher
extra budget to facilitate its needs and
some areas would simply crash due to
running out of memory before even
hitting the one gig budget it was about
halfway through the development that the
team began to stress about memory and
started getting flashbacks of previous
projects so how did we solve the problem
we set the texture streaming budget to
unlimited problem solved and now
everyone is happy instead of relying
upon a fixed memory budget the texture
manager monitors the total available
memory by querying the platform page
allocator in acting appropriately if
there is extra memory stream in more
textures if no more textures are
required then there's nothing to do and
if memory is low start releasing
textures I should note that this does
require a bit of balancing to avoid ping
pong effects and to determine proper
memory thresholds however these
thresholds are game specific not area
specific so they only needed to be
calculated once this was really a major
turning point for stability as out of
memory issues basically disappeared of
course we still had to deal with memory
usage issues where memory was scarce
however these areas only needed to be
pushed to an acceptable level and the
texture manager would Auto calibrate
moving forward we are pushing our
texture streaming solution beyond what
was achievable in shadow by prioritizing
and loading even higher resolution
textures when memory allows so here's a
screenshot showing texture streaming
stats at the top of the screen here you
can see that we have over two gigs of
textures loaded however the scene only
actually requires about 680 Meg's of
texture data as the texture manager
tries to maintain a buffer of about 220
Meg's free here's a more demanding area
there is 1.3 gigs of textures loaded
however only 1.1 three gigs are required
during development the art team would
play the game watching these statistics
to verify that the budgets were
satisfied by the end the
questionable area in the game was the
final cutscene where the secret garden
is exposed and of course the minute you
tell anyone that the texture budget is
exceeded you need to be able to show
them why and what is loaded it is only
then that you can properly debate how
big must the eye texture be now get that
and good so once again I would like to
thank you for your participation I'm
flattered that I've had this opportunity
to ramble on and yes Blue Point Games is
hiring basically if you are a badass and
you know how to shit how to get shit
done we want to work with you use the
email address on the slide and mention
my name I'll let everyone know that we
are best friends so I can get the
recruiting bonus and then perhaps it'll
be time for me to buy you a beer
also remember to fill out the speaker
evaluation forms so I know whether or
not I should ever do this again with
that I believe I'm done I believe we
have eight minutes for questions if
anybody has any questions
[Applause]
I love it the perfect talk there's no
questions that's perfect let's wrap it
up get the hell out of here I had a
question yeah are you talking about
being locked free using Atomics and your
allocators aha so we still live in a
world where there's no standard lock
free container library that you can like
do you guys did you use a third-party
one did you roll stuff in house yeah so
for the atomic locks we basically just
sort of rolled our own just using you
know the drawing a blank right this
second but basically just using the
Atomics that are available on the
different platforms it really comes down
to like four or five lines of code if
you're to send me an email like it sent
it to you it's real simple sure now I
mean did you didn't do unbounded
spinning though did you for some of your
locks they will spin yes okay
did you ever see stuff where like I feel
like spin lock is it's all fun and games
until you know your spinning core
scheduled and the work you need to be
done that it's waiting on to get the
lock back is it's not you know have you
did you see some of that not really and
in part of it as well as any time that
we use Atomics so we'll use the Atomics
anytime we want we've got a really small
portion that we want to get in and out
of really fast and we also try to lock
it or try to make it so that we're
really not calling other functions
inside of it so we don't run into this
case if I got a lock here I call this
function and then this guy wants to lock
and so they're not reentrant and so we
keep those sections very very small and
so we really haven't had any problems
with them or having problems of
something waiting for any duration of
time before it could get back in we also
do a lot of like performance profiling
looking to make sure that in order to
maximize CPU saturation across all of
our cores making sure that they're just
for that fact that there is nobody
that's stalling or waiting for too long
yeah I guess I've seen when spinning
goes bad is usually when you're over
subscribed did you work hard to avoid
oversubscription on the platform and
stuff like that yes okay I would say we
work hard to avoid it okay
you said your your next game is gonna be
more of a rien visioning I was wondering
if we have some color on like what that
kind of means what is what is a rien
visioning versus a remake sure so if you
were to take a look at like Shadow of
the Colossus like it's a ps2 game right
and so when you when that was built the
side of the hills were very low triangle
density like what was the environment
what was the atmospherics that they're
actually trying to achieve and when you
look at it a lot of it you're feel like
your mind fills in a lot of those blanks
you look at a green card that's filled
on the back and you're like yeah I
assume that's a grass plane right and so
for shadow there's a lot of us looking
at that and then building it out sort of
trying to remake it but sort of figure
out what the original was trying to do
as we move forward in some of our other
stuff we're taking stuff further with
not just you know trying to get the two
engines to run simultaneously but more
of bringing more and more stuff into our
engine and repurposing our technology so
that we have greater control of you know
redoing than complete animation systems
upgrading you know besides graphics and
other things but even starting to
upgrade more of the gameplay pass and
such which are all for us are always
very very tricky we've been very very
fortunate to work on games that we've
absolutely loved and foreshadow like you
wouldn't take that game and say I want
to make it like Assassin's Creed running
around doing stuff like you wouldn't do
that and so you want to maintain core
pieces so it's a very difficult question
but it's just take shadow and I feel
like we're about doubling the amount of
involvement that we're putting into it
thank you I am oh that's yes yeah how do
you deal with framerate issues
especially with doing the sort of hybrid
approach you to quit like two game
engines running at the same time so how
do we deal with framerate issues we as I
mentioned anybody a blue point is Herbie
rant about digital foundry at least a
bazillion times because I don't know if
you've seen their evaluations where they
do the
right graph and they show it we're going
straight across the screen and I tell
the team if we have a single dip they're
gonna find it and I'll lose my mind and
so we do a lot of profiling we our QA
Department is great they take our our
tools in our third party like ps4 tools
and they'll have graphs that are running
on their machine so they're playing the
game with the framerate graph and the
minute they see any sort of spike or
anything like that they go and they
capture it create a bug and off it goes
to engineering on shadow we have a 60
FPS mode on the pro kits in performance
mode that was not easy hitting that it's
a lot of testing a lot of looking at CPU
captures and exactly what jobs are on
what cores when and why and you know
trying to load balance that out and for
our next game the big thing that we're
really looking at is how do we maximize
our CPU saturation and I feel like we've
got a good job system to break stuff up
but you find at the same time like you
during your frame
you've got areas that here's the hot
spot everybody wants this last third
quarter of the frame and nobody's doing
anything over here so a lot of analyzing
and rearranging code in order to get it
to fill in all those spots yeah I guess
the second question is how when you
convert from like 30 FPS game to 60 how
do you deal with it
like game breaking issues oh yes so the
original game ticked the update at a
hard-coded you know 30 FPS and so on the
ps2 I don't know to be honest I don't
know if the game actually ever hit 30
and so yeah the minute we went to 60 or
the Myint we even locked it at 30 we
found that you couldn't can't climb the
colossi as well yeah your grip would
fall before getting around some of the
edges was twice as hard and so I really
got to you know give credit to that to
our QA Department they test the game
like no other and are looking for parity
issues they identify those sort of
issues and then we go back and we fix
them and a lot of times programmers love
when they look at a bug and they're like
I don't know why it's behaving this way
and it's like we'll go play it on the
ps2 on a ps2 dev kit and it's like I
don't even know how to make that thing
run anymore
right but it has to be done okay thank
you
hey Gretel there's a lot of issues that
speak close to my heart so I was
wondering if you have any solution and
utility libraries or whatever to deal
with the 32-bit to 64-bit address space
conversion any libraries that deal with
32 to 64 bit kind of like injection or
you know remapping of the 32-bit address
space into 64-bit or something like that
to ease the process of porting and
moving to a 64-bit platform yeah we
don't really use a third-party library
to do that it's a lot at the end it
comes down to us doing a lot of hard
work and updating code the one thing
that I would throw out that I didn't
really talk about but is mentioned in
the slides is what we use is we call the
tea offset pointer which basically
allows us to represent any address as a
32-bit number and then when you actually
do the look up when you dereference it
to a pointer we actually add an offset
to it we use that a lot in like memory
mapped files that have to do that so
that we don't break everything right out
of the gate but even then by the end
they get annoying because now every
single pointer dereference is basically
a dereference and it's adding and I know
that is a stupid shit like that just
like why are we wasting processor power
drives me crazy and we end up going
through them fixing most of them yeah
thanks yeah I'm curious about you have
the blue point engine and then you've
got the game engine and I presume you
take the blue point engine from project
project that is correct and you have the
the game engine does it call directly
down into the blue point engine and how
do you keep the blue point engine side
of things from not having references up
into the game that have to be cleaned up
from project to project right so you
could say I've been writing code for a
really long time and you can say there's
certain thing
like coding standards you know it's like
a religion what was I the fence do you
fall on there are certain things with
coding standards that I strongly believe
in and drive me crazy one of those is
dealing with library dependencies if
I've got a base library and I've got my
game library there's no way in the
codebase that I allow the game or the
core library to start calling up and so
we make very explicit rules about what
library is built on top of what so we
have a very clear hierarchy then it's
going through and we'll add layers in
the middle that's sort of like here's a
wrapper piece that is basically the
bridge between shadow code and the bpe
engine working with our components and
game objects and it can basically go and
do like reach into shadow code or shadow
code reaches into wit a lot and then I'm
I'm okay with that because at the end I
know that whole piece gets thrown away
and so I don't occur that debt so it's
essentially just a wrapper for it so
that the game can preserve its original
engine calls that wrap to calling into
the actual blue point engine and if you
think about it like any game engine it
all boils down to you've got a main loop
somewhere you've got a forever loop
that's looping doing whatever and then
the blue point engine similar has got a
forever loop that they're both basically
looping well now rather than looping I'm
just gonna make this one when it loops
it calls in over here does its thing
comes back out over here it goes back
over the top and calls and so it just
sort of ping pongs and the next thing
you got to do is come up with areas of
responsibility and so then you start
going to the original game engine you're
like no you don't do anything with audio
you don't do anything with rendering you
don't do anything you know dealing with
scene management that's all going to be
over here that's that's the the hardest
part is defining all those rules and
levels of responsibility thank you