thank you all for coming to my talk on
spider-man there's a lot of ground to
cover today so I'm gonna go really fast
you will be super stressed out if you're
trying to take pictures of every slide
you're not I will put them at that URL
sit back relax do not panic
also remember to send in those evals so
I can get on a deck of cards
and if you have anything that beeps
silence it today I'm gonna talk about
the biggest game we ever made our
biggest city our most populous city the
greatest responsibility we've ever faced
and well you know insomniac games is 2
Studios one in California one in North
Carolina we are about 250 people across
multiple projects Spiderman took us
three years to make the engine we use
began life more or less on fuse it has
been used on to open world games sunset
and Marvel's spider-man but one of them
is a much more open world than the other
our Marvel's Manhattan is about five and
a half kilometers from north to south
and that's not the only thing that's
bigger basically everything was bigger
about Marvel spider-man
our source assets about 3.3 terabytes
335 gigabytes of textures 151 gigabytes
of models a hundred and fifty Meg's of
source code somehow and that is several
times larger than sunset overdrive so
today I want to break down one frame for
you I'm gonna talk about how we built
the city how we stream the city how we
fitted on disk how we filled the streets
something else best place to start at
the beginning I'm sorry the whole story
would take forever to get through some
just gonna talk about these five rigs we
stepped on and the best place that
started at the beginning so 3 2 1
[Music]
alright let's break down a frame 30
frames a second 33 milliseconds of frame
PlayStation 4 has 6 CPU cores so we have
a main thread or render thread and for
workers we render a frame behind the CPU
of course I'm lying we can share half of
a CPU with the operating system we use
it for the loader and we render more
like 1 and 1/3 frames behind this in
because we do a post processing at the
start of the frame I'll get back to it
and instead of having 4 workers we
actually have 3 tiers of priority times
4 workers and we also have audio on its
own threads which we let run will
harbour it needs to because it needs to
interrupt things or else it pops here is
our frame from 3 milliseconds the top
one is the main thread the second one is
the render thread the bottom two are a
low and high priority worker thread that
share the same CPU and sorry and let's
start with the main right so the first
thing that the main thread does for
about 2 to 5 milliseconds of frame is
deal with streaming we are either
loading assets or getting rid of assets
or digitalizing assets so like spawning
actors or getting rid of actors we're
always doing something because spawning
everything that that needs to be active
when you come into a block takes many
many milliseconds it's too much to do in
one frame so we just amortize that work
also I've understated the godom where
people have a very interesting way of
dealing with this they should go look at
their talk in the vault that's the
pipeline one I think it's called we
spend about 10 milliseconds on gameplay
logic spread across 5 threads spend
about 6 milliseconds on physics and
audio also across multiple threads the
end of the frame on the main thread is
taken up with making stuff ready for the
render thread to draw up the next frame
so we have like a double buffering where
we copy let's say to the steena for it
to draw we also do like Laura last
skinning and our occlusion at this point
render thread picks up all that buffer
from the previous frame and basically
submits draw calls as fast as it can
renders the models the shadows all that
thing spends about 6 milliseconds on GUI
because
Scaleform and then if it has any time
left over at the end of the frame it
joins in on the workers just does
occlusion like anything else obviously
on some frames that has more left over
time than on other frames here's all of
our those other threads in priority
order audio gets first crack of course
we have a high priority and a low
priority tier in our workers the reason
for that is we have a bunch of jobs that
take a long time but can start early but
then while they're running they're
occasionally urgent things need to step
in and interrupt them for example during
gameplay there are some skeletal
animation that we have to do right away
because if it's like a locator at the
end of the hand of somebody who has a
gun or something so if we use a
cooperative multitasking model it would
be this big palabra where we have to cut
the big job up into little pieces and do
all the synchronization but by simply
having a high priority tier and a low
priority tier of threads the process
scheduler does the work for us
let's go ahead so like I'm whipping
through this but when you go back and
see it on the vault you can detect
vampires can see all of our timings and
numbers io is always ready to run it's
waiting for bytes to come off the disk
as they come in it decompresses them as
fast as it can once they're ready
they're ready and sends it off to the
main thread to initialize it's a
best-effort and we just deal with stuff
loading at variable times all right what
about that additional third of a frame
behind so like I said the main thread
ends its frame by submitting the scene
over to the render thread which draws it
the GPU executes those calls as they
come in at the end of its frame it says
I'm done rendering models meshes
textures anything that relies on an
asset so the main thread can go ahead
and unload all that stuff which means
there's a whole bunch of churn at the
asset heap in the asset heap at the
beginning of the frame like we're moving
things around getting rid of things so
it's not safe for the render thread to
start issuing draw calls and anything
that might touch an asset because those
pointers might invalidate so we had this
big bubble on the GPU for a significant
part of the project until suddenly some
parts park noticed well there is some
work that the GPU can do that doesn't
depend on meshes or models or textures
just on the pixels already on the screen
and of course that's post-processing in
tone mapping so the GPU is ending its
frame when this
he was beginning and that's like color
correction let's look at the GPU frame
and eraser we use a deferred rendering
model that means we put the Pauli's up
in the buffer and then go around and
light them later I've mixed up the pixel
shaders and the compute shaders in this
graphic because we don't really
distinguish them they're both you know
small programs that run in the GPU and
the graphics what's the difference on
the asynchronous compute pipeline the
first half of the frame we do our water
simulation which is fast Fourier it is
like our waves are actually waves and
screen size reflection and ambient
occlusion happens like vaguely somewhere
in the second half of the frame why
those things so on our last game global
illumination ran on a sink compute and
it's like a sink compute is this miracle
thing right it's zero cost stuff just
runs in the background you're not the
word but until like it doesn't fit
anymore and then all of a sudden there
is a cost cuz you're synchronizing on
this asynchronous thing so that got but
that got too big so we moved it over to
the regular compute plate line which
made room for reflection and ambient
occlusion why those things they happen
to be the things that fit it's just a
juggling act now I'm showing you the end
product here but we were like way over
frame for a long time and for once I
don't have any fancy optimization tricks
because we did them all on the last
project so this was just a long slog of
going through every component and making
it better doing the micro optimizations
most importantly trying to make things
more concurrent we had a lot of
concurrency before but there was logging
it wasn't efficient we made it better
concurrency is hard in our engine
creating actors isn't thread safe for us
creating components isn't safe for us
creating event handlers isn't thread
safe for us allocating memories also we
have a batch component update model
which means that we do all the animation
components in the world and then all the
a components in the world and then all
the movers in the world and then all the
sound in the world so we have a really
good way of saying which classes of
component come after which other classes
so we can say that movers always come
after AI but we don't have a good way of
ordering individual actors within that
component class so like if to act a if
to a eyes are running at the same time
and they're trying to read data on each
other's AI components that data may be
changing if they're both moving then you
literally step on each other so in those
cases we couldn't fend them out but if
we had components that didn't touch
component classes we could run like all
of the pedestrian components on one
thread while we ran all of the sound
components in another thread for example
those jobs would be huge but they would
be huge in parallel in fact for some of
our components we said do they need to
do components at all a bunch of them
like the pedestrians in particular we
just turned them into this big global
system we took that work away from the
actor because then it can have a very
big simple for loop and it just does all
the things and there's no stepping on
each other and there's no there's no
ordering problems again it ends up being
this gigantic job but for gigantic jobs
at the same time
so that's the TLDR in addition to all
that we did love what we have to do
level of detail very aggressively I
don't just mean geometric level of
detail although that matters were
skinning but also just like things that
are farther away we had to go through a
lot of work to make them think less tick
less shut stuff off as soon as it's out
of the camera on the other hand this old
model that we've been using for years of
like the main thread and the render
thread and workers it's fine it's fine
it's still good right we come to the
first moral of the story simple is good
I mean we could have done the cooperated
multitasking it really would have just
complicated things but also like the lod
thing before it leave it I just it was a
continuing struggle for us because the
game is so big stuff keeps falling
through the cracks for example here's
the body on dr. Morgan Michaels on the
left we have the cinematic asset which
is 90,000 triangles on the right is the
gameplay asset which is 5,000 triangles
can you tell the difference at this
distance all right there is a gameplay
scene that uses him would you like to
guess which LOD we used for a while yeah
there he is and all his 90,000 polygons
glory
obviously the fix for this is simple
once we found it right we just fix the
LOD distance but this like happened over
and over again somebody modeled cars
that had these beautifully detailed
motors and animation on them but the
hood never opens so
I mean like there's no lesson here other
than yesterday it's a continual struggle
and you have to stay on top of it
especially for project this big let's
keep going
[Music]
all right we built this city on mostly
Houdini Manhattan's pretty big and our
team was not that much bigger so for the
details go see Santiago's talk seven
hours ago well go find it on the vault
the overview is we divided Manhattan
into districts neighborhoods each of
which was tiles which are like a couple
of city blocks and every one of those
are stacked up zones a zone is just our
atom of streaming it's a list of scene
objects in the assets that go with them
the TLDR is that Houdini made a whole
bunch of stuff relevant to this talk are
the streets the ground and the buildings
the thing is after Houdini exported the
final art for the bill that well it's
notion of the final art for the
buildings we then of course have to go
and do it by hand because you can never
just procedurally adjust everything I
mean you can never count on the
procedure allottee however we also use
Houdini to make the markup for a lot of
stuff the navigation in particular the
traversal markup like the swing points
and the ledge grabs and all that stuff
and that depends on the final position
of the geometry right look the artists
go in and they push around a skylight
and now we have to fix that markup so we
needed a process where we could
procedurally make art edited by hand in
the level editor and then feed it back
into Houdini for its am up for its into
the markup here we got lucky our common
data interchange format for levels and
entities and a bunch of stuff is Jason
why Jason because a few years ago we had
this dream of getting into web
development and we decided we decided to
to try to write our tools at our level
editor in particular as Chrome apps we
backed off of that idea for reasons that
you can see in ondrea's talk on the
vault and Ron's talk tomorrow
but the the short of it is that because
all those things were written in
JavaScript Jason was a natural format
and we stuck with that for example this
means that Houdini doesn't have to admit
any kind of complicated binary asset we
don't have to teach it how to do
anything complicated it just emits
prefabs as JSON objects and for us a
prefab is just the structure that we
copy and paste into the level
automatically in the Builder and our
prefabs contain actors and models and
decals and other prefabs all of which
are other Jason objects so this was
super useful for the tech artists and
for us
gameplay systems right every scripting
language can read JSON without trouble
so we kind of looked into this very easy
way to move things between different
kinds of tools also it let us write new
builders for new kinds of asset pretty
easily crime like mugging it's the first
class asset in our game because Houdini
could make that prefab then we had a
like a special crime builder that went
through all the crimes in the world and
then offline computed the data structure
with a tree that knows how to spawn them
under what conditions so that the crime
system in the game didn't have to go
digging through the level and numerating
all the volumes like that was a small
optimization but the point is writing
these kinds of custom gameplay builders
for custom things specific to this game
was super handy some numbers about our
Houdini it takes like seven hours and
generate just the grounds every one of
the overlays like the traffic the props
the crimes the vignettes all that about
forty minutes each better part of the
day to regen the city if we have to not
counting lighting we have eight thousand
three hundred buildings three hundred
fifty storefronts three thousand random
crimes three thousand vignettes that's
stuff like traffic accidents five
hundred forty four roads and twelve
hundred alleys which of course we need
in a game because we have to have
crime-infested alleys for crime to take
place in but by contrast the real
Manhattan has one two three four five
six seven seven alleys and lower
Manhattan
if you have seen movies shot in New York
and whatever wondered like why do alleys
in New York look this thing it's the
same alley Cortlandt alley in Chinatown
that's the only one you can film in TLDR
this whole shtick of like creating
procedural content and editing it by
hand voting it back in it's fine
it's cyclic and annoying but because we
had this convenient format it was much
less annoying than it could have been
and then go see Santiago's talk for the
rest as we got a lot of stuff to do we
come to moral number two fit the text of
the context because we control the
format of all of our data we were able
to create custom gameplay builders for
stuff that was super specific to this
game like a set of crimes traffic the
ambient vermin system all of this was
like stuff that is specific to making a
game set in Manhattan but it was cool
also if you care about the art go see a
hickie stock tomorrow let's keep going
all right let's speed this up a bit all
right so we go at breakneck pace through
the city and that's a lot to sprint to
stream so let's talk about our streaming
technology actually I've done that in
this room before so you can go take a
look at that the gist is that we
tessellate the city into regular
polygons regular polygons on a plane you
have three choices rectangles triangles
and hexes on sunset overdrive you went
with hexes hexes are definitely the
right way to go if you're making a
streaming city game right you definitely
don't want to go with a rectangular grid
because then you end up with these long
streets that are from one side of the
city together and there's nothing
occluding yeah you know what it's it's
fine it's fine
there were never hexes it's just like a
point and a radius and within when
you're within the radius of the point we
load all the zones that are associated
with that point so if you arrange your
points like this it's a hex grid you
arrange them like this it's a
rectangular grid whatever let's talk
about Washington and Mercer here is the
ground plane we divided the environment
art up into multiple zones on sunset cuz
we had multiple actors working on them
at the same time here Houdini he did the
heavy lifting for us so on the arts in
one zone game plays don't overlay it on
top of that mostly navigation and
script hooks we also have an overlay for
stuff that is different at different
times of day or different points in the
plot so for example day time at the
beginning of the game we have a hot dog
cart at nighttime it's a closed top card
at the end of the game it's a
post-apocalyptic hot dog card traffic
global zone for the entire city because
it's less than the megabyte and might
complicate things did or the pedestrian
grid did of the pedestrian spawner
volumes did a whole bunch of stuff so
here's everything that stacks up to make
tile m34 go so like I said we decided to
build our city on a grid well no
Alexander Hamilton did but that's the
city that we have so on a rectangular
grid typically about 9 tiles loaded the
one you're standing on the 8 that are
adjacent if you're close to a border
it's more like 12 if you move from one
to the other three tiles are now
adjacent three begin toss overboard so
we load them in move diagonally it's 5 &
5 still fine missions random crimes and
basketball games and stuff like that are
also zones the only difference is the
ground tiles load when you're within a
geographical point missions load when
script tells them to load and because of
that they can poke out geographically
like car chases obviously go right
across the city all right some system we
have on Sunset pretty simple we didn't
really complicate it works fine
streaming that can keep up with the
player is a much bigger challenge on
spider-man go so the sunset overdrive
character moved about 30 miles an hour
or a Spider Man is possibly the only
human being who can move through
Manhattan at 70 miles an hour and the
tiles in spider-man where Marvel
Spiderman was slightly bigger and
because we're on rectangles we might
have to load five instead of three okay
so now we can figure out how long do we
have to load a tile and how many bytes
big can but I'll be alright 32 meters
per second hundred 28 meters by gives us
a little under a second to load each
tile isn't that a lot however we really
only need the three tiles that are ahead
of you we don't you're not going to the
ones on either side and if you decide to
turn and go to the ones on either side
it takes you about a second to do that
so we have enough time to go and pull
the switcheroo alright so we don't have
to load five tiles we only have to load
three it's one in a third seconds that's
more time of course ground tiles are not
the only thing coming off the day
Audio is streaming emissions are
streaming you might be recording a video
you might be streaming like your game up
to twitch
I'm also did you know that you can
replace the hard drive in your
PlayStation without voiding the warranty
which is neat right bigger hard drive
but some people use not very good hard
drives so we have to come up with a min
spec and like yada-yada-yada the the
budget we came up with was 20 megabytes
per tile that's as much as we could
confidently stream with all of the other
stuff that was happening that includes
geometry terrain actors now decals
shaders and most but not all the
textures the very highest tier of
textures we stream on-demand when you
get close to them in this video
I've replaced the high nips with solid
red so you can see them coming in and
out about 700 megabytes of memory given
over to them but like they're always
churning it's about 360 textures alright
in 2015 we looked at a tile Brandon won
in Hell's Kitchen but why is Hell's
Kitchen always this crime infested
neighborhood and comic books it has not
been that way in years the worst robbery
I've ever seen in Hell's Kitchen is what
Mickey Spillane is borrowed charges for
a martini anyway I'm kidding it's
actually very affordable twenty eight
megabytes was what we had in 2015 which
is more than 20 so we started to ask
well we don't want to cut content
because then the game will look ready so
what else can we postpone Frank is like
if we load all those 20 megabytes to the
front and then you stop moving we're
wasting bandwidth nothing is nothing is
coming in so do we need all this stuff
up front like I said we have multiple
MIPS for every texture like everyone and
we delay the biggest two MIPS and we
obviously have to have the lowest the
smallest MIPS loaded in when you come
into the tile otherwise the buildings
are all solid color but what about the
ones in the middle do we really need
those right away yeah it turns out when
you're going really fast everything is
blurry so like I actually I took out the
middle MIPS in this screenshot can you
even tell the difference so that gave us
two budgets for the tile 20 Meg's
upfront and then 20 Meg's of detail that
we load when you're not going at a
breakneck pace so like when you slow
down if you stop moving me pulling all
that stuff so we introduced this idea in
2016 but then the next spring we saw
there's a whole bunch of other detailed
stuff that we can also postpone like
storefronts right we have these lovingly
models cafes with tables and baristas
and customers
never gonna see it from five stories up
so we can wait until you're on the
ground and not running around or not
swinging around to load all that stuff
in so in this video the top left is our
delay loading pipeline while I'm
swinging at maximum speed nothing really
comes in because it doesn't have a
chance
but the moment I stop moving boom look
at all those textures and potted plants
it's about 12 Meg's a second just a
delay loaded assets
so by procrastinating half the tile we
got our base zone down to 20 megabytes
mostly by saving space on textures which
gave us more room for models materials
and what we call zone which is like
script and actors and terrain and
collision this is another place that our
handcrafted artisanal bespoke engine
worked for us we changed the idea of
what a level was for this specific game
right Manhattan come came with a very
specific challenge and we were able to
like change the notion of what even
loading means for this specific problem
and we will probably do something
different for a game that doesn't have
this problem while I'm talking about
streaming we originally hoped to have a
game with no load times ever right we
could given this technology use a bunch
of air locking and smoothy magic to
completely hide transitions even for
coming from an exterior space into an
interior space that is instanced so
here's an early example of that the FISC
emission coming in from Broadway going
into interior to our bottom rights what
you see in the game pulling the debug
camera out are we doing getting rid of
that city as fast as we can here comes
the fist interior twice the size of
hundred meters in this guy
you okay if you mix out of that building
we're gonna lose it camera cropping the
hides the dirty work do your thing
coming into the exterior building the
lighting change
[Music]
and we're inside yet seamless right
really cool took us some month to do if
we did everything like this we would
still be making the game so we bit the
bullet and said load screens
oh also like if you've wondered why we
have some ones capable cutscenes but not
cutscenes they're animated load screens
so that's what it took to fit into the
streaming budget that getting it onto
disk was another story right like I said
wearables Manhattan a little bit more
than six times bigger than sunset city
sunset overdrive came on one disc does
that mean that we would need to come on
like it's a really big box to take home
so let's back off when it let's look at
one of the technologies that we used on
sunset to speed low speed uploads which
is duplication we've relied really heavy
on duplicating act assets and the reason
for that is a disk is a spinning
mechanical device of the momentum and
the best way to get good bandwidth out
of that is to have a consistent read
size and never seek never move that head
back and forth it is often better to if
you have a piece of data you don't need
sandwiched between two pieces of data
that you do need to read over thing you
don't need and then throw it out minute
is to seek so I'm very simple we can
find all the assets that has own needs
and then if asset B is needed at both
zones x and y we just put it on the disk
twice because we didn't know which one
you'd load first and we didn't want to
move that head back and forth so here's
where we were in 2017 we decided to
finally measure how big our package was
44 gigabytes which is about 90% of a
blu-ray and we had not yet made a 90% of
the game where is this number come from
well we've got 700 tiles in this city
that's not even counting the interiors
in the raft and 40 megabytes of tile
like we just said so that works out the
28 gigs
this was our budget per tile
however like more talking about lighting
up there that's one time of day and
contrary to popular belief the Sun does
set on Manhattan in fact we have four
times of day some daytime sunset
nighttime and overcast that's four times
the lighting data that gets us up to 40
and a half gigs right there what's
missing from that number
well those detailed textures the
characters the cinematics the animation
basically most of the game so with that
added in we are at about 6070 gigabytes
which is more than one blu-ray but less
than two we could
on two discs considered it raise your
hand if you like games that come on to
disks yeah it kind of stinks right
because like swapping discs for installs
annoying if it's a digital I live in
Burbank it's like a three day download
for me and even even when you're
developing right installing that
gigantic package onto your dev kit every
time you want to get timings huge pain
so but let's back off the duplication
thing so on the left are the assets that
we duplicate on the right we at our
assets that we do not duplicate remember
how I said for the metal net textures we
wait until the disc is quieted down and
then read them that means we've time
enough to seek the drive head over there
so we unduplicated those saved us a
couple gigabytes anything bigger than
four megabytes we just arbitrarily
doesn't don't duplicate that because
well cuz it's big and anything that was
duplicated more than four hundred times
anything that appeared in more than 400
places again a disproportionate cost on
the discs we unduplicated those besides
if they're in every other tile they'll
probably always stay in memory until you
go to an interior Oh like if you're
wondering what we had in more than 400
tiles you know usual New York stuff like
pavement and bushes and piles of trash
because we duplicate about fifty six
thousand assets they would add up to a
gigabyte if we didn't duplicate them but
with duplication it's 11 gigabytes of
space on disk so it's about a 10
gigabyte cost for the duplication which
is a price we pay for faster loading we
also considered switching to a better
compressor than lz4 it's fast but it's
not the compressed but it has a bunch of
nice properties its byte wise you can
interrupt it it's really compact and
memory nothing else we tried compressed
that much better that it was worth doing
away with LZ except for textures
obviously which used lossy compression
jazz hands simples good but it doesn't
mean that we couldn't rearrange our
assets so that they compressed better
for example models at that point in the
project took about seven and a half
gigabytes of disk space so we figured
like Kamiya at least get them to
compress better so we looked at our
biggest model in Peter Parker's body
100,000 triangles about four point eight
six megabytes after an intensive weight
loss program we got him down to three
point two megabytes
the the details here I put them in the
appendix cuz they're there Gordy but
like I want to talk about the one thing
that made the biggest difference which
is in the index buffer that's the thing
that says for every triangle what verts
add up to the triangle we move towards
storing every index there as a delta
from the previous one because 72 usually
comes after 71 and is then followed by
73 so our buffer ended up with these
long strings of just ones repeated over
and over again which is exactly what LZ
for compress is really well it's
basically a glorified run-length
encoding so like it was mental how much
this one change did for our compression
that was like a we actually thought
there was a bug in our measurement code
so here this is a graph of our package
size across March and April 2018 the
y-axis is zeroed at forty five gigabytes
at one blu-ray so all the orange bars
represent how much over target we are
you can see it getting bigger and bigger
and bigger and you can see the point
where we put in our model optimizations
for that and for like a bunch of other
stuff and like that's like five
gigabytes just from better compression
so made a huge difference lighting
lighting is big our lighting model uses
two terms diffuse term which is like
incident light and a specular term which
is basically an environment map that we
use for gloss and reflection
I'll get back to the specular term let's
talk about the diffuse term first we
call them light grids because
essentially there's just a form of
computers that divides the city into a
grid and marches down it like with an
incident light meter takes a picture in
every direction so that if there's a red
wall if there's a brick wall to my right
and there's a concrete wall to my left I
know when I'm standing in that block
that the light coming from over here is
a little bit redder we just store that
information for every voxel in the city
that is her ambient light solution so
all right we're cutting the whole city
up into cubes and Foxx losing it yeah it
turns out that empty sky does not cast a
lot of bounce light there's not much
there to bounce anything so we just
stopped capturing cells that were not
near anything and it saved an
embarrassing amount of memory for the
details go dig up x-rays talk that he
did a couple hours ago after all that
the budget for a tile was determined not
by streaming bandwidth but how much
space we had on disk it came in at just
under 12 megabytes which is way less
than 20 so we now have a lot more room
to stream stuff in like we go bigger
mission overlays and we can have
those high net textures turning in and
out and we can do more audio and of
course like I said we paid for this by
putting more Sikhs on the like by adding
more seeks to are loading so some of
what we saved in space we paid for in
speed which is our third moral of this
talk revisit speed space trade-offs our
choice to duplicate assets and previous
games was based on our having not a lot
of CPU power and not allowed to CPU time
but having a whole lot of slack disk
space on this project realized belatedly
disk space was it much more of a premium
and we actually had a little bit more
slack on the CPU because we do my other
stuff pretty a little bit better so we
had to basically redo recalculate that
trade-off in the context of this
specific game if you're wondering what's
on the disk it's this the interesting
thing here is every one of our local
ization is about a gig and a half which
is why we had to have multiple SKUs for
different regions it has to be literally
could not fit all the languages on the
same disk in retrospect the better
compressor would have let us fit maybe
one more language on the disk but still
not all of them let's keep going
rendering New York is kind of a deal
it's big it's shiny it's full of stuff
that's moving around and you can see
everything from the top of the Empire
State Building rendering could be
multiple talks in and of itself
luckily digital foundry has done one of
them for me they've done this amazing
video just from like looking at the
retail product yes I'll go see that I
want to talk about a couple of things
that we changed for this specific
project from our previous one
so remember when I was talking about
like loading tiles obviously there's
more city out there that you can see
like so what's what floated out there so
we have a single low red zone that's the
background for the entire city exploded
all the time you can say from everywhere
and it's about 110 megabytes and models
and eighty five Meg's textures now I'm
about to start using the word Impostors
I just to warn you it is not the same
way that everybody else means Impostors
which is the thing where you render a
model onto a billboard and then show the
billboard for us an imposter is like
this super low optimized shader piece of
geometry because you know it's confusing
to me to if you can think of a better
name please tweet it dual think I'm
something better for us an imposter is a
lightweight scene object the minimum
that we need to render something
basically a transform a mesh and some
flags our imposters are automatically
generated from the full res geometry by
the zone builder they are essentially
like super decimated things that loads
up the final version like the full res
version of the building simplifies it
decimates it takes a picture of it from
every side to create a cube map we
project the cube map onto the impasto to
create the illusion of detail and that's
that all of the textures go into a big
atlas per block it's about four K tops
and because they're so simple we don't
need a bunch of information for this
texture that we would need in other
things for example the imposters don't
have any movies per face it's just
pixels per meter so we just know how
many how many bytes to go down how many
pixels to go down and the texture map we
have an access align the city so we
don't have to store normals and
basically it's just smaller however we
do need to store a higher source stream
a high-res version of that atlas when
clothes because you can tell even at a
distance also flat diffuse textures and
that super simple shader are not always
enough some stuff is emissive especially
in the city some stuff is reflective we
have those neat window windows that you
can see inside so for that we have a
notion of a high quality imposter which
is just initial model with a little bit
smarter shader tacked onto the side of
the building when you get close enough
all right not everything is an axis
aligned bounding box trees are
notoriously not box shaped also stuff
like water towers AC units stuff like
that they really affect the silhouette
of a building and you can notice them
from very far away if they're missing on
sunset overdrive our solution to this
was to just mark these things as always
loaded right all the water towers in
that game were loaded all the time
because there weren't many of them we
have 600,000 of them in Manhattan and
the minimum that we need to store just
the instance data for something for a
rendered model I'm not talking about the
geometry just like its position and all
that stuff the minimum is 384 bytes per
each of them which would be 230
megabytes of memory which is more than
like our entire gameplay heap obviously
not going to fly so we decided to come
up with something called hibernates on
this project to save on the runtime cost
of those things we still have the mesh
loaded but by storing only the bare
minimum because these things are not
gonna animate when they're at that
distance we don't need any clever
shaders in them at the distance the
player obviously can interact with them
at that distance so we can get away with
storing only 40 bytes through using a
bunch of trickery like all these things
are associated with a city block so we
don't need an absolute position for them
it can just be an offset in the middle
of the block fits into a 16-bit fix
point and when you get close to them we
then we basically swap out the hibernate
with a full instance with all the
animation and the stuff all told 600,000
hibernating objects takes up 24
megabytes of memory at runtime and the
geometry is really just 13 megabytes
because it's mostly the same water tower
and fire escape alright so TL DR in the
backgrounds let's review we use
billboards and low poly models in the
distance for Brooklyn Queens Bronx New
Jersey you know all the
the skybox is an ordinary skybox
imposters are as I've said the high
quality imposter stuck on those are the
hibernates everything else is not
background once again this is all about
fitting the text of the contact fitting
the tech to the context this whole
notion of hibernates is because
Manhattan has the water towers in the
antennas and the billboards and those
weird fire escapes that you bolt to the
side of the building they're a very
specific solution for this very specific
product oh yeah specular lighting forgot
about that so it would have taken 8
gigabytes to put on disc our specular
model is based on something we call n
probes and because their environment
maps and probes because their locations
that you put throughout the city the but
we build environment Maps like anyone
does we go to every one of those probe
locations load at the level take a
picture in all six directions make it
into a cube map then when you need to do
a reflection you just look at the cube
map instead of actually reflecting let's
step back a moment and ask why do we do
it this way all right if you wanted a
reflection we could just rerender the
scene from the point of view of the
reflective surface and then kalam that
onto the surface the reason that we do
environment maps in the industry in
general is to speed things up we do that
offline for all the world and then just
load that when we need to reflect
something but couldn't we maybe create a
little bit of runtime to get back a lot
of disk space the answer is yes we
decided to calculate our environment
maps kind of at runtime we have 64 of
them in memory it's like the cache if as
you're swinging through the city we
ditch the ones that you've left behind
if there's a nearby one that doesn't
have an environment map on it yet every
frame where we have spare time we will
pick one face from one probe and render
that face because we only need to use
the background geometry for our
environment Maps you know animating
things are not captured in our
environment maps it takes about a
millisecond to render one of the faces
so you know six of them and then a tomb
it's it's 12 mil twelve successive
frames to populate an environment map
you'll never notice and after all about
they take zero bytes of disk space look
about animation
writing your memoirs don't forget the
hyphen between Spyder and man get the
chopper ready I won't be long
so as you can see the visual fidelity
target for our facial animation is very
different from our previous games on
Ratchet we can get away with using
scuttle animation for the faces because
that was a deliberate part of the
aesthetic right we wanted that kind of
friendly cartoon look but on Marvel's
spider-man we used facial capture for
almost all of our cutscene performances
the thing where you've got the camera
and the dots and the actors face and you
record them acting the bodies and and
some of the head or skeletal animation
like the Jon the eyelids and stuff like
that but the really fine detail like the
movement of the muscles and the creases
and the wrinkles in the skin it's very
hard to do that just with skeletal
animation so for that we decided to use
morphs and to get it right we needed
between 600 and 800 morphs per character
which is a little autumn morphs to
composite at one time so we decided to
do all the morphs offline and then just
stream verts at runtime but we do the
joints but then we'll like the vert
distant the vert offsets from that ended
up actually being less than two
megabytes a second so we thought it was
crazy talk at first but end up being
okay to get there we worked with a
couple of partners three lateral scanned
our actors fours
very high quality textures really nice
models rigged the faces for us with 240
joints and in the case of Peter Parker's
620 blend shapes so there's the joints
going
there's the blend shapes for the really
fine stuff and then you put them
together you can get basically an
expression Union out of him cubic motion
then took the recorded performances and
worked out what combination of joints
and morphs and so on we get the animated
face to the same position as the actors
face so from that we can figure out how
much bandwidth will consume to stream
these faces every one of our faces is
about 18,000 verts I'm not counting the
body is in the hair and the whatnot just
the face that we're doing the verts
streaming on so from that we can work
out our bandwidth per face metric Peter
Parker's face 18,000 375 verts exactly
so of course we're gonna store three
vectors pervert position in the normal
for the lighting three plus floats per
vector at 30 frames a second it's 20
megabytes a second how about no
turns out skin does not move more than
about a centimeter from your face while
you're still alive so we can get away
with using fixed point notation for them
ten bits for every component thirty bits
per vector also the normal and the
tangent right like what if I told you
there was something called a cross
product that you could use to work at
the normals just from the positions Y
let's step back like why do we store
normals and tangents in meshes and
textures at all it's to save on run
times because calculating all that stuff
would be redundant except in this case
the bandwidth was really this scarce
thing and we actually had a lot of CPU
time left over because you're not
there's not a lot of sin going on during
a cutscene you're not swinging to the
city nobody's dying at you so we ended
up just as a really as a compression
tactics streaming just the positions and
doing the normals and tangents at
runtime that gets us down to two
megabytes a second with careful
arrangement LZ gets us down to half a
mexican't per face here's the format
every frame and a screen animation first
has all the joint positions for all the
characters followed by all the verb
positions for all the characters that
first screen we do the structure of a
raised thing so that's like all the X
components for everyone and all the Y
components for every vector and then all
the Z components because it compresses
better that way
also we Delta compress these things so
for frame two every verb salute position
it is the difference from its position
in frame one because most first actually
don't move from Frank the frame which
means we have these long runs of zeros
which LZ Ford eats up obviously this
makes screaming this makes scrubbing
really annoying but the player doesn't
scrub Oh also like the diploma this is
that there's a zillion places for things
to go wrong all you need is a single off
by one error and it's just this total
Horror Show of like joints or verts and
verts are joints also if you have like a
black spot and the characters face is
that what's going on there it is it is
it missing isn't missing from the meshes
at the wrong position
is the lighting wrong I'm not gonna talk
over this yeah so someone belatedly we
realized we needed a tool where we could
render just like one streamed face in an
app of its own and then point why did I
do this
pointed a single polygon and say you
Paulie what's all the math that made you
once again we revisited our speading
space trade-offs a lot of the things
that we've done in cinematics previously
were based on the assumption that CPU
was scarce and disk space was plentiful
that became not the case on this project
and especially not for bandwidth so we
realized we could actually spend more
CPU to make up the space let's talk
about populating the city please don't
ask me to do the hand thing all right
our original plan for pedestrians is
that they were just window dressing
interactive you just see them from up
above so we could get away with some
simplifications of course here's the
problem yeah so in 2015 we realized that
was not gonna fly the number of
pedestrians we began with was not enough
to fill out the streets it made them
look really weird when they weren't
responsive to the world they needed to
look diverse because that's kind of New
York's thing and they needed to interact
the spider-man when you were down there
so our original original prototype just
had actual ai's bots that we drove
around the world for pedestrians and we
can run 30 of those at once and we
needed a whole lot more than 30 after a
whole bunch of chicane array we got it
up to about usually more than a thousand
pedestrians on screen at a time I'm
gonna spare you the details and that one
because it's been discussed at GDC for
many GDC before many times 2015
apparently the year of the crowd at GDC
so there's a bunch of great talks
hmm-what all right someone say something
so you can go dig those up the one thing
I do want to show you is how
aggressively we LOD so in this video
I've left the LOD pivot back there and
I'm pulling the debug camera without
updating anything we stopped animating
the pedestrians of the moment they're
off camera to save on skinning also when
you get beyond a certain distance they
are not peds at all they're just like
these totally unanimity on the skinned
creepy sack people things
because they're like four pixels tall
you'll never see the dist the difference
right we can save all the time and
skinning it's just enough movement so
that they don't look like they're static
we could have probably used billboards
also we realize kind of late in 2017
that window dressing isn't enough if the
peds aren't interactive you're not they
don't invite you to spend time on street
level and see all the cool stuff you put
down there and it seems really robotic
and weird if they don't acknowledge with
you don't acknowledge you and all some
people just enjoy messing with the peds
and seeing them interact so late in the
game we came up with a couple of
interactions for them to do the low
intensity ones were simple right you're
just go down and they would do a canned
behavior take pictures of you calm on
wave whatever also we noticed it was
kind of weird if you put hit the punch
button while facing a pedestrian and
took a roundhouse of their face and they
didn't react so we just replaced the
animation kind of became a thing we also
came up with some more high-intensity
interactions every so often we body
snatch a pedestrian and turn them into a
full AI then they perform one of the few
canned interactions when you walk up to
them and hit the triangle button in some
cases the synchronized animation in some
cases they all point out like crimes or
collectables for you in some cases
they're in cahoots with one of the
villains and you get close and like aha
I work for Fisk you're going down in
spider-man just to give a little bit of
surprise and variety on the streets so
looking back it's the classic GDC story
we should have started on this soon no
no we did we we had this from the
beginning we knew that we needed this we
just had the wrong prototype and once we
realized we needed more we just kept
iterating on that one prototype but then
going back and rebuilding it from the
beginning so we have like this fully eye
thing and then we just kind of like
added those non-skin things on top of it
we never went back and built a system
from the ground up that had better
concurrency a real crowd sim we would
have liked a more unique interactions
really the lesson I want to communicate
here is if you put a human being in the
game the players gonna try to interact
with them and you just can't fool
yourself that they won't one last thing
that I want to talk about before we get
into Q&A is to talk about photo mode and
the most important thing I can say about
photo mode is that programmers
undervalue it but the success of photo
mode is not a tech story it is a social
story it was a story a basically like
Stone Soup that folktale what everybody
comes and contributes and it made a much
bigger difference than we suspected
beginning so we began working on photo
mode about five months before ship date
but we did a lot of work in those five
months an original plan was something
very simple just basically so we could
claim that we had done photo mode basic
3d cameras and controls rotate the
camera around and some stickers all
right well we're getting the stickers
from so we had the little social media
display in the pause menu so we just
grabbed the textures from that and use
those as stickers and this was a stroke
of luck because those things have a
specific aesthetic that immediately
encouraged other texture artists to come
along and start contributing their own
stuff other characters in the books and
it's a comic book right we can start
throwing in the onomatopoeia and and
some speech bubbles and okay what did
you take there when he would take a
sticker but the set of it being opaque
in the middle and transparent everywhere
else
what if it's transparent in the middle
and opaque everywhere else and that's
where frames came from and then someone
looked at that one movie poster from
that one movie with a superhero taking a
selfie and said how about we have
selfies
yes selfie mode is a thing and as when
people really started to see themselves
in the product because people when we
were even developing just loved taking
selfies of themselves with the content
that they had made and bugs that they
encountered and when people were doing
that on the team they started to see
opportunities for themselves to
contribute like an animator said I can
make Spiderman and make funny faces and
the leiter said it's not that hard to
put it light on the on the locator and
get a selfie light and the shade artist
said yeah you know with just some simple
color look-up tables I can totally make
a whole bunch of photo filters behind
the scenes it's really simple we just
duck out the game play character replace
it with the cinematic character drive in
a matrix with a thumbsticks have
different poses in the cinematic
character depending on whether you were
standing or purchasing or running the
cameras attached to the locator there's
no fancy okay I mostly just wanted to
show you this video
but it turned out to be a huge
promotional win for us to have this in
from day one photo mode is an
intrinsically creative thing right
people want to express themselves in our
game and then immediately to share it
with their friends and the sharing of
other friends was a really nice thing
and if we had done photo mode as I could
deal C weeks later it would be cool like
people have gone on Twitter and said
look at this photo mode thing from the
game from last month but for us because
we had it in day one it hit right at the
peak of the hype trend like right as her
advertising spend hit went in this stuff
was all over Twitter so it made a huge
difference for our launch momentum we
were kind of nervous about photo mode at
first because we thought people would
take pictures that made the game look
bad but it's not a problem one people
like the silly photos and two no one
cares so here's the morals simple is
really good still to this day fit the
text of the context I was talking about
the context of our specific engine but
really anything you're doing reevaluate
the specific game that you're making and
build the tech around them right we have
that's why we have programmers on tape
there's nothing everything nothing has
to be completely generic continually
revisit the speed and space trade-offs
we have to think over all the choices
that we had made on previous projects
for the specific context of a city that
has everything that Manhattan does and
if there is no fun than what is even the
point
right we are toy makers for a living if
you're not having at least a little bit
of fun making the game you're not gonna
find creative ways to contribute it and
you're gonna have a hard time putting
fun into the game so I am the one up
here talking but this talk was a team
effort
so here's everyone who contributed
mostly I just punched it into PowerPoint
I really want to thank Sony and Marvel
for letting us play with their toys and
make this game and i SuperDuper want to
thank my talk advisor andreas who
carried me across the finish line
and the start line and he just carried
me and thanks to everybody at insomniac
who actually made the game that I'm
talking about
oh yeah we're hiring please alright
appendix appendix for those of you
watching on video get your pause button
ready here's the numbers BAM
[Laughter]
all right QA
[Laughter]
he'll be fine
people can't you can line up while this
is going on like I put these in for a
while you can follow people we're
milling around alright I actually have
something alright thank you
what's the funniest glitch like the
things that are displayed here that you
personally have encountered in this gap
I put them up there which of them do you
like the most
oh the one where the guy gets like
punted over the girders into the
distance that's that it gets me every
time you know it's like it's like a
football goal it's good it's good
yeah also we got a Brooklyn Bridge
Brooklyn
yeah then we were off by one on the
model index all right seriously come
I don't know question so as the second
clip demonstrated through the game there
are a lot of characters to get knocked
off buildings I wondered if you wanted
to I expect that being able to show
spider-man did wasn't just knocking
people hundreds of feet to their death
was part of the design do you want to
talk about some of the challenges that
came about trying to save those people
oh yeah so the question is spider-man
toxin bunch people off the building is
how do we keep them from dying the
challenge basically the reason that we
that they don't dies spider-man doesn't
kill people
gratuitous Lee and to make them not die
we simply just do a rate cut like when
they're going over we do raycast in all
directions find the nearest vertical
surface and just stick them to that it's
really simple
thanks oh sorry hi first thanks for the
great talk on the slide that you
mentioned the vertices are the
compression and you need specific tools
would you be able to talk a bit more
about those tools and what was needed
what was really interesting in that
sense oh the debug tool yes okay so we
have a notion of a viewer app which
loads basically just the engine but none
of the gameplay it's just like enough to
view models in game and we can turn a
bunch of stuff on and off in that to
test specific engine features so
essentially what we did there is we just
can pick a single animation stream the
animation and get a whole bunch of
Diagnostics on all the math that's going
into making the face from that it's
really just we have a mouse selector you
can click a polygon and then it just
freezes the the stream and then just
shows all the calculations that happen
from the decompression through the
lighting through the normals to the
tangents to make that poly happen thank
you yep hi so you mentioned about the
feeding text detect to context and right
now like game engines like unity unreal
are used for several things but I was
wondering if you have any thoughts on
when it's a good trade-off the start
investing in tier on technology for like
your own game engine and like taking
like when is a good time to start
investing in them
the question is when is it a good time
to start investing in your own engine um
that that is a philosophical question
that I'm not sure I can answer for you I
mean the the short of it is when you
have enough of space on your payroll to
write your own engine and when you know
what you need from your engine write a
very small engine is a lot easier to
build in a big one if you're making a
very specific game I guess you know fit
the text to the context again I don't
know if I have an answer for you on that
one I'm sorry yeah no worries thanks
because it's a platform exclusive and
very much 100 percent likely to remain
so how beneficial was it to know that
you were just using that platform's
profiling tools for optimization and is
that different from other projects it is
super useful when you're optimizing to
know that you have a fixed platform that
is like the advantage of consoles over
developing on PC also both of the
console makers have really good
profiling tools they're different but
they're both awesome in their own ways
so I mean I guess that's the answer is
relying on the tools of the platform
maker gave us was absolutely fine
because they weren't very good did I
answer your question sufficiently I
think so thank you so you are studio
making multiple games at the same time
but you fit your engine completely to
the game so how does that work out
because they need to strip out
everything for the next game or just the
game where you're prototyping on has to
just live with what you do so we that is
an excellent question we have this know
we have sort of tiers right we have a
core engine we have the gameplay code
and then we have this notion of like a
shared library which is the thing that
sits in the middle we work on engine
features and they just grow by accretion
so when an engine feature that comes
available to one game it becomes
available to all of them so the core
stuff like the hibernate renders are
actually going to end up in some of the
other games just because it's
conveniently there we do need to be very
cautious about when we release the
engine to which projects so that we
don't inadvertently break the project's
but we do make sure that the same engine
and the same tools go out to all the
projects eventually is that they're on
the same because the streaming system is
probably very specific probably used by
the other things yes I mean so if we if
we mark the zones of they'll opener than
a geographic space then it's like
screaming like you know like I've shown
you here if you mark them so they're
loaded by script it's the exact same
thing we're using the exact same
screaming logic it's just it's not
happening all the time and you may
notice that our games stream more and
more as this technology becomes
community evil to every project another
small questions like you have everything
set up that it just fits with the tiles
and things like that
did the game designs all of a sudden
wanna like put a cutscene on the other
side of the world and I just have to
stream and doesn't fit in memory or you
were just like very strict and it's not
possible so I I dress this in the sunset
overdrive talk the gist of it is we have
to be careful not to set cutscenes and
faraway locations unless we can load in
like a very small Cyclorama behind them
Billy basically loaded and set quickly
enough for that cutscene to work however
once wearing that little thing with a
narrow camera right we can hide all
kinds of ills while we're streaming the
world around it sorry one lost and so
you do duplicate nips and everything
yeah you put that into things when you
for example make textures change what is
iteration time then because you need to
rebake everything to make an individual
texture of seconds yeah but you need to
re big then older basically files
loading the loading that date oh I think
you're saying so we have two ways of
loading assets basically that we use
during development the stuff that I
showed you is like the final packaged
version when we're trying to optimize it
we can also loose load files in fact we
have a thing where for the most part
when you're editing assets on your PC
the PC is actually running a little
service little web server that when the
game goes to load assets it actually
hits that web server on your PC the
reason we do that is so that if you
change an asset it updates in the game
immediately okay thank you bye thank you
excellent talk excellent game but I I
have to ask why did you dear as the
puddle
all right so if you don't know what he's
talking about
there was a scene that we showed in our
e3 trailer that had this big old puddle
in it and then when we showed that scene
same scene in the final trailer that
public was like not there and the
internet lost its mind because they
thought we had taken that reflective
puddles for performance reasons I'd so
we gave this really confused answer then
because nobody actually knew once I had
to do an investigation to the figure
that I talked to like 14 people and got
a whole bunch of different answers but I
finally got to the bottom of it are you
ready we switched from hand painting
them to procedurally putting them in
it's just too much of a pain to paint
all the puddles in that's it
it's a different procedural scene
now you can have all the puddles you
wants and they also got two more
questions to amendments for Q&A I can
talk about one thing I wanted about oh
yeah so um traffic we had a really
simple traffic system the basically
pedestrians the wheels so the in our
system like we don't have a nice tube
global navigation so every car just
looks immediately ahead of itself and
then stops if something is in the way
and pedestrians don't look before
crossing the street so you know the
realism is amazing we have at any given
time about a hundred and eighty cars
I know driving of which 25 are actually
moving in the rest are stuck in traffic
so again realism and it parking is
really hard in Manhattan that's it oh
sorry
can you watch the blooper really Kim
only if there's no weather quit want
people coming up if they have any other
questions this is a super naive question
but I'm really curious like you you said
that you guys did this awesome streaming
tech for this you know early on in
production where he he gets into the
interior and everything is seamless
everything is brilliant it took a whole
month calendar month I assume of
development time it seems like with all
that technology you built why was why
did it take so long so it's essentially
to do all the nitty gritty the question
is why did it take so long to set up the
the the interior transition in the first
place the answer it's not like a tech
thing it's just the content of like
framing the cinematic carefully and
being really careful about the budgets
because you know we still have limited
by how much we can get off the diskens
we're loading stuff in and balancing the
things and making the art the right size
so we can load it in I long for her it's
just a continual balancing act and
spinning that plate was a bit of a pain
where we're going we don't need roads
see ya
don't this one work yep
so when you were dealing with moving out
and like de-rezzing the people so they
didn't have really nice textures on them
what did you have to do for cars and you
also do that since the cars do look
really nice in game the yes so the cars
lod less than the pedestrians do simply
because we didn't have the program
resource to do it quite as much it's
okay though because the cars aren't
skins to the same level and they're just
less poly they do disappear the one
thing that they do is like the moment
they're out of view and they go around
the corner we just like move them back
in from someplace else oh one thing with
the cars is so a cart natural cartridge
would obviously not be possible in
Manhattan you'd get like three blocks so
whenever there's a car chase in progress
we actually look ahead on the spline
where the car chase goes and just
despond all the cars there it can
actually happen okay so you when the car
goes are on the corner you just kind of
like remove it so like you don't to
worry about it moving around and okay
thanks just a quick question
when you're looking at Manhattan from
far away are you simulating any kind of
traffic or pedestrian like I'm looking
right down the road all the way across
like multiple streaming levels okay so
the question is how much of the traffic
and the peds do we stream from far away
the pedestrians like I showed you it's
the little sac people more than a few
blocks away the traffic is simulating
out as far as you can see and that's one
of the reasons why we didn't lod the
cars down so much so you're simulating
like the whole city we're simulating
like the whole street that you can see
ahead of you right if something is
obscured into the side so that we're not
simulating the traffic look to the left
and right when we're doing that as soon
as you turn those cars come into
existence thank you very much
you
[Applause]