I I think it's 5:45 so we're gonna get
started hi everybody this is weird tech
we're going to be telling you some
stories that we've had from working with
novel hardware this is well first let's
go ahead and introduce ourselves so my
name is MJ I am a game designer and
programmer I founded a studio called a
thyroid games here in Austin
I've worked with a lot of different
hardware that I'm gonna be showing you
all today but I've also I teach at the
Art Institute
I teach game design and gameplay
scripting there and yeah that's me hi
I'm Steve Lucas I'm with across
realities most recently I was a venture
capitalist for three and a half years
investing in such great VR air companies
as alchemy labs and now we're working on
building a symmetric multi user
experiences for VR and AR hi I'm Andrew
key I my official title is developer and
producer extraordinaire and alleged
certified adult TM at alchemy labs
before that which is a lot of what I'm
going to talk about I used to work for
government consulting companies and did
a lot of work on military and other
government hardware so you'll see a
bunch of that hi I'm Adam great and I'm
the studio GM and director of
development for panic button we do a
bunch of different projects for hardware
that is not out yet and did that for
switch before released or VR or those
kind of things and I am the weird
technology I'm Tristan Parrish more I'm
the designer and co-founder of broken
window studios who has a couple of
different games in development some that
are on Steam and some that are coming
I've done design for a few years and
we've had a lot of fun with VR and some
random goofball technologies that we've
enjoyed working with so I'll be talking
about that
so let me tell you a little bit about
what we're going to do we're going to
tell you a series of stories and these
are going to be success stories and
failure stories and what we learned from
those experiences and we're going to
tell you about the struggles and rewards
we've had when working with cutting edge
technology and we've all worked with a
lot of different things individually we
work to different things but we've also
worked with different things from each
other so we've got a pretty broad
spectrum of technology to tell you about
so I'm going to kick this off with a
success story so I'm gonna tell you a
story of when I worked with the z space
have any of you ever used the z space
before a few of you cool so the z space
for Anna doesn't know is a semi a
holographic tablet it's a it's a very
it's a large tablet and you wear special
glasses that have trackers on them so
you can look around and you see a
hologram coming out of the tablet and
you have a stylus to interact with the
world that is being displayed and this
project that I did is quite a few years
ago and one of the things that we were
trying to explore with it is a lot of
the things that were available for the
zSpace
at that time you sort of looked down
into the the hologram as if you were
looking into a box and we really wanted
something that would come up into the
world that you could interact with and
we wanted also to we were we were one of
the first teams to make a game for the z
space and so we wanted to have a pretty
big open playing space so some of the
interesting things we did was we we
decided to have floating islands as our
our play environment and we made it so
that if you slid your head side to side
it would rotate through the sequence of
islands so we opened up the play space
to a very large world that you could
then move through and see different
parts of the world we also I thought was
a really interesting use of the stylus
we made it a pair of chopsticks in the
digital display so that when you click
the button with the stylus it's actually
pinching the chopsticks and a lot of
people sort of felt disengaged from from
the stylus initially because it was just
sort of floating point in the world but
once we added the chopsticks it grounded
it in reality to an interaction that
people were accustomed to
so that's my story of working with the z
space and up next Adam will have a story
for you thanks MJ so my success story is
the Nintendo switch and it's a lot of
these success or failure things are only
that way in hindsight we were working on
the hardware for for a long long time
and any time there's new consumer
hardware coming out people want it to be
a success right there people aren't
building things because they expect it
to to fail there's a whole bunch of
things outside of anybody's control as
to whether a piece of hardware is going
to be the thing that people go after so
we were working on this Hardware for
quite a while and Nintendo is very good
about protecting confidentiality and
keeping things close to the vest as they
let out features for developers or
consumers for upcoming Hardware and one
of the challenges we had and a lot of
developers had was not knowing exactly
what the spec was going to be for
different bits of hardware so there was
a lot of prototyping we did four at the
time the next Nintendo hardware that we
didn't know whether was gonna be in or
not so we did gameplay mechanics based
on different features because one of the
things that's important for Nintendo is
they do not want things to be gimmicky
they want them to be very tight and I
don't either as a gamer I want I want
features tied in that makes sense for
the form factor and for the delivery and
target platform so what we ended up
doing it because we didn't know what
things would land or what things
wouldn't even though we were working
with the hardware very early what we did
is we identified a bunch of different
gameplay mechanics that were fun in and
of themselves and we did that both in
our own titles and then the other titles
that we were identifying that we're out
there and then once features became very
very clear for what was going to be
available in the consumer hardware what
we did is we moved very very quickly we
took a title Astro dual deluxe that we
launched in May and from the decision to
do it to launch was three months the
development and everything else and what
it does is it lets you play six player
same screen multiplayer in any
combination so taking off the joy Kahn's
using them in dual mode flipping them
sideways using an horizontal or any of
that kind of thing so we were able to
create a very good experience for gamers
that not only leverage the hardware but
really fit in with that Nintendo
ei that sitting on the couch with a
bunch of people and playing together and
then the hardware is doing phenomenally
well so that's one of those things
that's outside of our control we did a
product for it that we were really happy
with and then the hardware does all the
rest as far as the success goes thanks
next slide Tristan
I guess how do I play this video there's
a play button okay uh yes I think close
enough no I don't see a play button here
okay oh okay there it is okay
just on the wrong screen okay so this is
a clip from my game reflections if it
plays it doesn't look like it wants to
Oh is there another another way of
viewing it okay this is this is a weird
tech actually so okay so the circuit hey
you're up
sorry I didn't mean to wake you I came
by early and your mom just let me in now
the doctors got up so the character
responds differently based on where
you're looking so if you look away from
her is this a bad time
so she gets annoyed and then you select
which options you so I want to make sure
we had some time together
I'm just looking you're busy we can talk
more later so this was a kind of an
interesting collaboration with Toby
eye-tracking so how many of you have
seen Toby eye-tracking before okay so a
good number of you okay awesome
so yeah in in our game we had initially
come up with the idea that basically
when you're comparing normal human
communication to video games a lot of
times it falls flat and the example that
I always give is like when you're
talking to dr. kleiner in half-life 2
and he's telling you about the things
that you need to do to save the world
and instead you're busy like teleporting
a headcrab and like breaking pots in his
his office there's a lot of disconnect
with players who are just actually not
paying attention to what's happening in
the game environment so we had already
come up with the idea that we wanted to
be able to
judge the amount of attention that the
player was giving to these AI characters
in our game and so when we talked to
Toby we found out that we could actually
do that an even more interesting way
because Toby's little eye tracking bar
gives us the ability to figure out where
a user is actually looking on the screen
so we'd previously been thinking of it
in terms of frustum checks of like is
something on camera or not but having
the ability to use actual eye tracking
data gave us a lot of facility to
actually make that feel like a genuine
interaction where the experience was
added and improved but we didn't lose
anything in the process and I think that
that's a lot of what we do when we're
judging the difference between maybe
kind of an experimental or gimmicky
piece of hardware and something that we
actually think is worth investing some
time in is are we going to be giving the
player a better experience that doesn't
sacrifice something and so I think that
Toby eye-tracking has been a really good
partner because they've helped us do the
integration it was all very smooth it's
very stable technology and we got to see
a lot of value out of it so that was my
success story there you got it
so typically with all weird tech and
unusual tech the first question that
comes to mind is what do we do with it
because you'll see it-- racking leap
motion and when I was working in in some
R&D labs then the first thing that comes
to mind is how do we use this in and
what component does it take and how did
how can it play in with our current
ecosystem and so having a project is
always really good to be able to say
here's a use case for the weird tech and
in 2010 Robert Downey jr. in the movie
Iron Man 2 used the most advanced
sophisticated augmented reality
interface that was mind-blowing was
phenomenal that were no wires no headset
no controllers and it was all fake it
was all CG so with VR and AR tech which
is relatively new technology we're
seeing it being developed by a number of
different manufacturers and each one is
finding something good that they're
doing something really unique that
they're bringing to the table but we
haven't yet seen an interface that is
analogous to what Robert Downey jr. was
doing so we said let's take a look and
see if we can take what we like best
about each different piece of hardware
and put it together and simulate that
experience today Flay so here we have
the hololens which is one of the most we
have the hololens so it's completely
tether free the person in in the headset
is walking around and being unrestricted
by wires and holding a vibe controllers
because the hololens we've noticed is
the best air tracking while the vibe
controllers are the best VR tracking
that we've experienced but just about a
month ago then the new revelations with
AR core and AR kit for iPhone and
Android are allowing for these mobile
devices to be a really cool mixed
reality viewer
so putting these together hololens also
has good voice control so you'll see
that here to pass
the image is replicated on the iPad and
using the hololens voice control i'm
actually able to zoom in on the map and
zoom out as well and change the location
so that's using matte box technology
because we wanted to be able to
visualize something very similar to the
ironman to your story and what we found
here is when you take take a look at the
different types of technologies that are
available you need to identify what is
it that this really brings to the table
that we can use and how do we utilize it
and here we are taking the same
digitalization model and being able to
create a limitless power point where you
don't have to be restricted to a 50 inch
screen and all the same principles apply
in VR you hear you may hear a lot of VR
vs. AR stories but everything in VR such
as laser pointers can be used in this
environment and here you'll see a
picture in picture of myself in the
hololens on the top right that's my
point of view while the AR kit is
filming what everyone else in the room
sees none of this is is post-processing
none of this is done with CG this is all
real time being filmed with iPhones and
iPads as you can see right here but then
of course when you're working on all
these disparate technologies you'll see
that in some cases things aren't
necessarily designed to work together
the way that hololens AR tracking works
as well as the vive basestation tracking
it's very similar they both use lasers
so what you're gonna see here is some of
the tracking gets lost when the hololens
looks directly at the controller the
controller loses tracking and then
suddenly the map falls to the ground and
that's going to happen a few times here
and so there's still ways you'll need to
work around getting systems to work
together but we found success in that we
were able to approximate
the the imagery that we were looking for
while still building towards something
new
I brought my own mic making this alright
so I'm gonna talk about some weird
military shit this is the wind T which
is the warfighter information network
tactical and this is the tactical
communication node the actual hardware
is that truck on the left that's it
so we had a problem where we had to
update the we had a little box inside
there it ran a version of Windows XP
embedded so this is 2008 and it was like
a touchscreen which was super novel at
the time and we were like ok this is
great we put this touchscreen in
basically controlled the radios and the
phones there's a VoIP and an ro IP
system and so this was awesome because
now you don't have to dial 8 radio you
just have this magic touch screen it
runs our software great problem being
that we had to update that software and
we had to update it and that was out in
Iraq and Afghanistan in the on the
actual battlefield so we're like ok how
do we go ahead and update that software
and to further complicate things we
bolted the thing into the vehicles so we
can't get it back and we can't ship a
vehicle back to Columbia Maryland where
I worked so they assigned me this
project I said you have a month and all
you have is one USB port in Ethernet
port and there's another computer inside
this vehicle so best of luck go for it
and so what I so what we ended up doing
is this was a this was a great success
so we we ended up using the computer
with a bootable CD to boot either a
version of Linux or basically a little
server that did something called Pixy
booting which is sending an image so we
pixie booted then that booted up its own
version of Linux which ran a set of
open-source software called Clonezilla
had an image file in it and
on the little device and that worked
great that's awesome because it doesn't
involve shipping a thumb drive into the
battlefield we can just send an ISO file
it burns onto a CD everybody's happy
so all that being said there was one
funny technical thing that happened
which is every time we upload it updated
it we because of the limitations of
clones L at the time we lost 20
megabytes of space but it was a two
gigabyte hard drive so what's the point
of all this like this isn't unity this
is some weird fighting video but what
the point of this is unity is an
incredible tool but all hello oh there I
am all too often I see developers
relying too heavily on the c-sharp you
know scripting back-end and digging too
deep into trying to reinvent the wheel
and in this case all those tools I
talked to are off-the-shelf open-source
things that we use to deploy this and
all those tools are available to all you
and it doesn't matter whether it was a
weird C++ application or it was a you
know a special unity application all
this stuff works and to kind of go
further on this even today even as late
as last week I was still doing batch
scripting to help us you know deploy and
to help us with our with our processes
so these things are tools that any of
you can pick up today and if you can run
with and they're all available to you
and that was a success story where we
couldn't install anything and we just
used everything that we could find
don't have cloud between every story you
guys will applaud way too many times we
have many stories left to tell you so
now we're going to go into a series of
failure stories so we all have
experienced failure at different times
and we have some particularly
interesting stories of times when we
failed to use hardware in a certain way
or the hardware failed us or the
situation in some way failed so we're
going to tell you what we learned from
those experiences and what we did to
overcome those failures so first up is
Andrew I'm back alright
more so virtual worlds let's talk about
virtual worlds so they were like a
really hot thing in like 2009 and at the
time the government the military like so
I worked for all these government
agencies and military were like we're
doing everything virtual worlds we don't
need offices anymore figure it out this
is gonna be amazing
and so you're like okay that's awesome
like we're gonna figure this out and
they were terrible and my favorite
example is we worked with this one
company this one a very large defense
contractor and they were like they're
like put all our products in the virtual
world and we were like are you sure and
they're like yes we're sure do it and so
we're like okay so we set up a meeting
room with like the in because and they
always had to populate these rooms with
like random people so it was like the
CTO and like the CEO and then like the
I've been like ten interns who are like
these are our fast risers or like junior
employees junior employees are all
gamers so like immediately what happens
is they're sitting in a room and we have
this thing it's like this but we can
like lock you in your seats so we lock
you in your seat and then somebody
spawns a hand grenade and throws it in
and then the check goes nuts saying like
oh my god there's a hand grenade there's
a hanger date and then they we had like
virtual death in there because it had to
be used for training - so then half the
participants died and they can't use
their camera anymore and then somebody
figures out how to unwedge themselves
from the seat and spawns a fighter jet
and begins flying over the entire like
office setting strafing and like
shooting as the instructor is screaming
like get down from there come back
down here right now and there's like m16
spawning and like all these weapons or
I'll shoot you so you like yeah it was a
complete disaster
needless to say like that deployment
didn't go so well
like at one point an executive like
stood up from his chair and like spiked
a headphone on the ground so there's a
total failure so what what's the whole
point of this is basically there's a lot
of things I haven't with technology
where people aspirationally are going to
put every hope and dream in virtual
worlds was everybody was like we should
make an application that does everything
and that turned out terrible in fact a
lot of what ended up coming of all of
this and each of these is actually a
different virtual world on the screen
but a lot of what came out of all those
studies was that we figured out that
like it was better to build a
purpose-built application that just did
the one thing people wanted to do and
let that grow into something larger than
it was to try to build a simulation of
the entire world and I think that that
goes for like when you're looking at new
technology just be wary of the people
who say that like this technology does
literally everything but they don't have
a use case where it does that one thing
well all right cool
all right so so my my failure story is
deals with Microsoft Kinect we were
really early on with Kinect one of the
the first developers with it back when
it was natal and limited kits and
proof-of-concept and all this kind of
thing and built several titles on it we
did the flight mission modes for Kinect
Star Wars and and some of those things
Hulk Hogan's main event we did a
wrestling Kinect or excuse me a rodeo
Kinect game which is about as niche as
you can get so we were doing quite a bit
and what we want to do is we wanted to
make a high quality Kinect title so this
is at the tail end of the original
Kinect and then the new Kinect had been
announced so we're moving forward with
this title and this the screenshots that
are up here this is a fully playable
demo that we had that worked for Kinect
where you could choose your fighter you
could box against the
and then you got your whole recap screen
and a bunch of stats and some other
stuff that played through a very robust
technical demo and proof of concept and
identity kind of thing that we had going
on and then very early on Kinect was
unbundled from the Xbox one so you had
this guaranteed install base that was
then no longer guaranteed we had
relationships on the business side on
the partner side and he's like that's we
had very early visibility that this was
going to become unbundled and so we
stopped this particular title that was
aimed at again when I was saying earlier
we didn't want something gimmicky for
the hardware we wanted something that
leveraged the hardware and made sense
for the hardware so this was a title
that wasn't as compelling outside of the
Kinect integration so it's a failure in
the sense that on the business side we
couldn't guarantee the platform would be
there but because we had the business
relationship and that's a lesson learned
that we were able to fish and cut bait
instead so that was important for us the
other lesson that's in there and it's
really applicable to VR a rmr and things
like that there's a lot of the
challenges and I did VR stuff in the in
the late 90s and early 2000s and then we
did a lot of this Kinect development and
yet some of the same problems when you
turn around and you lose arms and
appendages and and things like that I'll
hear people talk about challenges that
you know this has never been experienced
before and it's exactly what we dealt
with with Kinect as far as the fidelity
and turning around and all this other
stuff there's a lot of lessons learned
to be learned from previous tech and
it's super important because I like the
idea of innovating on top of innovation
because then I get to like springboard
as opposed to spending six or eight
months or several years trying to do
something that's already been figured
out last time we did VR so I think
there's some important lessons to be
learned there and then it's also the
business side of things it's not just
about the tech or the design or the
innovation those partnerships are super
important because they inform what's
going to be there or what's not going to
be there all right so I have an
interesting failure story it's not it's
a it's a bit shorter so try to keep us
on time
here so so I worked for Forex the makers
of the Omni which is a three cent
360-degree treadmill and at the time
when I started working there they mostly
just had shooters and they had a plastic
Xbox gun that they used with it and so
one of the things that I learned that
was my first time working at a company
that was developing hardware and one of
the things that I learned there was that
when you're working on hardware that's
not done there are always risks that it
will just stop working and it might stop
working at a very inappropriate time for
it to stop working and I had made this
demo that was a virtual tour of
Amsterdam and I was trying to since the
company had only made shooters I was
trying to do something that was a little
bit more approachable it didn't have any
violence in it you just walk around
Amsterdam and so the problem with this
was that you weren't holding the Xbox
gun and I hadn't realized at the time
that when when problems happen with the
hardware whoever was demoing would use
the joystick on the gun to resync the
game so that it didn't look like it had
stopped working and it was an important
lesson for me to learn that sometimes
you have to be able to fake things when
you're showing new hardware because it
doesn't work as continuously as you
would like it to work but I did actually
we found we found a way to fix this what
we did was we added the ability to take
pictures and while you're walking on
Amsterdam so it gave you a reason to
hold a controller and so this gave the
player something to put in their hands
so they actually could walk around and
take pictures and then actually view
those pictures afterwards as well which
was really cool for a lot of people and
this fixed the problem that when we were
demoing in important places we had the
joystick there available in case we
needed to fix things on the fly so that
was my lesson learned there hi Tristan
okay so this may be something that is a
little bit contentious but I wanted to
say it anyway because I think it was a
lesson that I needed to learn
so who here worked with the original dk1
for the oculus okay who did that and
then also got the dk2 who did that and
then also got the CV one who did that
and then switched to vibe who did that
and then also got an OS VR okay I
finally found the thing that nobody did
okay so when I was initially thinking
about VR I really I was looking at it as
a monitor replacement is the way that I
kind of approach it and so I thought
that I was going to be this like super
genius that did this thing that nobody
ever thought of and I was going to write
basically conditional statements for if
we were running VR we would change some
aspect of our gameplay and then if we
weren't running VR then it would just
run the kind of the standard set up and
so initially I thought that would like
was gonna be very easy because I was
thinking okay we changed the way that
the the arm moves when you're in VR or
we change the way that the camera
Orient's your character but then we
would start noticing oh well also the
menu system doesn't really work anymore
so I'd add another if statement into the
menu system so that it would be doing
the thing with VR in the menu and then I
would say oh actually half the post
processes need to be turned off so I'm
going to put another set of if
statements for all of the things that
don't want to be running the post
effects when you're in VR and then they
like had an API update and then at none
of my code worked anymore because I had
to actually go through and update all of
the integration of the VR pipeline for
the previous version to now work with
the new version so I don't know if you
know this but unity actually has this
thing called prefabs and the way that
prefabs are supposed to work is that you
make a prefabricated set of content that
actually is setup for specifically the
kind of thing that you're trying to do
and so what I had accidentally done is I
had made it that if I wasn't constant
getting version updates for all of this
hardware that the core game even without
VR would stop working and so I would
have to go through and make sure that
all of these kind of conditional
statements just kept being maintained
and maintained and maintained and it was
something like I would go from toggling
one script on and off to like having to
write dozens of lines of code to process
this slight change and so what we did
moving forward is we started saying that
we're going to set up prefabs for each
kind of major change in how the platform
is gonna be configured we still have
some options for how the user would tune
it but it was a much smarter idea for us
to say this is the VR prefab of the
player this is the non VR prefab of the
player every time we're doing those
things we're just swapping out which
version we're trying to work with and I
thought that initially was going to save
us all this time to have everything
embedded but the reality is the amount
of things that actually overlap is much
less than then you think initially when
you set out to make something compatible
with these different kind of hardware
devices so we ultimately just realized
that we didn't want to create hundreds
of wasted hours by trying to be super
clever and making all these conditional
statements and I would definitely
strongly advocate that for anybody who's
working with interesting tech in the
future is if it's an optional control
configuration make sure that you're
keeping those somewhat separated so that
your main game never becomes dependent
on functionality that's subject to
change while you're experimenting with
new API so so that's that's my my major
regret but okay so Steve so I'm not
talking about the oculus rift
controllers but years before they had
finalized hand tracking controllers
there was before vive controllers before
oculus touch before well before all
those new systems then VR was being with
still in its early days and when you
need to figure out what's the best input
system for VR
and we experience this magnetic tracking
technology with these hand controllers
where you could hold the controller
outside of your view and it would be
able to function and we could point out
if you had a weapon and you were
shooting at a zombie for example then
you could shoot in that direction
without having to look there and we said
that is something that we can't
replicate right now with like a regular
VR a regular non VR game so we said this
is interesting this is great and with
this type of tech any anything new then
there are debugging issues that you may
not be aware of so how many of you have
worked on a our kit or a arc or this
summer right so I'm sure you've gotten
into a situation where you're you hit a
bug and you're thinking is this because
it's my device or is it the Apple beta
or is it the Xcode beta or is it the
unity beta or is it something else
entirely and it's hard to pinpoint and
in some cases it was Apple didn't flip a
switch on their back-end server even
though everything else was up-to-date so
same thing with all new untested
technology and here we had purchased one
of these tracking systems and we had
seen the demo was great and you were
able to move around about nine feet from
the base station it was fantastic you
can move around in a lot of just free
movement and I took it home and it
looked great and I brought it back to
the office and suddenly my tracking was
only about five six feet and I thought
what did I do wrong
is it something when I unplugged it and
plugged back in and so I had to check
the laptop because it was a PC which may
not have the right version of Windows I
might not the right drivers and we're
going back and forth on it and I had to
finally take the laptop and the system
and start walking around my office
complex going from garage to garage and
plugging in the the power into whatever
outlet I could find to see is it
something else that's going on
and I brought around a measuring tape
and I was concerned because this is a
company big enough where people look at
you and say I need to see your badge
because I don't know if you work here
and here I was looking like a hacker
from a 90s movie going in to the garage
and being on the floor opening a laptop
and trying to use some crazy system that
no one had seen and finally I found that
one specific garage had so much
interference that it was down to less
than two and a half feet of range and
that's when I closed the laptop and
shout it it's not me
it's something else so when the big
takeaway is sometimes everything else
can be a perfect world everything else
is working but you still have to
identify that it could be something in
your environment in this case magnetic
interference from girders in a garage so
watch out for that with rear tech and
maybe something that you've never
expected and it may not be you alright
so the next set of stories we have is of
the weirdest hardware that we've ever
used so we're each going to tell you
about the strangest things that we've
done with hardware or the strangest
hardware that we've used and Adam is
going to kick this off for us
now alright so this was I was thinking
through a lot of things and a lot of
tech that I've worked with in the past
or things like that I was trying to get
permission to talk about some weird tech
and because I can we we don't want to
talk about that ever but so I but one of
them that came to mind was the letter B
architecture from Intel so this is a
time where game engines are being
solicited to do everything from visual
simulation to you know the next
generation of games to real-time movies
to everything else you've got this kind
of arms race that's going on with the
cel processor and Larrabee has its
multi-core device Ford's gonna do cache
coherency across all cores and and and
all this stuff and then you had nvidia
and AMD before acquisitions and it was
just kind of an insane insane time and
people were trying to figure out okay
how do we get more from these processors
how do we how do we do more with the
knowledge that we have in building these
combo cpu GPU things and this was one of
those things and no disrespect to
anybody you know if you were at Intel
and worked on Larrabee but this was one
of those things that came in it was
super secret people were individually
NDA there's all this time that we're
trying to figure out okay what can we do
with this thing we had this locked in a
room where people were ki badged access
to be able to work on this and I was
working for for a gaming company at the
time and it was it was a mini
refrigerator when we initially got it
and it was heating up the room so much
so that in North Carolina winters people
get their coffee and lean against the
Larrabee vault because it was warm felt
good and I would have these
conversations with people that were very
high up in this project and I would talk
about look I'm doing math on the physics
side of things and I know there's
there's only a certain amount of
coolness you can get to this thing I
know that you can't get below it was
roughly seven hundred and eighty watts
for powering this thing which is
untenable for consumers and I would get
into these round-robin discussions where
someone would say yeah that's fine it's
not a problem on the on the commercial
sector I said yeah but you're aiming
this at consumers and they said yeah but
it's fine
this is no problem on the commercial
sector and I would say we're aiming this
at consumers and they would say no it's
fine were that's
not a problem right like I like are we
in a different world is this something
where does commercial now mean consumer
is there something I don't know but it
was one of those things where it was
supposed to be the next big thing in
this arms race for scalable processing
and not just Intel as they're going
after this all these different players
were trying to figure out how to be the
next massively distributed processing
solution and and nobody on any front
want to say man we could be wrong
because this this wasn't all eggs in one
basket kind of thing and I would get
diagrams like this or some others that
were even funnier but and I would have
these conversations and it was purely it
was a math physics thing where you knew
what things couldn't couldn't be done
and that's that's far down the road
right when these projects start people
don't start these things to fail right
this is another one of those hindsight
things but what it was for me is is it
as new technology comes out I spend a
lot of time just asking some very very
fundamental questions everything from
audience and I do a lot of math I do a
lot of computational modeling I do a lot
of stuff to figure out okay what has to
happen for this thing to be successful
and this is the thing that honestly
taught me that because this was one of
those things on paper with who's
involved and everything else should have
been amazing I was talking to some
people on the competitive side earlier
today and they said yeah this thing had
a sweating bullets we were really scared
of this thing and then then we were
totally not scared at all so this was
this is one of those weird ones where
you know it was the big dog is the inner
pound gorilla that was gonna take over
the everything we were all in and and
they just kind of flamed out horribly
okay so many years ago I attended a
program at Carnegie Mellon called the
entertainment Technology Center anybody
heard of it so if it's a there are a few
of alumni here I think I've seen a few
of them so it's a it's a very
interesting program where we explore a
lot of different things we explore
different technology in different ways
of interacting with technology and one
of the projects I was on there
an experimental exercise game and we
kind of decided to just push the bounds
as far as we could we decided to see we
had picked up different kinds of
hardware that could be used for exercise
games and we liked all of them and we
just decided to use all of them so we
have in this game we have a Wii Fit
balance board each of the players are
standing on a Wii Fit balance board and
there's a Kinect that is tracking just
the upper body so the arms and the torso
and then the each player is wearing a
polar heartrate monitor and so we have
this exercise game where the players are
on a balance board steering a hovercraft
in the world so you lean to steer your
hover board and it's a cooperative game
so it's for two players so to balance
boards one Kinect two polar heart rate
monitors and an Arduino on each side is
capturing the data from the polar heart
rate monitors and pulling that into the
computer and the game is actually
responding to the player's heart rate so
as we take a baseline right at the
beginning and as your heart rate
increases you become more powerful in
the game so by actually engaging in the
exercise mechanics you get stronger in
the game and stronger in real life you
know from exercising so so this was a
really bizarre thing that we did where
we combined all these pieces of hardware
together and it's totally not something
that could ever be shipped and
distributed to people but what we found
was that this is the perfect kind of
experience to have set up at at an
arcade or at events or at carnivals or
things like that where there's a lot of
people milling around who can take turns
using it so we ended up making it as
sort of an installation piece that could
be set up at different places and it was
a ton of fun to work with but it was
also really bizarre because we had so
many different things that all had to
work and if one of them didn't work then
nothing worked and so sometimes we
didn't know which thing was was causing
the whole thing to stop working so this
was a really bizarre and exciting
project and up next is Andrew
so I talked about a bunch of weird tech
but I wanted to talk about a weird
devrel problem I had so we're gonna try
to like wipe all the names so that way
no one's incriminated in this but back
some years ago we were reached out to
buy a new hardware partner and they said
hey we want you to come test this
headset and we want you to make stuff
for it we love what you do and I was
working for a giant government
contractor so like we want we want you
and so we're like okay what do we got to
do to do to get this done and they're
like well we're not gonna give it to you
unless you come out for a three-day
exploratory jam with which you sit with
our engineers and we teach you all the
intricacies of this headset and there's
like unreleased to the public
nd aid stuff so we're like okay cool
yeah we'll do that so we fly out myself
and my business partner at the time we
fly out and we're like we get to the
hotel and we're like man the address for
the for the summit thing looks like it's
super far away and so we had to come up
with like a code word where it's like if
we think they were going to murder cabin
we got to have a we gotta have a word I
think was like bananas where it's like
we're gonna leave right now like hey do
you have any bananas and so we drive out
to this this place and we're driving up
winding roads over and over and like
mile like we drive like 15 miles into
like woods to the top of this hill and
it's a house and when we get there were
greeted by a guy wearing like no shoes
that's just like hey what's up welcome
to our dev cabin house place and we're
like oh holy shit we are we're actually
in their studio okay well that's cool we
can roll with this and then throughout
the experience they kept coming back to
us like after they had like showered and
it was like clear that we were like kind
of in their living space it was super
weird because like we're you know they
they invited us and we're part of this
large consulting firm were like we're
here this is gonna be very professional
we're ready to go and it was like super
different than what we expected as these
people are like coming out of showers
I'm like their tech is like taped
together and then we like started to
make so we're doing this game jam and
they're like interviewing us at the same
time and and all this stuff is taped
together and we're you know we're just
trying to piece something together in
there and we're like okay we're making a
game and they're like we didn't invite
you to make a game can you please go get
some government data and do government
things like all right so long story
short we ended up piecing together some
visualization of us with US Census data
and and put all this together but you
know every night on our way out and it
was a three-day thing they would be like
be careful going back to your rental car
there's Cougars out there we were like
alright so we're what does any of this
mean right what does this mean for for
Deverell or for anything well this is my
my pitch to you if you are doing
middleware or hardware or anything like
that which is which is don't like don't
do a jam in your house like it should've
just been in the hotel like you just go
rent a room and yeah like I get that you
want us to integrate with your like
super cool studio sitch but like I
really don't want like your engineers
dripping wet hair like hanging out over
my computer so yeah that was super weird
would not do again
all right so this is a video I stole
from online which is just a running
hamster ball so I'm not gonna play it
but imagine people getting run over by a
giant hamster ball it's kind of a
metaphor for this experience but so
basically we we went to an unnamed event
and saw an unnamed developer so it's a
similar type of situation where they
were doing VR where you could walk in
the VR space because you were literally
in a giant hamster ball size size for
humans so it was quite an interesting
experience because it didn't work and
then once the hamster ball started
moving you had to like try to stop it
because it would continue to roll past
the threshold of the movement that you
thought you were making and so then they
gave us their API so that we could work
with this so that we could integrate it
into our game and what we saw was at all
that it was doing was that the hamster
ball was like triggering oise d--
commands and so like so that the hamster
ball was somehow telling it which one of
those four directions was supposed to go
and and i was like wow this looks the
same as the default unity code for like
making a character move but thanks for
the example so then we talked with them
a little bit about like what they saw as
a plan for collaboration because we were
already getting a little bit suspicious
and they were like well this will give
you great exposure and I was like okay
do you have any money or a financial
arrangement or like anything that we
would get out of this and they were like
well we've got five Facebook followers
but we're working on a sixth and so I I
realized that like the problem with some
hardware relationships fortunately I
didn't have to say bananas in our
situation but the the people who are
trying new interesting hardware
sometimes aren't quite all the way there
towards the new or the interesting or
the hardware part and so
they they will often try to get you to
spend a lot of time in energy doing
something that would have absolutely no
value to you ever and then tell you that
it will probably pay off for you in the
future and so we decided that hamster
ball VR was not really for us and we
moved in the direction of finding more
reliable partners so just something to
think about when you go into your next
partnership all right we're on to the
last set of stories and we are getting a
bit short on time so we're actually
going to do these stories a little bit
faster
but I wanted to close with something
inspirational so we're all going to talk
about something that either we're
working on now or that we worked on
previously that we're either inspired by
the work itself or inspired by the
hardware and what the hardware has to
offer and Adam is first all right so my
dad is a lifetime medical doctor my
older brother is a doctor that's
literally working to cure cancer my
mom's a certified registered nurse and I
make video games so and and I and I say
that tongue-in-cheek because I believe
what we do is is amazingly important it
elevates people's narrative needs are
their souls and it's it's a very very
important thing but I make this joke all
the time that you know hey I got all
this medical family and I make games and
you know Thanksgiving is really awkward
I made this joke one too many times with
a friend and he grabbed me and drove me
to his children's cancer ward where
they're playing Disney Epic Mickey on
the Wii which is a game out to producer
on and it was a really good reminder of
how important what we do is and that
those things have life beyond what we're
trying to do for people who are trying
to tick the Metacritic checkbox or get a
certain number of sales or anything else
and then you have these things where
people are using these tools for very
real-world therapy situations and not in
a hey we want to experiment a little bit
but we're we found these things to work
the image on the left is is for a
physical therapy for and from Getty
Images for people who elderly folks who
end up getting serious serious physical
injuries
one on the right is for wounded soldiers
and getting uh motor skills back and
things like that the other thing that
people don't always know is we went too
far alright so Nintendo wants to now say
hey we're gonna do with the sleep
monitor thing and people are like this
is crazy they're making a sleep monitor
thing and they had this this slide that
talked about you know where they fit
into the hell thing what people don't
realize is Nintendo is hardcore involved
in physical therapy and restoration and
the medical community to the point that
I saw a controller that was built it's a
chair it's it's for quadriplegics that
has the control for breath and chin and
things like that for a Super Nintendo
that Nintendo built so that people could
play they were way ahead of anybody else
for people with disabilities being able
to play games and they have a whole
thing that they do that they never
talked about would probably be upset
that I'm talking about they have a
massive effort that's a formal
commercial effort where it's it's not
for making money it's for making lives
better and I think that's something to
forget as we look at these companies as
commercial consumer entities there's a
lot of good that's being done by
companies like this that's that's even
more than just hey we're giving money
for this good thing they're actively
innovating for tools that help people
with physical therapy and restoration
very inspiring to me
so Arlington National Cemetery is the
pride and joy as much as you can say
that of the Military Cemetery system in
the United States and very early on in
my career
they went to we you know during the time
they did a review of Arlington National
Cemetery and they said all right let's
let's see if we can figure out kind of
who is buried where and they found out
very quickly that they didn't know and
many times they would exhume remains and
they would actually find out that it was
something different than what they had
and their old system was a car catalog
in a basement so if you had a relative
who served the country and either died
or had served long enough that they were
part of a select group of individuals
who could be buried at the cemetery then
you would have to go there go down to a
card catalog pull a card photocopy it
and hope you can figure out the grid
system with which they layout the
headstones so this was a congressional
level scrutiny that went that happened
and it kicked in it was all sorts of
stuff and the company I worked for at
the time Booz Allen actually stepped in
and said you know what we can fix this
the iPhone 2.1 had just released and we
can now make apps and we know some
people I worked for a group called
analytics who can make an app to fix
this so we took we bought back when
iPhones were brand new we bought 200
iPhones we bought 200 mo fees before
anybody knew what the batteries like
external batteries were and we hooked
them up and every night for two months
something like 50 different basically
privates went out and took photos and so
we built the app one of the challenges
was we couldn't get any data into the
database because it needed to be secure
so we actually used emails and we ended
up just like whatever we could do to
tape it together the whole development
cycle of this process was the first one
was released in two weeks and then we
would just do two-week iterations
sending it out figuring out how to load
200 iPhones in a time when app
development was novel
and we did this and this turned out to
be something incredibly powerful and
while we didn't make all the money in
the world and yeah we used it to show
off our capability we didn't do it to do
that we did it because we felt a
responsibility especially considering
that one of our biggest customers was
military we felt a responsibility as
citizens to jump in and do this service
and today if you go to Arlington
National Cemetery the data we collected
can be used for the ANC Explorer app
which has a bunch of really great
features including if you have a
relative buried there you can go locate
them and it will GPS you right over
there so it was really powerful and it
was a it was a really interesting
project and I had to visit Arlington
many times and I had a visit Arlington
many times and work with them while
active funerals were happening and that
was really it puts a lot of emotional
weight on the importance of the work
you're doing and I think that sometimes
we need to remember when we're doing all
this work that it's not just in a bubble
that we're affecting real people in real
lives
so my story's a little more
self-absorbed than Andrews but so I had
a pretty low point in my development
last year because I don't I don't know
if you've encountered this but indie
development can be hard sometimes so
we've gone through a number of struggles
to make sure that we can release the
games that we want to on our schedule
but also still you know eat food and so
at the middle of 2016 I felt like it
seemed like there was a no light at the
end of the tunnel and we ended up going
to e3 and meeting with people who are
part of the steel media network and they
wanted us to participate in this global
VR challenge and they told us that you
could work with this exciting new
Chinese headset that nobody had heard of
at the time and so we had this thing
sent to us in like it literally looked
like somebody just wrapped it with saran
wrap and jammed it into a small box and
then shipped it 7,000 miles to us we
opened it up it like didn't even work
properly at first we had to like get
help to do it and then I had to pay for
a visa to be able to go to China and
there's part of me thinking that I was
singing that maybe I was going to China
to like never return and become some
kind of like slave laborer or something
but I actually didn't have the money to
pay for the visa so I asked if they
could maybe try to cover me or something
like that and they actually said yes
which I was amazed by so I went to this
10-day trip in China and our game ended
up being pitted against 200 Western
developers and a similar number of
Chinese developers it came down to 10
Western developers and I think about
five Chinese developers and we actually
won the competition and the grand prize
and so I literally went from not being
able to afford a $300 visa charge to
having $17,000 in both a fake and a real
check
though I tried cashing that one and it
actually didn't work very well but so
what what I thought was interesting
about this was just a couple of things
like so when you look at like the IGF
IGF has I think over a thousand games
submitted now but we were competing
against a couple hundred games because
we were in a category of weird tech and
a lot of times groups will actually
provide incentives like this to people
who are working with weird tech because
they want to see somebody actually make
a success out of it and so if we hadn't
been in that ballpark of being willing
to stretch ourselves in that direction
and do something that was like super
goofy and strange we never would have
gotten this kind of prize and so I think
it's important to always look for
opportunities where you can find
something that might be a novel or
unusual idea that could actually pay
dividends for you as a developer but
yeah so also I I wish that I had
something to say about soldiers but
okay so matter port is anyone familiar
with matter port this camera great what
matter port is for the rest of you is
its it's a way of being able to take a
essentially 360 scan of a property like
your house and be able to reconstruct it
into this beautiful Google Street View
type of viewer that you can see online
one of the other things they allow you
to do is be able to get a 3d model of it
so we thought this is great let's take a
3d model of a property and be able to
navigate through it in VR unfortunately
if you look at the top right image
you'll see that some of the triangles
break up a little bit it's a good
approximation for a single 360 imagery
at a time but for those areas that
you've scanned but the overall 3d model
isn't as crisp so looking at it another
way instead of doing a VR walkthrough
what if we use this to remodel a
property so we threw it into a game
engine and then actually remodeled the
house and put down different floorings
and different pain until we found a
scheme we liked and we did this actually
to remodel and house that we were buying
and moving into and this was going to be
in one of the toughest days of my life
because any type of change is difficult
and it was time to say hey children the
house you've known for your entire life
we're going to move to another one and
we walked through the house and I forgot
what had happened moments earlier months
earlier which was up on the right photo
but as I say yeah we're moving to a new
house and they walked through then they
started looking around looking kind of
funny and then my older son says wait a
minute this is the house from the video
game and he starts running around he's
like and our beds are gonna go over
there and then you're gonna be over here
and yet and this over here and they were
and they were super excited and running
around and it just I just looked at
their mom and I said look at that like
that was that was easy that was a great
day
so that was to us
the power of the game engine and that
I'll always remember what what that did
for us as a family and just how special
it was that we live in this time in this
age where we can create simulations of
living spaces that children will see and
then be able to inhabit so that's my
story all right so I just want to say
thank you all for coming out to our talk
and thank you to all of our wonderful
panelists for sharing their stories
we've ran a little bit over time so I
don't think we can really take questions
unless they let us stay in here but you
can all come up and chat with us
afterwards if you want so thank you all
for coming
[Applause]
[Music]