>>
LANDES: Hi, everyone. I'm Matt Landes with NGDU and we are very happy to have Katherine
Isbister fresh out of GDC as part of the series games at Google TechTalk series. Katherine's
a professor of Computer Science and Digital Media and the Director of the Innovation Game
Lab at NYU among many, many other things. Katherine and I have been in an ongoing conversation
over the last year about games and their potential impact on society, education and human interaction.
And what I love most about Katherine's perspective is the way she's--she combines the reflective
academic and the serious gamer. It sort of comes together in one person. So, please join
me in welcoming Katherine Isbister to Google. >> ISBISTER: Well, I'm super excited to be
here. It was definitely worth sacrificing one day of GDC and I hope you'll agree with
me. I'm here to talk today about one of my passions as a researcher which is how to bring
more sophisticated understanding to the design of movement-based interaction. So, I want
to start with the question of whether people think this is really the future of interface.
How many people remember seeing this movie? Oh, it's no sound. Remember Minority Report?
Right? And this kind of like data glove thing and this sort of really somewhat sinister
gestures. And is this how you guys want to--want to interface with the computer of the future?
How many people think that would be really cool and fun? It's okay if you think so. Yes,
raise your hand and wave it. Excellent idea. Okay, I saw a few people. I'm going to show
you what I think would be really cool. The audio on my laptop, so I hit mute. Okay. Here's
what I think would be really cool as the future of movement and interface. Let's see if I
could get the sound up. >> Oh, yes, I'll shake that booty. Oh, my
God. Don't make me laugh. >> ISBISTER: So, you can hear her friend egging
her on. And they are having the best time and it's a--I'll turn off the sound. It's
a--it's a really happy, silly, wonderful social experience and I actually think movement has
a potential to bring this to interacting with computers. And I'm not totally opposed to
the Darth Vader-like approach. I think that can be really intense and cool and fun, but
I think that approach is a little bit overrepresented in how we think about movement and technology.
And here's another--here's another idea. How many people played Dance Central on the Kinect?
A couple of people. Okay, cool. I saw the most amazing talk about this game at GDC this
week. And I highly recommended it. If you guys have access to the GDC vault, go check
it out. They go through, I think like 10 or 12 prototypes of Dance Central and how they
figured out how to teach people how to do the dance moves, how to have moves that weren't--weren't
too hard, what was fun about it and so forth. Really, really sophisticated. What I liked
about it is it showed that it's actually possible to get people doing really interesting full
body complex movement as part of engaging with technology. And, you know, as we have
things like the Kinect but with a lot better resolution and ability to track us, we will
have access to that kind of motion, right? And I think this game shows that it's actually
possible to make movement like that accessible and really fun and really interesting. And
that's another paradigm and I'll get into why I think that's interesting a little bit
as we go on. So the truth is that the future's already here, right? I mean, now is the time especially
since last year with the advent of the Move and the Kinect when almost everybody has in
their home very sophisticated sensor-based ways to engage with the console machine sitting
in their living room. So, how many people here have a Kinect? How about a Move? Anybody
have The Move? How about a Wii? More people have the Wii. Right. Okay. And also, this
is scaring the bejesus, actually, out of the games industry, which is that these kinds
of platforms now have really sophisticated sensors and multi-touch and cameras and so
forth, and are starting to have really amazing experiences like games and game experiences
themselves and they are a much, much broader market. And they're even more of a bridge
into everyday interaction with technology, right? So, essentially, the future's already
here. You already have a palate of movement-based devices that you can use to create interactions
with software, right? And I have to show this. How many people have seen this video? This
is really impressive. Okay, one person. It's amazing. This is a cat interacting with an
iPad. I don't know if you're going to laugh at me about the idea of trans-species opportunities
with platforms but, I mean, think about pet insurance. Does any body here have pet insurance
now for their cat or dog? You wouldn't believe the giant market that pets have become, okay?
So, I mean, I'm kind of being tongue and cheek here, but on the other hand, I'm not. It's--what
has happened with the iPad and the iPhone is that there's become a tremendous number
of applications for little children, okay? I have a four-year old and she grasps the
touch-based interface right away, okay? Just like this cat, basically. This cat is playing
with this imaginary stuff on the screen just like it would with, you know, a real ball.
And that's pretty astounding, right? There are also researchers who've been studying
how the elderly interact with touchpads. And they found similarly that people who get really
lost with the mouse and keyboard and really complex menu structures have a much easier
time dealing with multi-touch interfaces. So, this is just so--going to be so ubiquitous
and people are going to be doing more and more of their everyday computing on these
devices; people and their pets. Okay. So, that's just pretty awesome.
>> My dog ate my homework gives a whole new meaning to it.
>> ISBISTER: Truly. My dog deleted my homework, right? So, you might think because it's movement
and movement is something we do everyday that this is really intuitive and easy to do, but
the truth is, it's really not. It's actually really hard to design good movement and gesture-based
interfaces. And it's really easy to lose the games that we've made and designing all these
consistent rules for things like graphical user interfaces. If you want to see an excellent
rant about this, I highly recommend Don Norman's rant about how your touch-based interface
is a step backwards in terms of consistency and all kinds of things. And also, I would
refer you to this really great talk by the Harmonix Team at GDC about how many mistakes
they made along the way in trying to teach people how to do these dance moves and how
complicated that turned out to be. So, you guys are really data-driven people. This is
a concept that Will Wright uses a lot that I find really useful, which is this notion
that, you know, if you just go into a space without really understanding it, you can,
without realizing it, design towards local maxima in the design space. And you can achieve
something really great for this little design territory that you've decided is the ultimate
and the best possible choice. But you could just be way off base in terms of what would
actually be optimal and thinking about designing the solutions. And so, I'm going to present
you to these that I think people often fall into in designing what are now called Natural
User Interfaces or NUIs. It's kind of the buzz word. One of them is making the mistake
of thinking that the person should act like a machine. In other words, if you look at
even some of the gestures with the Kinect or if you looked at that Minority Report video,
you can see this sort of robot-like--does anybody remember the robot dance? Right? Like,
some of those gestures are very robotic and linear, right? And part of that is because
that's easier to recognize. So, how many people have ever seen "Very Nervous System" by David
Rokeby? Anybody? Let me show you the video while I tell you the story. So, David Rokeby
is a really brilliant Canadian artist and he was one of the first people to design a
reactive system that would--that would track your movement. Let's see if I can just cut
right to his interactions with the system which he looked pretty cookie. He's kind of
a cookie-looking guy. Okay. So, basically, what the system would do is it would generate
dynamic music, and this was in the '80s, in response to David's movement. So, he created--he
handcrafted this whole thing. And he--and he's a gallery-style artist. So, he told this
wonderful story. I was at the Banff Institute and he told a story about this horrible thing
that happened to him, which is that he took this piece and he installed it in the gallery.
He tested it, it was working great. He was ready for opening night. All the people came
in and it didn't work at all. It wasn't responding to their movements at all, and he was like,
"What happened?" He was completely freaked out. He tested himself, it's still working
fine. So, he goes back and he looks to his videos. He was videotaping himself interacting
with the technology. The system had trained him to move in these bizarre ways, okay? He
had entrained himself to move in really strange ways that the tech picked up extremely well.
It happened completely unconsciously. And so, he was able to use the thing brilliantly
like a--like a master of this instrument. But no one else could play it because it wasn't
actually intuitive at all. And I think, you know, it's great that we're really plastic
as human beings and we've definitely taken advantage of that over the years with these,
you know, with all the limitations we've had with processing power. We simply had to behave
more like machines for the machines to understand what we were doing. But that's really no longer
the case. We have the processing power. We have commercial grade sensors out in people's
living rooms. And we don't need to move like robots anymore, but I think we still carry
this strange kind of local maxima of act like a machine and how we think about crafting
and designing these interactions. Okay. The other one that I think a lot of people fell
into in designing the early Wii games that I think Nintendo was masterful of avoiding
was it's the same thing as the problem with endlessly chasing the ultimate VR augmented
reality situation, which is being overly literal about creating highly realistic movement.
And the Harmonix guys actually told a really interesting story about this. They said that
they were pretty close to locking down for development. And they had come up with all
these dance moves, they worked with a choreographer. They are ready to go. And then, they started
doing testing, like really extensive testing and they realized that how they had thought
about the progression was they would sort of gradually unlock moves and there were certain
moves they were putting towards the beginning that they thought were really easy, but even
those moves were incredibly hard for people who weren't good dancers to do. And they had
gotten all caught up in having this highly realistic and really engaging choreography
just like real dance and hadn't been thinking enough about what was accessible and approachable
and engaging for this tremendous market they were trying to reach. So, they actually did
a really funny thing. They had--they posted internally and asked people who had, like,
two left feet, who were, like, white males who hated to dance, like everybody who totally
hated to dance was always traumatized by going to any school dance, they got them all, like,
in a bus an took them over to the dance studio where they've been working on the game. And
they had an emergency reengineering of the choreography. Like, basically a sort of like
a basics, like a dance for dummies kind of training session where they quickly came up
with some moves that were super accessible to somebody who really felt like they had
two left feet. And they said all of that really, really saved the game because they realized,
you know, it's not about being Michael Jackson or whoever, Lady Gaga, right, playing this
game. It's actually about offering people this kinesthetic feeling of being a wonderful
dancer. And you--and you probably do that better by giving the moves that are kind of
easier that they could really pull off because then they'd feel really graceful. So, it's
the same thing with designing, you know, game worlds. You want to make people--you want
to always think of like a hyper-reality where you're magically graced--more graceful and
more interesting and more flawless in your performance than you are in everyday life,
right? You don't--I mean, it's like even in the Sims, it's kind of bizarre that you have
to, you know, take them to the rest room. I mean, it's kind of like a funny joke, but,
you know, you rarely see that in movies. They're parts of everyday kinesthetic reality that
we really don't need to imitate and we certainly don't need to do that in interface either.
I think people get excited about that as a challenge and go chasing that. And certainly
from a technical perspective, it's really interesting to see how precise centers can
be, but it doesn't mean we always want to design in exactly that way. So, how do you
tackle this design challenge? Well, one way you can do it is you can actually, you know,
offer a cash prize, which is what happened. There is open in EyeConnect Library and PrimeSense
sponsored this competition and they wanted to--people to create a natural web browsing
interaction. And this one is pretty cool. I actually tried to see a demo of it at GDC,
but it wasn't working, which is the bane of all demos. Let's see if I can skip to the
swimming part. This is really kind of a cool idea, which is basically that you swim through
the web. And this is actually a free library that you can directly hook into. You can go
via Kinect. You know, we use it with Unity 3D, which is a game engine. But you could
use it pretty much with anything. And this guy hooked up his web browser. So, you swim
forward to open the link and then you sort of backpedal to back out of a link and kind
of swim to the right, swim to the left, right? It's a very dreamy kind of, you know, pleasant
and relaxing way to interact with the web. Now, I'm not--I'm not saying this is the be
all and end all. And, of course, even if you had, like, 10,000 applications developed in
this way and people voted on them, you still wouldn't really understand why do they work,
why are they successful. So, my answer to tackling this question is to wear many hats, some of which are very silly.
We actually made these hats in my lab. But what I do is I combine thinking about this
problem from a human-computer interaction approach and a very designee approach and
using my background in social science and having a deep respect for what games--commercial
games have already begun to do in terms of making the user experience really important
the moment to moment experience. So, these are some of the weird things that we do in
my lab. We build strange evaluation tools. This is called the sensual evaluation instrument.
Ask me later if you want to know what it does. We build research prototype games that allow
us to pull data out of the games and do controlled comparisons and interesting things with them.
And I'll walk you through a project like that today. And I have books and other things you
can read later if you want to. So, I'm going to tell you today a little bit about this
Emotion in Motion project that we've been doing as trying to tackle this question of
how do you--how do you start to develop a design language for interesting movement-based
interaction. So, basically, my approach is I go to the social science literature about
how people work and I try to figure out where are the sweet spots for thinking about how
to--how to use a new technology that's going to work really well with people. And then
we've been creating games as research instruments because I think you have to have fairly rich
prototypes to explore some of these ideas. You have to have something that's dynamic
and engaging enough that you can see if it's working or not, all towards applying this
to interesting application scenarios including education, so. And that's how Matt and I actually
met. So, some of the Social Science we're using, and I can get into more detail in any
of this if you're curious, one thing is the physical feedback loop. How many people know
about this phenomenon? One person. So, let me describe this study. Basically, this social
psychologist told the people in their study that they were going to test out a hands-free
interface. They told half the people that they had to hold it like this and the half
like this. This activates your smile muscles and this activates your frown muscles, okay?
At the end of this interaction with this device, the people who had been unconsciously smiling
rated the device statistically significantly higher than the ones who had been frowning,
right? And then--and they didn't even really realized that they were doing this. So, this
is a really powerful thing in us and this is why your mom said "Stand up straight and
act confident" because that's part of how you decide how you feel. You're getting signals
from your body. So with the game mechanic, if you--if you have somebody performing as
if they feel a certain way, they're going to start self-labeling as feeling that way.
So, that's a really interesting and powerful effect that we can use. Another one is Emotional
Contagion. How many people have heard of this or maybe heard of mirror neurons, that kind
of thing? Okay. So--so the theory is basically that since we're social animals from way back,
we're--it's evolutionarily very viable for us to be very susceptible to one another's
feelings. So if--so if one of us gets really alarmed, the others are like, "Oh, what's
wrong?" you know? So, if there's some danger in the jungle then, we can all quickly respond
to it. So, what that means is that we do that all the time, even with media, even with things
like TV. If somebody looks really sad on TV., if you watch the person who's watching and
they're really into the movie, they start to look a little sad too. And then their brain's
a little, you know, to some degree, it's kind of the same as if they themselves were feeling
sad. So, it's super powerful. And this really comes into play with performative and physical
games like broad body games because it's so much more legible when you're witnessing the
game. It's a lot different than console play which sometimes has that zombie look like,
you know. You--I mean, there are intense feelings going on but you can't really see them from
the sides. But when somebody is playing adventure on Kinect, you can see them getting excited.
And so, you're much more likely to get excited too. And then kind of an extension of that
is this whole notion--has anybody ever done the stadium wave? Yes, a couple of people.
This is super fun. It's weird how good it feels to get in the synchrony with other people.
And there are all kinds of strange studies about this. Like, here's one you can try in
a meeting. Take a sort of a strange posture and then wait and see who also takes the posture.
I guarantee you probably one other person will. It'll be the person who agrees with
you the most and is kind of the most on your team. It's the weirdest thing. So you could--so
this is why, like, sales people they actually imitate your body language to get you to feel
like you're in sync. Now, think about game mechanics and movement mechanics. You know,
when you have two people doing things, you can actually have them doing stuff together
they'll feel--they'll feel more trust and liking for one another. It's wild. So how
we approached this in the lab was, first, we started to unpack the Wii games, that's
what was out there at the time, to really breakdown with more nuance like what was happening
in terms of movement in those games, what was engaging and not engaging about them and
whether we could start to create some dimensions along which we could design, basically. And
then we did a series of studies in our lab and also out in the school actually because
we're part of this Games for Learning Institute at NYU. And then we created this research
prototype. So, I'm going to just show you little segments of this process. And right
now we're building a math game, trying to make use of some of these ideas. So we played
a lot of Wii games and looked at, you know, what kinds of movements were fun and weren't
and why. And one thing we realized was that happy-go-lucky loose movement is much more
silly and positive and friendly than sort of tight, controlled movement. And I'll show
you these two videos. This is actually an actor performing along certain dimensions
of this Laban Movement Analysis. It's like a dance notation but it's a really--a really
great illustration. So this is floating, this is maybe more slow than a lot of the mechanics
we saw that people liked on the Wii, but just really light, and joyful, and mellow, and
happy, trippy kind of--right. And contrast that with this, the sort of slashing motion.
This is--this one on the--on the right is super fun if you're doing like a Star Wars
lightsaber game, right, because you feel like, "Whoa, I'm really powerful." It's kind of--it's
kind of has this whole different vibe, right? But both--both kinds of movements are pretty
flowing and allow the rest of the body to get into the game and that was something that
we saw that seemed to be really important. Just to, like, a little micro comparison,
here is a game called, We Cheer that people did like a lot. And it had--it's basically
a cheerleading game, but even the guys in my lab were like, "Oh, yes, I'm cheerleading.
This is fun," you know, because it had fairly free flowing movements and it allowed the
rest of the body to get involved. This was a game called Boogie Superstar and it was
also a dancing game, but basically they took this approach where you had to move like a
machine. You had to do your movements to a very strict metronome. So all of a sudden,
you were kind of doing the robot as you were dancing. It wasn't nearly as much fun. It
was much more mechanical and complicated to try to do. So we're starting to see, "Oh,
wow, there's actually some nuances of movement that seem to really shift the character of
how people feel about interacting." So, I'll show you some footage from my in-lab study.
And we did just video recording. These are some dancers doing We Cheer also because we
were sort of curious about what happens when there are people who're really super good
at the kind of thing the game is teaching as opposed to people who are novices. Do we
get really different kinds of motion? And one thing you can really see in this video
is if you look behind the two who are performing, you can see how engaged the women who were
just watching are and how they get so into the action emotionally. You can see it in
their faces. And they had really interesting things to say also about how you train movement
and how the various games did a good or bad job of training movement and that's another
interesting factor in terms of if you're going to innovate new interaction strategies, you
have to think about how you're going to teach them. So I won't to show this footage because
this is middle school girls and, you know, we don't usually show the video of classroom
studies. But essentially what we went out and did was I wanted to validate this intuition
that--well, like, everybody has this intuition that more movement is more fun and leads to
more energy and so forth, but I couldn't find any research that really validated that. So
we did a study where we had low, medium, and high movement Wii games. So, low was Mario
Kart, medium was We Cheer and high was Wii Boxing. Has anybody played Wii boxing? It's
like--it's like super vigorous, right? So we went into the schools and then we have
these little clickers. And the girls, after every round, would self rate how they were
feeling, like how happy, how frustrated, how much fun, and what were their energy level
was just so we could see what would happen. And we actually did, like, hours and hours
of video coding afterwards for level of movement so we could validate for ourselves was it
really low, medium, and high and, yes, it was. So our hypothesis was, well, you know,
everybody thinks that more movement leads to more positive emotion and energy and so
forth. What we found interestingly was it didn't seem to be related to fun or sort of
positive emotion, but it was definitely related to energy level. So the more you we're forced
to move in the game, the more energized you felt at the end which is, you know, I mean,
not surprising but it's good to check it out. And we actually have a Laban Movement Analysis--analyst
coding the video now looking--she's really interested in whether she can actually see
frustration as it forms in the body. And since we have all this self report frustration data
at a pretty micro level, she's going to use that to crosscheck her observation, so that's
sort of cool. So then we went on to create a research prototype--oh, and I have a paper
about that work. If anyone wants it, it's in submission right now, but I'm happy to
share it with you. So then we did a research prototype and, of course, you know, there
are lot of things that vary between Mario Kart, We Cheer, and Wii Boxing besides amount
of movement, right? So the most ideal way to test out this theory is to actually design
a game where the only thing you vary is movement and so that's what we did. And we used these
silly hats because we want to make sure that people would move a lot because sometimes
with Wiimotes people cheat. They figure out the accelerometer thing. They--and they're
less likely to do it when they have it on their head. Also, it was just really fun.
So let me just show you a brief little video about Wriggle. One of my students composed
this music. >> Wriggle! invites two players to put on
handmade Wiimote-powered hats to play a mood-management game. Wriggle's design is based on research
about how our emotions are influenced by how we move. The game includes mood ratings both
before and after play for comparison. >> ISBISTER: We also had them do pen and paper
measures to see how that compared with nodding your way into self-reporting your emotions,
which might be less subjective. We just thought it would be really fun to do.
>> This stand-alone game has two stages. In the first stage players collect critters by
imitating their motions. >> ISBISTER: My students hate this video.
They took a video of me playing the game and put it up on our website as revenge which
was fun. >> Grad students?
>> ISBISTER: Yes. These are grad students. >> In the second stage players compete in
a balance game in which each player struggles to keep his/her hat on in a wind storm.
>> ISBISTER: And people did start pushing one another when they played this game to
knock each other off balance which is sort of interesting.
>> And after three rounds, player users rate their emotional state again. The game also
captures Wii accelerometer data and play metrics that we'll use later to analyze whether our
movement and emotion research holds true for games as well.
>> ISBISTER: That's Chelsea's happy dance because she won. So I think you guys get the
idea. So this is actually a work-in-progress presentation at CAI. I don't know if anybody's
going to be there in May, but we'll have the game setup so you can come and play it if
you want to. So what do we found? We found the same pattern that we did with the school
study which is that--and you can see the little graphs here, you know, for valence, which
is positive versus negative emotion, it was equally fun, really, to play the game whether
you had the movement or not. But we got--we definitely got a significant effect for energy
level. People felt much more energized afterwards. And we also got this really interesting effect.
It was not--it was not a significant effect, but it's something that I'm planning to go
after which is people who played with the hats felt much more of a sense of social connectedness
after going through that experience compared to the people who played with the keyboards.
So it sort of underscores the interesting social aspects of thinking about this stuff.
Okay. So let me talk about how we're using this stuff in relation to, like, practical
concerns. While I was doing this research, somebody showed me this fascinating study.
How many people have heard of this power pose study? Okay, one person. This is just really
bizarre. It was some people from Harvard and Columbia and they had people either be in
this high power pose or low power pose for 10 minutes. And basically, they made them
spit into a cup before and after it and they measured their hormone levels and then also
offered them some fake money to, like, take--make a bet to see how risky of a bet they would
make. People who stood in the high power poses, they're--like, literally, their testosterone
went up, their cortisol went down and they are much more likely to make very confident
and kind of risky bets. And people who were forced to stand in this low power pose, just
for 10 minutes, had a significant--it had a significant impact on them, like, biochemically
as well as behaviorally, which is--I was like, "Whoa." So, I started thinking about that.
And I started thinking about kids who have math anxiety, because I've been in this Games
for Learning Institute and I was like, "I've already seen how I can change people's energy
level. What if I could literally make someone feel more confident while they're doing math?
And then maybe that would kind of break the cycle of them being really anxious, right?"
So we're actually--we have a little--this is super rough. It's much more polished now.
I would love to be able to show you guys the demo live, but you'll just have to see this
super crappy. Like, this was, like, in January. Basically, we came up with this very, very
simple math game idea which is you're landing fractions on a number line, okay? And so in
one condition of the math game, you have to keep expanding the number line in order to
catch the fraction. So you're inadvertently taking these really strong power poses and
I actually ran this pass the woman who did the research in New York City and she's like,
"Oh, yes, that'll be great." She's going to look at our prototype, so just to double check.
And the other condition, you have to keep the number line kind of small and you have
to constantly catch things a little bit lower to the ground, so you're being forced to stay
in the low power pose condition. So we're piloting this, this spring in a local middle
school. And if it--if it goes well, there's this really interesting experiment in New
York called the School of One which has highly customized math learning. And they've said
they're willing to actually pilot this to a much greater number of kids. So, I would
just be so excited if this actually works and I would have loved to learn math this
way. But, anyway, this is just to show, like, a super practical use of this in case you
were starting to feel like, "Oh, this is so Evil Empire," like, "Influence people's feelings."
You know, "Hmmm, very insidious." So it can be used for the forces of good, I guess, is
what I'm trying to say. And then in terms of next, next steps, I'm really interested
in building from these kinds of research-based building blocks to rethink how we approach
gesture. I feel like we have a window of opportunity here where it's not all totally set in stone.
It's--you know how it would, like, the whole story about the Dvorak keyboard and the QWERTY
keyboard and so forth? I think we have this moment now where it's not all completely locked
down, where there's a chance to actually get some really interesting social and emotional
considerations into play in these devices that we all interact with everyday. And I
think it has all kinds of implications for people's wellness, for education, for all
kinds of areas. I personally would, instead of sitting on my butt at my desk all day,
sorting email out of my inbox, like, why can't it be more like gardening or like cooking
together? Like, why can't my mundane data processing task actually be helpful and social
for me? And with these kinds of devices, there's not really a reason why that can't happen,
except for lack of imagination in my opinion. So this is like a–-like a recent open library
of multi-touch gestures. And, I mean, I think this kind of work is very valuable and fine,
but it just--it feels like that robot thing all over again. It feels like, "Okay, let's
all act like robots and come up with these very efficient sort of least common denominator
movements." I think there's a role for that kind of thinking, but I would rather see something
like this, right? Like, why not have Gmail tai chi, right? Like, why not? I mean, if
you have a Kinect in your living room, couldn't you sort your email that way into folders?
Wouldn't that be great? You could do that in the morning. Maybe it wouldn't be so horrible
to go through your inbox, right? You wouldn't have to declare inbox bankruptcy. You'd be
in so much better shape after sorting your email, like, "Ahh, sorting my email," you
know. I'm just saying. And personally, I think the way to get there is, first of all, to
really understand the science of what happens to people through movement and through gesture
and dive into that side of things, understand essentially how we're hardwired and how we
use these things in our everyday life and really think about moving with that material.
And it has huge impact on our emotions, on how we connect to other people and moving
around. There's this research that I read recently in the Times where you can't just
go get an exercise bicycle for 30 minutes and sit the rest of day. You're in just as
bad a shape as somebody who never got on the bike, right? So we need to find a way to incorporate
movement all day long into our lives. And so, I think that there's a lot of possibility
there. And so, I guess it's an incitement to think past the more robot sort of local
maxima into another space. It isn't necessarily gestural like hyperrealism, but it's something
that augments our everyday lives in really wonderful ways. And in case you think this
is not practical, I want us to tell you the story about the pretty ATM machine. Has anybody
read this in an HCI textbook? So basically, I think this research was done first in Japan
and then validated in Israel that for some weird reason, people would report identical
button structures as easier to use on more aesthetically pleasing ATM machines. I was
like, "Easier to use?" And it did seem like it was more efficient for them. They're actually
being more efficient. So the Israelis who replicated the study were like, "Ahh. We think
we figured out why this is so." And the reason is because of this thing called mild positive
affect. Has anyone heard of this? Basically, it's this idea that when you're in sort of
a mildly happy mood, you're much more creative, okay? You problem-solve better. So the Israeli's
theory was you actually make people slightly smarter by putting them into a little bit
better of a mood through the aesthetics, okay? So if that's true for graphics, I think that
would be very true for gesture. And I think that's part of the reason why Apple is kicking
everybody else's butt with their early multi-touch devices, okay, because they figured out this
gesture thing really fast. And I have a colleague who talks about how that initial--how many
people have an iPhone? Okay. That initial interaction where you--where you unlock it,
she compared that to petting a little kitty cat. She said, "The very first interaction
you have with that device is like a little caress. It's not like a button press. And
it's not like a--I mean, I love Android, too, but it's like, those, like, unlock the little
connect the dots thing. It's like [makes sound] right? But this is like, "Ahh," right? And
so, I think this is pretty potent stuff, right? So--and maybe this is partly because my husband's
allergic to cats and I can't have a cat, but I would like all my technology to be much
more like, you know, playing with a kitty cat. Then--or the kitty cat can play with
it for that matter, rather than me having to internally do the robot. So that is my
talk. I am happy to take questions. >> So, I mean, the second part of your argument,
when I heard that, that seems, "Oh, okay. Yes, that's right." So people are [INDISTINCT].
So I was trying to understand the second part of your argument connecting it to the first.
>> ISBISTER: Right. >> Is the pieces that even in areas, like,
if you're trying to design a Gestural UI not for fun purposes but for, you know, "serious"
purposes, having people have fun with it a little bit and being in that mild positive
affect is going to make it more useful user interface.
>> ISBISTER: Absolutely. So let me give you an example. So right now, I'm working with
a professor who does security applications at my university. And we're working on authentication
techniques. And he's really interested in the biometric aspect of it. But my argument
is people hate typing in passwords. They come up with crappy passwords they use over and
over again. If you design, like, kind of an awkward, annoying gesture, if you design a
gesture that feels really good to do, guess what, people are going to comply more. So,
there are very practical areas, not just sort of a subtle more positive affect, but also
just--we can change how much people want to perform certain--that task they may have to
perform anyway through the design of the movement. >> So would you say it's safe to say that
you could adopt a similar research approach even if you're designing a successful UX even
if your--even if your goal isn't to make a game?
>> ISBISTER: Absolutely. Well, I--so I think game developers have an evolutionary pressure
on them which is that consumers either buy or don't buy the game only based on whether
it's fun. So they're at the vanguard of understanding that moment to moment engagement and how to
maintain it. So I'm actually an HCI person. I came to games because I was like, "Why do
avatars in characters working games and nobody could make it work?" Like, remember the Microsoft
Paperclip? That's how I got to games. So I actually think games are a very interesting
space to watch more for that experiential expertise.
>> I agree with you. I think the games are on the bleeding edge of UX design and theory.
I really do. But I'm a hardcore gamer coming from the game industry, so. Does anyone have
some questions? >> Just sort of following on from that question,
so I've been reading a little bit about games. And one of the things that I've been reading,
which I'm not sure if this a mainstream view, is that one of the things that's fun for people
about games is that they have unnecessary obstacles that people actually like games
that are difficult and challenge them, but that necessary obstacles are not fun in the
same way. Like, if, you know, your boss puts a lot of obstacles in your path...
>> ISBISTER: Right. >> ...that's not actually fun for you.
>> ISBISTER: Yes. >> So, when doing experiments on how fun these
sort of movement-based games are, how can you separate out the part of the fun that
comes from an unnecessary obstacle that might not be fun in a real life interface?
>> ISBISTER: No, absolutely. >> You know the kind of fun that's like through
the grace and sort of happy aspect of the movement.
>> ISBISTER: I think it's--I think it's a deep consideration of mapping, you know? I
mean, if you think of, you know, Hutchins and Norman and like--I think you want to map
that arc of experience of kind of the knot--knotty problem. Like, not naughty, like, you know,
A-U but K-N-O-T-T-Y. Like, that's fun to unravel, too, and that is a big part of what games
deliver. They sort of deliver this, "Whoa, I'm really frustrated. I'm doing something
really hard." "Oh, that was so great," this kind of arc of experience. Like, I don't think
it's about easy fun. I think that's only like a first layer of what makes games compelling.
So, definitely what I presented here is more that surface of ease people's way into the
movement experience, but I think there's also--like, if you think about the Move, which is Sony's
device, it's probably the most accurate of the three devices, and I could really see
people developing very refined ways to move through data or to investigate things that
are very kinesthetic and precise. And there might be some kind of, like, pulling weeds
in the garden. There might be some very laborious parts of that that are strangely satisfying,
right? So, it is not just all about, like, ease. Yes. I didn't mean to give that impression.
>> Actually, I have two questions but one was the cheating you mentioned. I have a couple
of friends who bought the Wii. They have all the gaming consoles and they bought the Wii
and the guy beats us all kind of, like, laying on the couch just like flicking it because
he figured out all the motions. ISBISTER: Yes.
>> And then it was not really any fun. >> ISBISTER: Yes.
>> What have you seen in terms of is it a type of user who cheats or is it a type of
game that causes the user to cheat? Are there ways to bring them back and get them engaged?
>> ISBISTER: Yes. I think it's a combination. I think if you're a hardcore gamer and you're
used to the precision of a console device, like a handhold controller, the designer has
to really win you over with the movement-based interaction, and the Wiimote is not that accurate.
So, I actually gave a talk at GDC last year about the nuances of designing that kind of
kinesthetic realism and I contrasted, like, Star Wars: The Force Unleashed which was this
lightsaber game which had some really great movement stuff, but also some really complicated
combinations of buttons and movement. And people would sometimes kind of back off to
the flicking thing with another game which now I can't remember the title. But, basically,
it was all button stuff, so it maintained the tight control except for, like, going
in for the kill which was like this. And so it just isolated the most fun, crazy, like,
crescendo and the hardcore gamers love that because they didn't lose the control and they
still got kind of the big pay off movement, right? So, I think it's a combination, right?
Whereas if you come to these games without any gaming experience, something like--have
you seen Boom Blox? It's sort of like Jenga, like, you pull out the little--the little
tiles. Like, that one works really well. It's very natural, intuitive and fun because there's
not all that baggage about precise control embedded into the genre and the audience and
the expectation. So, I think it's kind of a mix.
>> Got you. So, it's not about the--well, it is about the type of person or there are
some people and you have to win them over, basically.
>> ISBISTER: Or you just--they're entrained in something and stuff. Also, I think, like,
in the--at least in the U.S., like, guys have kind of, like, performing physically, doing
crazy things physically is like, "Okay." Like, there's definitely that kind of tendency to
back off. >> That was actually my second question...
>> ISBISTER: Yes. >> ...was my favorite thing about the Kinect
is the menu navigation. >> ISBISTER: Yes.
>> Really intuitive to just learn how to work your way through a game, but then I really
didn't like the dancing unless I've had enough drinks.
>> ISBISTER: Yes. >> And I have the same thing on my tablet.
So, I'll dock my tablet to a Bluetooth keyboard and I type my email away.
>> ISBISTER: Yes. >> But I like touching and swiping the major
actions. >> ISBISTER: Yes.
>> But I haven't seen a lot of games that support that or do you know of games or if
developers are aware of that--of that combination of more buttons and then menus for just sort
of swiping and partial elements? >> ISBISTER: I think you're going to see more
sophisticated mixes in the next year in how Kinect games operate. I mean, I think that
was such a radical evolution and I think Microsoft put out certain standards. And I think everybody's
still getting the hang of that blend. Plus, with Kinect, you've got the voice recognition,
right? So, it's really hard to figure out what the right combination is. But I think
we'll have a hybrid world. I mean, we will still have to type sometimes, but if language
recognition gets really, really good and, you know, we're not in a crowded environment,
we may talk to our computer a lot of the time, you know what I mean? I think we'll have to
think very multi-modal and maybe that will allow these gestural interfaces to do things
they do well. Like, I do think sorting--and it's the same thing as you see how the iPad,
the apps on the iPad are very different. Like, I'm really interested in seeing iMovie on
the iPad. Like, how do you video editing with just a touch-based interface and does that
feel good and--instead of control? >> You should try it on the Android soon.
>> ISBISTER: I should. I should. Yes. >> We have a nice sort of movie editor.
>> ISBISTER: Sorry. I don't mean to be an ad for Apple.
>> And, actually, Gmail on the--Gmail on the new Android tablet is actually pretty nice
because you can--you can just swipe your emails into boxes. So you have this nice flicking
motion... >> ISBISTER: Yes.
>> ...I'm done with that email. >> ISBISTER: I will definitely check it out.
>> And--but then you type. >> Hi. So, to start with having a great deal
of trouble connecting with what you're saying, I guess, you know, I'm white, I'm male, I'm
English, I don't drink. >> ISBISTER: Right.
>> My favorite computer games are logistical games.
>> ISBISTER: Right. >> I like managing buses and trains.
>> ISBISTER: Yes. >> And I get really frustrated that the interfaces
don't have macros because I want to type a program that explains what I want the train
to do. >> ISBISTER: Right.
>> And have it execute... >> ISBISTER: Yes.
>> ...and I do not want any real time elements at all, thank you very much.
>> ISBISTER: Right. >> So, you come from another planet. I'm sorry.
At the same time, I actually work on Gmail and I came to hear you talk because I was
thinking, "So, how can Gmail be better? How can we..." Because I've gone--all my life,
I've been waiting for physical manipulations to happen to data. I've always wanted to send
files by dragging them across to my friend and letting my friend have them. So--but I
still can't put it together and maybe it's because of I have trouble with the emotional
connection. >> ISBISTER: Yes.
>> But at the same time, I need my mail interface to be something that will work when I'm on
the bus. I need it to be something that works when I'm in the office.
>> ISBISTER: Yes. >> I need it to be something that works when
there's someone else in the room with me. >> ISBISTER: Yes.
>> And dancing isn't going to do that even if it possibly provides some of the ideas.
So, this is a pathetic question. Help me put these things together.
>> ISBISTER: Well, so, I think you have to--I think you have to imagine that this is a modality
for interaction and that it would be valuable in the future to bring people's bodies into
play and engaging with their everyday digital work life, right? So that, sure, I don't think
it'll ever go away that you want that, especially with something like email where it's very
text-based and you're reading it and you need to have all kinds of different screens and
that kind of standardization. I think you should see it more as another avenue into
engaging with that data, right? And in--I mean, maybe another way to look at it is to
consider all the work that's being done with data visualization and exploration and thinking
about, you know--I think about people like--you know, I know people who work at places like
Pixar who do, like, 3D modeling. And touch is a very promising way to help those people
who are--who are manipulating things to do their work in a more fluent and approachable
manner. And that may turn out to be the case with data management and visualization to
some degree for all of us including things like email. You know, like, how many things
do I have from this person versus that one. Those kinds of big picture, global view things,
things like sorting, I mean, I don't think it would ever replace you directly coding
an application. I just think it's another layer, it's another lens that can come into
play. It's just like how we were all struggling at first with how do you put the web on a
small mobile device? Do you completely change the interface, you know, things like online
banking that look really different? Do try to find some way to have some standard--I
mean, I see this as the same struggle. And, I mean, I think there's a really wonderful
book by this woman called Maxine Sheets-Johnstone that's all about getting into your physical
body and trying out strange exercises to make you aware of the manner in which you move
through the world physically. Like, she has things, like, "Make up your bed as if you're
frantically late." Okay? Now, make up your bed as if you just got back from a two-month
vacation in Hawaii. And then she's sort of walks you through, "Notice how you tuck in
the sheets." It's like--and noticing how that makes you feel like. So, I don't know. I mean,
I think that you can get it as a--I have it on my kindle reader. So, like, I would encourage
you to explore that if you're kind of like, "What? What is she talking about?"
>> Okay. So, I realize there's--I realize there's a second thing that I should be asking
which is you're talking about things that people do for fun, but what about intellectual
pursuits? What do we know about how people behave and how they move when they're doing
something intellectually? And then when I'm talking to someone, I lean forward, I lean
back, there are changes in my facial expression. >> ISBISTER: Yes.
>> If I'm French, I move my hands a lot. And I think if I took a movie of myself playing
with Lego, which is a physical pursuit I greatly enjoy.
>> ISBISTER: Yes. >> You'd see a lot of those things going on,
the sort of not conscious things that I'm doing in order to manipulate the world but
they're presumably exploitable cues. >> ISBISTER: Absolutely. Yes.
>> Do you think there are prospects perhaps for...?
>> ISBISTER: Oh, definitely. Like, there's these parallel systems of reproach and avoid
that are really well documented in how people behave physiologically and that sort of approach
movement, right? Like, you lean in, you're interested, you're engaging versus the kind
of shrinking back. I mean, there are some very practical pragmatic sort of survival-based
areas of this work, so. >> I just want to jump in and see is there
anybody on the remote locations who has a question?
>> So, I have a question that's more for, like, power users or hardcore users going
back to, like, both their points. So, for instance, like, in my case, when I play games,
I'm, like, very competitive sort of like almost competition level. And so, I'm wondering,
like--so for all these things, it's very convenient and it's very nice when you try to introduce
it to new people who don't know anything about it.
>> ISBISTER: Yes. >> But then for me, for instance, like, I
can't--like, the reason why I don't have Wii, a Kinect or, like, any of those is because
it just doesn't work for me. Like, it's--it doesn't work. Like, I can't get--you know,
how many of their reactions I need in order to...
>> ISBISTER: Right. >> ...feel like I'm doing what I need to do.
So, do you have any, like, plans to address or try to help, like, hardcore users really
transition over? >> ISBISTER: Well, I guess, I think that hardcore
game genres that have a evolved on console controllers are a really beautiful thing that
I would not want to mess with. It's sort of like--I've skied my whole life. And when I
was like in my early 30s, I tried out snowboarding. And snowboarding is great, but I was like,
"Ugh. I suck at this. I don't really want to master this. It's kind of drifty. I can't
ski super fast on a snowboard," and I sort of abandoned it. I was like, "I'm not interested
in this. I already have this other thing I'm doing." So, I wouldn't think of it as a substitution
like, "Oh, you as a hardcore gamer now have to use this thing." I think some of the developers
made that mistake. I don't think that's true at all. It's like telling somebody who plays
a violin, "Okay. Now, your violin's going to have a lot of buttons and you'll blow into
it like an oboe. Wow. Isn't that cool? Don't you do that with your mom?" It's like, "No."
>> It's probably cool for some people. >> ISBISTER: Okay. Well, for some people it's
cool, but you know what I'm saying. Like, I don't--I think that that's a confused message
that somehow propagated in the press. And, I mean, I think--I think of console gaming
like that is sort of, like, virtuoso piano playing. Like, you wouldn't just change a
piano or creating different instruments for engagement.
>> You mentioned a couple of examples where the motion to play the game makes you feel
better such as the smiling when you have to... >> ISBISTER: Right.
>> ...smiling versus puckering or frowning. Have you studied at all how much those factors
lead to repeat play or, like, more play? >> ISBISTER: No, I haven't. I think that would
be a really interesting question. And I'm actually starting to think after my research
that what physical gaming really does is it changes your arousal rate and then use other
cues to decide what the valence of your emotion is. So, the physical feedback stuff with the
pen was your face. And your face has a more hardwired kind of positive-negative valence
stuff. But I sort of think with the body, I don't know, I think there's the approach-avoid
access and I think there is energy level, but I think we do a lot of contextual labeling
of, "Oh, I feel really..." and then it's either terrified or excited based on what's happening
in the world, so. But I think that's a really interesting question. I definitely want to
follow-up on that. >> So has there been much work on using motion
interfaces when there isn't visual feedback? Which it seems almost all games rely pretty
heavily on that. >> ISBISTER: Yes.
>> I've been wondering about, for example, in cars where you shouldn't be looking at
all the controls and... >> ISBISTER: Right.
>> ...everything else or on your phone. Is there some useful results there that maybe
can help us there? >> ISBISTER: I haven't seen useful scientific
results, but I've seen some pretty interesting, strange, independent games. And then there's
a real movement among the game--independent game community to get away from social gaming
where you're both still staring open-mouthed at the screen. Like, they are like, "Wow,
we really--we want people to look at each other while they play." So, I think it will
be an interesting space to watch. But I don't know of any sort of scientifically grounded
non-visual literature. >> Hi there. You talked about avoiding the
local maxima in your research base. >> ISBISTER: Yes.
>> So, I was very interested in your example of the dance central where they did all this
polish and then at the last minute, they brought in what I guess was their target users and
they couldn't use it. Is that...? >> ISBISTER: Well, it sort of crapped up on
them, I think. I mean, they tried really hard to make it super accessible. They did realize
early on that someone who absolutely hates dancing, they could never really make get
up and dance. They were like, "Well, we have to have someone who's at least willing to
try." But I think what they did was they got so caught up in working with the choreographers
and planning out the moves and they themselves became more expert at the moves that they
had that classic thing where they kind of lost their beginner's mind. And then when
they--when they went out and tested it, they're like, "Oh, no. It's too hard." And that happens
all the time with games. It's a big problem in gaming.
>> I guess the follow up was, you also talked a little bit about, like, perhaps you could
say, how much polish do you need in, like, a prototype? Do you think if--when you can
get something valid out of the testing. >> ISBISTER: Yes.
>> Like, it sounds like they went really far before they did.
>> ISBISTER: No. >> No?
>> ISBISTER: Actually, that's why you should get a hold of their talk.
>> Okay. >> ISBISTER: Because it literally showed all
of their prototypes. And one of their main points in their talk was you have the moral
responsibility with these dynamic systems to fake your way into a good enough prototype
as early as possible when you can still make changes. And they were super creative about
it. They did Wizard of Oz stuff. It was really cool and I was like, "That's why they're really
good at what they do." And, I mean, I've got really intrigued by the culture of design
at Harmonix. I was like, "How did they do that?" because lot of HCI people don't do
that, right? And--but, yes, I mean, I think the trick is the dynamics have to be tight,
but the visual of fidelity doesn't always have to be so tight. And that's not how we
usually think. So, I mean--and that was the nugget I took away, but I don't think you
have to have a highly--I think you have to have highly-resolved timing to see if one
of these things works, but you can fake that a lot of the time.
>> Really? >> ISBISTER: Yes.
>> They said one interesting contrasting comment. So, I'm a project manager who loves working
on projects that have engineers like Steven--is that your name--who thinks so critically about
every little bullet. And--but what I do is I get up on the board early on a project and
I gesture. And I gesture how the data is going to flow and I would love it if my gestures
just turned into the program one day. >> ISBISTER: Yes.
>> Or if there was a way to--us to both sit around the table with Lego Box and build the
application flow and hit submit and, you know, off it goes. So, I think a combination of
the two really works. >> ISBISTER: I think it's a very cool app.
I hope you guys build that. >> Thank you very much.
>> ISBISTER: You're welcome. Thank you.