okay so my name is under car I think
I'll be speaking about automated image capturing with comments and a car let's
and this is work done in collaboration with my advisor pick elliott Stanford
University
and thanks in part to to the stock before I dive into the talk I just want
to give you a bit of a brief description of my back route because
its importance some other approach day take to
are data and modeling so this is a screenshot from my website
arm so I started off at your own risk get run over I did my
undergraduate degree I did a double major in computer science and physics
and 2005 to 2009 from there I went to university in British Columbia where I
worked on
learning controller Sports Inc physically simulated articulated figures
is basically means our robotics and simulation roughly
from there I thinking here to Stanford and I am now import your PC don't
working a machine learning a computer vision and
with a particular focus on deporting now in between these
are I managed to squeeze in two wonderful internships a Google research
and so both times I work on large-scale on people earning apply to Bo's
the first time around it was in terms of unsupervised learning so we're training
all kinds of are on course
on YouTube data and a second time around it was a supervised learning
and so we're training large publish all that works to our special temporal
beatrice
based on labels up from YouTube and so I thought I should have a cheesy quote
on my website as well so here it is it says I like my data large my other than
simple
and my labels week what I mean by that is really when you approach these
problems
their problems and practice you have this trade-off between the quality of
the data and the quantity of the day a
and so I like to be on the side we have lots of data and its weekly labeled
in terms of a simple approach on
up I like to this probably informed by my pics background for all the
fundamental equations are very small simple and sweet
as I like my models to be in the same way very small compact
simple and still very expressive and I train them a large datasets
that's how I like a to do my model and you'll see that throughout the stock
okay so now I'll get to on the court the image capturing work first let me show
you ever going to get to by the end of the stalks let me show you some results
there are some results that our model will produce the model will be looking
at
image so for example this image the model take a look at that image
and is going to describe it with the sentence and this generation will happen
on the word by word basis
so the model will look at this image and it will say this is a man
in black shirt s playing guitar period
so model the sides when it should and went said enough
and then that becomes the sentence and so we'll see how this works by dee-ann
and I'll open I'd like to make here is that
on many of these images are kinda almost right
and we'll see this at the end when I present you with more results but this
man in particular is not actually playing the guitar his
tuning the guitar and he doesn't have a black shirt he has a dark blue shirt
but close enough on this image here the model looks at it and says it's a
construction worker and Orange City West
is working on the road and so I don't think this is a road
I think it's more generally just a construction site but relatively close
two young girls are playing with Lego toy it's impressive that I got the Lego
but I don't believe that their two young girls I would not pass to the slightly
whole all the lady still very young lady and
the last one boy is doing a backflip on a wakeboard so it's very impressive that
got the Backflip
but this is not a week or two troubling and some but you might
see why and what think that it could be awake or as I have a few more examples
here
I but up but I'm not going to go into detail on this
our first before I go into the model and describe how this all works
let me try to zoom out first and give you and I D bout
why are we interested in this problem rate ranks all
what is the problem we're trying to address here and the problem is that
we're
overflowing with Mitchell the up so my advisor faith and she likes to compare
visual data on the Internet to the dark matter
all internet in a sense that visual the ice everywhere in terms of I'm just and
video it occupies most of the space on the Internet more importantly
occupies most of the attention of people because people up images
and get our algorithms have basically berry little idea but what's inside
those images and of course we'd like to deal to fix that would like to be able
to and exit
like to be able to understand it and would like to make that data useful
to people
and so not only do we want to attack these images
are not only do we want to protect some number X categories for the summer just
were actually more I more ambitious and we'd like to actually
in general are devise on algorithm that can enable
natural language interactions between people and natural
and up image data image collections so and I mean that
in both terms up the input and also the output as a let me give you an example
of what I mean
here's an example of something I would like to be able to do I would like to be
able to dispose a query like this
im just ami scuba diving next to Turtle and
I would expect that a good result would look something like this: so
years and image of me scuba diving neck sweater or this happened roughly
two years ago and I Y as the time on
actually touched a startling to change my life but that's a different story
and so I would consider this to be a good result
and I would like to convince you that this is actually an extremely difficult
problem at to retrieve this image for this query
first of all these images are just giant three-dimensional blocks numbers tried
we have numbers that indicate the brightness every single XY position
and it every single color channel I'll just have literally
this image could be a million numbers arranged in a three-dimensional grid
and we have to understand patterns up this in the image moreover when I look
at the screen
it's for example talking about turtle you when I understand the semantics
after all we know what it refers to
we know a lot of things about turtles but when ogre them when given this query
this might as well look like this
right replaced every single character with a different character and
so not only does the algorithm have to understand what all this stuff is
not only does it have to understand how changing the order of these things and
how it matters and what it and plus
not on the understand these millions of numbers in terms of what they represent
has to also understand the relationship between these two
what model remains in East understand that this thing to get here
is actually referring to a particular pattern of that total
and of course that Terrel could be warranted and so many possible ways and
positions and scales in
just a complete disaster right how can we ever hope to achieve in a task like
this
and so when I am faced with a very difficult task like this and
complex multimodal task I like to use neural networks to solve this problem
and r2 motivations to this talk from my perspective first
I'd like to convince you that it's actually really important to establish
these bridges between natural language and visual data
and second I like to kinda I'm be a champion at the neural networks approach
to problems and us let me cut but give an idea of what I think up
what I think applying and look training
and using neural networks looks like in practice it looks something like
this aka so here we have a neural networks practitioner
and she is building what looks like to be accomplished on a roll at work
and it looks to be about nine layers deep at this point and
so the help but point here is that basically it's a very flexible
composable kind of structure and we're building up expressions and training
them
and two and India now we've seen your own networks become relatively pervasive
on over the last few years and I think the major innovation is not really in
terms of any of the details apparel that works and how they work
but the innovation i think is kinda end up innovation in neural networks
is the mindset when you approach to be a problem I get
in particular suppose that you're tasked with
describing image just what sentences what might a normal mindset be
if you were to task if you were to do that um seem I think okay well this is
John three-dimensional Walker pixels
so I need to somehow extract some understanding of what's inside it
see my tweet for open CB because obviously is very good with me just
and you think okay well with one each in what kinds of things would you like to
describe the sentences would like to maybe describe people
so I should run a object a person defector
I should run a face detector maybe I want to classify some actions maybe I
want to classify
some scenes soak up with all kinds of intermediate representations that I want
to extract
el cumple the pipeline to feature processing pipeline for tracking
all these things about this image and it will be written in C bus bus
now I want to extract on I also want to describe that stuff and sentences
and so I need some understanding up syntax sentences and how things
follow each other parsed response a speech tagging and so on
as I reach for coronal P which is very good on natural language processing talk
at
and understand sentences and how words are put together and understand the
language model
and
then I come up with the wage basically serialized like arbitrary process going
to any representations and to file
I love them in Java I use corn LP to somehow create all the stock
and my point is that you end up with this giant unwieldy pipeline a
preprocessing and
intermediate representations that you think might be useful and instead the
neural networks approach
is to is to kind of pressure from different mindset in a sense that
we're going to do is we're going to step back and we're going to thank were
tasked with the problem of describing
images were sentences what is the input what is the output
rights right to function from inputs to up but let's make it
entrepreneurs the let's get some data at the train those parameters
and then let's crush it was because the great incent as I in particular and this
particular
in this talk I'm going to present this model we're going to take accomplish on
your own artwork
which we know cassini ever going to take a recurrent neural network which he
knows is very good
at modeling sequences and then we're going to do is just as if we were
playing with LEGOs
as we're going to just stick them together that corresponds to that a row
over there
and you might think that this is kinda hockey thing but in fact in terms of
neural networks
this is a very very on natural thing to do
because these neural networks can speak vectors to each other and they can also
speak gradients to each other
so what's the wire them up then you establish the bridge
and can train the whole system and two and on data
so not owning so not only
so these two components can learn to work together to establish or and type 2
towards her and ask so for example this worker network
and order to generate the correct sentences for this image it can tell the
commercial networks with a gradient signal
how it should change that speech representation to complete the task
better
think about what it would look but this would involve in terms of if you work
with open CV and if you're working kernel p
you have these arbitrary API's and think about it when
involved if coronal t1-weighted tune some hype renders
based on your feature extraction pipeline is just something coble there
is a very natural thing that you can do in
if you're building your stuff our networks
arm yet so that might that's my cell and you're going to see you're going to see
some about by the in the school
by the end of this summer watcher talk so first let me describe these
components independently and then I'll show you how we can put them together
and how the whole system works
together suffers others cry published on or what works and give you a whirlwind
tour of how these
systems work so here's an example california roll at work
from roughly 1998 on the system particular from the look was one of the
pioneers in this field
and a neural network is really a single function
from images to class course in this case
so in this case for declassification so say we have about 10 classes at the end
and we're creating a single the principal function from the inputs
to the outputs and then we're training at on the a and
these neural networks have become very popular as a result in last two years as
a result of some recent success
on the image not challenge with a challenge as
run by that a lot that I come from and pick his lab and it has been compared to
cut up the World Cup computer vision
so every year we hold the challenge and people from around the world summit
their organs and the label some data and then look at the progress on the field
in terms of recognition and detection and so on as a function of the year
and of course we're making progress what's interesting is there's a
kink in here and 2012 and that cake response to the introduction to neural
networks and roughly 2012 and this was the
network of I projects key Oscar just got a skier and gentle in 2012
so this was a large published on our tree with deep use
on large dataset in 2012
this was the only college will not work submitted to the challenge in 2013
roughly half of them
were published on that works in 2014 all of them for commercial networks
and so this has really taken the field by storm
and the last two years have been very exciting because the entire field upside
down
they're looking pretty everywhere and its kinda been an exciting time in
applying these networks
across the cross in computer vision on
we're looking at these errors when the result in 2014 came in
and this is a 6.6 percent pop by pair and when you go through when you step
through the sky just looking at the predictions are the commercial network
you start to get the sense that this works almost magically well this is
predicting correcting one by one by one
and we got to thinking what is there a gap between the six by six percent error
from Google and human performance so this is something I became
a special interest in couple of months ago so I devised
on web interface that would allow a human to annotate
images these are image let me just among 1,000
was not categories so this was an interface for you scroll around
and you're trying to annotate but you might think this is for example here
this is a hot dog
and then we found that
this was actually not the very simple task at all we found that the human
error was
after you go through 1502 just a human or is about bike 11 percent so there's
still some got too close
and by human a running my air in particular because I had to go through
fifteen hundred images an entity demands was extremely brace
and i couldnt on comments anyone else to do this so this is just mean
and I'm not sure maybe it's a bit noisy and someone else to do it I but I so the
web interfaces
online and can all compete against the comment and see how well you perform
now there the reason this was interesting is that we're trying to get
an idea about work on social networks still haven't got too close
and a so we have some results on this and you can read all about it on my blog
I have a more detailed analysis but just to give you a just
of what comes out of this is no it works are still not very good at handling
very strong felt worse they're not very good at reading
so the South Sea salts that struggle for human but for a computer that might be
difficult
they're not very good when the objects are very tiny there's a real
over there it's very difficult to see and they're not very good at dealing
with abstract objects so here there's an abstract shape
a patch at you have to have 3d understanding to understand that
accomplish all that work works mostly on that extra level and so it
doesn't see any wooden and metallic picture so it doesn't think it's a
hatchet
but conversely they're very very good at
fine-grained categories for example here we have an image of
a dog and saw the commercial ever looks at this 30 more seconds it's estoy at
the shipper key
and me as a human labeling guess I would have to be sweating for five minutes
trying to find a great class as I'm scrolling through this crazy interface
and so counselors are very good with capture Bay stop picking on that by docs
extremely well
and they were probably all so I'm trying to go through kind of a whirlwind tour
of how these
models work
here's an example commercial neural network from we call this commercial
network
be judging at all Oxford not it was introduced by Karen Sloan and it was one
of the submissions
into the emission of 2014 challenge and
what I like about this neural network is that it's a it is extremely simple and
homogeneous Architektur
normally what you see is that when people introduce a model somewhere let
people keep reading papers to make the model more and more complicated
but in fact we've seen the commercial networks is that in fact for making them
simpler
first time to throw away stuff and that's encouraging so this network is
extremely homogeneous
in terms of a how it works they told us three by three completions in two by two
polling
are going to that in a bit but our throughout so
Inc so basically week but and image here in the top
for example this is an array of 224 but you trying for my 3 if you haven't
image up with and I 224 and we're going to transform the three single the
personal function into 1000 numbers
at the end and these one thousand numbers at the end I apologize to people
and back to camp maybe see this but these are the class probabilities
for one of the one thousand different classes and management and so when you
squint at this
a network when you look I more closely you'll see that there's these three
different types of players that were stacked on top of another
one another they are comedy show airs there are pulling wheres
and there are political heirs someone to go very briefly over
these layers every one of these layers is a very good for a simple as a very
particular type a function
so the API for every single layer and accomplish all that work is extremely
simple
every single layer takes a 3d model numbers and produces a tree model
numbers
the only constraint is that this function is the principal because then
we have to later back propagate through the pork picture
in other words we need to know how every single input influences
wanna be up its about layer and so here's an example commercial network
the consists of only four layers and every single layer is a 3d volume
and I'll be speaking about three dimensions with height and depth
the for example image is a to 24 by 24 by three bombing
with Heights 224 and ep3 for the three color channels
and by the end we'll work our way to one by one by one thousand
so one by one specially because we would have had we would have merged
all the information specially and 1,000 for the different process
initially so first I'll begin with the political heir
one of them is for example here at the end
political it here's an example for the collector where that takes is some ice
on bike patrol bottle
producers and one by one by 4,096 calling
and you can think of every single number in this bombing as a firing up in your
own
and opening your own in quotes just in case our neuroscientists and
audience because otherwise the tricky but
every single year on it but what it does as it has connections
to all the inputs and is balling and
very simple and it also has a set of weights doubling
every single year old has a different set of weights and just being a dot
product with the following
of something and buy some bias and this W&B those are learnable
and so it could be used as the late response experts be and then that
becomes the activation
there's also use being activation function that
are the neural Comput right after it takes the dot product and in practice
what works very well
ish just brush holders and there're okay
as every single year on this out the bombings independent
independently computing the late response time sucks this operation is in
fact just the giant matrix multiply
so will take all the numbers here and the bombing which stretch them out
and then we have major publication with the wait matrix wherever single row is
are the way to put it on your own so just do w transpose X
and we special detail very simple operation
commercial airs are very very similar to looking at the glass
but except for these neurons that looking at the entire bowling
they're actually looking at the small our local region
Indian but so here are some there are looking at only a tiny patch of that but
and the idea is that these receptive field start out small you're only
looking at 10
actress but we're going to stack the scholarship players one on top of
another
and always with a slightly bigger receptive field in the image so we're
building up from
small adjust to bigger and bigger on kinda up
receptive fields where we can recognize more complex patterns
and this operation but we can be implanted very efficiently with
collusion
so I'm not going to go into details I'm just trying to give you an idea that
this can be all very what a very efficient on-chip use pgp friendly
the last layer is the polling layer and this is only doing
down sampling so if you're at the bowling is a size 2 24 but to try for my
64
then the up it would be half of I'll the width and height
and the same number of that as a really what's going on as there's a Dell
settling on a per
depths likes basis independently so let me give you an example
10 the depth slices here could be 220 from a 224
activation lapse well we're just going to resize the don't want well
as they can't get the bottom in particular the functional form that
if you're doing max pulling into by two strides with two by two filters
is just separate out your input slice into two by two regions intake and Max
in every single a cell and those become your outfit
and you're just talking sup and the reason I'm going through all this as I'm
just trying to give you a sense for
there's nothing very magical about what's going on and cut & Publisher
Network
all that's happening as dot product fresh from 10 as a max operations
and they're wired in a specific way that makes sense
in terms of I'm just a but that's all that's going on throughout
from the raw image to class probabilities thank you look at what
these girls are sensitive to
as you go deeper and deeper in the architecture then we find that neurons
in the beginning
once you train network well end up being sensitive to different little
edges locally because they're such appeal is still very small in the
beginning
then you start building up these larger and larger sup the fields
and more and more complex patterns so we start building little circles
and the little pieces of objects and so on and by the end we have an entire
place near on and so on
and that's what happens in practice I can also show you what this looks like
in terms of the
activation firings up a network here is a small
Asian Network it takes an image and then we have a set I'll
layers throughout so all these things are three volumes throughout
3d bonds are very difficult to visualize what I'm doing here is i'm taking out
every single slice
up those three volumes and I'm laying them out and Rose but every single
column here is a different ball game
along the way to the class probabilities
and this function here is parametrized by
a set number stumbling and we're going to take data and we're going to
tweak those W's so that's the predicted probabilities that come out
are consistent with the ground troop labels in the dataset
so and I should mention also that this network I'll put it on the very end
this network actually training come the GS which is one of my pet projects
it lets commercial networks and JavaScript so this was trained overnight
I'll see part n you keep tab open during the night and come back in the morning
at last all these features crackle useless but it's cool
our her so in terms of how the
backpropagation work just to give you an intuition for what's going on suppose we
take an image like this image with the data
24 by 24 by three array we forwarded through the commercial networks
so lotta dot products and so on and get 1000 numbers
and then we look at those 1000 numbers here I'm always showing 5 them
I suppose that network basically think that this is 40 percent chance of being
a dog
30 percent chance of being a better or something else and then of course we
want this to be a banana
so we're going to basically desire of the network
that this up but here for banana is higher then all these other up its
and we have a way of encoding that and that back in the gradient
and then we can send a gradient and not propagated through chain rule
through this entire the fresh will function in fact we're getting through
that region signal
is every single wat along the way in this commercial network
we basically computer back propagation what happens when you wiggle
that weight a tiny bit what is it's a fact on the banana probability
so for every single wat we now know how it influences that probability
and i was just going to wiggle it slightly in the correct direction
to make the likelihood of banana higher when it sees this particular image
when you do this over an overall beam adjustment training set eventually the
network parameter string themselves so that they are all consistent
probabilities
in what is in the train India so the training basically it looks as follows:
you lou until you get very tired you sample batch of data
you forwarded you find what the air as and you find a way to back propagated
and then get all the gradients not the weights and do a proper update
and in practice the same take a very long time so what you do is you
distribute this across cheap use
so I think air Simonian and this particular model for image that
he used for cheap use and it took about two to three weeks so these models take
a long time to train
protest time they're extremely efficient on orders of tens of milliseconds per
image
so the summary so far is cultural networks are just 8
single relatively simple function from inputs to outputs
their function awaits and then when we have data we can tweak those weights
so that the up with that come out are consistent with what's in the training
set
so that about the commercial network and I'm going to try to present
a bit of a car network before that I would just like to plug
a briefly that pic and I are actually teaching see if you try
213 231 and I at Stanford
and the class uncommon for neural networks and all the materials are
online
all the notes are online all the signs are online so if you'd like to know much
more about cultural networks and I encourage you to
I go through the notes the silence by the way we're using term a lock on
and with this allows us to do is that we're basically setting up virtual
machines
in the browser it some dark magic that I still don't understand
but what it amounts to from your side is that you go to URL
you have a snapshot of a virtual machine and that you forget
in the browser and right there you have a I cannot book running
and all the day I set up all the requirements are set up and just start
coding right away
Lisa simon's in can implement southwest of the farce
and I'll when Europeans and commercial networks and so on
and we got the threat throughout the silent so
let me know tell you about recurrent neural networks recurrent neural
networks are extremely good
with sequences they can't remember sequences very very well
I'll just give you some idea about this from previous work Alex graves has done
a lot of
very good work with every car networks throughout this beijing for example
he has become a network that can learn to write international plans
to give them some particular text and then the recurrent never goes and Wright
said
overtime you can also try it for example on Wikipedia
on a character level so you just read the entirety of Wikipedia is just a
giant string of characters
in just but a job a car network on it in this particular case and tell us the end
which is one
for mobile recurrent network and you can train all these I'll
I'll on the data and then you can sample novel
which BB articles basically from the models so the model can learn to write
Wikipedia
it figures out words it figures out that are
there's different like syntax for HTML figures out that any statement opens a
bracket needs to close it
it figures out all of this just from looking at characters overtime
on another example as we can trade on we can train sophisticated level
language models whether car networks there's some nice work from
must be caught in this direction he's our recurrent network language models
I don't think Google users don't but that's just an example to give an idea
where basically getting the car network to model the probability
distribution over the next word given some previous words
so when you go to google and type cat sat on me
that's the language model it's predicting what you will say next what's
next characteristic was or next word in a sequence
a lot example as relatively successful example
up from the obscure or Albany else and quickly from 2014
this is a thinking translation model using recurrent neural networks
it was really fun actually to hear any I speak about this model admits
a few months ago he said that he described as model as
minimal contribution for maximum impact and
that's a Corsa controversial statement but what he means by that
is that this model is relatively simple but it works very well
what you do with this model and it's kind of surprising that actually works
is you feed its French words word by word so this worker network sees word by
word a French
and building up an internal representation of what has written a
sentence
and then you'll just after to start generating english words at some point
and so the network has some kinda but controller presentation of what it has
red
start to generate word by word up English translation so just
a and every single step along the way is just a matrix multiply
and thresholding or something like that so it's surprising that this works but
you try and a large corpus
up French to work English up couples and this actually seems to work all that
well
from then on people have but all kinds a bells and whistles on this model and
making it
a better
but at this point I think these are competitive with the best on machine
translation models
I would like to also plug a nother on our side project of mine
I'm like the recurrent networks in JavaScript it's called the current JS
and so you can train a car networks JavaScript a few
a lot of time as the asterisk the for example here's a Tulare
on long short term memory LTM it's a form of a recurrent network
I have a fun day more on the website on
because I want to run out of steam sauna character level on Sunday to
and so the task I chose to use side not merely probably require an essay
speakers they're very
knows what up with demand those essays and I likened to the problem with those
a sense that you need to wait for the next essay and I don't want to wait
so I took all Koreans essays and i thought id all too to where else team
JavaScript
and I basically tried to encode poultry house was them into the weights this
network
and so we can learn to read all that to say and then we can learn to sample
novel essays on the character by character basis so here are some example
result
preliminary up sources
care travel program was the generator and slutty
and this is only after training for one night so someone should run this for a
long time but
the most can start up ideas are that they can a company was the worse
aka that's not very not very good but you have to remember that
the network has to learn to spell words first of all and has done that
relatively successfully
as a summer those are numbers but some ovett for example start to get kinda
deep
the startup is a meeting to think to investors and
not bad right or a they wants to succeed
ok at sites that sister do something and you can also not only sample from this
model
some random character strings but you can ask it was the most likely think
that program would say
the single most likely thing and it's things that
that is the startup has a lot of the startup has a lot of the startup has a
lot of the startup is a lot about
and I had to cut it off because they're just got to continue to grow forever
but basically a program talks about a lot of startups
and so I think this works personally aka
so basically these recurrent networks what they're doing on what they're very
good at
is that we can train all kinds up I'll character level or
word level sequences so were basically predicting
distribution over the next word given a sequence a previous ones
so for example if you have a training example up the task at sat on a mat
then really what we want as if this is a training sample
then we want to basically I'm right out the function
that expresses the probability of a word what we want is the probability of cat
given that we have assembled anything at so as the first word we want that
probably
to be high the property of Sat given that we have generated cats to be high
the property of on given that we have generated cats at to be high
a problem at to be given that we generate catsup on to be high
so every single one of these training examples was really given us multiple
training examples
for the next word given the previous work this case report written examples
so we're going to again take the same approach we're going to right down
a set of equations that express a probability of the next were given
previous words
we're going to condition on these words and we're going to need ask
the network to June its weights so that it's predicting the correct
next word so here's what that's going to look like in terms of the graph
representation
and make this if you haven't seen it before it might look a little
intimidating but I'll step through it slowing
they're going to do here is we're going to set up a graph structure
every single box here is a better and every single arrow here
indicates arm and dependence so this vector
y0 is a function of the specter h0 for example
and buy a function of I mean this is well-connected layer
so we just have a bunch of neurons connect it fully to all the girls in a
previous
volume again these are old pictures and we're doing matrix-vector multiplies to
get from doctor to doctor
so we're going to do is we're going to associate every single word
with a 300 level with three hundred
numbers vector that represents that word
because River known that works in general they like to work with better so
we have to turn
are things like strings into actorse so we associate 300
the National better with every single number will lay them out here
on the bottom
and they were going to rest but this right here and in here
at why we're going to ask we're going to have basically a classifier over the
next word
so little but this up the way this will work is that
every single why as Italian ten thousand and one
numbers and 2001 because we're assuming and 10,000
word capillary plus one because we want to have a special
and talk and that's basically the dot at the end of the sentence and so what
we're going to do is
at every single step of the way we have an expression for the public every word
the distribution and every single number tells us the likelihood of
that particular word following the previous context
so what we wanted in this case as we want says the examples cattle at
we want the highest number for y0 to
be associated with the index at which cat is in the dictionary
at Y-one Y-one as a function of cat
as a function of I'll start start ok and as a special start talking to always use
started generation so want Y-one
are right now we just have a bunch of random numbers but we want
the the number that corresponds to the index of Sat
to be high and so on arm
so the hen representation here mediates
the context up
in this network and along every at every step of the way we're just doing majors
what the vice
our times I'll something but so for example to compute h4
were taking h4 as I'm activation function of
WxH times explore so that's
a matrix multiply times this vector plus matrix multiply times
history and WxH in WH age both parameters that we're going to choosing
so that I love this works out in other words
we're going to do is we're going to take the learnable vectors up
for every single word just 300 numbers that are learnable
the progress as well we're going to like this up
and then here we can establish the gradients because here we want it
most likely were to be capped here we want most likely were to be fat
on the Matt here and then the and talk and at what for
so for example here we want the end token because white for as a function of
cats at on Matt and at this point we want the most likely were to be
been talking let me show you what this looks like a test I maybe this will
clarify
but this is kinda hard to explain type of test but at this time the way this
will work is that we end up training this distribution over next working
previous words
so we start up generation with an with the start talking
and then we forward and get a distribution over the first word
wiser so this is 10001 numbers indicating how likely
we are to see the first number in the sequence so we can separate from the
property distribution
I suppose that we a sample capped because cats could have had hi
ask or if we have trained and model properly in the previous step
so simple cat and now we forward and we get another distribution
and we can sample
we get Sat for example we forward begun distribution again on the next word
and now we sample and we forward and
sample and forward and sample and we might get the and
on the sequence and so we've generated caps at on a mat
one by one and of course normally don't have just a single example capital a map
Matt you have an entire training set up something says and we're training them
to up well I can't go back to the or just like because too many steps
or training them one at a time to always predict the next word
and the previous word as unmediated directly
and the context from the previous for will promote has been generated
beforehand is mediated by this and a lawyer has said that recur network can
keep track of some context vector
to help predict the next word in a sequence got
so I described very vaguely what recur network as
at describe cultural at work ever been to describe how we can put them together
because we don't only want to generate arbitrary sentences
we actually want to condition does sentences on the information
in the image so that's where this line will come in so for example suppose we
have an example that
has an image and its labeled as straw hat in practice will have bigger and I
just
and will have entire sentence s but suppose the training sample
so during training will proceed as follows: I'll take the image
and will pack it through the commercial network now we're not interested in some
run random 1000 categories
so we're going to actually not care about the and about commercial network
we're going to take the representation at the four thousand at just before the
possible are
sources 4,096 neurons and they somehow on a higher level represent the content
of that
image not in terms of the raw pixel values but in terms of where are the
faces where are some high-level concepts and so on
and then we're going to just like that into their car network condition
degenerative process under a car network
on the information in that image support sample
normally a return that worked sees the start talkin and compare the properties
of y0 the first word
but now those properties y0r a function of the image
to accomplish all that work support sample we're seeing some straw texture
here
so you might think that the weights will tune themselves overdid course in the
training
to bias the probabilities up y0 so that maybe we
we make the properties of strong higher
true those interactions that are learned here on the vector
so alt amounts to to convert in are and then
a sentence model to a model that generates descriptions I'm just
is we have to tweak this first and only representation
from just I'll being influenced by
the start cooking we just have to make it also depend on the image
and one of the simplest ways to do that is to just plug it in through a bias
term
attraction so we have another set of W's Wijk
that are learnable and V as the sport 1000 nicest much elector
so a very small change in equations
an hour actually conditional degenerate process at the art and on
the sentence and it can modulate the entire generative process so
this is an ex this is a the figure from my paper but actually
its equivalent to exactly what it described as just using different kinds
of
up visualizations so it s time the way this will look like us follow
as follows: with a test image and will ask network to describe this test image
so proceed as follows: we take that image the pipette into the commercial at
work
we get the 4,096 the National vector that represents that image on a high
level
and now we're going to start sampling words but that representation of these
into the personal air
it conditions the generate process and you might expect that may be wiser will
be
are conditioned to actually say strong I am
and then we continuing we settle expert you might think that it could be hot
and so on until the network samples the and token the and token
ends up being the most likely and then network will finish
left generate process at the end that we arrive at
from here commercial at work understand what it saw
and then influenced generate process under a car network
and we train this all end to end on data and we treat the primers W
so that the sentences that are generated in that recur network are consistent
with what
and training data
by the way we don't have to also go some other technicalities here
is we don't have to actually go word by word we can actually have a moron
optimal process in terms of the generation process we can use something
called Bing search
so that we're actually looking had effectively
and I also wanted to briefly address are this difference between recurrent
networks and ISDN switch that for long short term memory
on basically turns out that Ellis teams work better than our nest for many
problems
but nothing changes about anything I described the only thing when you want
to use nellis damn
is the functional form of how you compute this H
normal you computed as just metres not apply with the vector from below
I may just not apply from the comp X Factor and now we're just going to
replace that age 1 and we're going to make it a slightly more complicated
function
that has more complex multiplicative interactions and the end product is that
the gradients in the network flow a bit nicer
Anderson mathematical reason for that and fact that
network an explicitly decide to reset its head in states the problem with the
car networks is that they kinda just
keep the ball down the interstate and if you make a mistake this just can go all
over the place
and Ellis teams have an explicit for get paid and asked him if it detects that
something's going wrong it can be set itself to a fixed point
and this may be helpful NY it performs better
are not going to go into too much detail about this I turned up the datasets
our dataset that I used to treat my networks called Microsoft Coco
it contains roughly 120 thousand images and every image
as describe a fight sentences and the sentences
are written by as in Mechanical Turk workhorse so basically we get them to
look at this image
and say describe it innocent us as we end up with roughly 600,000 sentences
and roughly 120 thousand I'm just in terms of training these networks and
practice
I'll go a slightly more technical but it turns out that is very important
out to put the gradients as your back propagating through these networks the
gradients like to explode sometimes
so you just got them I'll and up
five works OK her I found that arms props and adaptive learning rate that
works
are very nice and practice and when you train these networks Armas prop
as some is basically works like at a grad if you're familiar with at a grad
at a grad keeps a running some up to squared
up some up the gradients gradients as you train
and arms property exactly at a grad but this some upskirt gradients for every
single wat
is actually key so decay summertime and so you're still modulating how fast
every single wat is being updated and network
but you're doing that not but you're learning rate is not the King money
publicly throughout the training they're still going to modulating effect that's
very useful
but you're not actually completely decreasing alarming rate as your
training
I find it useful to i make there some technicalities renderings things that
you at least likely careful with
when you look at the grading expressions and so on up for example devices are
thus of Max
if to be careful with the fact that not all words are equally likely so when
initializing properly then
I'll that's good if you don't initialize improperly than the network has to
relearn
the frequency distributions of words and you'll see that a loss function is like
a hockey stick
and that's not very good you getting large gradients storing up network
so that's like one trick that I found works nice and you have to train for a
long time
so that's but that's obvious well yes
I'll actually get to it in a bit I think I depends on whether or not you're using
sipes argues
I view CPU's I hope for somebody's expense and
up I left it running over entire winter break so that was roughly
actually like one and a half months or even two months they want to use GPS
I'm yet to go all I don't think that
would take so long it looks like it's kinda leveled off anything that is done
but then come back like you know three weeks later and you get much better
performance
and so this just keeps going on and on you get very good performance very fast
and then it just takes forever to get the last 10 percent
and I'm not even sure if it ever stops raining I just ran out of time
bpm okay one other thing I want to mention is that
a very nice feature of using neural networks is that transfer learning as
extremely trivial to do so for example here we are we don't actually have
a lotta examples up images and synthesis I turns out that
here we can take this entire thing I would like to learn it from scratch
we can take this entire network desperate rain from a different datasets
such as image that
we take all those weights or just transfer them into our model so we don't
have to relearn it and we don't have to over fed
and similarly for my word met representations I don't actually learn
them I use work to back
work to back is a algorithm that learns these were doctor embeddings
by reading Wikipedia and other sources on the Internet so basically
these were doctor embeddings I they have some nice properties
from the get-go for example all the horses the workforce
in camel are nearby in the space there are all kinds of nice properties about
these
as I take offense to that it turns out that you don't need this part actually
works well anyways
are even without doing this but it's nice that you can something different
components from from parts
right so you can imagine extrapolating this and having these kinda
Lego systems and neural networks you can imagine large neural networks very kind
of swapping in and out these weights and making a better over time
as all these talk to each other and adjust each other and this multi task
setting
on different datasets so in summary the approach is
you want to describe images were sent us this so what did what did we do
we part about what's the impact was the up but you came up with a function from
the input and
to out but and wheat was parametrized by some separates the police
we initialize part to the networks a possible from different sources because
that always helps
and then we got some data to train the rest of the network and then we just ran
with stochastic gradient ascent
for a long time and we end up with something that can describe I'm just
wasn't assist
so I'll show you some examples of what you might end up with the end
and a structure these examples the results into three categories:
the first category will be holy crap I can't believe that worked category
the second will be I can kinda see it category
and the last will be I'm not sure what's going on. category so there's this
gradient throughout these
results sluts workforce at where works very well
and you might had so out let's look at for example this immature
its s a group of people standing around a room with remotes
like nice seems to work although these are we controllers
it's not clear young boys holding a baseball bat nice
cow is standing in the middle of a street holy crap
there is actually a cow industry it seems to work you here we have a cat is
sitting on a toilet seat
display cases display case filled with lots of different types of donuts
a group of people sitting at the table with
wine glasses so likes it seems like it works but then you start going through a
lot of these examples and find things like
a man standing next to a clock on the wall
so this there's a clock in a man but the clock is not on wall
a young boy is holding baseball bat you
not quite a kappa sitting
on the couch with a remote control pretty close right
I baby laying on a bed with a stuffed bear few
almost a table with a plate of food and a cup of coffee
so there's a cup of coffee in table but there's no plate I guess
and the young boys playing has been a park it's actually interesting because
this network when you go through the result keeps confusing
are genders all the time I suspect that there must be a bug that it's looking
man and woman somewhere because it's always the other way around
somehow into the results where it seems like
not sure what's going on here we have a toilet seat with seat up in a bathroom
but there's no toilet seat here a woman holding a teddy bear premier
not sure exactly what's going on here good
and maybe think that this is a teddy bear because I'm that texture up a
sweater
not not clear and here a horse
sending in a middle of the road so I'm not sure where it seemed a horse
I can see it
one kinda on nice property but there's no and the reason I'm
printing here a lock probability is that were forcing the model to say something
about
every single image but the model can tell you through that lock probability
that's actually not very certain
in this case like all these are very low like -13 is very low
so the model knows it's not getting it right but reported to say something so
it well
but the course is not getting these things bro it's not getting these things
right necessarily so looking at the Rock probability at
gives you some kind about notion of how sure the Majlis
you can look at many different examples of these predictions
so 1000 Coco images predictions are at at least last
neural talk tomorrow second to scroll through these not all of them are pretty
I warn you and I guess that actually brings me to %uh something I want to
address
arm when we released a lot of this stuff
I think somehow this was miscommunicated and I think people
it's very easy to nooo rise these algorithms and use your confirmation
bias to think that this really works very well
and in it's gonna be injuries because
it doesn't really work very well and I for example I received an email from
at doctor somewhere and was like this is amazing we can describe
images were sentences I want to but this is exactly what we've been looking for
for a very long time I have like all these a edges of cancer and would like
to describe them to
like doctors and I'm just like no no no you cannot
will not work and so there's on
and then you end up with you know people think that a lot of the stuff works just
by looking at some other results and
are and then we get all the stuff about Super intelligence and AI in Terminator
and this is how this happens
seem to be very careful let's think very clearly about that
but this doesn't do okay it seems like it works and we can act applies but
personal there's absolutely no reasoning you and I we look at the image
with a cart around from excessive but we understand the three-dimensional
structure of the
on the environment we understand will be in tractions we understand everything in
the scene and why it's there and how its contract with all the other parts at the
scene
we do complex reasoning task to figure out what's going on and these models you
saw what it does is just a whole bunch of matrix multiplies down the calm that
and into the art and which right away starts turning words there's no time for
any reason whatsoever
this is just the testicle pattern matching based on pictures seen in the
training data
its mimicking the training data based on pictures and
quickens distributions that season the train and it seems like it's working but
that's all is doing I am sickness and second of all we're only looking at the
global
image so this model is not very good with what's very small in the image it
won't find it
we are currently not dealing with any object detection or anything like that
so are now that is happening and second of all we can't
we can just describe any arbitrary image there's a particular training set
this is Coco images and we can we have some competence with things that appear
in the training set
but you can just take this on any arbitrary images it's not going to work
especially they're completely up to the main for example Coco images have lotsa
drops in them
so this model it sometimes my doesn't know what to say is just
such that loosely drops enjoyed like a sure like a forest scene and it's just
like oh yeah
drop is next to adultery and that's because your abs entries are very
correlated in the training data
there's all these kinda salty student how these models don't really work
and we can get we can work on this right but it's just it's just not even close
in terms of what a human can do and no human can describe visual since
aka so all this code having said all that if you still want to play with all
this
all the code doesn't get how on I have both and a conditional
arlen elementalist a.m. and that could be his country in Python
and Mumbai and it works a CPU I have to write the Jeep YouVersion
whenever I get time because right now this takes months to train from scratch
I also have models who will have a lot of pre train models see you can
try to play with those if you want to run this a separate images
right now that's not trivial you actually need MATLAB and kept a comment
premark
because you to extract those images and MATLAB in to extract the
features the senior pictures so you can put them into this code base on
I'd also like to mention common question I get is why didn't you use
are piano or something like that to implement this because this is just not
by
and I have a good answer and a passer the bed answer is that I don't like
anything touching my gradients
maybe old school but I don't like the idea of something computing my gradients
are and out because I want to for control over everything that's going on
in the pipeline I like to understand everything
I'll and so that's bad will see the reason a good reason
is when you're implementing these models are running out the back propagation
equations
explicitly I think actually really helps you work with these models
because you get intuition about how the gradient flow through the network
and it hopes you'd be bugs that works because you can't think or dynamically
about the system and how it behaves the dynamics of the system
and I find that to be much more effective approach in debugging these
models and thinking about loss functions
in like million commercial spaces I'm much more like to think about the
gradients one back
and thanks pushing and pulling on each other in order to accomplish some task
at the end
and are yeah I also like to mention that
um I don't only have a lot of work on
going from I'm just a language also have a lot of work on going backwards
so here for example I was about to shoot slightly separate model but also related
ranking model in a sense that you give it 8 query sentence
as such as dog plays catch with white ball your wooden fence
and the model will say okay this is the most compatible image what this
sentence query not only that but it will actually ground every single individual
word
in that image so for example I think this is the dog this is the white ball
and as a defense so that gives you some Chris some companies that this is
actually doing something reasonable
unlike the model I presented before this this actually does look at the objects
in the image
arm and then playing accordion mall
public area so it finds the man added grounds the accordion
I think this is what public area looks like and so on are so this ranking model
it slightly different but also involves for by Drexel recurrent neural networks
and you can look at some of these results and online as well
i have a web interface recanted go through all the rankings you can click
them
and it shows you exactly was grounded to what so this is about
establishing correspondences between between natural language and visual data
and us that's a separate model I want to briefly mention now
in summary I try to get you
the fact that I to motivations for this stock right one was in terms of modeling
I was in terms that task
purse I tried to convince you that building neural networks top
or your I'll for your data problems as
a good idea you try to think about your
in the coming year a domain establish a function make that function be
octial some primers a chain that I and you can stack and compose these like
Legos we've taken a collisional at work and Riker network
and with what the men and that is a very natural thing to do when you're building
I don't know that work stuff it easily supports transfer learning
and it works well in practice and turn to the motivation on task level
actually two models one going from an image to a sentence
and another from going purse from sentenced to retrieving images
as I hope that eventually this will help us establish a national language
interaction between us
and images and I hope they will let us try some light on the dark matter
up the Internet okay in