okay I think we all get started welcome
to my charge my great pleasure to
introduce Lawrence Sandra Martin is that
close surrogates yeah Lawrence comes
highly recommended by our good friend
Jeff Hinton
so that's sufficient introduction keep
the stop here but he has PhD in Tilburg
in the Netherlands it's wrong Peter
vision is a coming back to you CSV that
post-op care with Lawrence all he left
there I think to become a professor at
Delft and has been recruiting away to
Facebook AI research group in New York
City and reason I really wanted to here
I really like some of the work that he
actually started with Jeff and you know
as people were we're fundamentally
perceptual creatures and a lot of
representation and things is to make
abstract invisible things perceptual II
normal and does the work that or is it
going to talk about it really focuses on
taking higher dimensional spaces and
embedding them in the kind of spaces
that we can make sense out of 2d and 3d
kinds of spaces I think that's what's
really special and important and it has
just a really great finding spoodle so
Lawrence thanks very much
it's great to be back I did a postdoc
here in 2010 with Lawrence ol right my
office was right over there and it's you
know really nice to enjoy the San Diego
weather again it's a lot better than New
York I should move back here at some
point so I'm working now at Facebook AI
research which is a research lab that
sort of the final aim of which is to
build artificial intelligence but since
we don't really know how to do that yet
in the meantime we're trying to solve
problems in computer vision in natural
language understanding in speech
recognition all those all those fields
and
one of the sort of ingredients that we
think is important in the solution to
those problems are embeddings of symbols
of concepts of graphs etc and so that's
where sort of the relation between my
current work and the work that I'll
presenting certif is what I'll be
talking about in this talk is as Jim
already said is about using embed the
idea of embeddings of building low
dimensional representations for complex
high dimensional data for visualization
so I saw in the in the seminar series
there are a lot of great facilitation
speakers and you know if you're very
humble to be sort of one of them in
particular because I am not really a
visualization person I look at these
these kinds of plots and I think well
that is beautiful if only I knew how to
you know how to make this kind of cool
stuff but at the same time as a machine
learner I also look at this and I see
sort of one main shortcoming here and
that is you can you know you can use all
these kinds of plots to visualize your
data but you're always limited to
visualizing a few variables it's a few
parameters at a time right maybe you
know the best you can do is maybe like a
parallel coordinates plot where you can
you know maybe do eight to ten variables
and look at it at the same time and
still make some sense out of it but in
machine learning we routinely work with
you know hundreds or thousands or ten
thousands of variables and there's just
you know no good way to use these
techniques to do the visualization of
that sort of data
and so the question I'm basically
answering in this or asking in this work
is well how can we visualize high
dimensional data and the sort idea that
I've been working on and also a lot of
other people Loren soul being being one
of them one of the pioneers in this in
this area actually is to learn embedding
so what do I mean by that so let's say
we are given a set of n high dimensional
objects so let's call them X 1 2 X n so
you're right it's actually not working
let's golem let's call them x1 to xn now
how can we get a feel for how these
objects are arranged in this sort of
large high dimensional space and the
idea is to basically try and build a map
a two dimension or three dimensional map
of the data in which the distances
between the points in the map reflect
the similarities in the high dimensional
data right so the idea would be to build
a two dimensional map where you
basically have a point for each original
observation and what you want is you
want the the pairwise distances between
these points to somehow represents the
underlying similarities in the original
data so this would probably be a very
bad map right like the the right map
that you want in this case would be this
of course now in practice this will
never be really possible right because
we we cannot possibly unless our data
lies on like a two dimensional plane in
high dimensional data we cannot possibly
preserve all the pairwise distances and
so we have to make some compromises in
which distances we are going to preserve
and which we're going to sort of ignore
and all the techniques that that are
basically exist in this field are about
making making basically that choice
right but about like which pairwise
distances in the original data do I care
about and which ones are not so
important in my visualization so what do
we need in order to construct a map like
this well we need some sort of objective
function that basically measures the
discrepancy between on the one hand the
similarities in the original data and on
the other hand the distances or the
similarities in the map right and so
what we're going to try and do is we're
going to try and develop objective
functions that that measure this
discrepancy
now one technique I am I'm sure most of
you are familiar with that does this is
principal component analysis right for
just to sort of refresh
in case you forgot somewhat principal
component analysis does is it takes your
high dimensional data and it searches
for a two-dimensional or like a low
dimensional subspace in such a way that
if you project your data onto the
subspace the variance of the data is
maximized and I'm showing you an example
of a visualization that you could make
with principal component analysis here
so the underlying data here are these
images of handwritten digits and so in
the map here each of these points
corresponds to one digits what I did
here was I represented the digit just as
a vector of pixels so I think each digit
here has 784 pixels right so each of
these digits is a point in some 784
dimensional space and I've projected it
down to 2d and to make it a bit clearer
I colored the points right which
corresponds to the digits according to
which digit they represent right so
these red guys over here those are all
the zeros in this datasets and the
orange guys over here those are all the
ones and so this is the main the first
principal component and what you can see
is that the main source of variation is
actually the difference between zeros
and ones which makes sense if you think
about digits in terms of pixel values
right like a 0 and a 1 they have very
few overlapping pixels the second
principal component here it basically
goes from four sevens and 9s here to
three fives and eight here so both those
groups are sort of similar in terms of
the number of overlapping pixels so this
is a visualization you can you can make
and you know you see some structure but
really the reason you see the structure
is because I gave you the colors right
if you didn't know the labels of these
digits you would have seen something
like this and it would be very hard to
tell what is going on right like maybe
maybe you could tell well there's seems
to be slightly higher density here maybe
that means something but that's about
all right and so well why doesn't this
work so well right like why doesn't this
map sort of correspond with our own
intuition of digits which is that
they're sort of 10
different ones and you know that there's
some sort of that maybe for sevens and
9s are a bit more similar but but that's
about it so why is that not here well
there are basically two two reasons one
because we do a linear projection so
it's very limited in terms of what we
can actually learn and the other one is
that PCA makes makes a choice in terms
of which distance is to preserve which
is it says I want to make sure that
things that are dissimilar in the
original data are dissimilar in my
two-dimensional data right things that
are far apart need to remain far apart
that's sort of you know this this
maximizing variance idea and now the
question you can ask is well if my data
is somehow a bit more complex are large
distances are those actually are those
actually informative right and for a
while these kinds of datasets were very
popular in a machine learning because
they sort of convey the notion that well
maybe distances between those two points
are not actually that informative right
maybe you can only look at data sort of
locally on a manifold and those are the
only distances that you that you should
be trusting and in terms of physical
ization it's sort of the same right like
do you really care how different zeros
are from once or do you just want to see
that they're different right probably
you're you're happy as long as you can
see that zeroes are not the same as once
but you don't have you know very strong
intuition for exactly how different they
should be so in contrast to PCA what
what we're going to try and do is not
focus on trying to preserve distances
like this but we're only going to try
and preserve sort of small distances
right so basically we're only going to
try and make sure that if I put a point
in a map that the nearest neighbors of
that point in the indoor regional data
are also near neighbors in the two
dimensional map and what we will see is
that it leads to very different kinds of
visualizations if you have any questions
feel free to
interrupt me by the way okay so how do
we formalize that idea so here's the
here's the basic idea so this is a
technique called stochastic neighbor
embedding so what the idea is we are in
our high dimensional space and we have a
bunch of objects and what we're going to
do is we're going to take our our high
dimensional object which I call here X I
and we're going to Center a little
Gaussian on it right we're going to put
a little Gaussian bail on it and we're
going to measure the density of all the
other points X sub J under that Gaussian
right and so what we'll get is for
points that are similar to X i we will
get some you know quantity between 0 and
1 and 4 points that are somewhere over
here will basically get an infinite
decimal value right so we get a
characterization of the local structure
of of the data then the next thing that
we do is so we compute all these
densities and then we basically we
divide by the sum of all these densities
and this gives us a kind of probability
it gives us a distribution over pairs of
points I J where the probability of
picking a pair of points is somehow
proportional to how similar these points
are right so there's a good chance that
if I draw from this distribution there's
a good chance that I will pick two
similar points but the chance of picking
two dissimilar points is virtually zero
right so now we have these things and
now we're going to go to our low
dimensional space to our two dimensional
map the map that we want to create so
here we're going to make basically
represent each point in high dimensional
data by a point in this two dimensional
map which I will call Y i and y J so we
have this point why I it's it's
somewhere in our two dimensional map and
we're basically going to do the same
thing again we're going to center a
Gaussian over it and we're going to
measure the density of all the other
points y sub I Y sub J under that
Gaussian right and we're going to do the
same the same normalization here so
we're going to divide by the sum over
all
and this gives us another distribution I
will call this distribution Q and this
is the this q IJ here basically
represents the probability of me picking
picking a pair of points right but now
this probability is proportional to how
similar these two points are how close
together these points are in the map all
right so now what do I want well ideally
I want to lay out the points in this two
dimensional map in such a way that these
probabilities q IJ are as similar as
possible to the probabilities P IJ that
I computed in the original space right
so what we're going to be measuring is
some sort of distance between this
distribution Q here and the distribution
P that I computed on the original data
and the way so the way we're going to
measure that that discrepancy between
these two distributions is by means of
the kuba globular divergence which is
sort of the natural measure for
measuring measuring how different
distributions are and it takes this form
it's sum over all pairs of points of P
IJ log P IJ / q IJ right and so this
thing has a couple of properties I'll go
into a bit more detail later but one
property is the Kubik Leibel divergence
between you know two distributions are
the same is zero and it's only zero if
the two distributions are the same right
so now what we're going to do is we're
going to move the points around here so
we're basically we're going to play with
the Y's and why chase in such a way is
to try and minimize this function here
this go back like that for divergence
the way we implement that is with
basically a gradient descent over right
so we're going to do gradient descent in
this function with respect to the
locations of all the points right and
sort of intuitively you can think of you
can think about that as sort of moving
the points around until they reach a
configuration where the q IJ
probabilities are pretty much the same
as the as the pIJ probabilities that are
computed in the input space
does this make some sense all right I
see some not so that's good so what what
what is this you know why do we use this
cool backlight for they're virgins right
like you know it's there's some nice
mathematical reasons maybe wood but what
does it really do well you can you can
think of it as follows so let's say I
have two similar objects right so two
similar objects in the original data
they had a large value pIJ right so if I
have two similar objects I get a large
pIJ here I get a large pIJ here well
then I better make sure that this guy is
not zero right because otherwise I get a
large value divided by zero
I multiply by another large value so
that's going to be large right so I'm
going to pay a very large penalty for
having two things that are similar in
the original data far apart in the map
it doesn't work the other way around
right like if I have two points that are
very dissimilar I don't really pay a
penalty for putting them closer together
because this pIJ outside here is is
basically infinitely small right so if I
have a large you know if I have similar
points models by distant points in a map
I'll pay a large price and so really
what this technique is doing is it's
mainly trying to preserve sort of the
local similarity of the data right it's
only trying to make sure that you know
things that were close to you or still
close to you in the map now if you run
this algorithm and you run it on the
same data you get a very different
visualization you get something that
looks like this and you know it's it
still maybe not perfect but at least it
got some of the group's rights right it
figured out like that zeroes are sort of
a cluster of things and that they're
basically no other digits that are you
know it's very similar to two zeros and
so you see you get some of these groups
correctly you should keep in mind it
doesn't know about the groups right it
only sees the digit images but it
doesn't know that these are digits so it
basically doesn't have the colors so you
know it gets something right but at the
same time you see that some of the class
here they're sort of overlapping right
it seems like you had the right map but
there was sort of a guy who sort of you
know pushed everything a little bit
together and that's why things got got
sort of mixed up here in the middle this
is something that we called the crowding
problem and I'll try to explain what
happens here as follows so suppose I
have data that is intrinsically high
dimensional and suppose I also so what I
mean by that is basically it doesn't lie
on a two-dimensional plane right it's
really sort of filling up the high
dimensional space somehow and suppose
also that we try to model the local
structure of the data in this
two-dimensional map right which as I
just explained to you is what what
sarcastic neighbor embedding is doing
then the result of that is that
dissimilar points have to be modeled to
far apart in the map and this is not so
easy to see so I'll demonstrate it by a
very simple example let's say we have
our high dimensional space is now 2d
because that's you know the biggest I
can visualize and let's say you know we
call these two things we call those
small distances and we call this
distance a large distance right and what
we're going to do is we're going to try
and embed this data into one dimension
because you know well we cannot go lower
than one and we're going to try and do
it in such a way that you know the small
distances here are preserved right so we
want to keep the raw it's the same size
well we can do that perfectly right this
creates great embedding but look what
happens there's a large distance here
that actually became larger right it
just went from square root 2 to 2 right
and so this is basically also happening
when you make these kinds of when you
make these kinds of embeddings right you
have all these pairs of points that are
dissimilar and you know if you think
about all pairs of points and actually
most of them will be dissimilar right
and so they all they all want to be a
bit closer together right so you're all
sitting in that map there and they all
want to be a bit closer together and
you're sort of crushing a map right and
that's what is causing these crowding in
the
does this make sense all right so yeah
that one be embedding where the long
length went from root to two so far that
doesn't seem to be a problem so far no
but if you if you think about the the
cost function that we're trying to
minimize we so that the stochastic
neighbor embedding will not learn this
embedding because when it has this
embedding basically these two guys they
want to be a bit closer together right
and so the embedding that will actually
learn will be basically move this point
here and move this point here right so
you pay a little bit of a price in terms
of preserving these these small
distances but like this pair of points
actually got a bit happier right and now
if you have here we have only three
points so everything is fine right but
if you have a couple of thousand points
then there are lots of these points that
try and be a bit closer together and so
they will sort of start crushing a map
yeah so here's the solution that that
that we came up with to try and sort of
address this problem so this is a
technique called T distributed
stochastic neighbor meting or Disney and
the idea here is to in the map use a
different distribution to measure
similarities so what we're going to do
is we're going to redefine these q IJ
probabilities to not be measured by a
Gaussian distribution but in this case
by a t-distribution with one degree of
freedom and as you may remember a
t-distribution is a heavy-tailed
distribution right so what that means is
that if I have two points that are you
know dissimilar that are far apart in
the original data so let's say you know
they had a distance of 20 right and that
gave me under the Gaussian a density of
0.01 then to get the same density of
0.01 under this student-t distribution
in the two dimensional map I probably
need to put them 40 or 50 or parts right
so by redefining these similarities
using a heavy tail distribution
effectively what I'm saying is it's okay
for for dissimilar points to be modeled
too far apart in the map right and what
we will see is that this sort of
eliminates this crowding problem so if
we do this if we do this change in the
algorithm and we run the same gradient
descent procedure then we get something
like this so here you see the actual
gradient descent working it starts from
a random map and so this is the same
data right and keep in mind it doesn't
know about the actual you know the
actual labels and what you can see is
that it actually there's a pretty good
job of figuring out that's you know that
zeros are really very different things
from all the other digits right there
are some clear separation between zeros
and and all the other things at the same
time you know you can in this map
already very nicely see that sort of
large stations are sort of meaningless
right it puts zeros and ones you know
sort of next to each other but basically
like this distance is about as
meaningful as as this distance in a map
like this right
but you see sort of the the effect here
like if here I would have removed the
colors you could actually have made some
sense out of this data right you could
have you would have been able to
identify some of the clusters here and
now if we go and we plug back the
original digit images in the locations
that they that they got in a map we also
see that a lot of the local structure in
this data is preserved so wrens if we
zoom in on this cluster of zeros we see
that zeros are things that are you know
rounds or sort of more elongated right
and in here we see that they can have
slightly different orientations right
and so this is kind of the local
structure and that's the thing that is
really trying to preserve the same thing
if you zoom in here right so you see
that for sevens and 9s are still sort of
put together and the reason is that
they're sort of connected by digits like
this one is that a four or nine I'm not
sure and you know as a result it sort of
ended up in between right here also you
see these sort of orientations and you
see here like a group of sevens as
people
in Europe would write them with a little
horizontal crossbar and those are sort
of separated out so to give you an idea
of sort of how I use this in the
beginning was basically as a way to
assess the computer vision systems I was
I was building so one of the problems I
was interested in was in building
representations for texture images that
were invariant under basically different
viewpoints and before always what I
would do is I would you know design
those features I would throw them into
machine learning system and I would
measure some sort of classification
error and that gave me some idea of how
good my features were but now what I
could do is I could actually throw those
features into Tizen II and visualize
them and I would see clusters like this
right which really suggested which
really sort of told me that the
representations I had built the the
features I had built for texture images
we're actually invariant under under
view points right which is the thing I
really wanted to see in terms of
visualizations you can you can use this
for lots of different things right so
one example is trying to visualize
movies based on user ratings so if
you're a company like Netflix you have
you basically have for a lot of movies
and a lot of users you have ratings of
users that user gave to those movies and
a couple of years ago Netflix actually
had a large competition where they sort
of challenged people to try and predict
what ratings users would give to a movie
and if you were the best at that you
could win a million dollars so you know
that's how I got interested as well I
didn't win the million dollars but it
was still fun so the the basic way in
which techniques were for doing that
sort of prediction work is they make a
big matrix where you have in the columns
all the movies and in the rows all the
users and you put in the ratings when
you have them right so this is a sparse
matrix and now what you're going to
learn is you're going to do a matrix
factorization of this big matrix so
you're going to represent this matrix as
a product of basically two skinny matrix
matrices one where the columns
correspond to the movies and one where
the rows correspond to the users and
you're going to basically learn these
two matrices in such a way that if you
compute these products that the sort of
observed values are similar as possible
to the values you've observed here and
now for the non sparse parts of these
products that basically gives you your
prediction right like the you can use
this product also to predict the ratings
that you have not observed you can also
do other stuff right you can also
basically go and say well hey here I
have a column per movie and those
columns apparently somehow describe
these movies in a sense that if I have
two similar movies we're similar now
means basically the same users would
give them the same ratings then they
must have similar columns in this movie
matrix right so what I can do is I can
take these movie columns as sort of my
high dimensional features for movies and
I can throw them into t-sne and and try
and do a visualization if you do that
you get something like this so here what
we did is we did a three dimensional
embedding and we represent the first two
dimensions of this of this embedding by
you know the X and y axis and the third
one by the color of the of the plot of
the of the name right so what this
basically means is in this map two
movies are similar when they're close
together and they have the same color
right now if you zoom in on this map you
will see things like this right so for
instance there's a cluster on top which
is you know the Simpsons family Guy and
South Park kind of makes sense right if
you go down here you see all the Bond
movies and Mission Impossible apparently
you know those attract the same kind of
kinds of viewers other things that
immediately are visible are like these
colored clusters right here right well
if you look at that that orange cluster
there it's Indiana Jones Final Fantasy
Raiders of the Lost Ark in Star Wars
right so this is like you know nerd
territory basically the the rats
the red ones over here are friends
season 1 to 20 I didn't know how many
seasons there were but you know all of
them were there
and the orange guys are all the Star
Trek ones which are you know not
entirely with the Star Wars movies which
also makes sense so these are the kinds
of visualizations you can try and do
yeah local clusters make sense but what
about the distance between Star Trek
kids are wars so you definitely see so
you definitely see if you print out the
map really large you definitely see sort
of genres of movies right so you'd see
sort of a large group of old horror
movies or all romantic comedies and the
reason is that you know that they're
always sort of you know movies that are
connecting these things right so you
definitely do see do see that structure
the only the only time you don't like
the structure becomes meaningless when
the two when you basically have two
clusters that are really just disjoint
right like zeros and ones right like
those are you know completely disjoint
from everything else and then the
location of that cluster is meaningless
but in in this kinds of in these kinds
of sort of more real-world data
typically you don't have clusters that
are that clear
if you have classes that are they're
clear you should probably be worried
like you probably have a weird by using
your data or you know your your
measurement machine didn't work well on
a particular day or things like that
yeah good question so so in terms of
more serious research one of the things
we did together with people at the Allen
Brain Institute is use this for
visualizing genomics data so the data
here are basically you take a mouse
brain and you cut it up into very thin
slices and now a bunch of locations in
these slices you measure the full gene
expression profile right so at every
voxel in the in the mouse brain you
basically you get a 20
thousand dimensional gene expression
factor yeah so then you get exactly what
you don't want in the sense that then
you get an objective that sort of
behaves like PCA and there are people
that are who argue that this is the
right way of doing it because this is
sort of what you would measure when you
would think of it as a latent variable
model right but you know it doesn't give
nice maps yeah
good question so what we see when we
sort of visualize these gene expression
data is that you know gene expressions
in this case they really sort of cluster
by anatomic regions in the in the mouse
brain now I'm no expert on the electron
the anatomy of a mouse brain but these
are apparently sort of the main regions
and you can fairly clearly see them see
them show up right which is you know
interesting for for brain researchers to
see and here what we did was also a
different type of visualization where
what we do is we take the gene
expression we use T's need to create a
3d map and then what we do is we use the
3d locations to basically create a color
right so the first locate the first
dimension in the 3d map is going to be
like the amount of rats the second one
is going to be the amount of green and
the third one is going to be the amount
of blue right and so then we can make
maps like this where basically what
these what this mean are well if two
voxels
have a similar color that means that
they had a similar gene expression
profile right and so that that generates
pictures like this which are nice to
look at but they can also be they can
also show interesting things like one of
the first things we saw was actually in
one of the first runs that we did we
actually saw something like this and so
what is this well these are the slices
that people that people make right so it
turned out that all the variants in this
data which I
actually you know due to differences
between slices which you know are sort
of artifacts of the data right and so
like the first thing we realized
essentially was that the data was not
properly normalized and which is a bit
worried because they were already quite
a few publications that use this data in
you know all sorts of statistical
analysis but never actually you know
looked at these things and and and sort
of got their normalization right right
so the first thing we did was actually
you know sort of use these use these
visualizations to guide us towards
normalization of the data that would
actually sort of you know get rid of
these these artifacts now one thing I
didn't talk about yet is well how do you
make this scale right like I'm a
computer scientist so I immediately
start asking questions about like
complexity of algorithms right and the
way I explained the algorithm right now
I basically I have an algorithm that's
order N squared right if I have n points
I need to look at all N squared pairwise
similarities in order to produce a map
and so I can do that as long as I have
five to five thousand digits but you
know pretty quickly by informatics
people started showing up at my door and
they had like you know 1 million
measurements of whatever and and they
wanted me to do a plots and I couldn't
write it just wouldn't skill so what
it's interesting if you if you want to
try and solve that problem is to look at
the have a look at the gradient that of
the t's new cost function so the
gradients actually takes this form right
so what this is saying is well how
should I move point why I in order to
make this cool back light plot
divergence a little bit smaller and if
you look at it it basically it consists
of two parts so there's a spring between
point y i and YJ and then there is some
sort of you know exertion or compression
of that spring in particular you see
this term here p IJ minus q IJ right so
if i model the data perfectly in the map
then this term is 0 right and
spring will be completely happy right it
will not exert any force on on the
points but if they're dissimilar then
you know it will either attract the
points or repel them right and then
there's this sum which is the sum over
all pairs of points right so essentially
what I have is I have a bunch of stars
here and the you know I'm moving around
like let's say I want to move this star
a little bit all the other stars exert
some sort of you know gravitational
force onto this star and I need to
compute the resultant force right so now
what you can do is you can you know go
to the Journal of astronomy in the 70s
and look at you know how they how they
solve these things and if you're aware
if you're familiar with like force
directed graph layouts there you can use
similar tricks to basically get those so
scale well so the approximation we use
there is called a Barnes Hutt's
approximation and the idea is basically
that if you have three points over here
that are relatively close together and
they're relatively far away from the
points that they're exerting a force on
then maybe I can get away with saying
well I just compute one force right I
just compute the center of mass of these
three points here I compute the force
that the center of mass would exert on
this point I and I multiply by three
right this is the basic idea and what it
leads to is to an n log n algorithm
that's a lot more efficient and that you
can actually use the scale up to like
millions of examples so these are some
details on how you implement that I'll
skip skip over those for now ask me if
you're if you're interested offline so
now we can scale to much bigger data set
so these are variances like the full
digit data set this was for a long time
like the main benchmark in machine
learning it's finally abandoned luckily
and here you can you know you can see
similar things right like orientations
and all that stuff
here we also use it for basically trying
to assess how our computer vision
systems are doing right so this is a
visualization of features that are
learned by a convolutional net work on
on a bunch of on a bunch of images then
you see that you know they actually
group by you know cars or planes or
things like that so we really use it
sort of as an inspection tool to inspect
order algorithms or or systems or just a
fun way in this case to to visualize
Instagram photos so this is sort of the
Instagram universe and you can sort of
start going over it to see what people
actually post an Instagram while they
post a lot of pictures of cats it's it's
unsurprising they post pictures of dogs
they post pictures of in this case a lot
of foods not not really surprising
flowers so basically everything is sort
of in there right so this is you know
sort of you know all your all your
Instagram photos in a single in a single
plots one one other interesting
visualization I want to show is this one
so this is a visualization of words and
the words are visualized using based on
something called word effect embeddings
so the idea is that you
so word effect model basically it
represents each words by some factor in
let's say 300 dimensions and what it
wants to do is basically wants to make
sure that words that factors or words
that are co-occurring in a sentence that
those factors are close together and it
you know you basically you download the
internet you train an algorithm to learn
the to learn those factors and now you
visualize the word factors and so what
you didn't see is you can see clusters
like this right so you can you can see
that by just sort of you know reading
all of Wikipedia or reading all of the
Internet's you can actually figure out
that these words are all semantically
related right here's all your all your
favorite TV networks
here words related to you know holidays
or vacation here are words that are
related to you know folding the only
thing hasn't figured out is you know the
difference between Democrats and
Republicans it's still you know put some
in one big pot here are like all sorts
of words that have to do with you know
transportation things like that and you
know basically wherever you look in this
map you see this kind of you see this
kind of structure the reason I'm showing
this map is because it also highlights
one of the sort of limitations of the
technique I discussed so far and in
particular what I mean by that is the
limitation of using a single map so
let's say I really want to visualize
words like this in a map right or maybe
it's not words but maybe it's authors of
papers based on their co-authorship or
for a while was very popular to to
analyze the Enron email corpus to see
what you know what was going on there
who is emailing who and so on or more
general scale-free networks etc then you
run into the following problem how can
we model the words riverbank and
bailouts appropriately in one map
there's basically no way of properly
doing it right if I put River close to
bank which makes sense and I put bank
close the bailouts which makes sense
then River and Bill outs have to be
close together right like we're
basically you know we're limited by
destroying inequality and that doesn't
make any sense right River and bailout
have nothing in common other than that
they both have something in common with
the word bank so how can we get around
that so one solution that we worked on
is something that we call multiple maps
Disney and the idea is as follows
instead of learning one map we're going
to learn two maps we're going to learn
them simultaneously and what we're going
to do is we're going to give each words
a points in each of the maps but we're
also going to assign a weight to each
word in each of the maps and now we're
going to optimize both the locations of
the words and these weights and now all
of a sudden I can represent a
is dissimilarities appropriately because
what I can do is I can put River and
Bank close together in mAb one and give
them both high weights the words bailout
is also in this map but it has a zero
weight so I'm not showing it right and
in the second lap I'm going to do the
other thing I'm going to put bailout and
bank close together with high weights
the word River is also in here but it
has weight 0 I've learned that so I'm
not showing that word River and now all
of a sudden I have you know a much
better representation of the underlying
similarities right so in this way I can
actually visualize non-metric
similarities essentially that means so
in this case I mean you could also try
and say well we try and explicitly split
out the census I'll show you in you know
two slides why you know that will not
take you not take you all the way like
you will basically always fall short but
they're also you know they're also
datasets like if you think about it
scale free networks or anything like
that or like co-authors and papers like
they're actually the same person but
they you know they publish with people
at UCSD and they publish with peopled
Harvard's so they sort of want to be in
the middle between both and then you get
UCSD people next to Harvard people which
you know may or may not make sense right
so see it as an illustration I guess so
how do we formalize this well we
basically set it up like this again
we're going to we're going to redefine
these q IJ probabilities these
similarities under under a model and
we're going to make them a bit more
complicated so this why I super M is
basically the location of word I in map
M right and this P I super M here is the
weight of word I map M so what you have
here is the similarity of words I and J
in map M and this is the same thing as I
had before in the Disney objective right
it's just this student T kernel that I
puts over this this this point Y I and
I'm
measure you know the density of all the
points YJ under that's under that
student eternal but now I'm going to
multiply it by the weights of those
words in that map right so if one of the
two words has zero weights in this map M
then I'm basically going to set this
whole guy to zero right I'm going to say
well in this map at least these two
words are not similar and now I'm going
to sum that over all Maps right so my
final measure of whether two words are
similar is basically well you know
summed over all the maps how similar are
these words in each of the maps right
and now what I'm going to do is I'm
basically going to do the same thing I'm
going to again minimize the school bike
library diversions using gradient
descent but it's a little bit different
because now I'm moving all the points in
all of the maps around simultaneously
and I'm also changing these weights
right I'm all learning all those things
jointly if you do that so here's an
example where we did this on a word
association data set so this is a data
set I think from the University of
Southern Florida which is used a lot I
think in in linguistics research and so
on where basically what they did is I
can play with you Jim I will say a words
and you have to say the first thing that
comes to mind
I say coffee tea and so you do this
experiments a ton of times and now what
you get at the end is basically a
probability what is the probability that
given that I give as inputs as stimulus
coffee what is the probability that the
response is tea right and this so this
probability is a very natural input into
the seasoning algorithms right that is
basically going to be your P IJ now
right it's the pIJ is a probability of
coffee given tea right and so they get a
ton of this association data for like
five to five thousand different words
and we did visualizations of that data
so here I'm showing you two of the maps
that we learned there were in total like
20 maps but I'm showing you two here and
so I'm only showing the words that have
high weights right like words with zero
weight or not are not shown here
it's but keep in mind is all the words
are actually in all the maps now you can
see some interesting stuff right like
here's for instance a cluster with all
sorts of words related to sports right
like Irina and stevia stadium and and
soccer and coach and there's also the
word cheerleader well that makes sense
right to cheerleader to be close to
sports then if we look at this other map
here where cheerleader is also present
but here it's next to gorgeous and sexy
and beauty which is a completely
different type of Association that
people have with the word cheerleader
and by setting up the learning algorithm
in this way it was actually able to sort
of unravel these two types of
associations and I guess that that
answers your question you know it's
really the same word but it's a
different type of Association that
people have as a second example so for
instance here you see a bunch of words
that are related to you know sort of to
two states or like how states are
governed dry like monarchy and oligarchy
and anarchy
okay maybe not governed democracy
Republic and so on right if you look at
this word monarchy it also shows up here
but now shows up with all sorts of words
that are sort of related to the page
entry surrounding monarchy right so
England and royal and monarch and Queen
and King right which is a different type
of Association that people have with the
words monarchy yes
it's you know it's it's it's kind of
liking clustering algorithms it's a
little bit it's a little bit like this
there you you can do I mean one of the
things that you can do is basically you
can run you can learn a model with
different numbers of maps and you can
basically plot the objective function
value as a function of the number of
maps right and what you'll see is that
in the beginning it goes down a lot
right because there's you know a lot of
structure that was wrong in like one map
that the second map was able to fix
right and maybe the third maples still
out but at some point you know it starts
asymptoting out right it starts decaying
and typically that's you know that's
where you want to that that I mean
that's one way in which you can sort of
pick the number of maps that you want to
use any other questions okay um so one
last thing I wanted to show maybe
because it's ginger this is actually
stuff that people at UCSD here oh I've
also worked on in particular girdling
creats and and search Balaji
which is basically the problem of well
if you wanna if you want to visualize
things like human similarity ratings
then you always have this problem that
human similarity ratings are very very
unreliable right if I ask you on a
Likert scale how similar is A to B let's
say how similar is a is a Fiat to a
Ferrari then some of you will say five
because you know both are Italian cars
but some of you will say one because you
know Ferrari is awesome in the fields
not so much right so these things are
very unreliable and and also like the
other main reason I guess is that people
have sort of different internal scales
of similarity and so often in cognitive
science people instead as things like
well it's a more similar to be or more
similar to see right
and so what together is they gather a
set of similarity triplets of triplets
ijl which basically means that you know
the unroll that objects eye is more see
to object j-10 to object l so the
question we asked in this work is well
can we somehow use these kinds of
triplets to learn embeddings right like
can we learn a representation for a B
and C that we can show in a map based on
these based on these similarities and
basically the techniques we we came up
there are sort of the ideas similar in
this case what we're going to do is
we're going to define a distribution
over triplets ijl where you know we
measure the similarity between I and J
and we divide that by the sum of the
similarity between I and J and I and L
right and so this thing will basically
this will be large if I and J are very
close together right if this number is
large but I in L are very far apart
right that's basically when this
probably is large right so bij L is
large whenever you know I actually
satisfy a triplets in my embedding so
the way we're going to learn this
embeddings the way in which we're going
to learn these the values of these axes
here is by in this case just maximizing
the sum of these probabilities over our
set of triplets and so this is something
that we called stochastic triplet
embedding we tried here we tried the
same two versions so here we have like
this Gaussian version but what works a
lot better in practice is this sort of
version where you have a student-t
distribution so I'm just going to show
you some examples so this is like the
first experiment we did which is on this
digit data set we're richest from the
digits generated a set of triplets just
to see how different techniques would
behave so these are actually both
techniques that came out of of UCSD that
sort of inspired me to start working on
this problem and these are the kinds of
maps that they learn and then the final
algorithm we came up with cut to this
map which is you know it's not nearly as
good of course as the you know as the
map that I showed earlier we had all
pairwise distance between digits but
here you only have
you only had access to sort of very
partial information right you only knew
for a few trip triplets and you know one
of the problems is the number of
triplets is insane right it's n to the
power you know it's n cube and you
didn't even know like the exact
similarities for each n cube but only
well you know these guys are must be
closer together than these guys but it's
still able to figure out some structure
like it's still able to figure out that
that you know ones are really different
so the real experiment we did here was a
visualization of a data set of music
artists where people were actually asked
okay you know is this artists more like
this artist and like this artist and so
what we see here is that in this case
you can actually you know visualize some
of the structure in that map I'm not
really a you know a music expert but I I
presume this classifies as sort of you
know hard rock or something like that
this is sort of more more pop and
they're actually all female singers I
think there's like a hip-hop cluster
over here and things like that so this
is a visualization of these kinds of of
triplets so to conclude so I hope that I
have sort of convinced you that a you
know a good way to visualize complex
high dimensional data so it could be you
know high dimensional observations
that's how I started off but it could
also be non-metric similarities like in
the word associations or it could be
triplets like in this last example that
you you can take this kind of data and
try and visualize it in a map and it may
lead to insight into your data and into
more insight that you you maybe get in
order by other means and so the main
algorithm I showed you to do this is
this Disney algorithm which is today is
quite widely adopted in particular in in
bioinformatics and medical research it's
sort of I think today it's like the
standard way to analyze flow cytometry
data and stuff like that I sometimes
look at those papers I have no idea what
they are about
but you know the people tell me is
useful for it and what I've showed you
is that you can sort of develop
different variants of these kinds of
embedding techniques to suit sort of
your specific data type now to concludes
a lot of this was joint work this work
got started as Jim pointed out in
beginning as joint work with Jeff Hinson
some of the later work was done with
Killian Weinberger in particular to
stuff on on triplet embedding and the
brain analysis work was done with a ton
of people both at Delft and at the Allen
Brain Institute's and these are the
papers that I talked about in this talk
that's it Thanks
talk about triplets what I wanted to ask
is you got to put the result on the air
I missed with the weather this bears the
one you had goodness ly but then you
also mention that you do triplets with a
humoristic that's more a reliable task
right mm-hmm
so it seems there should be a trade-off
right like if the parents are noisy they
might not but but there are as many so I
like people to do just compared to
business but have more bits a signal to
noise ratio right right oh yeah it's
very interesting yeah I mean one one of
the main things one of the main problems
here is that you know there's so many
triplets and if you're going to as
humans you cannot possibly ask all of
them right so you have to be very clever
in how you select which triplets you ask
for right you want to pick triplets that
are somehow informative right so asking
is you know is a one more like a 6 or
like a 7 that's probably not a very it's
probably not going to give you much
information about the underlying
intuition that people have about digits
right and so I think a lot of the work
in this area also a lot of the work that
has been done here at UCSD is really
focused on you know trying to be clever
and and also sort of do this in an
active way right so take a set of
triplets learn and imbedding like this
and then sort of try and figure out what
is still missing
right like what information do we not
have yet and sort of use that as sort of
a guides as sort of a way to pick which
triplets to to ask for next you know now
is it very good point yes
Motors really calling in math and all
the stuff he use like write linear
mapping thinking of PCA or something
like that so is there any work that
looks explores kind of you know
architectures yeah so I've actually done
a study where I take like a deep neural
network and I basically train it to
minimize the teasing objective right
like you know a neural network is a very
flexible thing on on the top you can
define any loss function you want and so
it could also be the Disney objective
function and that actually works very
well yeah yeah so so what I do is I
basically I for I do a forward pass on a
batch of examples and then you know I
have the similarities in the original
data for that batch I get output
representations right on top of the
network now I can compute these
dissimilarities q IJ i can compute you
know the gradients that is induced in
each of these in each of these outputs
and then that that gradient I can back
propagate through the network and it
actually works very well yeah I mean it
needs to be flexible enough right like
it will be a bit worse than sort of the
nonparametric thing that I'm doing here
just because your model is a bit
constrained right like you're a bit more
constrained in terms of the function
that you can learn from the original
data to the two dimensional map but you
know as long as your model has capacity
enough it can work reasonably well at
the same time I know it was never clear
to me what the application of this was
because you know I would never suggest
using it as part of a classification
system because you know in the end
you're you're still reducing data so you
you lose some information that maybe is
important in the prediction that you
want to make so the really the only
value I see four techniques like this is
in visualization but there are people
who disagree with me on this yes
all the possible information
I mean sort of in general you can you
can sort of do learning in a multitask
way right where you have a bunch of
different loss functions that each of
them focus on sort of a different source
of information so one could be triplets
one could be some supervised objective
one could be maybe a TC objective or
whatever and you sort of try to minimize
get your network to minimize a weighted
sum of those things that so there are
some studies at least in sort of the
supervised learning context that suggest
that this can help so there's a there's
work and natural language processing
where people train networks to basically
solved and different natural language
processing tasks at a time at the same
time and there it actually helps but at
the same time like mixing different loss
functions is very difficult because you
basically because these loss functions
can operate on different skills right
and depending on you know like they can
have very different shapes and so
depending on sort of how good your model
is you may have to reweighed them in
different ways and that makes it really
really tricky right so so the only
successful cases I think of multitask
learning have been cases where sort of
all the losses are in a sense the same
in the sense that they're all
classification losses of the same form
or all so that would be my answer but be
happy to prove them wrong yeah
how sensitive it is so we there are a
few tricks that we do there that I
didn't talk about yet so one one of the
main problems is that you are one of the
main things you have to do is you have
to set the bandwidth of your Gaussian
right to measure similarity and what we
do there is we basically say well we
think it's unlikely that the same
bandwidth will work well across the
entire data space right because maybe
you know the clusters are once is
probably a much tighter cluster in data
space and a cluster of zeros right just
because there are some more variance so
what we do there is we're basically
going to say instead of computing these
these joints distributions we're going
to compute conditional distributions so
given point I what is the probability of
point J right so we're just going to
sort of do row normalization of this
Gaussian kernel and now what we have is
we have basically a different Sigma a
parameter for each of the gaussians
we're going to set that Sigma parameter
by saying we want each of these
conditional distributions to have the
same perplexity so they have the same
entropy basically and this perplexity
you can think of intuitively as the
number of nearest neighbors right so if
you want to make a comparison with
making a K nearest neighbor graph the
perplexity is essentially a value of K
so it's the number of points that you
want to sort of keep close in the in the
final embedding and the nice thing is
that you can find the sort of optimal
bandwidth for that particular points
that that you know the bandwidth that
basically achieves that perplexity you
can find it very efficiently with a
binary search so that's actually what we
do in in practice yeah good question
yes yeah
the number of maps and multiple times it
give you the singular demanded map
there be like intimates remember
combination which can
I mean yes and no right like no in the
sense that if you think about this loss
function it's an its non-identifiable
right just like clustering in the sense
that you know if I switch map one and
three I have exactly the same solution
in terms of the loss function right so
they're basically there are many
different solutions that lead to exactly
the same same loss value so in that
sense it will definitely every time
reach a different optimum sort of
qualitatively I think it's fairly
similar each time like each time you'll
run it you'll have the cheerleader
coming with the sports and with the
beauty cluster but that's you know
that's more of a qualitative statement
what you can do I mean what we do in the
papers of course is measure things such
as well if I for each words in the
original data basically look at what are
the five most similar words then what
percentage of those five most similar
words are also the most similar under
our model right so you can measure those
kinds of things to get an intuition for
how much of the information is actually
preserved in your map or in your maps
and I mean that's your very useful to do
but the I mean at the same time the
exact way to sort of measure the quality
of your maps really depends on your
problem
I I have not but you know you're welcome
to try I mean it will probably it will
probably work right like or it makes
sense in the sense that it will or it
has the potential to solve my sort of
Fiat Ferrari example rights like in this
in this stochastic triple and embedding
I didn't you know I mentioned the
problem that it didn't actually solve it
right so that the fact that people have
sort of different axes of say of by
which they measure similarity those axes
are in this visualization that I showed
they're sort of mixed right they're both
in there in some sort of convoluted way
and so you could try and unravel them in
in multiple maps so I mean I think it's
a very sensible thing to try thank you
thanks