so yes I'm gonna talk about understand it sort of led to a bunch of results of
drilling down into neural mechanisms underlying well-defined competitions and
we've been working on that with you know I spent a lot of time talking to your
physiologist especially at Stanford where I am so we have a whole bunch of
collaborations going on now involving understanding neural computation so for
example the retina itself is actually a deep neural circuit already because
there's interview there's an intervening layer of neurons the bipolar cells and
Amber consulted intervened between the photoreceptors in the ganglion cells and
oftentimes what we do is we we shine light on the photoreceptors and we
measure the ganglion cells we have no clue what's going on in the interior so
we've developed computational methods that can successfully competition we
reconstruct what's going on in the interior of this putative deep neural
network even though we don't have access to these things and we can actually
infer the existence of properties of intermediate neurons here and those
properties are similar to what's been previously recorded when people do
directly record from this bipolar cells in the clinton in lab we've we've sort
of been unraveling the motions underlying
competitions that the company's underlying motion visions that you know
when you swat a fly it's really hard cuz they can really quickly detect motion
coming towards it and they fly away so we've been you know there's been lots of
work
what kinds of algorithms might underlie motion estimation for example the right
card correlator the Barlow model and so forth
we've been applying systems identification techniques to bring
calcium imaging data will hold training from the flight visual circuit and you
know we find we just literally identify the competition and we find that it's
sort of none of the above its a mixture of all previous approaches so grid cells
we have some results in grid cells that these are these famous cells that
resulted in a Nobel Prize recently we can actually show that these grid cells
maintain their spatial coherence because the rat and mouse Arkham are always
interacting with the boundaries in the boundaries actually correct the rats
internal estimate of position and were it not for these interactions boundaries
the grid cells would rapidly Chico hair Chico here on the time scale of of
minutes like less than a minute and so it's quite interesting we can show that
whenever the rat and counters they encountered the boundary it corrects its
internal estimate position perpendicular to the boundary but it doesn't parallel
and it doesn't because it can't because when it hit the boundary and receives no
information about where it is perilous the boundary but receives information in
this direction and then with the shadowy lab we've been looking at I think a
really major conceptual puzzling neuroscience is why I can be record four
hundred neurons in the circuit containing millions of neurons and do
dimensionality reduction and infer these dynamical portraits of how you try to
infer the state space dynamics of the circuit and claim that we've achieved
success right we're doing under sound recording a hundred your own side of
millions how would how would the state space dynamics of room for change of
recorded more nuanced and we can show that essentially it will not change
because we've come up with a novel connection between the active year old
measurements in the act of random projections so we can show that the
active recording from a hundred neurons in the brain is like the act of
measuring a hundred random linear combinations that all neurons in the
relevant brain circuit and then we can apply random projection 32 to give us a
predictive theory of experimental design that tells us given the complexity of
the child's how many neurons which you need to record to correctly recover the
states based on
circuit and then in the Raymond lab at Stanford we've been looking at houses
enhancing synaptic plasticity can either enhance or impaired learning depending
on experience of for example we know that we think that synaptic plasticity
underlies the very basis of our ability to learn and remember you might think
that enhancing synaptic plasticity through various pharmacological or
genetic or genetic modifications might enhance our ability to learn learn and
remember that previous results have been mixed when you perturb synaptic
plasticity sometimes you enhance learning sometimes you impaired learning
so I believe in the rain glad that they're the first to show that in the
same subjects enhancing synaptic plasticity for example the cerebellum
can either enhance or impaired learning depending on the history of prior
experience and we can show that in order to explain the behavior learning curves
you need much more complex adaptive dynamics that people naturally assume we
have to really promote our notion of what a synopsis from a single scalar
like aw I J and in your own network to an entire dynamical system and its
alright so you're learning so this is sort of the low level stuff that we've
been doing I know that in this course you guys study higher level stuff so I'm
not going to talk about any of these each of them could be like a one hour
talk but I wanted to discuss some more high level stuff that we've been doing
and the other side the other sort of direction in our lab that we're looking
at is actually the mechanics of high-dimensional data analysis so this
is of course very relevant in the age of the brain initiative and so on where
we're developing very large-scale datasets and we'd like to extract
theories from this so-called big data so the entire edifice of classical
statistics is predicated on the assumption that you have many many data
points and you have a small number of futures right so then it's very easy to
see patterns your data but what's actually happening now a days that we
have a large number of data points and a large number of features so for example
where you can record from a hundred and Iran's using electrophysiology maybe
four hundred trials of any given trial type in a monkey doing some time so the
ratio of the amount of data to the the number of the number of features is
something that the older ones so you know the data threats to like three
points in a three-dimensional space that's the best we can visualize
so the significant challenge to do data analysis in this scenario so it turns
out there's beautiful connections between machine learning and data
analysis in the statistical physics of systems of quenched disorder so what I
mean by that is in data analysis you want to learn statistical parameters by
maximizing the log likelihood of the data given the parameters and
Psychophysics you often want to minimize and you know this can be viewed as
energy minimization and so this beautiful connection between these and
so we work on that so we've applied the compressed sensing we've applied this to
the problem of what's the optimal inference procedure in a written in the
regime in high-dimensional regime we know that maximum likelihood is optimal
in this regime but something else is better in this regime and we found
what's better it turns out to be a smooth maximum likelihood of course have
applied the string theory of neural dimensionality investment so you know
there's lots of beautiful interactions between physics machine learning and
neuroscience and and you know this school is a lot about that if you're
interested actually we we wrote like a 70 page review and statistical mechanics
of complex systems in high-dimensional data we cover a whole bunch of things
like spin glasses the mechanics of learning random matrix three random
dimension reduction compressed sensing and so on and so forth
it it was our attempt to sort of put some systematic order on the diversity
of topics viewed through the lens of statistical physics but what do I want
to talk about today and why did I decide to to branch out in the direction I'm
gonna tell you about what I think there's lots of motivations for the
alliance between theoretical neuroscience
machine learning that these opportunities for physics and math and
so on so and this is the question that should haunt all of us right the
question is what does that even mean to understand how the brain works or how a
neural circuit works that's an open question that we really have to come to
terms with a proximal
sort of more concrete version of this question might be or specifications
question might be we will understand this when we understand how the
connectivity and dynamics of a neural circuit give rise to behavior and also
having your own activity and synaptic learning rules conspire to self-organize
useful connectivity that's absurd behavior ok so
promising to give us recording some large numbers of neurons and even give
us the connectivity between those neurons know what what if theorist on
often what the arrests do is they they often develop theories of random
networks that have no function but what we would like to do is we'd like to
understand engineered networks that have function so the field of machine
learning has generated a plethora of learn neural networks that accomplished
interesting functions yet we still don't have a meaningful understanding of how
their connectivity dynamics the learning ruled that development experience ok so
basically can measure anything we want these artificial neural networks right
we can measure the connectivity between all the neurons we know the dynamics of
all the neurons we know the learning rule we know the entire development
experience the network because we know the training data that was exposed to
yet we still do not have a meaningful understanding of how they learn and how
they work and if we can't solve that problem how are we ever going to
understand the brain right in the form of this question
ok so that was sort of what was motivating me to look into this so so
this is the outline of the talk the original entry point was trying to
understand category learning in neural networks and then at the end of the day
we made actually several theoretical advances that led to advances in machine
learning and applications to engineering so for example we find we found random
way to nationalizations that make a network dynamically critical and allowed
very very rapid training of deeper let words we were able to understand and
exploit the geometry of high-dimensional era surfaces 22 against it up learning
like training new girl that works and we're also able to exploit sort of
recent work in non-equilibrium thermodynamics to learn complex
probabilistic generative models so what's a diversity of topics but I'll
walk you through them and you can relax because almost everything I'm gonna talk
about is is is published you can read about all of the stuff and most of these
papers are now in my website of the websites not entirely up to date but
these are the references and you guys will get the slides I wanted to again
thank my funding sources and yes the first part was work done with Andrew
Sachs who's sitting back there
NJ mcclellan the next the next piece of work was done with a collaboration with
NGOs lab and then Joshua sold 16 an excellent post op was was working on the
last part of the talk ok so ok so let's start with the motivation mathematical
theory of somatic development I think this speaks to some of the high level
stuff that you guys think about this particular be talked could could be
called the misadventures and applied physicist who found himself lost in the
psychology department store showed up at stanford Jays a great guy was talking to
him and I learned about his work and I realized I didn't understand it and this
is my attempt to understand that work with Andrew J
so what is the man to cognition so human somatic Commissioner rough definition of
this field is that we have an ability to learn recognized comprehended produce
inferences about properties of objects and events in the world especially
properties that are not present in your current perceptual stimulus so for
example I can ask you to look at how far into birds fly and you can answer these
questions correctly despite the fact that there's currently no cat or burden
the road right so you know our ability to do this likely relies on our ability
to form internal representations of categories in the external world
associate properties with those categories because we never see the same
stimulus twice so whenever we see a new stimulus or or we're trying to recall
information for my brain we rapidly identify the relevant category that
contains the information we use that categorical representation to guide
future actions or give answers right to a category formation is central to this
right so what are the kinds of psychophysical task that people use to
probe semantics of missions like this is a very rich psychologists working on
this all the time so one example is looking time studies to ascertain
whether or not an infant can distinguish between two categories of objects at
what age so for example they'll show up a sequence of objects from category 1
say forces and the first time the infancy of course the looking time will
go up and then it goes down over time it gets bored then you show a cow and if
the infant is old enough looking time will go up and then go back down and
that's from that we infer that the infant can distinguish between horses
and cows but if it's not old enough to looking time will not go up so it gets
old
and older can make more and more fine-scale discriminations between
categories it turns out for property verification tasks you can ask a canary
move and certain questions are answered quickly certain questions are answered
late which speaks to certain properties being central and peripheral to certain
categories a category membership cruises the sparrow aberdares an ostrich bird
again there's different relate Leighton seized and that's that suggests that
their typical and atypical category members and also very very important to
us and is inductive generalization we can both generalize familiar properties
the novel objects for example of Blake has feathers does it fly does its thing
and we can generalize novel properties the familiar objects a bird has gene
acts as a crocodile hunter you next as a dog you know so so so people have
measured these patterns of inductive generalizations and his various theories
that tried to explain all this stuff so jake has been working on this stuff from
a neural network perspective and you wrote a beautiful book called semantic
cognition where he uses neural networks to explain the whole variety of
phenomena especially for example the progressive differential progressive
differentiation of concept so let me just walk you through that and so this
was you know you know counter with a first encounter with a deep neural
network so they were they were doing deep neural networks before they became
popular and so what they were doing was they asked henry model infants the
development of se concepts and infants and so what they did was they had a
choice they decide where they had a bunch of objects and each object have a
whole bunch of properties so for example a canary can grow move clients with
right and so what they did was they exposed to this deep neural network to
training data of this form you know they had a whole bunch of features and
questions and objects and they just expose the network to training data
trained it using back propagation and they looked at the internal
representations the network especially their evolution over developmental time
we're training time right and this is what they found so initially the network
started with random weights so there was no structure so these are ok so what
what did they do here they looked at the distances between the internal
representations in this space
ass and they did harder clustering or multidimensional scaling and they found
these plots are these plots right so what you see is early in developmental
time the network's first makes the coarse-grained discrimination between
animals and plants right and then and then later it makes finer scale
discriminations and then eventually when it's fully learned it has it learns the
hierarchical structure that's implicit in the training did ok this is a
multidimensional scaling plot were initially the animals move away from the
plans and then you know fish move away from birds and trees move away from
flowers and then finally individual discriminations earlier
to predict exactly exactly yeah yeah yeah this is the field for a network
that goes from objects that ok so when I learned about this I was at once
excited and mystified and also mystified because this is sort of qualitatively
behaving like the way that instant but his yet it's a really stupid network
with like five players yet I don't understand how it's so I wanted a theory
of of what's going on here so by the way you know there's lots of reasons to
believe that semantic relationships are enclosed are encoded in the brain using
relatively simple metrics like Euclidean distance between your representations
for different objects so for example this is a famous study which I'm sure
you've seen what they showed with a whole bunch of objects to both monkeys
and humans and they clustered the objects are looked at similarity matrix
of the objects measured using Euclidean distance in Euro your laughter from the
audience base of firing rates of neurons here and voxel activity patterns in the
human and they showed the same type of object the monkey and human and the
matrices aligned essentially the basic the similarity structure of internal
representations of the month in humans the sane so we tend to encode similar to
encode semantic information using these similarly representations of this is a
hierarchical clustering to you so so this sort of you know the sort of seems
to actually happening in real life animals and humans ok there's actually
something else that happens is that different properties I learned on
different time scales so for example the network can learn that canaries can move
much more quickly than it learns that a canary yellow so it it learns the sort
of so some properties are much easier to learn than others and the properties of
our easy to learn it for the network also easier to learn for the infant ok
so these are the theoretical questions we'd like to answer what are the
mathematical principles underlying the hierarchical self-organization of
internal representations in the network
you know this is a complex system so what are the relative roles of the
various ingredients its Anderson on your input output response to learning rule
which is back propagation and there's the input statistics is the network
somehow reaching into complex input systems in in the in the training set or
or or can it really rely on her second order statistics you know what is a
mathematical definition something called category currents and how does it relate
to the speed of category learning so what determines the speed at which we
learn categories
why are some properties learn more quickly than others and how can we
explain changing patterns of inductive generalization over these developmental
time scales so so how do we get a theory well it turns out if you look at the
activations of this network as a training over time so these are sigmoid
all units and activations don't really hit the saturation regime that much
during training because you start from small waves so we started with
annotations proposal that maybe even a linear neural network might exhibit this
kind of learning dynamics ok now it's not at all obvious that it should
because of the simple linear neural network in this learning dynamics is
highly nonlinear right but it turns out that even in a linear neural network the
dynamics of learning on synaptic weights basis nonlinear and so there might be
hope that it might work and we might be able to get a coherent theory ok so what
we did was we analyzed just a simple linear neural network that looks like
this that goes from there to hit in there to output layer so the composite
map is linear ok so so we can write down dynamical equations on weight space for
the learning dynamic so this is the training data and we can adjust the
weights using back propagation and these are the backpropagation equations and if
we work in a limit what the learning is slow relative to the time it takes to
cycle through the data say you can take a continuous time limit and you
essentially get a nonlinear set of equations and weight stays right and the
equations are cubic in the weights right and that's because the Eris cortical the
weights right the error is the output minus W W times the input squared so the
error scored again the weight since you can't differentiate we're supposed to
get
the waves on the right hand side the gradient descent two questions about you
become the weights but there is one simplification that happens because the
network is linear it's learning dynamics of sensitive only to the second order
statistics of the data right so in particular the input input covariance
matrix and the input-output covered major ok so essentially this network
knows only about second order statistics in our work here that the impetus was
white so it's really only the input-output 262 drives learning ok so
this is a set of coupled nonlinear differential equations there in general
hard to solve but we found solutions to them so what we we can express the
solutions in terms of the singular value decomposition of the input-output
covariance matrix so so any any rectangular matrix has a unique singular
value decomposition in this context we can think about the input-output
covariance matrix matrix that not input objects to feature attributes and the
singular vectors have an interpretation where these singular vectors essentially
map objects into internal representations the singular values
amplified and then the columns if you are sort of feature synthesizers the
columns are sort of modes in the outfit reaches its ok so this is the SVD but
the question is how does the learning dynamics how does this drive learning
dynamics so what we did was we found exact solutions to learning dynamics of
this form with a product of the input layer 12 leer tu weights and where to
deliver three waves are of this forum where essentially the system what it's
doing the composite system is building up the singular value decomposition of
the input-output covariance matrix boat by mode and each mode alpha associated
with single value alpha in the training data is being learned in the sigmoidal
fashion in a way that so so at times
08 is sort of small and random you know some initial condition 80 but over time
as as time goes to infinity training time goes to infinity the approaches the
actual single value in the introductory covariance matrix so basically this is
the learning dynamic so nothing happens for a while and then suddenly the
strongest single mode defined by the largest single valujet's learns and then
later on smaller Cingular mode gets learned and later on an even smaller
single mode get there and the time it takes to learn each mode is governed by
one over the singular value so just intuitively stronger statistical
structure as qualified but single value is learned first if that's the intuition
yeah
Soto is roughly the time it takes to go through one learning up hot and tea as
training time measured and you in learning a box that's a different yes
the SDSS are these these diagonal elements of the singular value
decomposition oK so that's a general theory yes this is for you one hidden
layer will talk about many many layers eventually and then we'll connect that
to dynamical criticality and dynamical systems ok so actually so you know we
train your on that works we see sort of these plateaus and performance where the
network does nothing and then suddenly drugs such shows in drugs and this
actually shows that you can see very very sharp transitions and learning and
you can actually show that the ratio of the transition period the ignorance
period can be arbitrarily small infants also seem to show these developmental
transitions where they they're pretty dumb for a while make it ten weeks
that's not true it's pretty smart said he does is he loves looking at these
three things like he has a little seat and there's like these three things that
hang over him and you'll still at that for an hour I read nursery rhymes to him
and you can only handle that for about half an hour is really weird these three
random dangling things are much more interesting than I am anyways it's
really bizarre we need a theory for that but okay so that's so yeah you you can't
arbitrarily shot transitions in the system
ok so the take-home messages or so far it is the network learns different modes
of cooperation between input inopportune time scale inversely proportional to the
statistical strength of that derision and you can get these sudden transitions
and learning now the question is what does this have to do with the
hierarchical differentiation of concepts but that's what we like to understand
first so now we've come up with a general theory of learning the nonlinear
dynamics of learning in these in these deep circuits now we want to connect
this back to Hartlepool structure so one of the things with JS work as they were
just working with toy datasets and we didn't have any theoretical control over
the store data
that's what we sort of understood implicitly these toy data sets of
hierarchical structure so we need a generative model of data controlled
mathematically well-defined generative model of data that encodes the notion of
higher ok so so can we move beyond specific data sets to general principles
of what happens when a neural network is exposed to hierarchical structure that's
we'd like to answer so we consider a hierarchical generative model and a
classic parka generative model is is essentially what we want to do is we
want to connect the world of generative models to the world of neural networks
and that you know that will connect the methods of competition neuroscience this
course eventually right so so so yes we have data generated by some generative
model we take that data and we expose it to neural network and we'd like to
understand how the dynamics of learning depends on the parameters of the
generative model ok so a natural generative model for defining hardcore
structure is a branching diffusion process that essentially makes the
process of evolution where properties diffuse down a tree and instantiate
themselves across a set of items so what do I mean by that
ok ok so so so basically imagine for example that your items are at the
leaves of a tree right and you can imagine that this is a process of
evolution where there's some ancestral state may be 41 properties we do one
property to time the properties are independent of each other this might be
an ancestral state like can move right and then each time it this this property
diffuses down the tree there's a probability of footing ok so maybe in
this lineage which might correspond to animals this doesn't flip and Indies
lineages corresponding to plant it does flip so these things cannot move in
these things can move and then maybe it doesn't flips all of these things
inherit the property of moving so these are the animals and these things cannot
move so these are the plans and then we do that for every single property
independently and regenerate a set of feature vectors so that's a generative
model so what are the statistical properties of general models essentially
because we know there were analyzing these deeply in your networks and we
know that the learning dynamics of such networks is driven only by the input up
governance metrics to understand the learning dynamics we just have to
compute the singular values and singular vectors of heart be structured data
generated in this fashion is actually quite easy and we did it so here's what
happens to imagine an isometric tree like this so these are objects if we
look at the similarity structure of objects measured by dot product in the
feature space generated by the features under this wrenching diffusion process
we get this nice blocks within blocks similarity structure where all the items
on this on this branch this item this item are slightly similar this item this
item or even more similar and of course each item is most similar to itself you
have this hierarchical hierarchy of clusters that naturally arise because of
his friends so what are the singular values and singular vectors of the
Associated input upward revisions major leagues
well it turned out these are the one set of singular vectors of so-called object
analyzes which are functions of cross objects there's another side of singular
vectors that are functions across features which I'm not showing you what
we can but there's of course the duality right so that you get pairs of singular
vectors for each single value ok so what's the singular vector associate
with the largest single value will it's a uniform though that's constant across
all the objects but the next the most interesting one the next largest one is
the most lowest frequency function essentially it's constant along all the
ancestors of this branch and a different concert along all the ancestors of this
reg so this singular vector essentially makes the most coarse-grained
discrimination in this parking structure data that the next set of singular
vectors as a pair of them
discriminate between this set of objects in this set of objects and don't know
about these ones and the next one discriminates between this set of
objects on the set of objects as you go down to the smallest singular values you
get fine-scale you you get individual object discrimination so this is how the
hard core structure is reflected in the second order statistics of the data and
the season the singular values so this is the theory for the Single values in
in a in a tree that has five of his party and you can see that the singular
values 2k with the harkin level of the singular vectors ok so there's a general
theory for this and which singular vectors are associated with levels
tree ok so now you can see the end of the end of the story if you put the two
together you automatically get the results that we we were trying to
explain right so essentially the general theory of learning says that the network
learns input output modes on a time scale given by 10 percent whatever the
singular value when the data is hard to be structured singular values of broader
hierarchical distinctions are larger than single adviser finer distinctions
and the input uppermost correspond exactly to heart disease in the trees so
that essentially said the network must learn broad-scale discriminations before
it can learn fine-scale discriminations so then actually what we did was just
analytically worked out that the dynamics of learning for harkin
structured data and we computed the multidimensional scaling and this was
theory we never does single simulation to get this plot we generated we
generated a branching diffusion process which was essentially this one and we
just labeled these nodes arbitrarily with these labels and this is what we
get right so we see the multidimensional scaling part that we sort of see hear
and essentially just to compare this is what was done with a toy dataset over
which are no theoretical control in a nonlinear neural network over which we
have no theoretical control and this is a well-defined mathematical generative
model under linear neural network and we see that we qualitatively explain the
results so this is the difference between simulation and fury right now we
have a conceptual understanding of effectively what was going on in the
circuit and we really a sort of now it's no longer a mystery so now I think I
understand what she was doing
Jane collaborators were doing it would be lovely if for all of the stuff that's
going on in this course we could obtain such a deep rigorous understanding as
much more challenging but it's a goal worthy of pursued I think ok so
conclusions progressive differentiation of hierarchical structure is a general
feature of learning and Eberle networks it cannot be any other way
ok interesting enough cheap but not shallow networks exhibit such stage like
transitions during learning so it's a few just dunno hadn't there's you don't
get this actually you need it
later to do this and somewhat surprisingly it turns out that even only
the second order statistics of semantic properties provide powerful statistical
signals that are sufficient to drive this nonlinear learning dynamics you
don't need to look at the high-order census data to get this dynamic second
order statistics suffice which was not obvious where we started
ok to an ongoing work we can explain the whole bunch of things like a loser
correlations early in learning so for example infants will initially say that
they don't even know that for example pine trees don't join have leaves then
it intermediate point they think that pine trees have leaves in an hour later
point they they correctly know that countries don't have so so so so we can
explain these non monotonic learning curves we can explain these familiar
into the ecology facts we can explain inductive property judgments
analytically we're looking at basic level affects me have a theory of
category and so on but in the interest of of moving forward I wanted to give
short shrift to this stuff so we can answer why are some properties learns
faster basically properties that look that are low frequency properties in the
leaves of the tree gotta learn faster properties whose inner product with sync
with the single largest single value get learn faster that's the story why are
some items more typical we have a theory for that how is inductive generalization
achieved by neural networks we have a theory for that and so on and you know
what is a useful mathematical definition of category coherent so for example you
know there are some things that are just into two-week-old incoherent categories
instead of all things are blue is a very incorrect category in fact it's so
important we don't know the name for such a category the set of all things
that are dogs seems to be a very coherent category and soaker
a well-known name trade name is quite short actually to actually I wonder if
there's a theory were shorter words corresponding more parent categories in
the flick and informative efficient representation of categories to evacuate
so we have a natural definition coherent category that's precisely enough to
prove a theorem that coherent categories learn faster and actually does this also
relates the size of the categories so frequency effectual anyway so so there's
a lot of stuff there but that was sort of the entry point now you know we
wanted to understand that
sure yeah
so we haven't yet analyzed it later network with multiple layers rights were
gonna get their attention
exactly I totally agrees we haven't said anything about that yeah yeah we might
be able to say stuff about that as well
huh
what you know yes so so interesting there is sort of a particle structure at
least in this set of images show the monkeys and humans right so but I agree
like so high construction just one possible notion of structure right and
what the general theory says that it that these linear networks whatever
structures throughout it it learned Single modes at different rates we don't
have a theory for for what aspects of the statistical structure are encoded in
nonlinear neural networks we don't have that 30 what we're thinking about it
moment but I think actually remaining in geometry of internal positions might be
the way to think about that we're starting to think about that but you're
asking much more difficult question but nevertheless we can say stuff we can
actually give you practical ways to better trained on the new neural
networks that's what I'm gonna go next
based on theory ok so so here's sorry I skipped so many slides but there's lots
of talk about ok so now what about a theory of learning in much deeper
networks that have many many layers ok so again I'm gonna make a long story
short because it's all published you can read all the details but I wanted to
give you the spirit of the essence of the intuition behind the work ok so the
questions we like to answer our house training time scale with depth how
should learning rate scale with how do different weight initialization impact
learning speed and what we'll do is once we understand these theoretically will
find a certain weight initialization that correspond to critical dynamics
which all define a deep learning and generalization so the basic ideas in a
in a very very deep neural network right you have you have a vanishing exploding
a gradient problem and that's one of the issues that makes deep neural network
learning hard so if you're gonna back propagate the error through multiple
heirs the backpropagation operations a product of Jacobian some layer to layer
and that product of Jacobian since fundamentally a linear mapping right so
if the singular values associated with that linear mapping
so so so essentially if the singular values of the Jacobian at each layer are
large bigger than one the product of such matrices will lead to a product
matrix that has singular values to grow exponentially with similarly if the
singular values are less than 12 K with right so that's that's a vanishing
gradient in the latter case and exploding gradients in the former case
that seems to be one of the major impediments to understanding deep
learning so what people often did was they tried to scale the matrices to
avoid this question right so what they often do is they they picked up the
initialize the weights randomly so that w is a random matrix with the elements
Wijk are iid Gaussian with a scale factor scaled precisely so that the
largest eigenvalue of the Jacobian or the backpropagation operator is one okay
so it's like scaling the system so that if you place a random ever victory here
right the desired output minus the actual output and back propagated
through a random network a random vector will preserve a rare of actual preserve
its norms its back propagated across and so such as this is the famous sort of
glow and NGO initialization and it works pretty well 4454 or whatever
right okay so we would like a theory of that for the learning dynamics of that
and I said there's no hope for a complete three at the moment with
arbitrary nonlinearities ok we don't know of any situation where an
analytically solved for the dynamics that way through your networks I know
your theory which is a different
generalization so there's very good theories of yeah oh yeah I wasn't here
yesterday so I totally agree so things like the VC dimension your networks
Rademacher complexity all that kind of stuff the dynamics yes I'm talking in a
few of the dynamics the moment I'm well aware of the beautiful theories of
generalization about the generalization error in terms of training error and
complexity that's a beautiful subject in itself
ok so what we're going to do is to analyze the learning dynamics just will
get rid of the nonlinearities right so again it might seem like we're throwing
the baby out with the bathwater but we're going to learn something that
helps us a trained on the new networks ok so the basic idea is that we have a
network which is linear so why output is a product of weights right so then the
the back propagation of cases so then the backpropagation again just a product
of of of matrices right the greatest comics is nonlinear and couple the
nonconvex and to actually even in this linear network you see plateaus and
sudden transitions right and actually interesting enough even in this very
deeply in your network you see faster convergence from pre trained initial
conditions right so basically if you start from random galston initial
conditions you get slow learning for a while and then sudden sudden learning
here relatively sudden learning here if you pre train the network using greedy
unsupervised learning right so this is the time it takes to preach train you
get sudden learning and a drop here remember if you go back to the original
hinson paper this was the phenomenon that start a deep learning 3d
unsupervised pre-training allows you to rapidly trained very very deep neural
networks so the very empirical phenomenon that led to the genesis of
deep learning was present already in deep linear neural networks so deeply
neural networks in terms of their expressive power a crappy because the
composition of linear operations lender they're not good they're not a good
model for details
network in terms of input-output nothings but there are surprisingly good
model theoretical toy model for modeling the dynamics of learning in nonlinear ok
because because very important phenomenon also arise in the deep words
and we're focusing on learning dynamics here ok so we can build intuitions for
the nonlinear case by analyzing the linear case ok so we went through the
three-layer dynamics already what about the multiple their dynamic so it can to
Jacobean can back propagator explode right so what we gonna make a long story
short but what we find is that if you take what we tell you the final result
what we find is that we find a class of weight initialization that allow
learning time to remain constant as the depth of the network goes to infinity
now measuring learning time in units of learning a box right so obviously to
train a deep your own that work very very deep neural network it just takes
longer to compute each gradient in terms of real time of course the time will
scale with the death of the network but you might imagine in terms of number of
gradient evaluations as a network gets deeper and deeper it might take longer
and longer to train it and we show a class of initial conditions which that's
not true as the as the network gets deeper and deeper the number of gradient
evaluations you need to train the network can remain constant even as the
depth goes to infinity even in a nonlinear network let me give you
inspiration for why ok so so for example we and the classical initialization this
this glowing enjoy nationalization doesn't have that but our initialization
does so basically what we did was will start off with a linear networks we
train deepen your networks on Emma's and we scaled the depth like this right and
we started with random Gaussian initial conditions and then ran back propagation
but scaled random gunshots
and we found that the training time as you might expect room with this is
trading time measure number of learning about your number of creating
evaluations but here what we did was we initialize the weight using random
orthogonal weights and then we found that the learning time didn't grow with
death and also if you pretended it doesn't grow with death dramatically
different scaling and learning time between random Gaussian initialization
and random thought analyst position why and the answer is the following Wed
let's think about the backpropagation operator let's say you want to back
propagate errors from the output to the right so that the backpropagation
operational in your network right is just the product of weights throughout
the entire network ok so so you're you're so the back if you do a random
random Gaussian weight initialization here then the this is a product of
random guess in major cities should you understand the statistical properties of
backpropagation you need to understand the statistical properties of the
singular value spectrum of renegades in major cities there isn't really a
general theory for that but we can look at it numerically and get into action
for it so the basic idea is if you have one random Gaussian matrix right the
singular values of W W transpose W that's a famous
distribution marcinko pastor distribution and you know they're a very
in a range that's order one so if you back propagate through one layer your
fine but you don't get the radiance ok but if you look at the product of five
random dozen major sees a single gets very distorted you've got a large number
of similar values that are that are close to zero and that long tail that
extends up to four but if you if you do a hundred layers you get a very very
large number of things about use of a close to zero and a very much longer
tail ok now this is a product of random Gaussian matrices so if you feed a
random factor into this on average it's normal be preserved
the factors length will not change but we know that preserving the length of
the vector is not the same as preserving angles between all pairs of actors case
was actually the way that this network that the way that this product of random
matrices preserves the norm of the gradient is it does it in very
anisotropic way it takes an error vector at the output and IT projects it into a
low-dimensional space corresponding to the singular values to a large and then
it amplifies it in that space so the length is preserved all era factors get
projected onto a low-dimensional space and amplified a lot of air information
is lost in a product of random Johnson majors so that's why the globe in
judicial Commission's work well up to date five or six or seven but they don't
work well up to 200 or in recurrent neural networks as well so so what can
we do with a simple thing we can do is we can replace the matrices these random
matrices with us autonomy jerseys ok so we know that all the singular values
have no imagery for one every single one and the product of orthogonal matrices
orthogonal so therefore the backpropagation operator has all of its
singular values equal to 1 and their generalizations of orthogonal matrices a
rectangular versions when players don't have the same number of neurons in each
layer ok so this is fantastic so this works really well for linear networks ok
but how does this generalized nonlinear networks because then you have a product
of Jacoby Jones right so what happens here what time is it ok so what happens
here
the product of what you have is you have a case or if we imagine how if we
imagine how either errors back propagate to the front or how input perturbations
back propagate so that the end right it's the same things easier to think
about for propagation imagine that you have an input interpreter but slightly
how does the perturbation grow idk well what happens as a linear expansion or
contraction due to W
and then this this nomination is usually compressive so there's a nonlinear
compression due to the Jacobian passes out the diagonal Jacobean passing
through the point was not years and then again
linear modification on the new york impression when your modification under
compression so what we could do is just simply choose these again today random
orthogonal matrices and then what happens is the growth or decay of
perturbations it is Cale Gale the random Gaussian random of other major cities
but scale factor to combat the nonlinear compression and then the dynamics of
perturbations is like this you rotate annually scale compress rotate scale non
linear compressor and so on
that's essentially the type of dynamics that occurs in in dynamically critical
systems that are close to the edge of chaos you get this alternating face base
expansion and compression in different dimensions at different times so now you
can just compute numerically under that initialization how does the product of
japan is a single value spectrum the product of Jacobian scale and its gills
beautifully so this is the scale factor for the type of nonlinear that we used
hyperbolic tangent on the meert the optimal scale factor in in front of the
random orthogonal matrix is one and you see when you choose that even for this
was a hundred mayors i believe even 400 layers to enter and Jacobean from the
input-output has a single value spectrum that remains within a range of order one
if G is even slightly less than 12 singular values exponentially vanish
with death if she is larger than 1 the single girl but actually not not as
quickly as you think so this is the the critically dynamical regime that at
least preserves not only the norm about propagated gradients but all angles
between pairs of radiance light so it's an isotropic isotropic preservation of
error information from the end of the network all the way to the beginning ok
so it doesn't work and it works better than the other nationalizations even in
on your network so we trained 30 they're not mean you're not
works and the initialization works better and and so also interesting the
enough at this critical factor you also achieved better generalization error and
we don't have a good theory for that the test over of course goes goes down so
that's an interesting situation where a theory of linear networks led to a
practical training advantage for nonlinear networks ok yes definitely you
wanted you've won it turns out that I'm we don't have a three for the
generalization error right it just turns out an empirical observation is a
generalization error has a minimum close to 10 but slightly larger 438 may go to
one of largest don't know but what we do know is that I G of one is what you need
for this dynamic less amateur property in the back propagation ingredients
have a general theory for an arbitrary nonlinear d how do you compete she and
so for you give me the nonlinear d I'll tell you which is basically it's a slope
of the non-linearity of the origin right you want you to be won over that slowed
the tanagers the slope of one of the origin is one if the non-energy strongly
compressive right it's slow but one it's very small you want you to be large to
counteract that we have a theory for why we have a three for that as well and you
get another answer was with two or something like that
so if you compute the variance of activity in the input layer and you
propagated through the new expansion and nonlinear suppression and compute the
variance of the Opera clear you find the G that preserves the variance at a fixed
point that that's essentially it's classic mean field dynamics of
propagation of variance through nonlinear circuits yeah so we just one
we just want to wait to back propagate gradients without distorting right
so I think so what's the role of the rotation right yeah I agree we should
think about that more case you're going to lose certain dimensions and you don't
want to lose the same dimension every single time you want to rotate so you
you randomized directions which you lose yeah you could cycle cycle right we know
of course in in high dimensions in high dimensions of randomness you get
deterministic structure out so you can mimic that determine the structure using
a deterministic construction in your fun yet that has an analog and compressed
sensing were initially random measurements were considered to be good
but then you try to design deterministic measurement matrices that enjoy the
properties of random measurement matrices so it's an interesting
directions deterministic initialization ok ok but I we started a little bit
leads so can you guys indulge me and give me 10 more minutes we're over time
that I won't cry out I won't cry here I'll cry afterwards alright so so weird
under two parts of talk in the next two parts are quite quick action they're fun
ok so here's another question that we had ok there's a hole like World of
convex optimization we want your machine learning algorithms to correspondent
convex optimization so we can find the global minimum and there are no local
minimum impedance replacing the global minimum that's a conventional wisdom yet
the DVR network people ignore this conventional wisdom and trained very
very deep neural networks and don't worry about the potential impediments to
local minima they seem to find pretty good solutions y que is the intuition
that local minima are really an impediment and on the New Year
nonconvex optimization in high-dimensional spaces really true ok
and you might think that it's not true for the following intuitive reason so
again it's often thought the local minima at some high level of error in
training our stand as a major impediment and nonconvex optimization okay and you
know this is an example of a two-dimensional caricature of the
protein folding energy landscape and it's very rough so there's many many
local minima and and and the global minimum might be hard to find and that's
true if you sort of drawing random generic surfaces over low dimensions
those random surfaces will have many local minima but of course our intuition
about geometry derived from our experience with a low dimensional world
is woefully inadequate for thinking about geometry in high-dimensional ok so
it turns out that random nonconvex are functions of high-dimensional spaces
local minima are sort of exponentially rare and the dimensionality relative to
saddle points just intuitively imagine you have an error function over thousand
variables thousands not equate Sandeep network that's a small people at work
but anyways let's say there's a point at which the gradient and wait space
evaluations so now there's a thousand directions and wait space you could move
away from that from that extreme what are the chances that every single
direction you move has positive curvature if it's a fairly generic
landscaping answers exponentially small and the dimensionality some directions
will have negative curvature some directions look positive curvature
unless you're critical point is already at the bottom in that case most
directional positive or unless you're critical points at the top of the most
directions of negative curvature right so statistical physicists have made this
intuition very precise for random landscapes and they've developed a
theory for it so this is a paper in Physical Review Letters by Bryan Dean so
what they did was they imagined just a random doubts inner landscape
yeah I guess I'll tell you exactly what
ok ok I agree you're skeptical and I'll address your skepticism soon but let me
tell you what the theory tells you for random landscapes and let me give you a
polemic between the tension between physicist Peter scientists and I'll
resolve your your your skepticism soon so so here's what physicist it right we
often random functions and so so what they did was they looked at in arid
landscape ok that's a continuous function over N dimensions but it there
is a correlation that's correlated over some length scale so it's like a ran its
a single draw from a random Gaussian process where the kernel of the random
girls of the Gaussian process is falling off with some length scale so the Ararat
point one is correlated with the arid point to over some length scale and that
the correlation smoothly so it's a random Smith landscape ok but the
correlations of local essentially and then what they did was they asked the
following question let X be a critical point the point with a gradient vanishes
ok we can play every single critical point in a two dimensional feature space
what is that feature space while the horizontal axis of the era level of the
critical point that how high on the air axis was critical points that and then
this is the fraction of negative eigenvalues of the Hessian at that
critical point so it's the fraction of directions that curved downwards and the
front so now a priority critical points could potentially sit anywhere industry
dimensional feature space it turns out they don't they concentrate on a
monotonically increasing curb that looks like this so the higher that era level
with critical point the more than negative curvature directions you have
to be in order one distance away from the s-curve the probability of that
happening is exponentially small and the dimensionality of the problem
ok now what does that mean it automatically implies that there are no
local minima I ever or at least 20 points of a given index so basically you
typically never encounter local minimum hire
ur that would be stuff that sits here and there's nothing here in a second if
you are a local minimum which means on this access you're at the bottom then
your era level must be very very close to the global minimum so if you're if
you get stuck in a local minimum you're already closed a narrative the global
Minh if you're a local minimum your era level be close to the era level the
global men because what does it mean to be a local minimum 20 the fraction of
the 20 and this is the distribution of error levels of such critical point
they're strongly peaked at this value which is the value of the global minimum
essentially there's nothing out here right now in physics there's this
well-known principle called universality there are certain questions whose
answers don't depend on the details for example certain critical exponents in
the liquid gas phase transition
exactly the same as critical exponents in the ferromagnetic phase transition
because the symmetry in dimensionality the order parameter density in the case
of liquid and position in the case of magnets are the same so there's certain
questions whose answers don't depend on the details they only depend on the
symmetry dimensionality of a problem so one might think that this qualitative
prediction is true and in just generic high-dimensional arid landscapes the
computer scientist would say no no no no no no you're random landscapes are a
horrible model for our era landscapes of neural networks trained in Afghanistan
and so on and so forth
you're completely irrelevant to us because we're doing something special
we're not doing something random we have a lot of structure in the physics of my
counter well you know you just have a high-dimensional problems just the basic
intuition that in high dimensions it's very hard to have all directions should
also hold you and your problem of course we'll never get anywhere if we stop
there
right we have to move over into your land
and just simulated system so often times you know biologists your scientists
don't believe it through until they see the simulation ok so what we'll do is
we'll search for critical points in the air landscape of you girl that works and
that's what we did so what we did was we used this method ok to find critical
point so it turns out that this method is attracted to settle right to Newton's
method will descend in the positive direction but it will ascend in the
negative curvature direction because you can't method is gradient descent
multiplied by the inverse Hessian negative eigenvalue you take the
negative gradient multiplied by the negative eigenvalue turned back around
you go up to a new method on corrected as attracted to settle points ok so what
we did was we looked at the arid landscape of deep neural networks
trained on amnesty and see 5 10 and we just plotted the prediction of random
landscape theory right and what we found was exactly qualitative either
prediction we took each critical point and plotted in the street dimensional
feature space and we found that the critical points concentrated on a
monotonically increasing curve which again shows that there are no local
minima high error and if your local minimum your areas close to at least the
lowest ERA minimum but we found we can't guarantee that the lowest are minimal
funds the global minimum but qualitatively the structural ok now the
issue is what can we do about it so what this is telling us that even in these
problems of practical interest saddle points might stand as the major
impediment optimization because that'll points can trap you know you might get
you might go down here and then there might be a very slow leak Irving
negative curvature direction that might take you awhile to escape in fact in the
learning dynamics that I showed in these transitions in learning hierarchical
structure the thing controlling the transitions with the existence of saddle
points in Weight space of the vineyard
and so the part of no learning correspond to to sort of falling down
this direction slowly and the rapid learning
eventually coming out this way
ok so how do we how do we do that we'll what we can do it you can do a simple
modification to Newton's method which instead of dividing by the Hessian we
divide by the absolute value and again I should say that this was done in
collaboration with lab and it's a fantastic graduate students in los
angeles la did all of this work on the training and testing of these
predictions ok so we we suggested was you know the offending thing is divided
by negative eigenvalues to just take the absolute value which by definition i
mean take the Hessian computed using replaced each negative one with its
absolute value so that will obviously get repelled by saddles and that
actually works really really well and there's a way to to arrive this
algorithm in a way that makes sense even far from settled using by minimizing a
linear approximation within a trust region in which the linear quadratic
approximation agreed to let me show you first that it works so that this is the
most dramatic plot so basically what we did was we did stochastic gradient
descent right for a while and then it seemed like the error as a function of
training time plateaued those for a deepwater encoder and a recurrent neural
network problem so when the error as a function of training time plateaus
that's sort of interpreted as a fact that you're stuck in local minima right
but actually we switch to this what we call the saddle for Newton method the
era suddenly drops again so this was an illusory signature of local minimum was
actually a saddle point with probably a very shallow negative curvature
correction was hard to escape then we switched our algorithm we could escape
and and and you know what these crucial that we do do better in the final
training hours while the other out rooms ok so oK so that's that
now if you'll indulge me for just five more minutes we'll talk about the
reversal of time
so in the beginning there's not that much difference because if you randomly
initialized you're usually relatively far from the saddle so is more
computationally complex that's just great to see you start with it you can
see and even if you start with settle for noon it doesn't give much of a
difference initially it really helps when you get close to settle
so so we looked at problems besides 5,000 weights or so completely compute
the second derivatives there's well-known algorithms like lbs and stuff
that can approximate the Hessian using the history of Radiance we tried that
its exact same algorithm change so it reduces a positive definite quadratic
was a good model for your function which is the entire theoretical justification
for Newton's method because Newton's method assigns a quadratic Bowl in one
step at a quadratic more awful model for general nonconvex our landscape because
it's riddled with negative curvature so the entire theoretical justification for
Newton's method is irrelevant to the problems we care about instead what does
so just to emphasize this instead what we do is we use the quadratic course or
curvature information very very differently we don't approximate the
function using the quadratic term we approximate the function only using the
linear term a linear term is always a good model locally but if you minimize
of linear approximation your minimums that offered infinity right so we don't
go that far we only go as far as you're willing to trust a linear approximation
to use the difference between the linear approximation the quadratic
approximation to bound how far we go in minimizing the new approximation to
Newton's method you use the quadratic approximation approximate the function
which is a bad idea in this method you use the quadratic approximation to find
the trust region which is a good idea and use a linear approximation which is
always a good idea that's the essential intuition ok so now ok so now how do we
trained with thousands of those actually how do we how do we model complex
probability distributions so we wanted we want to sample from very very complex
probably distributions and too complex distributional learning so this was done
by a fantastic post doc of mine Josh so we were going to Berkeley for this
monica Breeze mechanics meetings and things like that and you can show that
roughly speaking the second law of thermodynamics which says that things
get more and more disordered what time can be transmitted violated and small
systems for short periods of times you can spontaneously generate order ok so
I'll just go through this again very quickly so so here's the basic idea
let's say you have a complicated probability distribution lets just
destroyed let's feed that probably distribution through diffusion to turn
it into a simple distribution may be an isotropic Gaussian and we keep a record
of the destruction of structure and then we try to reverse time in that process
by using deep neural networks to reverse time and then essentially create
structure from noise and then you have a very very simple way to sample from
complex distributions if you can train the neural network which is you just
sample noise and you feed it to a deterministic neural network and that
constant just sample from a complex distribution and so this was inspired by
recent results back to the basic idea again it's let's imagine that you are
very complex distribution corresponding to this density of die you defuse for a
while it becomes a simpler distribution eventually become uniform you can
destruction if you if your reverse process of diffusion you'll never go
from this back to this but if you're refers processes and neural network
trained to do it you might be able to do it so that's the basic idea so so that's
what we did and I'll just show you some nice movies to show that it works
but it still learning distribution it's it's a sample from a complex
distribution so let me show you examples of it working and then we can discuss
potential theories for it so so here's the basic idea okay so this is a
classical toy model will go to more complex model this is
distribution two-dimensional space and so what we do is we just systematically
have the points to fuse under Gaussian diffusion with restoring force the
origin of the stationary distribution of that of that destructive process is an
isotropic Gaussian ok and that's that's that's what happens so that's our
training data higher movie is the training data then what we do is we
train a neural network to reverse time in that movie so it's a it's a neural
network with many many layers hundreds and hundreds of layers right so
classically training a network of hundreds of layers you have the credit
assignment problem because you don't know what they are supposed to do
credit assignment problem because each layer going up to the next layer just
has to go from time to time 2-1 in the in the training data so you have targets
for all the intermediate layers there for you
circumvented the credit design so it's it's relatively easy to train such
networks and so once you have such a network what should be able to do you
should be able to feed that neural network and isotropic Gaussian and then
have that yeltsin be turned into the data distribution so that's what happens
this on the right is a different calcium and we just feed it through the train
deterministic neural network and out pops the
perfect there's some data points that are over here but this is roughly the
distribution that it learned which is similar to what it was trained on ok so
now we can look at slightly more complicated distributions so ok
so now we can trade it on a toy model of natural images right so classic toy
model of natural images of the dead leaves model where the sampling process
as you just throw down circles of different radii you get a complex model
of natural images that encounter that has long range edges occlusion coherent
so far longer length scales and so on and support so we can train the neural
network on such distributions we train it in a convolutional fashion by working
on local image patches and we can false information propagate over long ranges
and so again we take these natural images and turn them into noise keep a
record of that movie and then reversed the flow of time using a TV network and
ok so once we trained right we should be able to train noise into the network's
best guess as to what a dead leaves model would look like
so this is what happens it's taking noise and it turns it into a gas oK so
it's not a perfect model but it turns out blog probability of dead leaves
under this generative model that consists of a thousand layer deepen your
own network that's higher than any other model so far so this is currently state
of the art ok and as you can see it gets long range for herons and sharp edges
and moreover against long-range coherence in the orientation
hard to do in in generative models in natural images on the input the input is
so it is again exactly that's right so the importance of random noise white
noise and pixels pixels yeah it's not because the Gaussian samples we put in
our never guessing samples that came from the training did you can never
memorize many of these images and ensemble oh yeah so it's not exactly so
that's the analog here right
many points we have many many points in this distribution and and that's the
trick to it many analog here is each point of the dead leaf image and that it
diffuses into white noise ok so now we can actually do something somewhat
practical with this is week sample from the conditional so then we also traded
on textures ok so so so for example textures of bark right and we can also
sample from the conditional distribution so what we can do is we can clamp the
pixels outside of some range replace the intruder with
make it blank and because the network operates in the convolutional fashion
information from the boundary should propagate into the interior and fill it
right so if we look at that so that's white noise and the network is filling
it in and it fills in it fills in the best guess image ok and so again it's
not identical to the original image but it does get long-range structure
parents and the orientation of the edge and smooth structure as well again this
is a thousand there nearly at right now there's some lessons here ok oftentimes
when we model complex data distributions what we try to do is we try to create a
stochastic process was stationary distribution is the complex distribution
not your distribution has multiple nodes you're going to run into a mixing
problem because they can take a stochastic process a long time to jump
over energy barriers that separate the multiple modes so you always have a
mixing problem that oftentimes your train system trade publishing models you
have to sample and the news
train the model that makes training a long long training take a long time so
what we're also doing in addition to circumventing the credit assignment
problem in training very deep your letter words are circumventing the
mixing problem in training the generative model because we're not
trying to model the day-to-day description as a stationary distribution
of us
have to run for a very long time to get to the station is reason we're demanding
that during training the process get to the the data description in a finite
amount of time because during training we demand that we get to the date
attrition of fine animal a time where circumventing in mixing problem during
training and that's the idea that that's
idea that there's lots of results now that show that you can attain
information about stationary equilibrium distributions in nonequilibrium
trajectories ok so that now I'm done so let's see what ok so there is that ok so
again you can read about all of this stuff in the set of papers again and I
think my funding and just that the key players so Andrew Sachs you know do the
work with me on on on your learning dynamics and learning hardcore category
structure
Joshua selection into the work deep learning using knowledge and Truman
Annex and the the work and saddle points as a nice collaboration
fantastic graduate students ok so I think there's a lot more to do in terms
of unifying your sides machine learning physics math statistics all of that
stuff it will keep us busy for the next century