this is a panel on micro electronics
we're we're looking at for the - all
through the foundation of our digital
society and we have extremely qualified
panel to discuss this Carver Mead
electronics industry visionary Dan
hammer strum who did who ran programs
inside of DARPA on novel technologies
and Doug Berger the guy at Microsoft who
you want to talk to if you've got a new
technology you want to put into data
centers first a bit of housekeeping
please mute your cell phone do not turn
it off because you need your cell phone
to ask questions so I if I look up and
people looking down fiddling with their
thumbs I know you're not doing email
you're sending in your questions and I
appreciate that so let's get started I'm
Steve trim burger I am recent DARPA
program manager in the micro electronics
technology office so I come to DARPA
from Xilinx where I was responsible for
bringing new technologies into
programmable logic so a little bit
familiar with new technologies but also
very familiar with technology scaling
because that benefited FPGA is quite a
bit I started I started my my career as
a graduate student I was at Cal Tech
studying with under Carver Mead and Ivan
Sutherland and interning summers with
Lyn Conway I've had the privilege of
living my entire career with Moore's Law
it's been great what technology scaling
gave us was but give it gives us more
transistors they're cheaper they're
faster they're low-power they're more
reliable it gets wonderful as I said
several times I love being a design
engineer because every two years the
process engineers make me look like a
genius
maybe they need to get to work for
myself well maybe not so we're looking
at maybe it's the end of Moore's law and
then that kind of if the technology
scaling is slowing down what are we
going to do but that's not the only
trend that's really interesting there's
the in there's the environment of our
computing if we were here 20 years ago
and talking about computing we would
talked about computing on the desktop
personal computers 10 years ago we
talked about mobile computing now we
talked about edge and cloud if the
computing is being done in a different
place
maybe the computers ought to be
different the second major trend is what
we're doing
it's the workload computers were
invented they were doing ballistic
trajectories a lot of nice math they're
used many years very successfully doing
database operations we started doing
computing putting pretty pictures on the
screen and now we're looking at
computers who want to do machine
intelligence learning inference making
decisions if we're asking our machines
to do something new maybe we ought to
have new kinds of machines to ask
it certainly has worked before digital
signal processing GPUs as the kind of
work change the kind of processor
changed to so now we're going to look
ahead we look at neuromorphic processors
different different kinds of compute
technologies are these the next step
actually what does it take to make it
the next step so that's what our panel
is going to discuss what does it take to
make it and should DARPA do it what does
DARPA role in bringing out the next from
the next revolution in compute
technology and so to begin answer those
questions I'm going to call up Carver
Mead Carver Mead is the the Cal Cal Tech
Gordon and Betty Moore
professor emeritus emeritus means he has
time to come talk to us and so we really
want to take advantage of that he is the
one who did the first study of
semiconductor scaling he wrote with
Caitlyn Conway the textbook that
generations of designers learn to do
VLSI system design and wrote the book on
neuromorphic computing that's led to
generations of now newer more quick and
artificial intelligence carbon Medan I
am I came to neural networks as they
were called back then from spending
fifteen years doing digital VLSI designs
and teaching students how to do that and
out of that process came the notion of
optimal designs how do you make an
optimal design on a silicon chip and
what that boiled down to as there's a
cost vector that you want to minimize in
some sense and the cost for a silicon
chip the three of them there's the area
that's obvious there's the power it
takes or the energy per result and then
there's how much computing it can really
get done per time and
if you think about silicon as a medium
then there's some obvious things you can
say one is that that the power and the
area are interrelated closely because if
you're going to ship information far
away you have to charge up a longer wire
than if you can use the information
locally and charging up the longer wire
takes more energy so the energy and the
area are related and also it takes
longer to charge up that wire so the
time is related so anytime you have an
architecture where you send a lot of
information a very long way all the time
it's very inefficient guess what that's
the definition of our ubiquitous single
instruction stream computer it just does
that well can can we do better and and
in the process of doing better can we do
useful computation it's easy to help
come up with something where you can do
a lot of stuff but is it useful to
anybody so for me those questions became
a quest for is there an existence proof
for something that computes differently
than we do on our silicon medium and it
doesn't take long to realize that the
brains of animals even very simple
animals are exquisitely set up to do
really quite daunting computation on
very low power just for example ordinary
fly does all the time a set of tasks
that are
at the same general level as an
autonomous vehicle was faced with and it
does it on a mill awhat in a very small
package so there's something going on
there that isn't just business as usual
well so you start looking inside nervous
systems of animals and the first thing
that becomes apparent is that the neuron
which sort of the the restoring element
in a neural system as typically
thousands of inputs well in those days
we had no idea what to do with a
thousand in place except choose one up
like a memory does well then about that
time there was a movement in the area
called
neural networks and what they were in
those days were computer programs that
emulated somebody's model of what they
thought that the brain might be doing
and that had been going on a long time
as you heard yesterday and not getting
very much back for it and then in the
early 80s the back propagation algorithm
got invented basically the notion was
you have planes of computation which
then project to the next plane which
then projects to the next plane and
those computing elements in those planes
were called hidden units and then you
get some result out here which you can
compare with the truth and there will be
errors and you just propagate those
errors back through where they came from
as weighted by
the computation elements they came
through and that ended up starting to
work but it worked on toy problems but
there wasn't enough there somehow and we
went through another winter as we heard
of the AI section of nobody cared about
neural networks because they can't do
anything real but there were a few
people that kept at it and they just
kept at it and it turned out there were
a few technical things at the very
bottom level that we didn't get quite
right and it took a lot more scale of
computation and a lot more scale of data
truth data that we knew the answer and
then could propagate the errors back
it took bigger scale than anyone had
imagined so in the last five or so years
it started to be able to do problems
that people actually want had done and
you all know by now that what's now
called deep learning it's exactly the
same stuff with a few important tweaks
is now big big deal and it's solving
great big problems for great big
companies and they make great big money
doing it so it lives on that's gonna be
part of our landscape for a long time
but let me remind you that it employs
basically the first observation you
would make about the nervous system and
that's all so now I talked to people and
they sort of have the opinion oh now
we've discovered a secret of the nervous
system and we're doing it so done
well not so fast there are a lot of
other secrets have come computation in
the nervous system to be thought about
and to understand whether it does or
does not map onto silicon let me give
you just a few example of flotsam we all
do restoring logic we go through and
we'd go through a NAND gate or an or
gate or a decoder or something and then
we amplify the result of one rail or the
other of course that's a very small
amount of analog computation before it's
restored all it does is to get over the
threshold or under the threshold of an
analog amplifier and then it gets
smashed against one rail or the other
nervous system does a very different
thing
it has between the inputs of its
elementary computing element it
typically has a thousand or several
thousand inputs which turned into little
analog signals that travel down the
dendrites as they're called of this tree
at and they don't grow and they don't
die out but when they merge together
they get bigger so now you start and
when the whole thing at the bottom grows
big enough it'll fire the nerve and if
the neuron is getting a lot of those
things there's an exquisite gain control
system that turns down the gain so that
it still fires not too much faster and
turns the output into a a digital bit
well it's not quite a bit because the
neural code that comes along is encoded
in the relative time
of arrival of different inputs and you
can imagine that if you have a thousand
inputs and you have different analog
times of arrival if you can decode the
analog time so that when they come
together you can pick out arbitrary
subsets of the thing you're trying at
any level of input and that's why the
nervous system has six orders magnitude
of dynamic range to where you can you
can act when you're driving down the
dark road on the arrival of a few
photons in your rat now that's something
we don't know how to do well should we
be doing it I don't know I tried to do
it fifteen twenty twenty years ago now
and failed well we didn't have the
technology we have today twenty years
ago and it was a lot harder than I
thought and that hasn't changed
should we be doing it I don't know
that's just one example of the kind of
thing that can be useful I'll give you
just one more than all about everybody
do it it's clear that the nervous system
has exactly the same kind of limitation
we do in terms of of communication that
it's it's cheap to do local computation
communication and it's very expensive to
do long-distance so what it does let's
take the visual system as a example it
takes a little little region of
photosensors and right behind the photo
sensors it puts some analog computation
very simple stuff there's some some
neurons there that average what's coming
in around there's some neurons there
that that take time derivatives there's
some simple stuff but by the time the
outputs go
on digitally digital in amplitude and
analog in time down the optic nerve
they're already processed through
several reasonably sophisticated levels
of adaptive decoding that's something we
could do right now today and make better
imaging systems and in fact there's some
people in Zurich that have done spent 20
years getting one like that to work
that's something DARPA should get deeper
into that's that that's I'll stop there
I could go on for hours we will have
some had some Q&A and you can add a few
more stories if you need to now next the
next panelist is Dan hammer scrum Dan is
it's a high-performance computing
he was a DARPA program manager from 2012
to 16 where he ran the up side program
and looking at some of these alternative
computation models is now Portland State
and we'll let let him talk about a
little bit history and a little bit of
present and a little bit of future thank
you well first of all I want to thank
MTO for inviting me and it's a great
pleasure to be with all these
distinguished colleagues of mine and I
to cut my teeth on Mead Conway many many
years ago and I also AM amazed still
that I've ridden an exponential my
entire career I think we're all very
privileged to been able to do that what
I'd like to do today is kind of pick up
a little bit where what Carver had
started again doing more neuromorphic
computing as most of you know me that's
what I'm going to be mostly focused on
anymore
if you look at the traditional deep
learning models it turns out that
they're solving some wonderful problems
but they also have a lot of shortcomings
they don't handle time very well they
don't actually do continuous real-time
learning they learned offline in a batch
mode so you don't have lifelong learning
as Ava's program is promising a look at
they use huge amounts of training data
there's no way to really transfer the
learning to another network and I think
most importantly they don't capture high
order structure in the data itself so in
discussing the shortcomings of existing
deep learning models Gary Marcus at NYU
said that the deep learning systems
learn correlation between sets of
features of themselves flat or non
hierarchical hierarchical structures
inherently were directly not represented
in these systems and as a result they're
forced to use all kinds of proxies and
approximations and they don't actually
get to it very well last year John
Lansbury who was office director and I -
Oh a very nice presentation we talked
about the three waves of AI the first is
using handcrafted systems so this is the
expert system the rule-based systems
that we'll remember from a number of
years ago the second wave is statistical
learning and that's where we are now and
these networks are accomplishing some
really amazing things to be sure the
next wave from idea I two O's
perspective is context adaptation and
explanation and it's my contention that
you really have to understand the
structure of the data and it's high
order relationships of the components to
be able to do that third wave okay so
I'm proposing that we need to continue
to look more thoroughly at neocortex as
a potential model of computation for
what we're doing what do we know about
neocortex well I'm gonna run through a
number of characteristics it's not an
exhaustive list and you pick any
neuroscientists they'll probably
disagree with half the
that I'm talking about okay the question
is fundamentally the question is do we
know enough about neocortex to be able
to glean characteristics and start
incorporating though those
characteristics and the algorithms and I
think there the answer is yes
so the cortex is a roughly square foot
and it's a in humans it's about six
millimeters thick it consists of a
two-dimensional array of modules which
are called columns
[Music]
it's laminar so each column has six
different layers that are customized for
certain kinds of computation and
communication they have both excitatory
inhibitory functions and it's massively
parallel they're roughly about a million
common columns in the human brain
hundreds to thousands of columns in
lower mammals it is roughly uniform for
all areas of mammals and for all areas
within an organism and for all or
mammals so it is closest nature gets to
a general purpose computing
architectures also it doesn't do
gradient learning it does more what's
called prototype learning where it's
taking snapshots and this allows it to
learn very quickly and allows transfer
learning and other things to eventually
occur cortical connectivity and
activation is very sparse communication
is also sparse neurons do not do inter
products with all their inputs a spike
arrives they do an update at an internal
state and they move on okay this is
vastly different from the linear algebra
that is used in the traditional learning
models that we're using today dendritic
trees perform far more complex
computations than inner products or
typical vector matrix inner products the
intuitive trees have computational
subunits some of them are active active
components often doing various kinds of
threshold and
a coincidence detection okay cortical
connectivity has certain statistical
properties of local long-range
connections if you want to know
something about cortical architecture
there's a great book it was published
about 20 years ago by Brighton Bergen
shoots called cortex statistics and
geometry of neuronal connectivity and
they go through and reanalyzed Mouse
cortex and what they concluded is that
cortical connectivity is bimodal that
there's a lot of local connectivity and
then there's some random non-local
connectivity
my research group and I are looking at
this connectivity patterns because we
believe that they say something
fundamental about what cortex can
represent so you have local regions that
do kinds of associate memories and get
local interpretation of the variables
that they're seeing and then at a higher
level these these regions then correlate
with each other creating hierarchical
kinds of representations so a number of
researchers are now developing
algorithms based on the neural science
of cortex most of you are probably where
Jeff Hawkins is group at Numenta
they're developing increasingly
sophisticated models and actually
starting to show some real results that
are making their models competitive I
think for this panel a key question then
concerns the essential hardware
acceleration for these kind of models
it's not likely that hardware developed
for traditional machine learning are
gonna do these models very well but way
too different my other things are also
low precision typically operating with
just a few bits of few bits of precision
but there is one fundamental difference
between silicon and
the neural models and that is that
biology has large numbers of low
bandwidth connections there's a lot of
dis in that correlation silicon has
small numbers of high bandwidth
connections and it's a fundamental
mismatch in the neuromorphic VLSI that
Carver and his group developed at
Caltech one of the concepts they
developed is something called the
address event representation which is a
way to multiplex large numbers of
virtual connections onto small numbers
of metal connection
another issue concerns the computations
themselves cortical structures are
significantly recursive and such
recursive models basically make
nonlinear energy services these become
what are called attractor models
computation then becomes a process of
parallel constraint satisfaction through
a nonlinear energy space and there's a
lot of speculation is that how cortical
circuits may act CD doing that kind of
computation so I don't like to diminish
the importance of the work that's going
on in the last 20 years in algorithms
certainly for the the deep models but I
think it's important to keep in mind and
cart Carberry referred to this as well
Moore's law has played a huge role in
the success of machine learning we have
computers now that a thousand times
faster and have thousand times more
memory than they did in the 90s however
we haven't solved the fundamental
problems of intelligence and we don't
have a thousand decks left anymore you
know Intel can debate how much scaling
it really might be but the general sense
is that Moore's law is slowly coming to
an end so the question is how do we
proceed from here and I think the
important point is that we need to cast
a wide net we need to look at different
kinds of representations and
computations and help us approach a lot
of these problems and
official intelligence so Moore's law has
not been kind generally to
unconventional computing because we just
wait two years and it's 2 times faster
so there's no reason to worry about more
unconventional architectures however I
like to joke with my students that
unconventional computing has now been
freed from the shackles of Moore's law
so machine learning cortical models 3d
VLSI asynchronous computing resistive
Ram computation and memory hybrid analog
digital circuits carbon nanotubes
quantum annealing coupled oscillators
kermy ons
I think the future of semiconductor
industry engineering is exciting and
wide open so our third panelist is is
Doug burger and Doug is his career in
computer architecture is won the ACM
where he's Wilkes prize in 2008 he
founded and leads the Microsoft catapult
project if you don't know what that is
that is putting FPGAs and data centers
. either guy did it and by doing that he
achieved the greatest reward an
industrial researcher can can gain and
that is being promoted out of research
into product development so he is now
technical fellow at Microsoft and will
talk to us a very different view of
unconventional computing go ahead thanks
very much Steve we appreciate the
invitation very much it's a great to be
here with colleagues that I'd hope to
meet someday and I've read several of
their books so it's it's really just
wonderful so I'm the technical fellow
and Asura's Hardware group that covers
our cloud and edge computing and given
the the capital we spend we think very
very hard about the roadmap and I mean
the cloud industry is spending
and only tens of billions of dollars and
soon-to-be hundreds of billions of
dollars a year on infrastructure and so
it really matters that we get this right
just as an industry and that our that
infrastructure to be computing utilities
is efficient as possible so I thought
I'd start with a just a few tactical
points and then think about some paths
forward that'll hopefully be helpful to
the government and DARPA so number one
at least my view is that Moore's law
ended in 2015 so it's nice round number
gives you exactly 50 years but I didn't
pick it because it was 50 that's the
year that industry realized they
wouldn't be able to stay on the two-year
cadence I mean Moore's law is about a
rate and a predictable rate and that's
the that's the point at which they said
well we're not gonna make the next year
for that next node so we're still seeing
scaling but it's attenuated and slowing
down and the gains are less predictable
but that predictable constant rate has
stopped so Moore's law is done and and
scaling is slowing and that really
there's in the underlying technology
there's been a four-stage fall-off so
first we saw frequency gains cap off and
that was over a decade ago and then
power efficiencies stopped improving at
a rapid rate so we lost speed and we
lost power they're still gains but
they're much smaller that historical
gains with each node and then the one
we're facing now is cost so when you
when you move to a new technology node
and the transistors don't get any
cheaper if you want to put a new feature
in the silicon you have to take
something out at fixed cost and that's a
position we've never been in before as a
computing industry now there's a lot of
debate about when that's happening but I
you know the chatter is it's pretty
close and so watch that very carefully
because that changed is the economics of
the industry and then the last one is
density you might be able to pack more
in because they're smaller but they may
not be much faster or more power
efficient which means they have to be
dim or dark a lot of them and they may
not be cheaper which means you pay more
for that increased density you know
whether we get to you know more
expensive transistors and higher density
remains to be seen
it's a market but its frequency power
cost density in that order we don't see
any CMOS rescue clothes on the horizon
so I mean there's lots of research going
on great work that's but we need we need
to move to something new so that's an
important area of focus the commercial
growth and maybe this is a tactical
industry perspective since I'm the
industrial person on the panel the
commercial growth is now so large that
that burden of legacy is getting bigger
and bigger if you just look at Microsoft
you know we're primarily an x86 company
although we invest in a lot of things
but that's a legacy is a but that legacy
is a is growing exponentially every year
in terms of the legacy base we have
since our cloud is doubling
year-over-year so that that it's getting
harder and harder to move off to a new
new paradigm it can still be done but
sort of to compete with that legacy tale
the new things need to be kind of bigger
to be proportionately interesting and so
it's getting harder to move off of
legacy it's much harder than it was 10
or 20 years ago but Jen and general per
people talked about a lot about
accelerators and we did work in
specialization in the data centers the
general-purpose computing remains
incredibly important just because of the
capital investment you need to make and
the complexity of managing lots of
different architectures at the system
level and so what we need our new
general paradigms that are better that
are more than good for just one thing
that are applicable to a broad range of
things one of the reasons deep learning
has taken off is because it seems to be
pretty general you know it marched
through a lot of the traditional areas
like vision and speech and translation
semantics you know search and and
replace the customized algorithms that
had kind of plateaued in each area I
mean they're different deep learning
architectures but it's the same
underlying paradigm it's to cast a
gradient descent and and many layers of
compute so we need so that that's I
think why you're seeing so much
investment because there's not a lot
else out there that looks general FPGAs
or a different paradigm but there
general which is why we've invested so
much in them you have the programming
challenge and a lot of progress is being
made there so I thought I'd wrap up with
the five
paths that I see as good areas of
investment going forward you know what
do we need well so number one we still
need more efficient CMOS architectures
if you take a general-purpose processor
and carver alluded to this as well it's
not ten percent efficient it's not one
percent efficient it's you know half to
a third percent efficient if you look at
the amount of energy that goes into
executing a math operation versus the
amount of energy spend executing an
instruction so you know it's factor of
300 off so we can do better but we need
to maintain the programmability that's a
hard challenge and one thing we're
investing in something I call structural
computing which is building larger
structures and software and mapping them
down to the silicon as opposed to doing
everything an instruction at a time kind
of structured data flow I mean Bach has
talked about this in his Turing lecture
and I get around the von Neumann volume
neck sorry I think that will be one big
area of investment a second will be
moving to new compute paradigms so the
three are the 3 obvious ones are biology
neural networks and quantum quantum
still seems to be a ways out and we
don't understand how good the algorithms
will be commercially neural networks are
the ones having a lot of commercial
success but there I think limited for
the reasons that the other panelists
have described and I think more
importantly we're heading towards a
compute wall so what we've done is
picked all the low-hanging fruit
allowing us to move up that three or
four or five orders of magnitude in
efficiency but there's really three
things you know that you can do to
improve your linear algebra efficiency
you go to a new process node that's kind
of done or getting close to being done
you can put more of the energy and your
chip into the math and the new
accelerators that are coming out have a
significant fraction in the math there's
not 2x left there I don't think number
three you can reduce the precision of
the operators and we're you know down
now around 3/4 bit then doing pretty
well so I think we're we're gonna
definitely get a need to move to
something new and the biological
inspiration is the only existence proof
we have that there's something out there
that's much more efficient six orders of
magnitude
but it operates on completely different
principles as we've all said and and we
don't know how to exploit those
digitally yet or in you know silicon
analog systems last three just very
quickly then investment in the uncor you
know outside of the the computes of
memory storage networking those great
things happening there DNA storage
optical networks etc etc non-volley
non-volatile memories resistive memories
atomic memories so think there's still a
lot happening there and there will
continue to be from a systems
perspective this is I think sometimes an
underappreciated change Steve talked
about cloud and edge you know what
within Microsoft we really have
convinced ourselves what we're building
is a planetary computer in Amazon and
you know other cloud companies are
building them to you know mega data
centers around the world data centers
underneath the the ocean with massive
networks and then billions of devices
attaching to them and that's not just
marketing hype I mean when you have all
these sensors coming into a central hub
and you can do large-scale analytics and
then push control out it's a very
different paradigm than the sort of
standard client-server you know I have a
client talking to a server and it's at a
massive massive scale so that there's
something new emerging there that we're
sort of riding and we don't fully
understand although we're more certainly
selling it and then number five is is
security as a first principle I don't
think I have to say that here people
understand how important is better than
I do
one really interesting and this is my
final point I don't like to talk our own
book too much but one really interesting
thing that that Microsoft announced this
year was something called Asher's fear
that really when I saw it shifted my
thinking and the what the offering is is
a IOT microcontroller where we took the
security block from Xbox that's been
showing to be very very secure immune to
physical attacks put it a couple it with
the microcontroller and you can create a
trusted connection into our cloud with a
strong root of trust and ability to send
down up
dates and patches and what that does is
that secures the end point and brings it
under the security umbrella and so
really that's end-to-end security at the
distributed system level now I'm not
saying it's perfect but it's an example
of the sort of things that we need to do
to secure very large very distributed
systems and have roots of trust that are
not just a single point on a machine so
I'll stop there but I'm sure we'll have
a great discussion thank you so a
reminder send us send us your questions
and as moderator right get the first
question so what I'd like to do is is
start we've heard a lot about
neuromorphic computing that and that
model and I don't want to I don't want
this to be solely about that but I do
want to explore this a little bit more
and I think I've heard a little bit of
there's a distinction between
neuromorphic and neural nets and I want
to tease that out and what I really want
to where I really want to go is where is
the efficiency coming from give it is it
from this network representation are we
doing it from the analog is it from the
pulse based signaling and harbor I'm
gonna start with you well what Doug said
is is common to silicon and the nervous
system if you use lower precision for
representation let me just back up a
little bit if you train a neural network
to recognize things to give your results
out of of big data it's useless if it's
just a lookup table if you take all of
the things you've ever seen and memorize
them and give you back the answers of
what they were choose to be you give it
one new thing and it'll fail if it's all
new things they'll all fail
well the corollary to that is it's crazy
if that's what you're gonna do you want
something that will generalize to things
that are sorta like what you're looking
for what that means is it's crazy to use
64-bit representation if it depends on
that last bit
it'll never generalize so that's why at
the fundamental level you want to store
the parts of the network at lower
precision because you it's
counterproductive to be using higher
precision well the precision comes in as
something like the square or maybe more
of the energy it takes with with exactly
the same technology otherwise so that's
why people are doing it right that's
obvious we've known that for a long time
pointed it out in the 70s and nobody
paid the slightest attention because a
we weren't doing girl that works and B
nobody cared about power because what
the heck you plug your PC in the wall
and so it takes a little more power a
big deal so that's that's one of the
things that that's common to it and it
has a lot to do with now it's actually
better than that
we were the early people trying to get
people to scale the voltage down on the
LSI technology and they hung in at five
volts for way way longer and it had
reliability problems of mainly it had
energy problems it just dissipated way
way more than you want it goes like the
square of the voltage so nervous system
works on about 100 millivolts
and and that helps so there's a few
orders of magnitude we just got to there
and then the other one I mentioned
already is that and the thing you talked
about that if you build a hierarchical
system you can find ways in which in the
idea space or the representation space
things are more or less local and then
you can combine your short-range
communication with the computation in
smart ways the things that are that are
doing the the computation are also doing
the communication locally and we started
working on the retina because that was
the most obvious example of that and we
knew what the inputs were when you get
to to cortex a little harder to tell
what although a lot more snow now when I
was doing it 35 years ago it wasn't much
known there it's much better now that
you're I'm glad to hear you're doing it
again
but the the architecture in
representation space is probably not
totally different in cortex than it is
in the retina and I think there's
wonderful work to be done there in the
exploitation of locality and
representation space the columns that
that Dan talked about so there's there's
three things and there's more but but
that'll give you a few orders of
magnitude yeah I'll take a couple orders
of magnitude and run with that for a
little while let me add Linda that
passes to Dan and say well do we need
new new process technology should
you look at some of the novel
technologies or can I just build this
out of good old CMOS and and let the
process engineers make me look like a
genius again and that's going to be part
of the transition stage but as I was
trying to convey is that the
computational models are very different
they're low precision they create these
attractor models and sometimes most of
these models actually fit an analog
computing paradigm better than digital
and when you get into that space then
the question is do you necessarily need
CMOS there are the other technologies
that might might work as well I think
what's gonna happen is you know it isn't
like there's the machine learning with
the deep models and now we're gonna make
this sudden switch to biology
biologically inspired models it's going
to be an evolution little
characteristics are going to be
incorporated here and there
eventually we'll see that the mismatch
with the hardware is starting to grow
that will start to evolve and this will
be a ongoing process they go over the
next I don't know if I'll live long
enough to see all of this but certainly
over the next 10 to 20 years that I
should say that all of these comments
about neural computing assume that it's
going to learn at every level a lot of
the the problems with doing computing in
the analog domain are come from the fact
that people aren't doing adaptive
learning in the process floating gates
are a very nice local way to do adaptive
learning perfectly compatible with the
technologies we have have been for years
and they're limited in the number of
steps by how many electrons it takes to
represent full-scale that's
at the smallest things we can make that
still gives us more than five bits so
Steve can I jump in here for one jump in
one comment so I think I have some
pretty strong views on this so I
apologize if this is controversial so
good biology biology and neurofen
computing is here and digital deep
learning is here and heading you know
that yeah there may be a little bit of
move at lower precision but they're
really fundamentally different and I
think it's important to understand that
just list off a few things in a digital
deep network you evaluate all the
neurons quote unquote neurons all the
time for every invocation biology it's
incredibly sparse at a fundamental level
the you know the as Carver said that the
timing in the spiking model that sparse
model encodes an enormous amount of
information and the problem is that we
haven't been able to figure out how to
build you know sparse spike timing based
systems that can learn as well as our
digital deep learning systems and and
learn without tons of label data and
digital deep networks are incredibly
brittle you can change a few pixels and
trick them into saying anything you want
them to so I mean they're just they're
just totally different and so we have to
cut over or find something else for the
reasons we discussed but we have to
understand the model and what we can
extract from the biological model what
are the primitives we can extract it
gives us that emergent behavior you know
unsupervised training and incredibly
sparse activation and an incredibly
dense encoding and so you know before we
understand the model I think it's
premature to be building things I mean
we might build some things that work
okay but we're not gonna and we should
experiment but we're not gonna get to
that thing we really want to need and so
I really love for people to speculate on
what might line up in terms of you know
manufacturing systems but we gotta
understand them all
so it dug this a lesson from the project
brain wave that you're you're running or
is this something you know or how does
that our our brain wave system is
Microsoft's in-house deep learning
system which uses the digital hit every
neuron every time you know it's it's the
state stochastic gradient descent train
models you know we are we are using very
low precision datatype to squeeze
efficiency out of the silicon and that's
how we're able to get such high
performance in the cloud and and for our
partners on the the edge but you're
hearing some frustration because I see
that six orders of magnitude sitting
there and I want it and I don't know how
to get there right so we're just we're
just squeezing the precision down but
you know we're gonna get another 4x or
5x if we're lucky 10x and then maybe 20
with better silicon and maybe 30 with
exploiting some sparse indications you
know in my wildest dreams were might get
100x but there's still another thousand
ten thousand hundred thousand sitting on
the table yes right and and so we have
to understand that model we are doing
some investments down that path but it's
it's hard and it'll take a long time so
this is why you guys are here to tell us
how to get that sorry well I will say
that since I was in the game there's
been a vast amount more learned about
real nervous systems using the
technology that has developed in the
meantime and also there's a lot more
teamwork between biologists and
engineers that are trying to figure out
what the nervous system model is sure as
you're saying and that didn't exist we I
think were one of the early people to to
build an organization in the university
that was was whose absolute focus was to
try to get those people to to think
together and
figure stuff out and that's that's
become a worldwide trend now and I think
that's going to pay off I can't give you
any kind of scheduled I'm sorry that's
okay well if we have a few orders of
magnitude in between the biological
systems and our electronic systems
should we be talking about biologic
computers and stop making them out of
seem awesome and those guys are talking
in the other room by the way but Dan I
think you really do like this one I've
actually had a lot of people ask me
about that when I was at DARPA I had a
fairly regular basis people coming in
wanting to do that I can't imagine
working with live tissue in any
productive way you know it brings brings
new meaning to the phrase my computer
died natural trying to maintain living
tissue as part of an ongoing product
that's out in the field so no I am I'm
very pessimistic about that to be honest
I think all of them
everything is required to keep that that
biology alive and I think we can do
pretty well with with the silicon
technology absolutely set of life sites
in the cloud we have dead sites yeah I
don't want that this is the first time
I've heard a dyed-in-the-wool digital
guy say they want what's in the nervous
system first time in all these years I
read your book
so I'm gonna I'm gonna pull one from our
from our audience and just kind of throw
this up here
so what what offers the greater
potential newer morphic hardware or
morphic software where should we be
looking for this answer well I think
it's gonna start off as neuromorphic
software we're gonna explore the
algorithms hopefully we can even manage
to find applications where they work and
we can get by with the digital hardware
that we have today or minor variations
on or using for example FPGAs which
extraordinary powerful I think over time
as we gain experience with the
algorithms and and applications then
we'll start migrating to fully customize
hardware but it's not gonna happen
I think the hardware has to hold off a
little bit longer I was at a conference
in Europe
that we that my staff and I ran and we
had a number of people from the human
brain project involved and one of them
Steve Furber who some of you may know it
was a it was one of the founders of arm
as now in Manchester University Steve
got up and said you Americans you go off
and build silicon before you understand
the algorithms you know there's there
was an omen of truth to that I think
that's not to say you don't continue to
experiment I think that there's a lot of
synergy possible but I think in the
beginning it's going to be software yeah
thank you and we use Microsoft platforms
to do that well happily so one of the
things you find in the nervous system is
is counterintuitive to computer people
because computation has always been
based on sort of logical conscious
reasoning and in a an animal the amount
of the brain that's devoted to conscious
logical reasoning is a miniscule
fraction I use the neural iceberg
metaphor for this that there's a little
tiny tip that you're aware of your brain
doing and then there's all this stuff
down and it's vastly more dense closer
to the sensors and closer to the
actuators you heard yesterday that deny
AI session that you know just just
manipulation like your hand does
effortlessly there's a really hard
problem in a robotics well what seems
effortless eyes there's a huge amount of
actual processing in the muscles and in
the organs and in the nerves that are
connecting them and that's why we can
effortlessly pick up something pick up
an egg without breaking it and set it
down carefully well how smashing it that
sort of thing that's not something that
we build systems do but if you think
about the challenges facing autonomous
systems a lot of them are about that and
so maybe one of the killer apps is gonna
be making hands that can actually pick
this stuff up at Amazon Fulfillment
Center like we heard it's all done by
people now because you can't get
machines to do it without breaking this
stuff neural models and I'd like to talk
a little bit about some some others we
talked a synchronous logic adiabatic you
know should be good for a low-power are
the probabilistic kinds of stochastic
computation are these alternatives are
they complimentary do you say we should
rebuilding all of it at the same time if
we have to prioritize how to repair
archives
I think I alluded to this in my opening
remarks the the scale and the generality
are super important just because of the
the amount of investment that's been
made and CMOS and silicon it's so high
that that band gap that energy gap now
is it's a very tough wall to climb
because you're competing with something
it's sad hundreds of billions or maybe
even in the t's I don't know what the
aggregate investment has been over the
last 50 years or 30 years from CMOS but
it's been very high so we should be
investing in all of these things and
some of them can a synchronous logic can
can play a role in existing systems
adiabatic super interesting but but the
bar is high so it's gonna be really
tough for a lot of these things to get
over the hump and so that that killer
apt question becomes important you
believe that that neural network model
has achieved that or not I think the
neural network model in Hardin for
training hardware has achieved it for
large-scale commercial requirements you
just internet search is a killer app we
use deep learn now in Kent there's stuff
we can't do without it and and
increasingly inference so I I think I
think it's there I don't know where you
know all of the analyst reports have
this exponential growth and demand for
those cycles over the next six years I
don't know if we're gonna see that curve
that that's TDD it's a super important
question if we see the curve I hope we
can meet the curve yeah yeah that might
be the defining
are we even talking about the right
problem I mean we talk about computation
but it's communication right you know we
all know that what is computation it's a
lot of data we went in a tiny little off
we how do we look for altering the
foundation of our digital communication
where where is the where is this leap
forward there let me take a short answer
then then defer I we've seen enormous
progress in networks at the large scale
in the last five years I mean we've gone
down from a millisecond to a microsecond
regime and there's still a ways to go
until you get to true speed of light
Layton sees on ship the reason I was
pushing this structural conduct
computing concept we've been pushing is
exactly for that so that you you lay out
things so that locality is preserved but
you need different compute substrates
underneath it to support that so yeah
yes you need locality it's incredibly
important i I think but I think we know
how to get a lot of locality of the
macro scale and the micro scale not not
perfectly but there are things we can do
there but the general compute models on
these new paradigms there's a much
harder problem I don't think we know how
to solve yet I'm gonna write that one
down it's not to say we don't want lots
more work there because we're leveraging
the work that's been done I I would
predict that as the deep learning stuff
gets better and more understood there's
now impetus to really understand what's
going on there and as we do will we will
begin to understand things like I
mentioned locality and representation
space right we'll begin to understand
something about that yeah and you can
imagine a system where you know you
start with a retina because that's a way
of aggregating data and
moving it up a representation level and
then aggregating that and moving that up
and we've been doing that for like 30
years now and they're now quite good
ones out there that do the first few
levels that's a good place to start
because you can use it to look at real
things in real time and it doesn't have
to be in a better not be in the data
center right because you're gonna be
driving around with it so you can
imagine then that that that can start
rippling up into your cortex that's what
it does in the real system and the
paradigm doesn't change immediately all
at once it's it's a shift but it's
there's a lot of similarity so there's
something I mean just the fact that
there's a cortex there's something kind
of general purpose absolutely yeah so as
you learn more about that then you know
they're parts of it that you can build
into hardware like the retinas right now
that are took 20 years for the people in
system to get a real workable real-time
event-driven you know right now yeah
with reasonable level features coming up
but it exists now so now that's a place
that people could start doing the next
level of stuff but that in improved
representation understanding it I think
will move us away from stochastic
gradient descent which is what all of
the deep learning now is based on maybe
we'll reconcile the two but it's a panel
we should take strong positions so I'm
guessing there that we won't I'm going
to we're right about that time but I
won't want to ask the question
that we convened the panel for what
should DARPA do in this in this space
where is the where's that lever and I'm
gonna I'm gonna ask Dan to start since
you have DARPA experience so what you
know recent darpur expert
whatcha DARPA do yeah I wanted to axe it
as one last comment I wanted to make
gopanna previous but it leads into what
I would say anyhow and that is that I
think sometimes people forget that
information theory can play a really
important role because you're trading a
lot of computation and communication and
I think the cortex has found an optimal
trade-off
and that's another reason why we should
learn from them so the whole notion of
computation of communication I think is
very important but it's not something
that's easy to put your arms around as
far as what DARPA should do I think that
there should be more exploration in
these kind of models that we've talked
about I think DARPA is in a privileged
position of having great applications
that they can work on and a lot of great
groups at the disposal that can start to
look at some of these some of these
problems so I think it's kind of what I
said at the end of my statement is that
there's a huge number of models out
there most of them are probably useless
but we don't know that and we've got to
do some preliminary exploration of that
space and I don't think there's been
enough and that was the other point I
made about conventional computing being
freed from Moore's Law is now we're much
more highly motivated to look at these
more these alternative models and these
more unusual implementation so that
would be so bill that would be my
suggestion well I think we have a
consensus we need to figure out what we
can pull out of biology as computational
models without having to eat the whole
thing right I don't want to I don't want
to build live tissue with thousands of
connections and try to deploy that so
that anything there would be super super
helpful but I think some of the work and
new architectures were doing you know
was supported by DARPA and polymorphous
computing and other programs so
continuing to focus on what are new new
classes of computing that have some
generality would be it's still really
thats still a very very important
problem and I think the third is is all
that all the other computing components
continue to push on radical jumps there
in memory networking storage security
you know the there's just so many
candidates and it's going to take
investment to get them to production
scale
so those would be my three buckets my
three asks as well well I think one of
the questions is you know for example
their solution talked about using
machine learning as part of his tool
development and I think that there's a
lot of these models that may actually be
usable and a lot of the software but the
people that developed the software are
not gonna be looking at them obviously
if they have deadlines to have you but I
think that's another direction that we
haven't really looked closely at is what
are all the pieces of functionality that
are meaningful to these models and I
think there's a lot more than we
realized to be on well I want to thank
you the panel for going connector
innings pretty short and and and you the
the audience were also staying for the
little little extra time to explore the
topic I obviously did not get to all the
questions we still have panelist here
and we'll you're welcome to come up and
talk to us and we're but we're gonna
bring this session to a close
thank you all yeah thank you
[Applause]