- Hello everyone, welcome to CS231. - 大家好，欢迎来到CS231。
I'm Song Han. Today I'm going to give a guest lecture 我是宋寒。今天我要做客座讲座
on the efficient methods and hardware for deep learning. 关于深度学习的有效方法和硬件。
So I'm a fifth year PhD candidate here at Stanford, 所以我是斯坦福大学的五年级博士候选人，
advised by Professor Bill Dally. 由Bill Dally教授建议。
So, in this course we have seen a lot of convolution neural 所以，在这个过程中我们已经看到了很多卷积神经
networks, recurrent neural networks, or even 网络，递归神经网络，甚至
since last time, the reinforcement learning. 从上次开始，强化学习。
They are spanning a lot of applications. 它们涵盖了很多应用程序。
For example, the self-=driving car, machine translation, 例如，自驾车，机器翻译，
AlphaGo and Smart Robots. AlphaGo和智能机器人。
And it's changing our lives, but there is a recent 它正在改变我们的生活，但最近有一个
trend that in order to achieve such high accuracy, 趋势是为了达到如此高的准确度，
the models are getting larger and larger. 模型越来越大。
For example for ImageNet recognition, the winner from 例如，对于ImageNet识别，获胜者来自
2012 to 2015, the model size increased by 16X. 2012年至2015年，车型规模增加了16倍。
And just in one year, for Baidu's deep speech 就在一年之内，为了百度的深刻演讲
just in one year, the training operations, the number 仅仅一年时间，培训操作的数量
of training operations increased by 10X. 培训业务增加了10倍。
So such large model creates lots of problems, 所以这么大的模型会产生很多问题，
for example the model size becomes larger and larger 例如，模型尺寸变得越来越大
so it's difficult for them to be deployed either 所以他们也很难被部署
on those for example, on the mobile phones. 例如，在手机上。
If the item is larger than 100 megabytes, you 如果项目大于100兆字节，那么你
cannot download until you connect to Wi-Fi. 在连接到Wi-Fi之前无法下载。
So those product managers and for example Baidu, 那些产品经理和百度一样，
Facebook, they are very sensitive to the size of the binary Facebook，他们对二进制文件的大小非常敏感
size of their model. 他们的模型大小。
And also for example, the self-driving car, you can only 而且，例如，自动驾驶汽车，你只能
do those on over-the-air update for the model 做那些模型的无线更新
if the model is too large, it's also difficult. 如果模型太大，那也很困难。
And the second challenge for those large models is 这些大型车型的第二个挑战是
that the training speed is extremely slow. 训练速度极慢。
For example, the ResNet152, which is only a few, less 例如，ResNet152只有少数几个
than 1% actually, more accurate than ResNet101. 实际上超过1％，比ResNet101更准确。
Takes 1.5 weeks to train on four Maxwell 需要1.5周的时间来训练四个马克斯韦尔
M40 GPUs for example. 例如M40 GPU。
Which greatly limits either we are doing homework 这极大地限制了我们做作业
or if the researcher's designing new models is 或者如果研究人员正在设计新模型
getting pretty slow. 变得很慢。
And the third challenge for those bulky model is 对于那些庞大的模型来说，第三个挑战是
the energy efficiency. 能源效率。
For example, the AlphaGo beating Lee Sedol last year, 例如，去年AlphaGo击败Lee Sedol，
took 2000 CPUs and 300 GPUs, which cost $3,000 需要2000个CPU和300个GPU，成本为3,000美元
just to pay for the electric bill, which is insane. 只是为了支付电费，这是疯了。
So either on those embedded devices, those models 所以在那些嵌入式设备上，那些模型
are draining your battery power for on data-center 正在耗尽数据中心的电池电量
increases the total cost of ownership of maintaining 增加维护的总拥有成本
a large data-center. 一个大型数据中心。
For example, Google in their blog, they mentioned 例如，Google在他们的博客中提到过
if all the users using the Google Voice Search for 如果所有用户都使用Google语音搜索
just three minutes, they have to double their data-center. 仅仅三分钟，他们就必须将数据中心翻倍。
So that's a large cost. 所以这是一个很大的成本。
So reducing such cost is very important. 因此降低这样的成本非常重要。
And let's see where is actually the energy consumed. 让我们看看实际消耗的能源在哪里。
The large model means lots of memory access. 大型模型意味着大量的内存访问。
You have to access, load those models from the memory 您必须访问，从内存中加载这些模型
means more energy. 意味着更多的能量
If you look at how much energy is consumed by loading 如果你看一下装载能耗多少
the memory versus how much is consumed by multiplications 记忆与乘法消耗多少
and add those arithmetic operations, the memory access 并添加那些算术运算，内存访问
is more than two or three orders of magnitude, 是超过两个或三个数量级，
more energy consuming than those arithmetic operations. 比那些算术运算更耗能。
So how to make deep learning more efficient. 那么如何使深度学习更有效率。
So we have to improve energy efficiency by this 因此，我们必须提高能源效率
Algorithm and Hardware Co-Design. 算法与硬件协同设计。
So this is the previous way, which is our hardware. 所以这是以前的方式，这是我们的硬件。
For example, we have some benchmarks say Spec 2006 例如，我们有一些基准测试说Spec 2006
and then run those benchmarks and tune your CPU 然后运行这些基准测试并调整CPU
architectures for those benchmarks. 这些基准的架构。
Now what we should do is to open up the box to see 现在我们应该做的是打开盒子看看
what can we do from algorithm side first and see what 我们可以先从算法方面做些什么，然后看看是什么
is the optimum question mark processing unit. 是最佳的问号处理单元。
That breaks the boundary between the algorithm 这打破了算法之间的界限
hardware to improve the overall efficiency. 硬件提高整体效率。
So today's talk, I'm going to have the following agenda. 所以今天的谈话，我将有以下议程。
We are going to cover four aspects: The algorithm hardware 我们将涉及四个方面：算法硬件
and inference and training. 和推理和培训。
So they form a small two by two matrix, so includes the 所以它们形成一个小的二乘二矩阵，所以包括
algorithm for efficient inference, 有效推理的算法，
hardware for efficient inference 用于有效推理的硬件
and the algorithm for efficient training, 以及高效训练的算法，
and lastly, the hardware for efficient training. 最后，用于高效培训的硬件。
For example, I'm going to cover the TPU, I'm 例如，我要报道TPU，我
going to cover the Volta. 要报道沃尔特。
But before I cover those things, let's have three 但在我报道这些事情之前，让我们有三个
slides for Hardware 101. 硬件101的幻灯片。
A brief introduction of the families of hardware 硬件系列简介
in such a tree. 在这样的树上。
So in general, we can have roughly two branches. 所以一般来说，我们可以有大约两个分支。
One is general purpose hardware. 一个是通用硬件。
It can do any applications versus the specialized 它可以做专业的任何应用程序
hardware, which is tuned for a specific kind of 硬件，针对特定类型进行了调整
applications, a domain of applications. 应用程序，应用程序域。
So the general purpose hardware includes, the CPU 所以通用硬件包括CPU
or the GPU, and their difference is that CPU is 或GPU，它们的区别在于CPU
latency oriented, single threaded. 面向延迟，单线程。
It's like a big elephant. 它就像一头大象。
While the GPU is throughput oriented. 虽然GPU是面向吞吐量的。
It has many small though weak threads, but there 它有许多小而弱的线程，但在那里
are thousands of such small weak cores. 成千上万的这种小弱核心。
Like a group of small ants, where there are so many ants. 就像一群小蚂蚁，那里有很多蚂蚁。
And specialized hardware, roughly there are FPGAs and ASICs. 而专用硬件，大致有FPGA和ASIC。
So FPGA stand for Field Programmable Gate Array. 因此FPGA代表现场可编程门阵列。
So it is programmable, hardware programmable so its 所以它是可编程的，硬件可编程的，所以它
logic can be changed. 逻辑可以改变。
So it's cheaper for you to try new ideas and do prototype, 所以你尝试新的想法并做原型会更便宜
but it's less efficient. 但它的效率较低。
It's in the middle between the general purpose and 它位于通用和中间之间
pure ASIC. 纯ASIC。
So ASIC stands for Application Specific Integrated Circuit. 因此，ASIC代表专用集成电路。
It has a fixed logic, just designed 它有一个固定的逻辑，只是设计
for a certain application. 对于某种应用。
For example deep learning. 例如深度学习。
And Google's TPU is a kind of ASIC and the neural networks 谷歌的TPU是一种ASIC和神经网络
we train on, the earlier GPUs is here. 我们训练，早期的GPU就在这里。
And another slide for Hardware 101 is the number 硬件101的另一张幻灯片是数字
representations. 表示。
So in this slide, I'm going to convey you the idea that 所以在这张幻灯片中，我将向您传达这样的想法
all the numbers in computer are not represented 计算机中的所有数字都没有表示
by a real number. 一个实数。
It's not a real number, but they are actually discrete. 这不是一个真实的数字，但它们实际上是离散的。
Even for those floating point with your 32 Bit. 即使是那些带有32位的浮点数。
Floating point numbers, their resolution is not perfect. 浮点数，它们的分辨率并不完美。
It's not continuous, but it's discrete. 它不是连续的，但它是离散的。
So for example FP32, meaning using a 32 bit to represent 例如FP32，意味着使用32位来表示
a floating point number. 浮点数。
So there are three components in the representation. 所以表示中有三个组成部分。
The sign bit, the exponent bit, the mantissa, 符号位，指数位，尾数，
and the number it represents is shown by minus 1 to the S 它所代表的数字由S表示减1
times 1.M times 2 to the exponent. 次1.M乘以2指数。
So similar there is FP16, using a 16 bit to represent 类似的是FP16，使用16位来表示
a floating point number. 浮点数。
In particular, I'm going to introduce Int8, where 特别是，我将介绍Int8，其中
the core TPU use, using an integer to represent a fixed 核心TPU使用，使用整数来表示固定
point number. 点数。
So we have a certain number of bits for the integer. 所以我们有一定数量的整数位。
Followed by a radix point, if we put different layers. 如果我们放置不同的图层，则后跟一个小数点。
And lastly, the fractional bits. 最后，小数位。
So why do we prefer those eight bit, or 16 bit 那么为什么我们更喜欢那些8位或16位
rather than those traditional like the 而不是像传统的那样
32 bit floating point. 32位浮点。
That's the cost. 这是成本。
So, I generated the figure from 45 nanometer technology 所以，我用45纳米技术生成了这个数字
about the energy cost versus the area cost for different 关于不同的能源成本与面积成本
operations. 操作。
In particular, let's see here, go you from 32 bit to 特别是，让我们看看这里，从32位到你
16 bit, we have about four times reduction in energy 16位，我们的能量减少了大约四倍
and also about four times reduction in the area. 并且该面积减少了约四倍。
Area means money. 面积意味着金钱。
Every millimeter square takes money to take out a chip 每平方米都需要花钱才能取出芯片
So it's very beneficial for hardware design to go from 因此，硬件设计非常有用
32 bit to 16 bit. 32位到16位。
That's why you hear NVIDIA from Pascal Architecture, 这就是你从Pascal Architecture听到NVIDIA的原因，
they said they're starting to support FP16. 他们说他们开始支持FP16。
That's the reason why it's so beneficial. 这就是它如此有益的原因。
For example, previous battery level could last four hours, 例如，之前的电池电量可能持续四个小时，
now it becomes 16 hours. 现在变成了16个小时。
That's what it means to reduce 这就是减少意味着什么
the energy cost by four times. 能源成本降低了四倍。
But here still, there's a problem of large energy costs 但在这里，还存在能源成本高的问题
for reading the memory. 用于阅读记忆。
And let's see how can we deal with this memory reference 让我们看看我们如何处理这个内存引用
so expensive, how do we deal with this problem better? 如此昂贵，我们如何更好地处理这个问题？
So let's switch gear and come to our topic directly. 那么让我们切换装备直接来到我们的主题。
So let's first introduce algorithm for efficient inference. 因此，让我们首先介绍有效推理的算法。
So I'm going to cover six topics, this is a really long slide. 所以我将介绍六个主题，这是一个很长的幻灯片。
So I'm going to relatively fast. 所以我要相对快。
So the first idea I'm going to talk about is pruning. 所以我要谈的第一个想法就是修剪。
Pruning the neural networks. 修剪神经网络。
For example, this is original neural network. 例如，这是原始神经网络。
So what I'm trying to do is, can we remove some of the 所以我要做的是，我们可以删除一些
weight and still have the same accuracy? 重量仍然具有相同的准确度？
It's like pruning a tree, get rid 这就像修剪一棵树，摆脱它
of those redundant connections. 那些冗余连接。
This is first proposed by Professor Yann LeCun back in 1989, 这是由Yann LeCun教授于1989年首次提出的，
and I revisited this problem, 26 years later, on those 26年后，我重新审视了这个问题
modern deep neural nets to see how it works. 现代深层神经网络，看看它是如何工作的。
So not all parameters are useful actually. 所以并非所有参数实际上都有用。
For example, in this case, if you want to fit a single line, 例如，在这种情况下，如果您想要适合单行，
but you're using a quadratic term, apparently the 但你使用的是二次项，显然是
0.01 is a redundant parameter. 0.01是冗余参数。
So I'm going to train the connectivity first and then 所以我首先要训练连接
prune some of the connections. 修剪一些连接。
And then train the remaining weights, 然后训练剩余的重量，
and through this process, it regulates. 通过这个过程，它规范。
And as a result, I can reduce the number of connections, 结果，我可以减少连接数量，
and annex that from 16 million parameters to only 和附件，从1600万参数到只
six million parameters, which is 10 times less 六百万个参数，少了10倍
the computation. 计算。
So this is the accuracy. 所以这就是准确性。
So the x-axis is how much parameters to prune away 所以x轴是要修剪多少参数
and the y-axis is the accuracy you have. 而y轴是你的准确度。
So we want to have less parameters, but we also 所以我们希望减少参数，但我们也是
want to have the same accuracy as before. 希望获得与以前相同的准确度。
We don't want to sacrifice accuracy, 我们不想牺牲准确性，
For example at 80%, we locked zero away left 80% 例如，80％，我们锁定零离开80％
of the parameters, but accuracy jumped by 4%. 参数，但准确率上升了4％。
That's intolerable. 那是无法忍受的。
But the good thing is that if we retrain the remaining 但好的是，如果我们重新训练剩余的
weights, the accuracy can fully recover here. 重量，准确度可以在这里完全恢复。
And if we do this process iteratively 如果我们迭代地执行此过程
by pruning and retraining, pruning and retraining, 通过修剪和再培训，修剪和再培训，
we can fully recover the accuracy not until we are 我们可以完全恢复准确性，直到我们
prune away 90% of the parameters. 修剪90％的参数。
So if you go back to home and try it on your Ipad 所以如果你回到家里并在你的Ipad上试试
or notebook, just zero away 50% of the parameters say 或笔记本，只有零距离50％的参数说
you went on your homework, you will astonishingly find 你继续做作业，你会惊奇地发现
that accuracy actually doesn't hurt. 准确性实际上并没有伤害。
So we just mentioned convolution neural nets, 所以我们刚刚提到了卷积神经网络，
how about RNNs and LSTMs, so I tried with this neural talk. 那个RNN和LSTM怎么样，所以我尝试了这个神经话题。
Again, pruning away 90% of the rates doesn't hurt the 再次，修剪掉90％的费率不会伤害到
blue score. 蓝色得分。
And here are some visualizations. 这里有一些可视化。
For example, the original picture, the neural talk says 例如，原始图片，神经谈话说
a basketball player in a white uniform is playing 穿着白色制服的篮球运动员正在比赛
with a ball. 带球
Versus pruning away 90% it says, a basketball player 与一名篮球运动员说，比如90％的修剪
in a white uniform is playing with a basketball. 穿着白色制服正在玩篮球。
And on and so on. 依此类推。
But if you're too aggressive, say you prune away 但如果你过于咄咄逼人，就说你要修剪掉
95% of the weights, the network is going to get drunk. 95％的重量，网络将喝醉。
It says, a man in a red shirt and white and black shirt 它说，穿着红色衬衫和白色和黑色衬衫的男人
is running through a field. 正在穿过田野。
So there's really a limit, a threshold, you have to 所以你必须要有一个限制，一个门槛
take care of during the pruning. 在修剪期间照顾。
So interestingly, after I did the work, did some 有趣的是，在我完成工作后，做了一些
resource and research and find actually the same 资源和研究，发现实际上是一样的
pruning procedure actually happens to human brain 修剪程序实际上发生在人脑中
as well. 同样。
So when we were born, there are about 50 trillion synapses 所以当我们出生时，大约有50万亿个突触
in the brain. 在大脑中。
And at one year old, this number surged into 1,000 trillion. 在一岁时，这个数字飙升至1,000万亿。
And as we become adolescent, it becomes smaller actually, 当我们成为青少年时，它实际上变小了，
500 trillion in the end, according to the study by Nature. 根据Nature的研究，最终将达到500万亿。
So this is very interesting. 所以这很有趣。
And also, the pruning changed the weight distribution 而且，修剪改变了重量分布
because we are removing those small connections 因为我们正在删除那些小连接
and after we retrain them, that's why it becomes soft 在我们重新培训之后，这就是它变软的原因
in the end. 到底。
Yeah, question. 是的，问题。
- [Student] Are you trying to mean that it terms - [学生]你是想表达它的条款
of your mixed weights during the training will be 在训练期间你的混合重量将是
just set at zero and just start from scratch? 只是设置为零，只是从头开始？
And these start from the things that are at zero. 而这些都是从零开始的事情开始的。
- Yeah. So the question is, how do we deal with those - 是的所以问题是，我们如何处理这些问题
zero connections? 零连接？
So we force them to be zero in all the other iterations. 所以我们强迫它们在所有其他迭代中都为零。
Question? 题？
- [Student] How do you pick which rates to drop? - [学生]你如何选择降低的费率？
- Yeah so very simple. Small weights, drop it, sort it. - 是的，非常简单。小重量，下降，排序。
If it's small, just-- 如果它很小，只需 -
- [Student] Any threshold that I decide? - [学生]我决定的任何门槛？
- Exactly, yeah. - 没错，是的。
So the next idea, weight sharing. 所以下一个想法，重量分享。
So now we have, remember our end goal is to remove 所以现在我们有了，记住我们的最终目标是删除
connections so that we can have less memory footprint 连接，以便我们可以减少内存占用
so that we can have more energy efficient deployment. 这样我们就可以进行更节能的部署。
Now we have less number of parameters by pruning. 现在我们通过修剪来减少参数数量。
We want to have less number of bits per parameter 我们希望每个参数的位数更少
so they're multiplied together they get a small model. 所以他们成倍增加他们得到一个小模型。
So the idea is like this. 所以这个想法是这样的。
Not all numbers, not all the weights 并非所有数字，不是所有权重
has to be the exact number. 必须是确切的数字。
For example, 2.09, 2.12 or all these four weights, you 例如，2.09,2.12或所有这四个权重，你
just put them using 2.0 to represent them. 只是把它们用2.0来代表它们。
That's enough. 这就够了。
Otherwise too accurate number is just leads to overfitting. 否则太准确的数字只会导致过度拟合。
So the idea is I can cluster the weights if they 所以我的想法是，如果它们可以聚集权重
are similar, just using a centroid to represent 类似，只用一个质心来表示
the number instead of using the full precision weight. 数字而不是使用全精度重量。
So that every time I do the inference, I just do inference 所以每次我做推理时，我都会做推理
on this single number. 在这个单号上。
For example, this is a four by four weight matrix 例如，这是一个四乘四的权重矩阵
in a certain layer. 在某一层。
And what I'm going to do is do k-means clustering by having 而我要做的是通过拥有k-means聚类
the similar weight sharing the same centroid. 类似的重量共享相同的质心。
For example, 2.09, 2.12, I store index of 例如，2.09,2.12，我存储索引
three pointing to here. 三指着这里。
So that, the good thing is we need to only store the 所以，好的是我们只需要存储
two bit index rather than the 32 bit, floating point number. 两位索引而不是32位浮点数。
That's 16 times saving. 这节省了16倍。
And how do we train such neural network? 我们如何训练这样的神经网络？
They are binded together, so after we get the gradient, 他们绑在一起，所以我们得到梯度后，
we color them in the same pattern as the weight 我们将它们以与重量相同的模式着色
and then we do a group by operation by having all 然后我们通过操作完成一个小组
the in that weights with the same index grouped together. 具有相同索引的权重组合在一起。
And then we do a reduction by summing them up. 然后我们通过总结来减少它们。
And then multiplied by the learning rate 然后乘以学习率
subtracted from the original centroid. 从原始质心中减去。
That's one iteration of the SGD for such weight 对于这样的重量，这是SGD的一次迭代
shared neural network. 共享神经网络。
So remember previously, after pruning this is 所以记住以前，修剪后这是
what the weight distribution like and after 什么重量分布喜欢和之后
weight sharing, they become discrete. 重量分配，它们变得离散。
There are only 16 different values here, meaning 这里只有16个不同的值，意思是
we can use four bits to represent each number. 我们可以使用四位来表示每个数字。
And by training on such weight shared neural network, 并通过训练这样的重量共享神经网络，
training on such extremely shared neural network, 这种极度共享的神经网络的培训，
these weights can adjust. 这些重量可以调整。
It is the subtle changes that compensated for the 这是微妙的变化，补偿了
loss of accuracy. 失去准确性。
So let's see, this is the number of bits we give it, 那么让我们看看，这是我们给它的位数，
this is the accuracy for convolution layers. 这是卷积层的准确度。
Not until four bits, does the accuracy begin to drop 直到四位，精度才开始下降
and for those fully connected layers, very astonishingly, 对于那些完全连接的层，非常令人惊讶的是，
it's not until two bits, only four number, does the 它直到两位，只有四位数
accuracy begins to drop. 准确性开始下降。
And this result is per layer. 这个结果是每层。
So we have covered two methods, pruning and weight sharing. 所以我们已经介绍了两种方法，修剪和分配重量。
What if we combine these two methods together. 如果我们将这两种方法结合在一起怎么办
Do they work well? 他们运作得好吗？
So by combining those methods, this is the compression 因此，通过组合这些方法，这就是压缩
ratio with the smaller on the left. 与左边较小的比例。
And this is the accuracy. 这就是准确性。
We can combine it together and make the model 我们可以将它们组合在一起并制作模型
about 3% of its original size without hurting the 大约3％的原始尺寸没有伤害
accuracy at all. 准确性。
Compared with the each working individual data by 与每个工作的个人数据相比较
10%, accuracy begins to drop. 10％，准确率开始下降。
And compared with the cheap SVD method, 与廉价的SVD方法相比，
this has a better compression ratio. 这具有更好的压缩比。
And final idea is we can apply the Huffman Coding 最后的想法是我们可以应用霍夫曼编码
to use more number of bits for those infrequent numbers, 为那些不频繁的数字使用更多的位数，
infrequently appearing weights and less number of bits 不经常出现的重量和较少的位数
for those more frequently appearing weights. 对于那些更频繁出现的重量。
So by combining these three methods, pruning, weight 所以通过结合这三种方法，修剪，重量
sharing, and also Huffman Coding, we can compress the 分享，还有霍夫曼编码，我们可以压缩
neural networks, state-of-the-art neural networks, 神经网络，最先进的神经网络，
ranging from 10x to 49x without hurting the 范围从10x到49x不会伤害到
prediction accuracy. 预测准确性。
Sometimes a little bit better. 有时好一点。
But maybe that is noise. 但也许这就是噪音。
So the next question is, these models are just pre-trained 所以接下来的问题是，这些模型只是经过预先训练
models by say Google, Microsoft. 谷歌，谷歌，微软。
Can we make a compact model, a pump compact model 我们可以制造紧凑型号，泵紧凑型号
to begin with? 开始？
Even before such compression? 甚至在压缩之前？
So SqueezeNet, you may have already worked with this 所以SqueezeNet，您可能已经使用过这个
neural network model in a homework. 在家庭作业中的神经网络模型。
So the idea is we are having a squeeze layer here to shield 所以这个想法是我们在这里有一个挤压层来屏蔽
at the three by three convolution with fewer number of 在三乘三的卷积与少数
channels. 通道。
So that's where squeeze comes from. 所以这就是挤压的来源。
And here we have two branches, rather than four branches 在这里，我们有两个分支，而不是四个分支
as in the inception model. 就像在初始模型中一样。
So as a result, the model is extremely compact. 因此，该模型非常紧凑。
It doesn't have any fully connected layers. 它没有任何完全连接的层。
Everything is fully convolutional. 一切都是完全卷积的。
The last layer is a global pooling. 最后一层是全局池。
So what if we apply deep compression algorithm 那么如果我们应用深度压缩算法呢？
on such already compact model will it be getting even 在这种已经很紧凑的型号上，它会变得更加均匀
smaller? 小吗？
So this is AlexNet after compression, this is SqueezeNet. 所以这是压缩后的AlexNet，这是SqueezeNet。
Even before compression, it's 50x smaller than AlexNet, 即使在压缩之前，它比AlexNet小50倍，
but has the same accuracy. 但具有相同的准确性。
After compression 510x smaller, but the same accuracy 压缩后510x较小，但精度相同
only less than half a megabyte. 只有不到半兆字节。
This means it's very easy to fit such a small model 这意味着很容易适应这么小的模型
on the cache, which is literally 在缓存上，这是字面上的
tens of megabyte SRAM. 几十兆字节的SRAM。
So what does it mean? 那是什么意思呢？
It's possible to achieve speed up. 有可能实现加速。
So this is the speedup, I measured if all these fully 所以这是加速，我完全测量了所有这些
connected layers only for now, on the CPU, GPU, and 现在连接的层，在CPU，GPU和
the mobile GPU, before pruning 移植GPU，修剪之前
and after pruning the weights, 修剪重量后，
and on average, I observed a 3x speedup in a CPU, 平均而言，我发现CPU的速度提高了3倍，
about 3X speedup on the GPU, 在GPU上大约加速3倍，
and roughly 5x speedup on the mobile GPU, which is a 在移动GPU上大约加速5倍，这是一个
TK1. TK1。
And so is the energy efficiency. 能源效率也是如此。
In an average improvement from 3x to 6x on a CPU, GPU, 在CPU，GPU上平均从3倍到6倍的改进，
and mobile GPU. 和移动GPU。
And these ideas are used in these companies. 这些想法在这些公司中使用。
Having talked about when pruning and when sharing, 谈到修剪和分享时，
which is a non-linear quantization method 这是一种非线性量化方法
and we're going to talk about quantization, which is, why 我们将谈论量化，这就是原因
do they use in the TPU design? 他们在TPU设计中使用它们吗？
All the TPU designs use at only eight bit for inference. 所有TPU设计仅使用8位进行推断。
And the way, how they can use that is because of the 顺便说一句，他们如何使用它是因为
quantization. 量化。
And let's see how does it work. 让我们看看它是如何工作的。
So quantization has this complicated figure, but 所以量化有这个复杂的数字，但是
the intuition is very simple. 直觉很简单。
You run the neural network and train it with the normal 你运行神经网络并用正常训练它
floating point numbers. 浮点数字。
And quantize the weight and activations by gather 并通过聚集量化重量和激活
the statistics for each layer. 每层的统计数据。
For example, what is the maximum number, minimum number, 例如，最大数量，最小数量是多少，
and how many bits are enough 多少位就足够了
to represent this dynamic range. 代表这个动态范围。
Then you use that number of bits for the integer part 然后使用该位数作为整数部分
and the rest of the eight bit or seven bit 和其余的8位或7位
for the other part of the 8 bit representation. 对于8位表示的另一部分。
And also we can fine tune in the floating point format. 而且我们也可以微调浮点格式。
Or we can also use feed forward with fixed point 或者我们也可以使用固定点前馈
and back propagation with update with the floating 和浮动的更新和反向传播
point number. 点数。
There are lots of different ideas to have better accuracy. 有很多不同的想法可以提高准确性。
And this is the result, for how many number of bits 这是多少位数的结果
versus what is the accuracy. 与什么是准确性。
For example, using a fixed, 8 bit, the accuracy for 例如，使用固定的8位精度
GoogleNet doesn't drop significantly. GoogleNet不会大幅下降。
And for VGG-16, it also remains pretty well for 而对于VGG-16来说，它仍然相当不错
the accuracy. 准确性。
While circling down to a six bit, the accuracy 在向下盘旋到六位时，准确性
begins to drop pretty dramatically. 开始急剧下降。
Next idea, low rank approximation. 下一个想法，低等级近似。
It turned out that for a convolution layer, 原来，对于卷积层，
you can break it into two convolution layers. 你可以把它分成两个卷积层。
One convolution here, followed by a one by one convolution. 这里有一个卷积，然后是一个一个卷积。
So that it's like you break a complicated problem 这就像你打破了一个复杂的问题
into two separate small problems. 分成两个小问题。
This is for convolution layer. 这是针对卷积层的。
As we can see, achieving about 我们可以看到，实现约
2x speedup, there's almost no loss of accuracy. 2倍加速，几乎没有精度损失。
And achieving a speedup of 5x, roughly a 6% 并实现5倍的加速，大约6％
loss of accuracy. 失去准确性。
And this also works for fully connected layers. 这也适用于完全连接的层。
The simplest idea is using the SVD to break it into 最简单的想法是使用SVD将其分解为
one matrix into two matrices. 一个矩阵分为两个矩阵。
And follow this idea, this paper proposes to use the 并且遵循这个想法，本文提出使用
Tensor Tree to break down one fully connected layer into Tensor Tree将一个完全连接的层分解为
a tree, lots of fully connected layers. 一棵树，很多完全连接的层。
That's why it's called a tree. 这就是它被称为树的原因。
So going even more crazy, can we use only 所以更疯狂，我们可以只使用
two weights or three weights to represent a neural network? 两个权重或三个权重来表示神经网络？
A ternary weight or a binary weight. 三元重量或二元重量。
We already seen this distribution before, after pruning. 在修剪之前我们已经看过这个发行版。
There's some positive weights and negative weights. 有一些正权重和负权重。
Can we just use three numbers, just use one, minus one, zero 我们可以只使用三个数字，只使用一个，减一，零
to represent the neural network. 代表神经网络。
This is our recent paper clear that we maintain 这是我们最近的文章，我们保持这一点
a full precision weight during training time, 训练期间的全精度重量，
but at inference time, we only keep the scaling factor 但在推理时，我们只保留比例因子
and the ternary weight. 和三元重量。
So during inference, we only need three weights. 所以在推理期间，我们只需要三个权重。
That's very efficient and making the model very small. 这非常有效，使模型非常小。
This is the proportion of the positive zero 这是正零的比例
and negative weights, they can change during the training. 和负重，他们可以在训练期间改变。
So is their absolute value. 他们的绝对价值也是如此。
And this is the visualization of kernels 这是内核的可视化
by this trained ternary quantization. 通过这种训练的三元量化。
We can see some of them are a corner detector like here. 我们可以看到其中一些像这里的角落探测器。
And also here. 而且在这里。
Some of them are maybe edge detector. 其中一些可能是边缘检测器。
For example, this filter some of them 例如，这个过滤器中的一些
are corner detector like here this filter. 角度探测器就像这里的过滤器一样。
Actually we don't need such fine grain resolution. 实际上我们不需要这么精细的颗粒分辨率。
Just three weights are enough. 只需三个砝码即可。
So this is the validation accuracy on ImageNet with AlexNet. 这就是使用AlexNet在ImageNet上的验证准确性。
So the threshline is the baseline accuracy 因此，阈值是基线准确度
with floating point 32. 浮点数32。
And the red line is our result. 而红线是我们的结果。
Pretty much the same accuracy converged compared with 与之相比，几乎相同的精度收敛
the full precision weights. 全精度重量。
Last idea, Winograd Transformation. 最后的想法，Winograd转型。
So this about how do we implement deep neural nets, 那么关于我们如何实现深度神经网络，
how do we implement the convolutions. 我们如何实施卷积。
So this is the conventional direct 所以这是传统的直接
convolution implementation method. 卷积实现方法。
The slide credited to Julien, a friend from Nvidia. 这张幻灯片归功于Nvidia的朋友Julien。
So originally, we just do the element wise 所以最初，我们只是明智地做元素
do a dot product for those nine elements in the filter 为过滤器中的九个元素做一个点积
and nine elements in the image and then sum it up. 和图像中的九个元素然后总结。
For example, for every output we need nine times C 例如，对于每个输出，我们需要C的九倍
number of multiplication and adds. 乘法和加法的次数。
Winograd Convolution is another method, equivalent method. Winograd卷积是另一种方法，等效方法。
It's not lost, it's an equivalent method proposed at 它没有丢失，它是一种等效的方法
first through this paper, Fast Algorithms 首先通过本文，快速算法
for Convolution Neural Networks. 用于卷积神经网络。
That instead of directly doing the convolution, move 而不是直接进行卷积，移动
it one by one, at first it transforms the input feature 它一个接一个，首先它转换输入功能
map to another feature map. 映射到另一个要素图。
Which contains only the weight, contains only 1, 0.5, 2 其中仅包含重量，仅包含1个，0.5个，2个
that can efficiently implement it with shift. 可以通过班次有效地实现它。
And also transform the filter into a four by four tensor. 并且还将滤波器转换为四乘四张量。
So what we are going to do here is sum over c and do an element-wise 所以我们在这里要做的是总结c并做一个元素
element-wise product. 元素产品。
So there are only 16 multiplications happening here. 所以这里只发生了16次乘法运算。
And then we do a inverse transform to get four outputs. 然后我们进行逆变换以获得四个输出。
So the transform and the inverse transform can be 所以变换和逆变换都可以
amortized and the multiplications, whether it can ignored. 摊销和乘法，是否可以忽略。
So in order to get four output, we need nine times channel 因此，为了获得四个输出，我们需要九倍的通道
times four, which is 36 times channel. 四次，这是36次频道。
Multiplications originally for the direct convolution 最初用于直接卷积的乘法
but now we need 16 times C of our output 但现在我们需要16倍的输出C.
So that is 2.25x less number of multiplications to 这样，乘法次数减少了2.25倍
perform the exact same multiplication. 执行完全相同的乘法。
And here is a speedup. 这是一个加速。
2.25x, so theoretically, 2.25x speedup and in real, 2.25x，理论上，2.25倍的加速和实际，
from cuDNN 5 they incorporated such 从cuDNN 5他们加入了这样的
Winograd Convolution algorithm. Winograd卷积算法。
This is on the VGG net I believe, the speedup is 这是在VGG网上我相信，加速是
roughly 1.7 to 2x speedup. 大约1.7到2倍的加速。
Pretty significant. 非常重要。
And after cuDNN 5, the cuDNN begins to use the 在cuDNN 5之后，cuDNN开始使用了
Winograd Convolution algorithm. Winograd卷积算法。
Okay, so far we have covered those efficient algorithms 好的，到目前为止，我们已经涵盖了那些有效的算法
for efficient inference. 有效推断。
We covered pruning, weight sharing, quantization, 我们介绍了修剪，重量分配，量化，
and also Winograd binary and ternary. 还有Winograd二进制和三元组。
So now let's see what is the optimal hardware for those 那么现在让我们看看那些最佳硬件是什么
efficient inference? 有效推理？
And what is a Google TPU? 什么是Google TPU？
So there are a wide range of domain specific 因此，有广泛的领域特定
architectures or ASICS for deep neural networks. 用于深度神经网络的体系结构或ASICS。
They have a common goal is to minimize the memory 他们的共同目标是尽量减少记忆
access to save power. 获得节省电力。
For example the Eyeriss from MIT by using the RS Dataflow 例如，使用RS Dataflow从麻省理工学院获得Eyeriss
to minimize the off chip direct access. 最小化片外直接访问。
And DaDiannao from China Academy of Science, 和来自中国科学院的DaDiannao，
buffered all the weights on chip DRAM instead of having 缓冲芯片DRAM上的所有权重而不是
to go to off-chip DRAM. 去片外DRAM。
So the TPU from Google is using eight bit integer 因此谷歌的TPU使用8位整数
to represent the numbers. 代表数字。
And at Stanford I proposed the EIE architecture 在斯坦福大学，我提出了EIE架构
that support those compressed and 支持那些压缩和
sparse deep neural network inference. 稀疏深度神经网络推理。
So this is what the TPU looks like. 这就是TPU的样子。
It's actually smartly, can be put into the disk drive 它实际上很聪明，可以放入磁盘驱动器
up to four cards per server. 每台服务器最多四张卡。
And this is the high-level architecture 这是高级架构
for the Google TPU. 对于Google TPU。
Don't be overwhelmed, it's actually, the kernel part 不要被淹没，实际上是内核的一部分
here, is this giant matrix multiplication unit. 这里是这个巨型矩阵乘法单元。
So it's a 256 by 256 matrix multiplication unit. 所以它是一个256乘256的矩阵乘法单元。
So in one single cycle, it can perform 64 kilo 所以在一个循环中，它可以执行64千克
those number of multiplication and accumulate operations. 那些乘法和累加运算的数量。
So running 700 Megahertz, the throughput is 92 因此运行700兆赫，吞吐量为92
Teraops per second 每秒Teraops
because it's actually integer operation. 因为它实际上是整数运算。
So we just about 25x as GPU and more than 100x at the CPU. 所以我们只有大约25倍的GPU和超过100倍的CPU。
And notice, TPU has a really large software-managed 请注意，TPU有一个非常大的软件管理
on-chip buffer. 片上缓冲器。
It is 24 megabytes. 它是24兆字节。
The cache for the CPU the L3 cache is already 三级缓存的CPU缓存已经存在
16 megabytes. 16兆字节。
This is 24 megabytes which is pretty large. 这是24兆字节，非常大。
And it's powered by two DDR3 DRAM channels. 它由两个DDR3 DRAM通道供电。
So this is a little weak because the bandwidth is 所以这有点弱，因为带宽是
only 30 gigabytes per second compared with the most 与大多数情况相比，每秒只有30千兆字节
recent GPU that HBM, 900 Gigabytes per second. 最近的GPU表示HBM，每秒900 GB。
The DDR4 is released in 2014, so that makes sense because DDR4于2014年发布，因此有意义
the design is a little during that day, used the DDR3. 在那一天设计有点使用DDR3。
But if you're using DDR4 or even high-bandwidth memory, 但如果您使用的是DDR4甚至是高带宽内存，
the performance can be even boosted. 性能甚至可以提升。
So this is a comparison about Google's TPU compared 所以这是对谷歌TPU的比较
with the CPU, GPU of this K80 GPU by the way, and the TPU. 顺便提一下，这个K80 GPU的CPU，GPU和TPU。
So the area is pretty much smaller, like half the size of a 所以这个区域要小得多，就像一个大小的一半
CPU and GPU and the power consumption is roughly 75 watts. CPU和GPU以及功耗大约为75瓦。
And see this number, the peak teraops per second 并看到这个数字，每秒峰值teraops
is much higher than the CPU and GPU is, about 90 远远高于CPU和GPU，大约90
teraops per second, which is pretty high. 每秒teraops，这是非常高的。
So here is a workload. 所以这是一个工作量。
Thanks to David sharing the slide. 感谢大卫分享幻灯片。
This is the workload at Google. 这是Google的工作量。
They did a benchmark on these TPUs. 他们对这些TPU进行了基准测试。
So it's a little interesting that convolution neural nets 因此卷积神经网络有点有趣
only account for 5% of data-center workload. 仅占数据中心工作量的5％。
Most of them is multilayer perception, 大多数是多层感知，
those fully connected layers. 那些完全连接的层。
About 61% maybe for ads, I'm not sure. 大约61％的广告，我不确定。
And about 29% of the workload in data-center is the 数据中心大约29％的工作量是
Long Short Term Memory. 长期短期记忆。
For example, speech recognition, 例如，语音识别，
or machine translation, I suspect. 或者机器翻译，我怀疑。
Remember just now we have seen there are 记住刚才我们已经看到了
90 teraops per second. 每秒90 teraops。
But what actually number of teraops per second 但实际上每秒的teraops数量是多少
can be achieved? 可以实现？
This is a basic tool to measure the bottleneck 这是衡量瓶颈的基本工具
of a computer system. 一个计算机系统。
Whether you are bottlenecked by the arithmetic or 无论你是通过算术还是瓶颈
you are bottlenecked by the memory bandwidth. 你是内存带宽的瓶颈。
It's like if you have a bucket, 这就像你有一个桶，
the lowest part of the bucket determines how much 桶的最低部分决定了多少
water we can hold in the bucket. 我们可以把水放在桶里。
So in this region, you are bottlenecked 所以在这个地区，你是瓶颈
by the memory bandwidth. 通过内存带宽。
So the x-axis is the arithmetic intensity. 所以x轴是算术强度。
Which is number of floating point operations per byte 这是每个字节的浮点运算数
the ratio between the computation and memory 计算和内存之间的比率
of bandwidth overhead. 带宽开销
So the y-axis, is the actual attainable performance. 所以y轴是实际可达到的性能。
Here is the peak performance for example. 例如，这是峰值性能。
When you do a lot of operation after you fetch a single 在获取单个操作后执行大量操作时
piece of data, if you can do a lot of operation 一块数据，如果你可以做很多操作
on top of it, then you are bottlenecked by the arithmetic. 在它之上，然后你被算术瓶颈了。
But after you fetch a lot of data from the memory, 但是从内存中获取大量数据后，
but you just do a tiny little bit of arithmetic, 但你只需要做一点算术，
then you will be bottlenecked by the memory bandwidth. 那么你将受到内存带宽的瓶颈。
So how much you can fetch from the memory determines 那么你可以从内存中获取多少就可以了
how much real performance you can get. 你可以获得多少真正的表现。
And remember there is a ratio. 请记住有一个比例。
When it is one here, this region it happens to be the same 当它在这里时，这个区域碰巧是相同的
as the turning point is the actual 因为转折点是实际的
memory bandwidth of your system. 系统的内存带宽。
So let's see what is the life for the TPU. 那么让我们看看TPU的生命是什么。
The TPU's peak performance is really high, TPU的最高性能非常高，
about 90 Tops per second. 每秒大约90个上衣。
For those convolution nets, they are pretty much saturating 对于那些卷积网，它们几乎是饱和的
the peak performance. 最高性能。
But there are lot of neural networks that has a utlitization 但是有许多神经网络已经完全消失了
less than 10%, 不到10％，
meaning that 90 T-ops per second is actually 这意味着实际上每秒90个T-ops
achieves about three to 12 T-ops per second in real case. 在实际情况下，每秒可实现大约3到12个T-op。
But why is it like that? 但为什么会那样？
The reason is, in order to have those real-time guarantee 原因是，为了得到那些实时保证
that the user not wait for too long, you cannot batch 用户不要等待太久，你不能批量
a lot of user's images or speech voice data 很多用户的图像或语音数据
at the same time. 同时。
So as a result, for those fully connect layers, 因此，对于那些完全连接的层，
they have very little reuse, so they are bottlenecked 它们很少重复使用，因此它们存在瓶颈
by the memory bandwidth. 通过内存带宽。
For those convolution neural nets, for example this one, 对于那些卷积神经网络，例如这个，
this blue one, that achieve 86, which is CNN0. 这个蓝色的，达到86，这是CNN0。
The ratio between the ops and 操作和操作之间的比率
the number of memory is the highest. 内存数量最多。
It's pretty high, more than 2,000 compared with other 它相当高，与其他相比超过2000
multilayer perceptron or long short term memory 多层感知器或长期短期记忆
the ratio is pretty low. 这个比例非常低。
So this figure compares, this is the TPU and this one is 所以这个数字比较一下，这就是TPU而且这个是
the CPU, this is the GPU. CPU，这是GPU。
Here is memory bandwidth, the peak memory bandwidth 这是内存带宽，峰值内存带宽
at a ratio of one here. 在这里比例为1。
So TPU has the highest memory bandwidth. 因此TPU具有最高的内存带宽。
And here is where are these neural networks 这就是这些神经网络的所在
lie on this curve. 躺在这条曲线上。
So the asterisk is for the TPU. 所以星号是TPU的。
It's still higher than other dots, 它仍高于其他点，
but if you're not comfortable with this log scale figure, 但如果你对这个对数比例数字不满意，
this is what it's like putting it in linear roofline. 这就像把它放在线性车顶线上一样。
So pretty much everything disappeared except 几乎一切都消失了，除了
for the TPU results. 对于TPU结果。
So still, all these lines, although they are higher 所有这些线路，尽管它们更高
than the CPU and GPU, it's still way below the 而不是CPU和GPU，它仍然低于
theoretical peak operations per second. 每秒理论峰值运算。
So as I mentioned before, it is really bottlenecked 正如我之前提到的，它确实是瓶颈
by the low latency requirement so that it can have 通过低延迟要求，它可以有
a large batch size. 批量大。
That's why you have low operations per byte. 这就是你每字节操作低的原因。
And how do you solve this problem? 你是如何解决这个问题的？
You want to have less number of memory footprint 您希望拥有更少的内存占用量
so that it can reduce the memory bandwidth requirement. 这样可以减少内存带宽需求。
One solution is to compress the model and the challenge 一种解决方案是压缩模型和挑战
is how do we build a hardware that can do inference 我们如何构建可以进行推理的硬件
directly on the compressed model? 直接在压缩模型上？
So I'm going to introduce my design of EIE, the Efficient 所以我将介绍我的EIE设计，即Efficient
Inference Engine, which deals with those sparse 推理引擎，处理那些稀疏的
and the compressed model to save the memory bandwidth. 和压缩模型，以节省内存带宽。
And the rule of thumb, like we mentioned before is taking 和我们之前提到过的经验法则一样
out one bit of sparsity first. 首先出一点稀疏性。
Anything times zero is zero. 零的任何时间都是零。
So don't store it, don't compute on it. 所以不要存储它，不要计算它。
And second idea is, you don't need that much full precision, 第二个想法是，你不需要那么多的精确度，
but you can approximate it. 但你可以近似它。
So by taking advantage of the sparse weight, we 因此，通过利用稀疏的重量，我们
get about a 10x saving in the computation, 5x less 在计算中节省大约10倍，减少5倍
memory footprint. 内存占用。
The 2x difference is due to index overhead. 2x的差异是由于索引开销。
And by taking advantage of the sparse activation, 并利用稀疏激活，
meaning after bandwidth, if activation is zero, then 如果激活为零，则表示带宽后的含义
ignore it. 忽略它。
You save another 3x of computation. 你节省了另外3倍的计算。
And then by such weight sharing mechanism, 然后通过这种权重分享机制，
you can use four bits to represent each weight rather 您可以使用四位代表每个重量
than 32 bit. 超过32位。
That's another eight times saving in the memory footprint. 这是内存占用的另外八倍。
So this is physically, logically how the weights are stored. 所以这在物理上，逻辑上是如何存储权重的。
A four by eight matrix, and this is how physically 一个四乘八的矩阵，这就是物理上的
they are stored. 他们被存储。
Only the non-zero weights are stored. 仅存储非零权重。
So you don't need to store those zeroes. 所以你不需要存储那些零。
You'll save the bandwidth fetching those zeroes. 您将保存获取这些零的带宽。
And also I'm using the relative index to further save 而且我也使用相对索引进一步保存
the number of memory overhead. 内存开销数量。
So in the computation like this figure shows, 所以在如图所示的计算中，
we are running the multiplication only on non-zero. 我们只在非零运行乘法运算。
If it's zero, then skip it. 如果它为零，则跳过它。
Only broadcast it to the non-zero weights 仅将其广播为非​​零权重
and if it is zero, skip it. 如果它为零，请跳过它。
If it's a non-zero, do the multiplication. 如果它是非零，则进行乘法运算。
In another cycle, do the multiplication. 在另一个循环中，进行乘法运算。
So the idea is anything multiplied by zero is zero. 所以这个想法是任何乘以零的都是零。
So this is a little complicated, 所以这有点复杂，
I'm going to go very quickly. 我会很快走的。
I'm going to have a lookup table that decode the four bit 我将要有一个解码四位的查找表
weight into the 16 bit weight and using the four bit 重量为16位重量并使用四位
relative index passed through address accumulator 相对索引通过地址累加器传递
to get the 16 bit absolute index. 获得16位绝对索引。
And this is what the hardware architecture 这就是硬件架构
like in the high level. 喜欢在高层次。
You can feel free to refer to my paper for detail. 您可以随时参考我的论文了解详情。
Okay speedup. 好的加速。
So using such efficient hardware architecture 所以使用这种高效的硬件架构
and also model compression, this is the original 还有模型压缩，这是原始的
result we have seen for CPU, GPU, mobile GPU. 结果我们看到了CPU，GPU，移动GPU。
Now EIE is here. 现在EIE就在这里。
189 times faster than the CPU and about 13 times faster 比CPU快189倍，速度快13倍
than the GPU. 比GPU。
So this is the energy efficiency on the log scale, 所以这是对数尺度上的能源效率，
it's about 24,000x more energy efficient than a CPU 它比CPU高出约24,000倍的能源效率
and about 3000x more energy efficient than a GPU. 比GPU高出约3000倍的能源效率。
It means for example, previously if your battery can 这意味着，例如，以前，如果您的电池可以
last for one hour, now it can last for 持续一个小时，现在它可以持续
3000 hours for example. 例如3000小时。
So if you say, ASIC is always better than CPUs and GPUs 所以如果你说，ASIC总是优于CPU和GPU
because it's customized hardware. 因为它是定制的硬件。
So this is comparing EIE with the peer ASIC, for example 例如，这是将EIE与对等ASIC进行比较
DaDianNao and the TrueNorth. DaDianNao和TrueNorth。
It has a better throughput, better energy efficiency 它具有更好的吞吐量，更好的能源效率
by order of magnitude, compared with other ASICs. 与其他ASIC相比，数量级。
Not to mention that CPU, GPU and FPGAs. 更不用说CPU，GPU和FPGA。
So we have covered half of the journey. 所以我们已经覆盖了一半的旅程。
We mentioned inference, we pretty much 我们提到推断，我们差不多
covered everything for inference. 涵盖一切推断。
Now we are going to switch gear and talk about training. 现在我们要换档并谈论培训。
How do we train neural networks efficiently, 我们如何有效地训练神经网络，
how do we train it faster? 我们如何更快地训练它？
So again, we are starting with algorithm first, 再说一次，我们首先从算法开始，
efficient algorithms followed by the hardware 硬件遵循的有效算法
for efficient training. 进行有效的培训。
So for efficient training algorithms, I'm going to mention 因此，对于有效的训练算法，我要提一下
four topics. 四个主题。
The first one is parallelization, and then mixed precision 第一个是并行化，然后是混合精度
training, which was just released about one month ago 大约一个月前刚刚发布的培训
and at NVIDIA GTC, so it's fresh knowledge. 而在NVIDIA GTC，所以它是新鲜的知识。
And then model distillation, followed by my work on 然后模型蒸馏，然后我的工作
Dense-Sparse-Dense training, or better Regularization 密集 - 稀疏 - 密集训练，或更好的正规化
technique. 技术。
So let's start with parallelization. 那么让我们从并行化开始吧。
So this figure shows, anyone in the hardware community. 所以这个数字显示了硬件社区中的任何人。
Most are very familiar with this figure. 大多数人都非常熟悉这个数字。
So as time goes by, what is the trend? 随着时间的推移，趋势是什么？
For the number of transistors is keeping increasing. 因为晶体管的数量在不断增加。
But the single threaded performance is getting plateaued 但单线程性能正在趋于稳定
in recent years. 最近几年。
And also the frequency is getting plateaued in recent years. 此外，近几年频率也在上升。
Because of the power constraint, to stop not scaling. 由于功率限制，停止不缩放。
And interesting thing is the number of cores is increasing. 有趣的是核心数量正在增加。
So what we really need to do is parallelization. 所以我们真正需要做的是并行化。
How do we parallelize the problem to take advantage 我们如何并行化问题以利用它
of parallel processing? 并行处理？
Actually there are a lot of opportunities for parallelism 实际上并行机会很多
in deep neural networks. 在深度神经网络中。
For example, we can do data parallel. 例如，我们可以并行执行数据。
For example, feeding two images into the same model 例如，将两个图像输入同一模型
and run them at the same time. 并同时运行它们。
This doesn't affect latency for a single input. 这不会影响单个输入的延迟。
It doesn't make it shorter, but it makes batch size larger 它不会缩短，但会使批量更大
basically if you have four machines our effective batch 基本上如果你有四台机器我们的有效批次
size becomes four times as before. 尺寸变成以前的四倍。
So it requires the coordinated weight update. 所以它需要协调的重量更新。
For example, this is a paper from Google. 例如，这是来自Google的一篇论文。
There is a parameter server as a master and a couple of 有一个参数服务器作为主服务器和几个
slaves running their own piece of training data and update 奴隶运行他们自己的训练数据和更新
the gradient to the parameter server and get the updated 参数服务器的渐变并获得更新
weight for them individually, 单独的重量，
that's how data parallelism is handled. 这就是数据并行性的处理方式。
Another idea is there could be a model parallelism. 另一个想法是可能存在模型并行性。
You can sublet your model and handle it 您可以转租模型并处理它
to different processors or different threads. 到不同的处理器或不同的线程。
For example, there's this image, you want to run convolution 例如，有这个图像，你想运行卷积
on this image that is six dimension for loop. 在这个图像上是六维循环。
What you can do is you can cut the input image by 你可以做的是你可以切割输入图像
two by two blocks so that each thread, or each processor 两个两个块使每个线程或每个处理器
handles one fourth of the image. 处理图像的四分之一。
Although there's a small halo here in between you 虽然你们之间有一个小小的光环
have to take care of. 必须照顾。
And also, you can parallelize by the 而且，你可以通过并行化
output or input feature map. 输出或输入要素图。
And for those fully connect layers, 对于那些完全连接的层，
how do we parallelize the model? 我们如何并行化模型？
It's even simpler. 它甚至更简单。
You can cut the model into half 你可以把模型切成两半
and hand it to different threads. 并把它交给不同的线程。
And the third idea, you can even do 第三个想法，你甚至可以做到
hyper-parameter parallel. 超参数并行。
For example, you can tune your learning rate, your 例如，您可以调整您的学习率
weight decay for different machines 不同机器的重量衰减
for those coarse-grained parallelism. 对于那些粗粒度的并行性。
So there are so many alternatives you have to tune. 所以有很多选择你需要调整。
Small summary of the parallelism. 并行性的小总结。
There are lots of parallelisms in deep neural networks. 深度神经网络中存在许多并行性。
For example, with data parallelism, you can run multiple 例如，使用数据并行性，您可以运行多个
training images, but you cannot have unlimited number 训练图像，但你不能有无限数量
of processors because you are limited by batch size. 处理器，因为您受到批量大小的限制。
If it's too large, stochastic gradient descent 如果它太大，随机梯度下降
becomes gradient descent, that's not good. 变得渐渐下降，这不好。
You can also run the model parallelism. 您还可以运行模型并行性。
Split the model, either by cutting the image or 通过切割图像或拆分模型
cutting the convolution weights. 削减卷积权重。
Either cutting the image or cutting 切割图像或切割
the fully connected layers. 完全连接的层。
So it's very easy to get 16 to 64 GPUs training one model 所以很容易让16到64个GPU训练一个模型
in parallel, having very good speedup. 同时，具有非常好的加速。
Almost linear speedup. 差不多线性加速。
Okay, next interesting thing, mixed precision with 好的，下一个有趣的事情，混合精确
FP16 or FP32. FP16或FP32。
So remember in the beginning of this lecture, 所以请记住在本讲座开头，
I had a chart showing the energy and area overhead for 我有一张图表显示了能量和面积开销
a 16 bit versus a 32 bit. 16位与32位。
Going from 32 bit to 16 bit, you save about 4x the energy 从32位到16位，可以节省大约4倍的能量
and 4x the area. 和面积的4倍。
So can we train a deep neural network with such low 那么我们可以训练一个如此低的深度神经网络
precision with floating point 16 bit rather than 32 bit? 浮点精度16位而不是32位？
It turns out we can do that partially. 事实证明我们可以部分地做到这一点。
By partially, I mean we need FP32 in some places. 部分地，我的意思是我们在某些地方需要FP32。
And where are those places? 那些地方在哪里？
So we can do the multiplication in 16 bit as input. 所以我们可以将16位乘法作为输入。
And then we have to do the summation 然后我们必须做总结
in 32 bit accumulation. 在32位积累。
And then convert the result to 32 bit to store the weight. 然后将结果转换为32位以存储权重。
So that's where the mixed precision comes from. 这就是混合精度的来源。
So for example, we have a master weight stored in 例如，我们有一个存储的主要权重
floating point 32, we down converted it to floating 浮点32，我们将其转换为浮动
point 16 and then we do the feed forward with 16 bit 点16然后我们用16位前馈
weight, 16 bit activation, we get a 16 bit activation 重量，16位激活，我们得到16位激活
here in the end when we are doing back propagation 最后，当我们进行反向传播时
of the computation is also done with floating point 16 bit. 计算也是用浮点16位完成的。
Very interesting here, for the weights we get a floating 这里非常有趣，因为重量我们得到浮动
point 16 bit gradient here for the weight. 这里的重量为16位梯度。
But when we are doing the update, so W plus learning 但是当我们进行更新时，W加学习
rate times the gradient, that operation has 速率乘以该操作所具有的梯度
to be done in 32 bit. 以32位完成。
That's where the mixed precision is coming from. 这就是混合精度的来源。
And see there are two colors, which here is 16 bit, 看到有两种颜色，这里是16位，
here is the 32 bit. 这是32位。
That's where the mixed precision comes from. 这就是混合精度的来源。
So does such low precision sacrifice your prediction 那么低精度也会牺牲你的预测
accuracy for your model? 你的模型准确吗？
So this is the figure from NVIDIA just released a couple 所以这是NVIDIA刚刚发布的数字
of weeks ago actually. 几个星期前实际上。
Thanks to Paulius giving me the slide. 感谢Paulius给我幻灯片。
The convergence between floating point 32 versus 浮点32与之间的收敛
the multi tensor up, which is basically the mixed 多张量涨，这基本上是混合的
precision training, are actually pretty much 精确训练，其实非常多
the same for convergence. 收敛相同。
If you zoom it in a little bit, 如果你稍微放大一点，
they are pretty much the same. 他们几乎是一样的。
And for ResNet, the mixed precision sometimes behaves 对于ResNet，混合精度有时会表现出来
a little better than the full precision weight. 比全精度重量好一点。
Maybe because of noise. 也许是因为噪音。
But in the end, after you train the model, this is 但最后，在训练模型之后，就是这样
the result of AlexNet, Inception V3, and ResNet-50 AlexNet，Inception V3和ResNet-50的结果
with FP32 versus FP16 mixed precision training. FP32与FP16混合精密训练。
The accuracy is pretty much the same 准确性几乎相同
for these two methods. 对于这两种方法。
A little bit worse, but not by too much. 有点糟糕，但不是太多。
So having talked about the mixed precision training, 所以谈到混合精确训练，
the next idea is to train with model distillation. 下一个想法是用模型蒸馏训练。
For example, you can have multiple neural networks, 例如，您可以拥有多个神经网络，
Googlenet, Vggnet, Resnet for example. 例如，Googlenet，Vggnet，Resnet。
And the question is, can we take advantage of these 问题是，我们能否利用这些优势
different models? 不同型号？
Of course we can do model ensemble, can we utilitze them 当然我们可以做模型合奏，我们可以利用它们
as teacher, to teach a small junior neural network to have 作为老师，要教一个小小的神经网络
it perform as good as the senior neural network. 它的表现与高级神经网络一样好。
So this is the idea. 所以这就是主意。
You have multiple large powerful senior neural networks 你有多个强大的高级神经网络
to teach this student model. 教这个学生模型。
And hopefully it can get better results. 希望它可以获得更好的结果。
And the idea to do that is, instead of using this 而这样做的想法是，而不是使用它
hard label, for example for car, dog, cat, the probability 硬标签，例如汽车，狗，猫，概率
for dog is 100%, but the output of the geometric 对于狗是100％，但输出的几何
ensemble of those large teacher neural networks 那些大型教师神经网络的集合
maybe the dog has 90% and the cat is about 10%, 也许狗有90％，猫约10％，
and the magic happens here. 而魔法发生在这里。
You want to have a softened result label here. 你想在这里有一个软化的结果标签。
For example, the dog is 30%, the cat is 20%. 例如，狗是30％，猫是20％。
Still the dog is higher than the cat. 狗仍高于猫。
So the prediction is still correct, but it uses 所以预测仍然是正确的，但它使用
this soft label to train the student neural network 这个软标签用于训练学生神经网络
rather than use this hard label to train 而不是使用这个硬标签来训练
the student neural network. 学生神经网络。
And mathematically, you control how much do you make 从数学上讲，你可以控制自己的收入
it soft by this temperature during the soft max 在软最大值期间它被这个温度软化
controlling by this temperature. 通过这个温度控制。
And the result is that, starting with the trained model 结果是，从训练有素的模型开始
that classifies 58.9% of the test frames correctly, 它正确地分类了58.9％的测试帧，
the new model converges to 57%. 新车型收敛至57％。
Only train on 3% of the data. 只训练3％的数据。
So that's the magic for model distillation 这就是模型蒸馏的神奇之处
using this soft label. 使用这个软标签。
And the last idea is my recent paper using 最后一个想法是我最近的论文使用
a better regularization to train deep neural nets. 训练深度神经网络的更好的正则化。
We have seen these two figures before. 我们之前看过这两个数字。
We pruned the neural network, having less number 我们修剪了神经网络，数量较少
of weights, but have the same accuracy. 重量，但具有相同的准确性。
Now what I did is to recover and to retrain those 现在我所做的就是恢复和重新训练
weights shown in red and make everything train 重量显示为红色，使一切都训练
out together to increase the model capacity after 一起出来后增加模型容量
it is trained at a low dimensional space. 它是在低维空间训练的。
It's like you learn the trunk first and then gradually 这就像你先学习后备箱然后逐渐学习
add those leaves and learn everything together. 添加这些叶子并一起学习所有东西。
It turns out, on ImageNet it performs relatively about 1% to 事实证明，在ImageNet上它的表现相对约为1％
4% absolute improvement of accuracy. 精度绝对提高4％。
And is also general purpose, works on long-short term memory 并且也是通用的，用于长期记忆
and also recurrent neural nets collaborated with Baidu. 还有反复神经网络与百度合作。
So I also open sourced this special training model 所以我也开源了这个特殊的训练模型
on the DSD Model Zoo, where there are trained, all 在DSD模型动物园，有训练的，全部
these models, GoogleNet, VGG, ResNet, and also SqueezeNet, 这些模型，GoogleNet，VGG，ResNet以及SqueezeNet，
and also AlexNet. 还有AlexNet。
So if you are interested, feel free to check out this 所以，如果您有兴趣，请随时查看
Model Zoo and compare it with the Caffe Model Zoo. 模型动物园并将其与Caffe Model Zoo进行比较。
Here's some examples on dense-spare-dense training helps 以下是密集备用密集培训帮助的一些示例
with image capture. 与图像捕获。
For example, this is a very challenging figure. 例如，这是一个非常具有挑战性的数字。
The original baseline of neural talk says a boy in 神经谈话的原始基线说一个男孩在
a red shirt is climbing a rock wall. 一件红色的衬衫正在攀岩墙。
And the sparse model says a young girl is jumping 稀疏模型说一个年轻女孩正在跳跃
off a tree, probably mistaking the hair with either 在一棵树上，可能会误认为头发
the rock or the tree. 岩石或树木。
But then sparse-dense training by using this kind of 但随后使用这种稀疏密集的训练
regularization on a low dimensional space, it says 它说，在低维空间上正则化
a young girl in a pink shirt is swinging on a swing. 一个穿着粉红色衬衫的年轻女孩正在秋千上荡秋千。
And there are a lot of examples due to the limit of time, 由于时间的限制，有很多例子，
I will not go over them one by one. 我不会一个接一个地过去。
For example, a group of people are standing in front 例如，一群人站在前面
of a building, there's no building. 一幢建筑物，没有建筑物。
A group of people are walking in the park. 一群人正在公园散步。
Feel free to check out the paper and see more interesting 随意查看论文，看看更有趣
results. 结果。
Okay finally, we come to hardware for efficient training. 好的，最后，我们来硬件进行有效的培训。
How to we take advantage of the algorithms 我们如何利用这些算法
we just mentioned. 我们刚才提到过。
For example, parallelism, mixed precision, how are 例如，并行性，混合精度，如何
the hardware designed to actually 实际设计的硬件
take advantage of such features. 利用这些功能。
First GPUs, this is the Nvidia PASCAL GPU, GP100, 首款GPU，这是Nvidia PASCAL GPU，GP100，
which was released last year. 这是去年发布的。
So it supports up to 20 Teraflops on FP16. 因此它支持FP16上最多20 Teraflops。
It has 16 gigabytes of high bandwidth memory. 它有16千兆字节的高带宽内存。
750 gigabytes per second. 每秒750千兆字节。
So remember, computation and memory bandwidth are 所以请记住，计算和内存带宽是
the two factors determines your overall performance. 这两个因素决定了您的整体表现。
Whichever is lower, it will suffer. 无论哪个较低，它都会受到影响。
So this is a really high bandwidth, 700 gigabytes 所以这是一个非常高的带宽，700千兆字节
compared with DDR3 is just 10 or 30 gigabytes per second. 与DDR3相比，每秒只需10或30千兆字节。
Consumes 300 Watts and 消耗300瓦特和
it's done in 16 nanometer process 它是在16纳米工艺中完成的
and have a 160 gigabytes per second NV Link. 并拥有每秒160千兆字节的NV Link。
So remember we have computation, we have memory, 所以记住我们有计算，我们有记忆，
and the third thing is the communication. 第三件事就是沟通。
All three factors has to be balanced in order to 所有这三个因素必须平衡才能
achieve a good performance. 实现良好的表现。
So this is very powerful, but even more exciting, 所以这非常强大，但更令人兴奋，
just about a month ago, Jensen released the newest 大约一个月前，Jensen发布了最新版本
architecture called the Volta GPUs. 架构称为Volta GPU。
And let's see what is inside the Volta GPU. 让我们看看Volta GPU中的内容。
Just released less than a month ago, so it has 15 of 刚刚发布不到一个月前，所以有15个
FP32 teraflops and what is new here, there is 120 FP32 teraflops和这里的新功能，有120个
Tensor T-OPS, so specifically designed for deep learning. Tensor T-OPS，专为深度学习而设计。
And we'll later cover what is the tensor core. 我们稍后会介绍什么是张量核心。
And what is this 120 coming from. 什么是120来自。
And rather than 750 gigabytes per second, this 而不是每秒750千兆字节
year, the HBM2, they are using 900 gigabytes per second 一年，HBM2，它们每秒使用900千兆字节
memory bandwidth. 内存带宽。
Very exciting. 很刺激。
And 12 nanometer process has a die size of more than 800 12纳米工艺的芯片尺寸超过800
millimeters square. 毫米平方。
A really large chip and supported by 300 gigabytes per 一个非常大的芯片，每个支持300千兆字节
second NVLink. 第二个NVLink。
So what's new in Volta, the most interesting thing for us 那么Volta的新功能对我们来说是最有趣的事情
for deep learning, is this thing called Tensor Core. 对于深度学习，这个东​​西叫做Tensor Core。
So what is a Tensor Core? 什么是Tensor Core？
Tensor Core is actually an instruction that can Tensor Core实际上是一个可以的指令
do the four by four matrix times a four by four matrix. 做四乘四矩阵乘以四乘四矩阵。
The fused FMA stands Fused Multiplication and Add 融合的FMA代表Fused Multiplication和Add
in this mixed precision operation. 在这种混合精确操作中。
Just in one single clock cycle. 只需一个时钟周期。
So let's discern for a little bit what does this mean. 所以让我们分辨一下这是什么意思。
So mixed precision is exactly as we mentioned in the last 所以混合精度就像我们在上一篇中提到的那样
chapter, so we are having FP16 for the multiplication, 章，所以我们有FP16用于乘法，
but for accumulation, we are doing it with FP32. 但为了积累，我们正在使用FP32。
That's where the mixed precision comes from. 这就是混合精度的来源。
So let's say how many operations, if it's four 所以，如果是四个，那就说多少次操作
by four by four, it's 64 multiplications then just 四乘四，然后就是64次乘法
in one single cycle. 在一个单一的周期。
That's 12x increase in the speedup of the Volta 这比Volta的加速提高了12倍
compared with the Pascal, which is released just less year. 与Pascal相比，Pascal的发布时间更短。
So this is the result for matrix multiplication on 所以这是矩阵乘法的结果
different sizes. 不同的尺寸。
The speedup of Volta over Pascal is roughly 3x faster 伏尔塔超过帕斯卡的加速比大约快3倍
doing these matrix multiplications. 做这些矩阵乘法。
What we care more is not only matrix multiplication 我们更关心的不仅仅是矩阵乘法
but actually running the deep neural nets. 但实际上运行深度神经网络。
So both for training and for inference. 所以无论是培训还是推理。
And for training on ResNet-50, by taking advantage 对于ResNet-50的培训，通过利用
of this Tensor Core in this V100, 此V100中的Tensor Core，
it is 2.4x faster than the P100 using FP32. 它比使用FP32的P100快2.4倍。
So on the right hand side, it compares the inference 所以在右侧，它比较了推断
speedup, given a 7 microsecond latency requirement. 加速，给出7微秒的延迟要求。
What is the number of images per second it can process? 它每秒可以处理的图像数量是多少？
It has a measurement of throughput. 它具有吞吐量的测量。
Again, the V100 over P100, by taking advantage 再次，V100超过P100，通过利用
of the Tensor Core, is 3.7 faster than the P100. Tensor Core的速度比P100快3.7倍。
So this figure gives roughly an idea, what is a Tensor Core, 所以这个数字大致给出了一个想法，什么是Tensor Core，
what is an integer unit, what is a floating point unit. 什么是整数单位，什么是浮点单位。
So this whole figure 所以这整个人物
is a single SM 是一个单一的SM
stream multiprocessor. 流多处理器。
So SM is partitioned into four processing blocks. 因此SM被划分为四个处理块。
One, two, three, four, right? 一，二，三，四，对吧？
And in each block there are eight FP64 cores here 在每个块中，这里有8个FP64内核
and 16 FP32 and 16 INT32 cores here, units here. 此处有16个FP32和16个INT32内核。
And then there are two of the new mixed precision 然后有两个新的混合精度
Tensor cores specifically designed for deep learning. 专为深度学习而设计的Tensor内核。
And also there are the one warp scheduler, dispatch unit 还有一个warp调度程序，调度单元
and Register File, as before. 和注册文件一样。
So what is new here is the Tensor core unit here. 所以这里有新的Tensor核心单元。
So here is a figure comparing the recent generations of 所以这是一个比较最近几代人的数字
Nvidia GPUs from Kepler 开普勒的Nvidia GPU
to Maxwell to Pascal to Volta. 到Maxwell到Pascal到Volta。
We can see everything is keeping improving. 我们可以看到一切都在不断改进。
For example, the boost clock has been increased from 例如，增益时钟已从增加
about 800 MHz to 1.4 GHz. 约800 MHz至1.4 GHz。
And from the Volta generation there begins to have 从伏尔塔一代开始就有了
the Tensor core units here, which has never existed before. 这里的Tensor核心单元，从未存在过。
And before the Maxwell, 在麦克斯韦之前，
the GPUs are using the GDDR5, GPU正在使用GDDR5，
and after the Pascal GPU, 在Pascal GPU之后，
the HBM begins to came into place, HBM开始到位，
the high-bandwidth memory. 高带宽内存。
750 gigabytes per second here. 这里每秒750千兆字节。
900 gigabytes per second compared with DDR3, 与DDR3相比，每秒900千兆字节，
30 gigabytes per second. 每秒30千兆字节。
And memory size actually didn't increase by too much, 内存大小实际上并没有增加太多，
and the power consumption is actually 实际上是功耗
also remaining roughly the same. 也保持大致相同。
But giving the increase of computation, you can fit them 但是，增加计算量，你可以适应它们
in the fixed power envelope that's still an exciting thing. 在固定功率范围内，这仍然是令人兴奋的事情。
And the manufacturing process is actually improving from 制造过程实际上正在改善
28 nanometer, 16 nanometer, all the way to 12 nanometer. 28纳米，16纳米，一直到12纳米。
And the chip area are also increasing to 芯片面积也在增加
800 millimeter-squared, that's really huge. 800平方毫米，真的很大。
So, you may be interested in the comparison of the GPU 所以，您可能对GPU的比较感兴趣
with the TPU, right? 与TPU，对吧？
So how do they compare with each other? 那么他们如何相互比较呢？
So in the original TPU paper, 所以在最初的TPU论文中，
TPU actually designed roughly in the year of 2015, TPU实际上大致在2015年设计，
and this is comparison of the Pascal P40 GPU 这是Pascal P40 GPU的比较
released in 2016. 2016年发布。
So, TPU, the power consumption is lower, 那么，TPU，功耗更低，
is larger on chip memory of 24 megabytes, 在24兆字节的芯片内存上更大，
really large on-chip SRAM managed by the software. 由软件管理的非常大的片上SRAM。
And then both of them support INT8 operations, 然后它们都支持INT8操作，
while the inferences per second given a 10 nanometer latency 而每秒的推论给出了10纳秒的延迟
the comparison for TPU is 1X. TPU的比较是1X。
For the P40 it's about 2X. 对于P40，它大约是2倍。
So, just last week, 所以，就在上周，
in the Google I/O, 在Google I / O中，
a new nuclear bomb is landed on the Earth. 一颗新的核弹落在地球上。
That is the Google Cloud TPU. 这就是Google Cloud TPU。
So now TPU not only support inference, 所以现在TPU不仅支持推理，
but also support training. 还支持培训。
So there is a very limited information we can get 所以我们可以获得的信息非常有限
beyond this Google Blog. 超越这个谷歌博客。
So their Cloud TPU delivers up to 180 teraflops 因此，他们的Cloud TPU可提供高达180 teraflops的吞吐量
to train and run machine learning models. 培养和运行机器学习模型。
And this is multiple Cloud TPU, 这是多个云TPU，
making it into a TPU pod, 将它制作成TPU吊舱，
which is built with 16 the second generation TPUs 它是由16代第二代TPU构建的
and delivers up to 11.5 teraflops 并提供高达11.5 teraflops
of machine learning acceleration. 机器学习加速。
So in the Google Blog, they mentioned that 所以在谷歌博客中，他们提到了这一点
one of the large scale translation models, 其中一个大型翻译模型，
Google translation models, used to take a full day to train 谷歌翻译模型，过去需要一整天的时间来训练
on 32 of best commercially-available GPUs, probably P40 32个最好的商用GPU，可能是P40
or P100, maybe. 也许是P100。
And now it trains to the same accuracy, 而现在它的训练精度相同，
just within one afternoon, with just 1/8 of a TPU pod, 就在一个下午，只有1/8的TPU吊舱，
which is pretty exciting. 这非常令人兴奋。
Okay, so as a little wrap-up. 好的，还有一点点总结。
We covered a lot of stuff, we've mentioned 我们已经提到了很多东西
the four dimension space of algorithm and hardware, 算法和硬件的四维空间，
inference and training, we covered the algorithms for 推理和训练，我们涵盖了算法
inference, for example, pruning and quantization, 推理，例如，修剪和量化，
Winograd Convolution, binary, ternary, Winograd卷积，二进制，三元，
weight sharing, for example. 例如，分担重量。
And then the hardware for the efficient inference. 然后是有效推理的硬件。
For example, the TPU, 例如，TPU，
that take advantage of INT8, integer 8. 利用INT8，整数8。
And also my design of EIE accelerator that take advantage 而且我的EIE加速器设计也充分利用了
of the sparsity, anything multiplied by zero is zero, 稀疏性，任何乘以零的都是零，
so don't store it, don't compute on it. 所以不要存储它，不要计算它。
And also the efficient algorithm for training, for example, 还有高效的培训算法，例如，
how do we do parallelization and the most recent research on 我们如何进行并行化和最近的研究
how do we use mixed precision training by taking advantage 我们如何利用混合精确训练
of FP16 rather than FP32 to do training FP16而不是FP32进行训练
which is four times saving the energy 这是节省能源的四倍
and four times saving in the area, 在该地区节省四倍，
which doesn't quite sacrifice the accuracy you'll get from 这并没有完全牺牲你的准确性
the training. 培训。
And also Dense-Sparse-Dense training using better regularization 并且还使用更好的正则化进行密集稀疏密集训练
sparse regularization, and also the teacher-student model. 稀疏正规化，也是师生模型。
You have multiple teacher on your network and have a small 您的网络上有多位教师并且身材较小
student network that you can distill the knowledge 学生网络，你可以提炼知识
from the teacher in your network by a temperature. 来自您网络中的老师的温度。
And finally we covered the hardware for efficient training 最后，我们介绍了硬件以进行有效的培训
and introduced two nuclear bombs. 并介绍了两枚核弹。
One is the Volta GPU, the other is the TPU version two, 一个是Volta GPU，另一个是TPU版本二，
the Cloud TPU and also the amazing Tensor cores 云TPU以及令人惊叹的Tensor核心
in the newest generation of Nvidia GPUs. 在最新一代的Nvidia GPU中。
And we also revealed the progression of a wide range, 我们还揭示了广泛的进展，
the recent Nvidia GPUs from the Kepler K40, 最近开普勒K40的Nvidia GPU，
that's actually when I started my research, 那是我开始研究的时候
what we used in the beginning, 我们在开始时使用的是什么
all the way to and then K40, M40, 一直到然后K40，M40，
and then Pascal and then finally the exciting Volta GPU. 然后Pascal然后终于令人兴奋的Volta GPU。
So every year there is a nuclear bomb in the spring. 所以每年春天都会有核弹。
Okay, a little look ahead in the future. 好的，未来有点展望。
So in the future of the city we can imagine there are a lot 因此，在城市的未来，我们可以想象有很多
of AI applications using smart society, smart care, 使用智能社会的AI应用程序，智能护理，
IOT devices, smart retail, for example, the Amazon Go, 物联网设备，智能零售，例如亚马逊Go，
and also smart home, a lot of scenarios. 还有智能家居，很多场景。
And it poses a lot of challenges on the hardware design 它给硬件设计带来了很多挑战
that requires the low latency, privacy, mobility 这需要低延迟，隐私，移动性
and energy efficiency. 和能源效率。
You don't want your battery to drain very quickly. 您不希望电池快速耗尽。
So it's both challenging and very exciting era 因此，它既具有挑战性又非常令人兴奋的时代
for the code design for both the machine learning 用于机器学习的代码设计
deep neural network model architectures 深度神经网络模型体系结构
and also the hardware architecture. 以及硬件架构。
So we have moved from PC era to mobile era. 所以我们已经从PC时代走向移动时代。
Now we are in the AI-First era, 现在我们处于AI-First时代，
and hope you are as excited as I am for this kind of 并希望你像我这样兴奋
brain-inspired cognitive computing research. 大脑启发的认知计算研究。
Thank you for your attention, I'm glad to take questions. 感谢您的关注，我很乐意提问。
[applause] [掌声]
We have five minutes. 我们有五分钟。
Of course. 当然。
- [Student] Can you commercialize the deep architecture? - [学生]你可以将深层架构商业化吗？
- The architecture, yeah, some of the ideas are pretty good. - 建筑，是的，一些想法非常好。
I think there's opportunity. 我认为有机会。
Yeah. 是啊。
The question is, what can we do to make the hardware better? 问题是，我们可以做些什么来改善硬件？
Oh, right, the question is how do we, 哦，对了，问题是我们怎么样，
the challenges and what opportunity for those small 那些小的挑战和机会
embedded devices around deep neural network 深层神经网络周围的嵌入式设备
or in general AI algorithms. 或者一般的AI算法。
Yeah, so those are the algorithm I discussed 是的，所以这些是我讨论过的算法
in the beginning about inference. 在一开始就推论。
Here. 这里。
These are the techniques that can enable such 这些是能够实现这一目标的技术
inference or AI running on embedded devices, 在嵌入式设备上运行的推理或AI，
by having less number of weights, fewer bits per weight, 通过减少重量，减少每个重量的比特，
and also quantization, low rank approximation. 还有量化，低秩近似。
The small matrix, same accuracy, even going to binary, 小矩阵，精度相同，甚至是二进制，
or ternary weights having just two bits 或仅具有两位的三元权重
to do the computation rather than 16 or even 32 bit 进行计算而不是16位甚至32位
and also the Winograd Transformation. 还有威诺格拉德转型。
Those are also the enabling algorithms for those 这些也是那些的启用算法
low-power embedded devices. 低功耗嵌入式设备。
Okay, the question is, if it's binary weight, the software 好的，问题是，如果它是二进制权重，那么该软件
developers may be not able to take advantage of it. 开发人员可能无法利用它。
There is a way to take advantage of binary weight. 有一种方法可以利用二元权重。
So in one register there are 32 bit. 所以在一个寄存器中有32位。
Now you can think of it as a 32-way parallelism. 现在你可以把它想象成32路并行。
Each bit is a single operation. 每个位都是单个操作。
So say previously we have 10 ops per second. 所以说以前我们每秒有10个操作。
Now you get 330 ops per second. 现在你每秒获得330次操作。
You can do this bitwise operations. 您可以执行此按位操作。
For example, XOR operations. 例如，XOR操作。
So one register file, 那么一个寄存器文件，
one operation becomes 32 operation. 一个操作变为32个操作。
So there is a paper called XORmad, 所以有一篇名为XORmad的论文，
they very amazing implemented 他们非常了不起实施
on the Raspberry Pi using this feature 在Raspberry Pi上使用此功能
to do real-time detection, very cool stuff. 做实时检测，非常酷的东西。
Yeah. 是啊。
Yeah, so the trade-off is always so the power area 是的，所以权衡总是如此权力领域
and performance in general, all the hardware design 和性能一般，所有的硬件设计
have to take into account the performance, the power, 必须考虑到性能，功率，
and also the area. 还有该地区。
When machine learning comes, there's a fourth 当机器学习到来时，还有第四个
figure of merit which is the accuracy. 品质因数是准确性。
What is the accuracy? 准确度是多少？
And there is a fifth one which is programmability. 第五个是可编程性。
So how general is your hardware? 那你的硬件有多普遍？
For example, if Google just want to use that for AI 例如，如果Google只想将其用于AI
and deep learning, it's totally fine 和深度学习，这是完全正常的
that we can have a fully very specialized architecture 我们可以拥有一个完全非常专业的架构
just for deep learning to support convolution, 只是为了深入学习支持卷积，
multi-layered perception, long-short-term memory, 多层感知，长期短期记忆，
but GPUS, you also want to have support for those 但是GPUS，你也希望得到那些支持
scientific computing or graphics, AR and VR. 科学计算或图形，AR和VR。
So that's a difference, first of all. 首先，这是一个区别。
And TPU basically is a ASIC, right? TPU基本上是一个ASIC，对吧？
It's a very fixed function but you can still program it 这是一个非常固定的功能，但你仍然可以编程
with those coarse instructions so people from Google 那些粗略的指示让人们来自谷歌
roughly designed those coarse granularity instruction. 大致设计了那些粗粒度指令。
For example, one instruction just load the matrix, 例如，一条指令只加载矩阵，
store a matrix, do convolutions, 存储矩阵，做卷积，
do matrix multiplications. 进行矩阵乘法。
Those coarse-grain instructions 那些粗粒指令
and they have a software-managed memory, 他们有一个软件管理的内存，
also called a scratchpad. 也称为暂存器。
It's different from cache where it determines 它与确定的缓存不同
where to evict something from the cache, but now, 在哪里从缓存中驱逐某些东西，但现在，
since you know the computation pattern, 既然你知道计算模式，
there's no need to do out-of-order execution, 没有必要执行无序执行，
to do branch prediction, no such things. 做分支预测，没有这样的事情。
Everything is determined, so you can take the multi of 一切都是确定的，所以你可以采取多种方式
it and maintain a fully software-managed scratchpad 它并维护一个完全由软件管理的暂存器
to reduce the data movement and remember, data movement 减少数据移动并记住，数据移动
is the key for reducing the memory footprint 是减少内存占用的关键
and energy consumption. 和能源消耗。
So, yeah. 是的。
Mobilia and Nobana architectures actually I'm not quite Mobilia和Nobana架构实际上我并不完全
familiar, didn't prepare those slides, so, 熟悉，没准备那些幻灯片，所以，
comment it a little bit later, no. 稍后评论一下，没有。
Oh, yeah, of course. 哦，是的，当然。
Those are always and can certainly be applied 这些总是并且当然可以应用
to low-power embedded devices. 低功耗嵌入式设备。
If you're interested, I can show you a... 如果你有兴趣，我可以告诉你一个......
Whoops. 哎呦。
Some examples of, oops. 一些例子，oops。
Where is that? 哪里是？
Of my previous projects running deep neural nets. 我之前运行深度神经网络的项目。
For example, on a drone, this is using a Nvidia TK1 例如，在无人机上，这是使用Nvidia TK1
mobile GPU to do real-time tracking and detection. 移动GPU做实时跟踪和检测。
This is me playing my nunchaku. 这是我演奏我的双节棍。
Filmed by a drone to do the detection and tracking. 由无人机拍摄进行检测和跟踪。
And also, this FPGA doing the deep neural network. 而且，这个FPGA做深度神经网络。
It's pretty small. 它很小。
This large, doing the face-alignment and 这个大，做面对齐和
detecting the eyes, the nose and the mouth, 检测眼睛，鼻子和嘴巴，
at a pretty high framerate. 在相当高的帧率。
Consuming only three watts. 只消耗三瓦特。
This is a project I did at Facebook doing the 这是我在Facebook上做的一个项目
deep neural nets on the mobile phone to do 深度神经网在手机上做
image classification, for example, it says it's a laptop, 图像分类，例如，它说它是一台笔记本电脑，
or you can feed it with an image and it says 或者你可以用图像喂它，它说
it's a selfie, has person and the face, et cetera. 这是一个自拍照，有人和脸，等等。
So there's lots of opportunity for those 所以那里有很多机会
embedded or mobile-deployment of deep neural nets. 嵌入式或移动部署深度神经网络。
No, there is a team doing that, 不，有一个团队这样做，
but I cannot comment too much, probably. 但我可能不会评论太多。
There is a team at Google doing that sort of stuff, yeah. 有一个团队在谷歌做那种东西，是的。
Okay, thanks, everyone. 好的，谢谢大家。
If you have any questions, feel free to drop me a e-mail. 如果您有任何疑问，请随时给我发电子邮件。