你好再次 Hello again,
啊 ah,
又是丹尼斯·视窗 this is Dennis Windows again.
所以在看完社论的第二部分后 So after going through the the second part of the editorial,
侧重于分布式和联邦化 which focused on distributed and federates,
这是学习 it's learning.
在本部分中 In this part,
我将开始讲如何应用分布式机器学习 I will basically start talking about how we can apply distributed machine learning,
拥有无线网络 owned wireless networks,
持续的无线边缘网络 constantly wireless edge networks.
当然 Of course,
我会讲到无线网络中的分布式联合学习 I will talk about distributed federated learning across wireless networks,
但在那之前我会的 but before that I will.
我还想谈谈跨无线网络的分布式推理 I would like to also touch upon distributed inference across wireless networks.
并分发培训 And and distributed are training,
如果它在联邦设置中进行联邦 就不一定了 not necessarily if it federated in the Federated setting,
我将阐明这两者的区别 which I will clarify how that differs.
最后 And and finally,
我会讲一下我们如何通过无线网络与分布式学习相结合 I will talk about how we can do over the air are federated to distributed learning,
我会解释这是什么意思 which I will explain what that means.
好吧 Okay,
所以 so,
我们都知道 as we all know,
神经网络不能很好地应用神经网络算法 we cant neural network algorithms to apply neural network.
我们需要模型 Models we need to.
我们有两个相 we have two two phases right,
首先我们需要一个训练过的神经网络 so we first need to a trained neural network.
这就是我们在本教程之前主要关注的内容 so that's what we mainly focused on in the previous parts of the tutorial.
但是还有 but then there's,
或者是推理部分 or there's the inference part which is in a way,
比培训要求要低 a less demanding than the training.
的培训 The training,
当然 of course,
必须通过 has to go through.
整体 The whole.
数据集需要大量的计算 这在分布式设置中非常具有挑战性 data set a lot of computation and this very challenging to do in the distributed setting.
但当我们在下一个工作中谈到无线边界时 But when we come to the wireless edge next work in particular,
我们的影响力也具有挑战性 our influence can also be challenging.
首先让我告诉你 and so let me first you know,
就像我说的 as I said,
我首先要说的是 I would first talk about.
关于分布式推理然后继续讲分布式训练的问题 About distributed inference and then move on to the distributed training overwhelmed Surface OK,
所以当我们尝试使用神经网络来影响无线网络时我们面临的一个挑战就是Ios设备的计算资源非常有限 so one of the issues that we can have one of the challenges we can have when we try to use the neural networks for influence on at the wireless edge is the complex that's almost Ios devices have had limited computational resources.
当然 and of course,
我听说我有一个夸张的例子 你知道小Ios他现在的答案 I hear I have an exaggerated example that you know tiny Ios his answer to now.
如何在湖南型号上部署复杂的定界设备 How can we deploy complex at the Hunan models on such a complex delimited device?
所以 一个解决方案 So one solution,
当然 of course,
你知道的 you know,
就像任何复杂的设备 like depending of whatever the complex to afar device,
无论神经网络有多大 whatever the size of the neural network,
它允许 it allows,
所以我们可以训练这种规模的神经网络 so we can train the neural network of that size.
当然 Of course,
ice的表现将会受到限制 ice performs is going to be limited,
但这将是最好的方法 but that will be the best effort kind of approach.
还有一个 There is another,
越来越先进 more and more advanced,
更多的信息技术被称为我们的知识蒸馏 more informed technique which is called our knowledge distillation.
这个想法是 So this idea is,
这是基于我们可以训练空气和更复杂的神经网络的想法 用它来从网络中转移知识 this is based on the idea that we can actually train air and more complex at neural network and use that to transfer the knowledge from that network.
对它来说网络更浅显 to it were more shallow network.
所以这个知识转移 它完成了它 So this knowledge transfer and it turns out it performs it.
与直接训练和训练浅网络相比 它的性能要好得多 it gives a much better performance than directly the training and training the shallow network.
因此 这可能是一个更好的方法 并在实践中使用 so this could be a better approach and is used in practice.
当然 And of course,
另一个免费的解决方案必须使用您定制的硬件 another complimentary solution has to do to use you custom hardware.
这就使得 so this would allow actually day,
尽管设备的贫瘠程度有限 even though the device's impoverish are limited,
它会偷走你的表演 it could steal your perform much,
越来越复杂的网络 much more and more complex networks.
有许多商业产品周围基于FPgs的例子和微软洗脑或xilinx珠穆朗玛峰或埃塞克斯 There are many commercial products around based on F Pgs examples and Microsoft brainwashed or xilinx Everest or Essex,
谷歌提示使用 直到nirwana和我烧掉更多 Googles tip use until nirwana and I burn through more.
这些都是设计用来运行神经网络的设备它们都能有效地进行推理 So these are all devices that are designed to run a neural network are inference efficiently so they can do.
这些类型的设备可以在边缘节点上实现 而且很快就能实现 These type of devices can be implemented on edge nodes and still achieve quite soon,
不像 not as big as,
当然你和她一样知道皮尤关于我们的属性的状态 of course you know as much as did she Pew's state of the attributes we have,
但它们仍然可以运行相当复杂的神经网络 另一种可能是使用定量 but they can still run quite complex neural networks and another possibility is to use quantitation.
这是还原的 So this is the reduced.
权重的精确度 the precision of the the weights,
所以安 so Ann,
也是两个 and also both.
的内存需求 The memory requirement,
安迪 罗迪克或者操作复杂的操作处理在边缘设备 Andy Roddick or operation the complexity of the the operations there processing at the edge device,
所以我们可以把速率 激活和匆忙都称为束缚 so we can call ties both the rates and the activations and rush.
这是一个非常活跃的研究领域 This is a very active research area,
它们令人印象深刻 and they are quite impressive.
结果你几乎知道 Results almost you know,
在妥协的时候不会有性能损失降到很少的一些速率和激活 结果会更好如果你真的优化了珍珠层的定植 are no performance loss with while compromising are down to a few bits of you know the rates and activations and and the results can be much better if you actually optimize the colonization of Pearl layer.
另一个 And another,
我们的方法是网络培训 our approach is network training.
这就是我们可以训练一个复杂的网络我们称之为完整的网络然后逐渐地移除连接或权重 So this is the idea that we can train a complex network and what we can call it a full network and then gradually remove the connections or the weights,
或者信道是在网络的依赖和再训练的蓝图网络 or the channels are in the network of depending on an and retrained the blueprint network.
再一次 And again,
结果是 我们可以将权重削减到大约1% 而且几乎不会造成性能损失 there are results that we can get pruned down to maybe one per cent of the weights and almost without performance loss.
这是 So this is,
当然 of course,
这些入侵者 are these intruders.
引入它的大多数技术都增加了复杂性 因为通常需要训练一个复杂的网络 然后降低维数 Most of these techniques introduced it increase the complexity because you typically need to train a complex network and then reduce the dimensionality.
但只要我们能做到 But as long as we can do this,
训练部分在一个更强大的计算机上 然后部署结果在你和一个边缘设备 that the training parts on a more powerful computer and then deploy the result on thee and on an edge device.
这些都是圣经中的技巧 并在实践中使用 So these are all the Bible techniques and and used in practice.
Ios设备 for Ios devices.
好吧 Okay,
另一种可能性 so another possibility,
当然 of course,
如果你仍然 is you know if you still,
你知道 我不能真的 you know like I cannot really,
把他们的网络缩小到你那里 and reduce their the size of the network to thee,
达到我们想要的水平 to the level we want,
或者可能是北朝鲜还没有复杂的管理能力 or maybe they're the north hasn't even had the complexity to run.
这就是网络 That's network.
另一种选择是做联合装置一个深入的边缘推断 Another option is to do joint device an edge inference so deep.
所以这里的假设是你知道我们可能会有一些优势朝着更少的空军基地站有所需的计算能力和附近的传感器 So here the assumption is that you know we might have some edge towards less air base station with the required computational capable of capability and the nearby sensors,
例如 for example,
它们能做的就是运行神经网络的某些层然后停止运行神经网络的其他部分 what they can do is that they can run some layers of the neural network and they can stop and instead of running the rest of the neural network.
他们实际上发送了任何东西 and they actually sent whatever.
将这一层的输出输出到边缘 边缘运行其余的推理 Did the output of this layer to the edge which runs the remaining of the inference.
然后得到结果 And then I get the result.
这很有趣 因为实际上很多著名的非常复杂的神经网络 So this is this is interesting because actually many of the well known very complex neural networks for example,
居民五十岁 the resident fifty.
这里有个例子 如果你看一下数据 Here's an example If you look at the data,
它们通常的做法是取出输入然后将其扩展到高维 what they typically do is that they take the input and they expanded to a high dimension.
如果你观察剩余块的输出在早期阶段 So if you actually look at the output of the residual blocks and in the early stages,
实际上 您将不得不发送比输入更复杂的东西 you would actually end up having to send something much more complex than the input.
所以有很多 so you have many,
与输入大小相比 有更多的变量 many more variables company compared to the input size,
所以在博士阶段只做了几层 so in a way you know to be running only a few layers at Phd,
明智最终会需要大量的沟通 Wise would end up requiring a lot of communication.
但如果你运行了几次 通过更多的层 But if you run a few pass a more layers,
你实际上减少了通信需求 you actually reduce the communication requirements,
我们需要出售的信息量 the volume of the information that we need to sell,
但有 but there is,
当然是在这些上面 of course are on top of these.
这些特性并不一定是独立的价值 These features are not necessarily just independence are values.
有潜力 there's potential.
它们之间存在巨大的相关性 可以被压缩 there are huge correlations which can be compressed.
所以最初的工作是看这个问题 So the initial works are looked at this problem,
假设 assuming that,
例如 for example,
你有一些有限的泄露你可以从设备传输到观察者 You have at some finite are betrayed that you can transmit from the device to the to the observer,
我们的目标是降低我们需要传输的信息的复杂性通过分歧点和一些工作他们最初使用的工作 and the goal was to reduce the complexity of what did the information we need to transmit over the split point and some works the initial works they used,
例如 for example,
标准图像压缩技术jpgepic2008 the standard image compression techniques jpg epic two thousand and eight and this for,
例如 for example,
对于使用的谐振型网络如果目标是 for the resonant type of networks used for if the goal is to do,
例如 for example,
一些图像识别任务的图像处理 image processing for some image recognition task,
我们知道 通常这些中间层 we know that typically this intermediate layers,
输出也是电视的一些特征 outputs are also some features of TV.
所以它们看起来像图像的起源 So they looked like the origin of image.
所以这些技术草药在某种程度上是有效的 And so these these techniques herbs are effective to some extent,
所以他们做的 so they did.
他们能够减少必须传输到边缘服务器的数据量 they were able to reduce the amount of datum that had to be transmitted to the Edge server.
然而 However,
一般来说 in general,
当然 of course,
我们不知道这种为图像设计的压缩技术对未来的压缩是否理想 we don't know that this compression techniques designed for images are not ideal to compress this future so,
所以我们可能需要设计 and so we might need to design.
压缩方案是专门针对这些特性设计的 Compression schemes are specifically designed for those features.
但如果你仔细想想 But if you think about it,
这个问题 this problem,
假设你知道 let's say you know,
比如你运行一些神经网络得到一些中间特征图我们需要传输到ad网络 Like you run some neural network and get some intermediate feature map and we need to transmit to the to the ad network.
这实际上就是我们所说的联合源信道编码问题 This is actually what we call a joint source channel coding problem,
所以我们 so so what we,
在压缩if这个特性到尽可能少的方法的基础上 on top of your compressing the if this feature mapping to as few methods as possible,
我们还需要进行信道编码并传输这些比特 we also need to do the channel coding and transmit these bits out of the challenge,
我们知道一般情况下 and we know that in general,
即使在实际中我们也使用分开的或压缩方案和信道编码技巧 are even though in practice we use separate in or compression schemes and channel coding skills.
我们知道 We know,
尤其是在短街区 especially in the short block,
像政权 like regime.
这是一个微妙的地方 this is this a subtlety,
所以我们可以通过加入源信道编码和 so we can actually gain by doing joined source channel coding and.
他发现 如果他看着这个瓶子 And he saw that if he looked at this bottle,
净R加上做功 net R Plus plus work,
所以在这里我们看到他们所做的就是设计压缩方案只是设计一个基于未来压缩的神经网络或者在某种程度上了解了未来映射的最佳还原尽可能地符合他的适合度 so here we see so what they did was to design just the compression scheme just to design a neural network based future compressions or somehow learnt the best reduction of this future mapping to his fitness as possible,
然后通过信道传输 but then transmit them over the channel,
运用一些沟通技巧 处理瓶颈文件 using some communication skills and under the bottleneck papers.
基本上 Basically,
把这个作为联合来源 look at this as a joint sources,
编码问题和设计两个传输与特征维数降低一起实现了稍微好一点的性能 coding problem and designed both the transmissions came together with the feature dimension reduction and achieved a slightly better performance,
但是在我们的研究中 but in this work and we looked at this,
我们又解决了这个问题 we took the problem again,
把关节看作是记录问题 looked at as a joints which are recording problem,
但最重要的是 but on top of it,
我们也用修剪来降低未来网络的复杂性 we also used pruning to reduce the complexity of the futures networks or at that,
所以在设备上运行的初始网络 我们能够 so that the initial network that is run at the device and we were able to,
与协同效应或chunky编码方法一起 在一个数量级上减少计算自己设备的计算量 together with the synergies or chunky coding approach and reduce the amount of computing own device computation on an order of magnitude.
所以他站在同等数量的竞争对手面前 获得了巨大的利益 So he stood before the same amount of competition as a huge gain.
为什么我可以实现从一天的信道带宽 Why can I achieved from the day the channel bandwidth?
输入的数量是你需要通过信道传输的符号 The amount of coming in are symbols that you need to transmit over the channel.
所以 So,
是的 yeah,
所以基本的想法是 so the basic idea is to kind of you know,
再次使用 to use again.
你知道的 You know.
比如更先进的沟通技术 专门用于潜在推理的目的 Like more advanced communication techniques that are specialized for the purposes of the underlying inference,
图斯克 tusk,
也有局部的情况 not there are also cases that are local.
推断可能是不可能的 Inference might not be possible.
前一个也有 it also in the previous one.
局部推断是不可能的 因为复杂的计算限制的设备 the local inference was not possible because of the complexity of the computational limitations of the device.
但即使有时我们假设你有一辆智能自动驾驶汽车它可能搭载了非常强大的处理器 But even if sometimes let's say you have a smart autonomous car which might have actually quite capable processors on board,
但是由于缺乏完整的信息 局部推断仍然是不可能的 but it's still it might not be possible to do local inference because of the lack of full information.
例如 For example,
这款车需要做一个决定你知道 街上的行人可能需要考虑地形 this car to make a decision about you know pedestrians on the streets might actually need are the terrain,
附近基站可能提供的信息 information that may be available on a base station nearby,
或者真正从不同的角度来看 那是可以和其他车一起召回的 or the real from a different angle that's available and other car and have the recall.
在这种情况下 So in this case is you know,
比如需要将一些信息传输到边缘服务器这样才能做出更明智的决定 like maybe this is some information has to be transmitted to the edge server may be true to make a more informed decision and of course,
延迟成为一个主要的挑战 latency becomes a major challenge.
其中一个例子是 So one example of this is,
例如 for example,
不成熟的麻烦 所以它可以适用于任何一种水合物的邪恶 immature trouble so it can be applied to any kind of hydrates evil.
但这里通过安全摄像头进行个人身份识别的人都是人 But here for personally identification through a security camera that takes pictures of the people are people.
我们想要确定这些人这些图像是否属于某个人或者某个人存在于数据库中这个数据库是X服务器 and we want to kind of determine whether these people these images belong to someone or some other people that exist in a database that is an X server.
现在这个图像需要被传输 So now this image needs to be transmit.
一些关于任务的信息将通过无线信道传输 决策将在这个边缘服务器上做出 Some information about the mission is to be transmitted over the wireless channel and the decision is to be made at this edge server.
由于设备无法访问这些数据集 因此无法在设备上做出决定 The decision cannot be made at the device because the device doesn't have access to these datasets.
所以标准的方法是尽可能高质量地传输图像 So the standard approach would be to transmit the image as high quality as possible,
我让你们知道如何在边缘服务器上做格朗德检索算法 and I make you know to do gerande retrieval algorithm at the edge server.
但这是你们知道的 but this is you know,
就像这样 它的性能会很差 因为检索时 like this is this will perform very poorly because for the retrieval,
我们实际上不需要他所有的不相关的背景信息 所以你知道 we don't actually need all the kind of his background irrelevant information so that the ideas you know,
第一个想法是我们实际上需要一个面向检索的压缩方案 the first idea that we actually need a retrieval oriented compression scheme.
所以我们需要把图像压缩到尽可能少的位 So we need to compress the image to thee as few bits as possible,
也就是说 that is,
这些是检索任务中非常重要的部分 those are the bits that are really essential for the retrieval task,
所以我们使用 so we use.
你知道的 You know,
我们可以用神经网络来做这件事 we can use neural networks to do that,
但是这里 but here you know,
加上未来的编码 together with the future encoding,
我们还需要培训我们的员工来压缩你们的数量 we also need to train our have to compress the quantities you have.
补偿和压缩部分都增加了 both the compensation and the compression parts add,
其中也有哪家还用了暑期机器学习 which is also which also use a summer machine learning.
所以 So,
最重要的是 on top of it,
正如我们之前看到的 as we seen before,
我们也可以 we can also do.
欧盟也可以考虑这个问题 the EU can also treat this.
谢丽尔·科利的问题 as a joint saw Cheryl Corley problems,
而不是压缩成比特 rather than compressing into bits,
我们可以直接映射这些图像 we can actually directly map that the images,
该特征到信道符号和接收机上的此噪声子应符号可用于恢复 the features to the channel symbols and at the receiver of this noisy children shall symbols can be used to recover.
图像的概念和这种联合方法能更好地塑造形状 the idea of the image and this joint approach does much better shape,
正如人们所预料的那样 as one would expect.
事实上 在你提高表演水平的同时 as and actually on top of your improving the performs,
它还使用优雅的退化 it also use graceful degradation.
我的意思是 What I mean by that is that if,
由于某些原因 信道质量低于我们设计的系统 所以这种方法仍然会失去一些准确性 for some reason the channel quality goes below what we designed our system for so this approach still performs loses some accuracy,
而数字方法则失去了一切 whereas the digital approach loses everything.
所以如果你不能成为普通法院 So if you cannot be court the General Court,
你不能得到任何特征向量 也不能做任何事情 you cannot get any of the feature vector and you cannot do anything.
这就是结果 so this is the the results here,
例如 for example,
假定这种联合或某种编码方式对一个人进行了两种不同的操作方式 具有不同的复杂性 assume this joint or some coding a person on a two different ways of doing it with different complexity.
但基本上它的表现几乎接近完美的交流 But basically it performs almost close to the perfect communication,
就好像信息和图像的可用性 as if the information the images availability,
如果服务器直接 if server directly,
即使在地球上类似的深度DBS水平上 Even at similar levels of a few deep few DBS in Earth,
而数字方法即使假设咀嚼信道码的容量 whereas the digital approach even assuming capacity chewing channel codes,
我不能 I cannot,
你知道的 you know,
远远低于性能 我甚至没有它的结果 也没有发送图像和使用边缘服务器上的识别基线因为这实际上需要达到性能 is far below the performance and I don't even have the result for it or sending the image and using the identification baseline at the edge server because that actually took to achieve any performance here.
它将需要db间谍记者 大约是3040db it would requires db espionage journalists and are around thirty forty db.
所以这是不可比较的 So it's not even comparable.
好吧 Okay,
现在 So now,
在讲了一点我们对无线频道的影响后 having talked a little bit about our influence across wireless channels,
现在我想把话题转到分布式训练上讲一下我们如何通过无线网络来训练我们的神经网络 I would like to now switch to distributed trainings of how we can train our neural networks across wireless networks,
在讲联邦之前我想讲一下 and I would like to talk about before going to federate.
这是学习 It's learning.
我还想谈谈我们所说的分布式艺术培训 I would like to also talk a little bit about what we call the distributed art training,
所以我的意思是这是一些稍微不同的情况 so the so what I mean by this is slightly different scenarios.
我假设 I am assuming that,
与联邦设置不同的是 as opposed to federated setting where devices,
每个设备都有自己的本地数据集 each device has its own local datasets,
所以也有可能作为一种设备 so it could also be possible that as a device,
例如 for example,
监控摄像头可以收集大量数据 a surveillance camera collects a lot of data.
但它对帕默没有任何处理 But it doesn't have any processing to Palmer.
它想利用一些边缘服务器来进行计算 And it wants to utilize some edge servers for doing computing for it,
实际上可以有多个aix服务器 and that there can be actually multiple aix servers,
特别是随着无线网络在基础设施中的密集化 especially with the densification of wireless in infrastructure.
所以计算服务器可能有几个优势 so there might be several edge a computing servers,
而这个边缘装置利用他可能是有益的 and it might be actually beneficial for this edge device to utilize him.
这些服务器同时进行培训 这一天 Several of these servers simultaneously effort to do the training and here the day,
当然 of course,
这不会增加计算速度 this will do not increase the computational speed.
用更多的计算资源进一步编写 Further written with more computational resources,
它也可能眼睛 and also it might eye,
引入一些隐私和安全可能会有用 因为 it might be useful to introduce some privacy and security because,
而不是单一地理解所有的数据 rather than understanding all the data to a single,
我设计了以智能的方式传播你的数据 I devised by spreading your data in an intelligent way.
你现在可以保护你的数据不受单一服务的影响 You can actually protect your data against a single service now,
当然 but of course,
这带来了一些挑战 this introduces some challenges,
由此产生了一些有趣的研究问题 and so there are some really interesting research problems arising from that.
在切换到联邦学习环境之前 我想稍微讲一下这个问题 So I would like to talk a little bit about this before switching to the federated learning setting,
因此 一些挑战是服务器的分散 so some of the challenges are straggling servers.
这里 So here,
在这种情况下 in this case,
这些服务器都不会有 none of these servers will have.
当然 Of course,
你知道的 you know.
异构的计算资源或者其中一个就可以了 Heterogeneous computational resources or one of them can be happy,
可以有更具体的计算能力或者在那个时候有更多的资源 can have more concrete computational power or have more resources at that moment,
这些东西也会以一种讽刺的方式改变 and these things would also be changing in a sarcastic manner,
根据服务器在特定实例上的计算负载 depending on the computational load of the server at that particular instance,
所以当然 and so there is of course,
圣西罗的通信瓶颈 the communication bottleneck in the San Siro.
还有 There is also that,
如果这些是无线频道 if these are wireless channels,
信道可能在某个时候会褪色 阻塞 甚至进行计算 the channels might be at some point there can be faded blocked or even the computation is carried out.
它可能无法将结果一直传送到设备上 It might not be able to deliver the result to the device all the time,
当然 and there's of course,
隐私及安全方面 the privacy security aspects,
甚至 如果 or even if,
例如 for example,
我可以让它不受任何服务器的攻击 I can make it as secure against one of the servers,
的服务器 the servers,
我的同谋 好让我找到它 my collude so I can find it.
我可能需要引入更高级别的安全性 I might need to introduce a higher level of security.
所以现在我想知道 So now I would like to know,
就像简单介绍一下我们的想法你知道可以用来做那件事的一些技术 like give a brief kind of our idea about you know some of the techniques that can be used to do that.
因此 我想看一个例子 关于线性回归问题的争论 So for that I want to look at as an example the dispute with linear regression problems.
我在用户设备上有这个数据集 So I have this data dataset at at the user device,
X变暖延伸 X warmed to extend,
我可以把它们输入这个矩阵 and I can inform them into this matrix.
我的目标是 And you know my goal is to you know.
到最后 To the end,
李的平方线性回归 Li's squared linear regressions.
我想让这个参数向量最小化平方误差 I want to come up with this parameter vector that minimizes the squared error,
我们可以应用梯度下降法 and I have that we can apply gradient descent.
如果我听到梯度下降的声音 If I hear like right down the gradient descent,
所以我得到的结果就是每次我需要计算这个值的时候我们可以看到 so what I will get is at every time I need to compute basically this value now we can see that you know,
就像这个部分X转置Y在成分上是不变的不是仅仅依赖于数据的迭代 like this Part X transpose Y is invariant across ingredient isn't iterations with just depends on the data,
所以我可以假设这是计算了1美分 保留了设备 so I can assume that that's computed one cent and retained the device,
我需要计算的是 and what what I need to compute,
每次迭代都是这个乘法 that every iteration is this multiplication and,
当然 of course,
根据数据集 depending on the data set,
就像我说的 as I said,
你知道的 you know,
这可能是一项艰巨的are计算任务 可能超出了特定边缘设备的能力 this could be a formidable are computation task and maybe beyond the capabilities of that particular edge devices,
现在 and now,
如果它想分发这个 一个羊角面包边缘服务 if it wants to distribute this a croissant edge service.
一种看待它的方法是我们可以把它看作一个矩阵和一个向量的乘法 So one way to look at it is that we can actually treat this as a matrix and a vector multiplication or again.
它也是如此 So did it.
我们假设这个乘法是一次性完成的 Let's assume this multiplication is done at once.
现在我有这么大的我们要做的矩阵 So now I have this big that we'll do matrix that I want to.
每次我想传输的电流都是模态矢量 Every time I want to transmit at a multiply with the currents are modal vector and what one possibility,
例如 for example,
如果我有三个边缘服务器可用 if there are three edge servers available to me,
我可以把这个矩阵分成三部分把前几行修改成第一次的那一行 I can divide this matrix into three parts are revising the first couple of rows as the one the first time,
度量 等等 metrics and so on so forth,
我会把这些部件送到服务器上 最好每次你们成为一的时候 and I would send these parts to the servers and best every time you are one.
一旦它们到了茶叶里 它们就会繁殖 Once they get to the tea they can multiply.
每个都可以乘以磁盘上的1/3矩阵 Each can multiply with the one third of the matrix on our disk,
在某种程度上是这样的 this in a way.
在理论上 in theory,
这可以让我的速度加快三倍感谢使用这三个计算服务 this could give me three times speed up our thanks to using this three computational service.
当然 But of course,
你知道的 you know,
就像我说的 like as I said,
你知道的 you know,
比如有时可能是由于此时服务器实例不可用的组合能力或通道不好 like sometimes is possible that either because the composition capabilities at that moment that the instance of the server is not available or the channel is not good.
所以我可能无法从服务器获得响应 so I might not be able to get the response from the server,
这可能会增加延迟因为如果我这样做 and that would actually add delay might because if I'm doing this,
你在等待所有的计算来进行迭代 you're waiting all the computations to do the iteration,
这将延迟整个操作 so this will delay the whole operation.
屏幕上 so the screen again,
沟通将成为瓶颈 the communication will become the bottleneck.
所以另一种选择是我可以在计算中引入编码这个矩阵向量乘法的计算 So the alternative to that I can actually introduce coding into the compute this computation of this matrix vector multiplication.
它提出了这篇获奖论文 It was proposing this award winning paper,
似乎作者认为 as though the authors suggested basically,
不是把矩阵分成三部分 instead of dividing the matrix into three,
我可以分成两部分 I can divide into two,
但我可以 but I can.
你知道的 You know,
比如两个服务器各有一个部分 这两个部分的和 like you one part to each of the two servers and the sum of the two parts,
将编码版本发送到第三台服务器 a coded version in a way to the third Server and this way,
基本上 basically,
即使从其他两个服务器得到的结果足以恢复所有计算 even if the stories out the results from the other two servers are enough to computing to recover all the computation.
那么我们现在是什么 So so what are we now?
你知道吗 You know?
就像交易 Like trading off,
好像你知道 it seems like you know,
这种艺术 this kind of art,
健壮是免费的 the robustness came for free,
但事实并非如此 but that's not really the case.
如果你注意到 If you notice now,
每个服务器需要做一半的矩阵乘法 而不是想 each server needs to do half of the matrix multiplication rather than want,
左右 or so,
我们要交换的是 what we are trading off there,
我们在支出上引入了冗余计算 以回报一些健壮性 以应对分散的服务器 we are introducing redundant computations at the expenditure at the in return of some are robustness against straggling servers.
所以这个行动 So this action,
当然 of course,
这是一个非常简单的例子 this is just this very simple example.
这可以扩展 This can be extended.
所以有很多不同的密码人们一直在工作 So there are many different codes and people have been working,
所以这是一个非常 so it's a very,
非常令人兴奋的 very exciting,
很有趣 very interesting.
我可以研究区域吗 Can I research area?
我将展示一些 I will show some.
这里有一些参考资料 Here are very few references in a bit,
但还有另一种方法 but there is another approach.
我们也可以不把它当作矩阵乘法 We can also do not treat this as a matrix multiplication.
我们也可以把它写成今天的外积 We can also write it as the outer products of the day,
第一个矩阵的列 the columns of the first matrix.
与度量盈余的角色 With the roles of the metric surplus.
正如我们所知 As we know,
我们知道两个矩阵 we know that two matrices,
我们可以用内部产品 也可以用祭坛产品 we can either use the inner product or the altar product.
在超产品的情况下 In the case of ultra product.
现在我需要 Now I need to,
我需要作为回报的小动物和矩阵外部产品和他们的儿子会给你结果 I need as return the small animals and matrix outer products and their son with you would give the results.
这样的话 So this way you know,
实际上 我现在可以发送这些alter产品并收集结果 I can actually now send out these alter products and collect back the result.
但你知道 But you know,
如果这是一种引入健壮性的替代方法 if it's an alternative now to kind of introduce robustness,
我能解决的每个服务器的问题 what I can solve the issue of each server,
我不能对每台服务器进行两次计算即使其中一台服务器的计算结果是 I can give no two computations to each server and in even if one of the results so as to what the servers do,
他们发送了一些计算结果的线性组合你们可以在这里看到 They sent some linear combinations of the computation results than here you can see,
例如 for example,
即使这个是从这两个线性组合中出来的 even if this one is out from these two linear combinations.
我可以找回我想要的儿子 I can actually recover the son that I want.
所以在某种程度上我们引入了一个编码方案 so in a way we introduce a coding schemes or,
而不是 instead of you know,
比如返回他们的比赛 like returning their competitions,
清算 liquidated,
little引用了我们的结果 the little quoted our results,
这是一个代码 用来给我们想要的竞争 which is it's a code designed to give us the desired competition.
这被称为原料服装 So this is called an ingredient clothing.
另一种方法 Another type of approach.
就像我说的 So as I said,
你知道这是一个非常活跃的研究领域 you know that this is a very active research area.
有很多叫你手臂的声音 There are many calls your arm.
许多主要依赖于分布多项式外套的不同版本 many mostly depending on distributed polynomial coats are various versions,
另外一个我没有讲的方面是我们可以通过编码来提供 and so another aspect that I didn't really talk about is that we can actually into this coding to provide,
为了保护隐私和安全不受瑞士计算的影响他们可以在不了解任何基础数据的情况下进行计算比如矩阵 as privacy and security against the Swiss computing sort of some action they can do the computation without learning anything about the underlying data are matrices,
所以他们有很多有趣的问题需要研究 and so they're yanked the lots of interesting problems there two to study.
如果有人感兴趣的话 If anybody is interested in.
好吧 Okay,
现在让我们切换到最后的联邦边缘学习 so now let's switch to finally the Federated Edge learning.
所以对领域的了解是一个特殊的术语 so this is an acquaintance with fields is a special term,
这指的是现在 and this refers to the fact that basically now,
这些设备 these devices,
在联邦学习期间 每个带一个局部数据集都在操作 during the federated learning with each with a local datasets are operating,
共享一些相同的无线媒体 are sharing some of the same wireless medium right.
这里参数服务设置是安全的如果有服务器许可证 So now there's maybe in the parameter service setting safe if there's this permit of server,
这可能是一个基站和愿望 which there could be a base station and desire to be.
移动设备是同一物理物理中的不同传感器 Mobile devices are different sensors in the same physical.
区域 Area,
现在 and now,
每次他们想要发送他们的更新到服务器做平均 他们需要以某种方式共享无线资源 every time they want to send their updates to the server to do the averaging and they need to somehow share wireless resources.
这里再次强调 And here again,
D均值是每次基站需要以某种方式选择将参与特定迭代的设备并需要在这些设备上进行一些资源分配时所忽略的异常问题 D the mean are questions anomalies assumed missed every time the base station needs to somehow choose the devices that will participate in that particular iteration and needs to do some resource allocation across these devices.
实际上有很多方法来表述这个问题 So there are many ways to actually formulate this problem.
关于这个有很多论文 There are lots of papers on this.
我并不总是能够研究每一个可能的公式 I'm not always to be able to look into every possible formulation.
所以 一个人可以 So one can,
例如 for example,
为每个迭代设置一个固定的截止日期 并尝试在截止日期内获得尽可能多的设备来传输更新 或者我们可以设置的不是我们在每个迭代的设备需求的数量 试图最小化延迟 set up a fixed deadline for each iteration and try to get as many devices to be able to transmit their updates within that deadline or all we can set as certainly like not the number of our device requirement at every iteration had tried to minimize latency.
我们可以看看能源效率 and we can look into energy efficiency.
我们也可以考虑竞争 We can also take into account the competition,
比赛时间等等 a competition time and so on and so forth.
但我想强调一下 But I would like to just kind of highlight.
主要的因素仍然是它的核心成分 The main has remained core ingredients of this compared to,
比如说传统的资源分配 let's say the usual traditional resource allocation,
我们不会像以前那样做无线网络 we won't be what we were doing a wireless network.
现在让我们假设我想安排一定数量的设备来更新一次我想尽量减少保险的延迟或者我需要做的是尽量减少我安排的用户的延迟的最大值 So now let's assume I would like to schedule a certain number of devices for updated a reiteration and I'm trying to minimize the delay in insurance or what I would need to do is to minimize the maximum of these delays across the users that I scheduled,
梅塞德斯说 Mercedes said,
就像给每个人喝茶一样 as tea now for each one.
那么 它将解决哪些问题呢 So what issues it will do?
所以他们什么时候被安排取决于他们 So when they are scheduled are depending on their.
你知道的 You know.
就像这些资源的一个来源 Like the one source of these sources,
他们会有一些速率可以上传到基站 they will have some rate that they can upload to the base station,
这将取决于它们的内在力量几十年来 它们一直是永恒的存在和分配给它们的资源它们将做的是它们首先是你的计算机 which will depend on their inner power decades they've been the eternal stayed and the resources allocated to them and what they will do is that they were first your computer.
它们当然是建起来的 Of course they're built.
它们需要一些时间来计算 They take some time to do the computation,
他们得到了梯度更新模型的更新 and they obtained a gradient update model updates,
现在他们需要把它压缩成一些欠款使用我们在教程的第二部分看到的那种压缩方案他们减少盘子的更新来处理它 and now they need to compress it into some debits using one of those compression schemes that we have seen in the second part of the tutorial and they reduce the dish are updated to deal with it,
现在要花这么多时间 and now it will take them this much time.
基站想要做的就是安排设备 这样需要的时间就能最小化 Droplets of what the base station is trying to do is to schedule devices such that this time it takes for those many devices is going to be minimized.
好吧 Okay,
在传统的用户调度中 so now in traditional scheduling of users,
我们的目标通常是在相同的速率或可能强大之间最大化吞吐量 our goal is typically to maximize the throughput either the same rate or maybe between mighty.
施加一些大胆的约束 put into some fearless constraint.
但通常情况下 But typically,
你知道的 you know,
就像我们知道我们要做的是分配渠道 以最佳利用网站或引擎盖 like we know what we would do is to allocate the channel to the best use of sites or the hood,
谁能在每次迭代中传递尽可能多的利用知识机会 who can transmit at each iteration as much as possible to exploit the intellectual opportunistically.
他们在等待良好的航道条件 They await the good channel are conditions now here,
我报价 and I quote,
基本上忘记了联合式的海上喷泉训练 basically forget the sea fountain training in a federated manners,
我们有数百个设备随机分布在500米读卡器的半径范围内 where we have hundreds of devices randomly distributed in the soul of five hundred meter reader radius.
每一次我想要施展魅力 And every time I'm trying to charm,
选择任何用户并查看整个延迟 select to any users and looking at overall latency,
他自己的研究 花了一些时间 his own research and take some time,
我进行了多次迭代 and I'm doing multiple iterations.
大家可能会认为 Now one would expect that you know,
比如说你在这里 like say you're here,
我在随机比较 I'm comparing the random place.
选择这个给终端用户关于chinulaMayer调度 Choosing this to end users with respect to chinula Mayer scheduling,
也就是最佳用户的每次时间表 which is basically every time schedule of the best user,
所以我们可以看到空气的通道实际上是这样的 so we can see that the channel of air is actually were so.
一开始很顺利 It goes very good initially.
但后来他看起来就像每天表演者掉下来的样子 But then he looked like the per day the performers drops.
所以这里发生的是由于用户到基站的距离不同 So what's happening here Is that because the users are kind of a different distances to the base station,
所以离基站较近的用户会被安排得更频繁 so the users that are closer to the base station end up being scheduled more often.
即使这会结束这些调度最终会发送更多 So even though this will end up these generally very scheduling would end up sending much more.
就数据速率而言 You know some in terms of some data rate,
这对学习性能不好因为我们不想让紧张的模型偏向可能更接近基站的用户子集 对吧 it's not good for the learning performance because we don't want to bias are strained model toward the subset of users that are may be just closer to the base station right,
所以我们可以看到 实际上随机调度在频谱效率方面可能不是很好 so we can see that actually random scheduling maybe not that good in terms of the spectral efficiency,
但是学习前的表现要好得多 but it is much better for before the learning performance.
当然 Of course,
有很多方法可以改善这种情况 there are many ways to improve this to do something in between,
其中一个就是这项工作 so one of them is in this work.
作者调查了年龄 The authors looked into age,
意识到设备 aware devices,
或者即使他们没有增加支持的成本 or even though they don't added the cost of support,
向量机 vector machines.
所以这个想法基本上是为了确保每个用户每隔一段时间就被安排好 So the idea is basically to make sure that each user is scheduled every once in a while,
这样可以确保你知道 and this would make sure that you know,
模型是这样的 Like did the model is,
比方说 let's say,
更少偏向于更好的用户 less biased towards the better users.
好吧 Okay,
这是另一种改进方法 so another way to kind of to improve.
以及通信性能 and the communication performance,
而导致更好的学习将是使用和一些层次网络架构 while resulting in better learning would be to use and some hierarchical network architecture,
但是要做联邦学习分级 but to do the federated learning hierarchical,
同时也要让交流的层次非常分明 but also to do the communication very hierarchical.
所以 So,
正如我们所知 as we know,
和无线网络 and wireless networks,
我们可以通过将电池划分为更小的电池和利用小电池基站来提高销售量 从而提高频谱效率 we can increase the spectral efficiency by introducing a small sales rising by dividing the cell into smaller cells and exploiting small cell base stations,
我们可以在那里减少一些交通 we can offload some of the traffic there.
这允许我们首先 This would allow us first,
当然 of course,
通过减少交流 to by reducing the communication.
距离减少了小单元间设备间的干扰 distances reduces the interference across devices across the small cells.
它还能让我们提高整个网络的频谱复用和频谱效率 and also it would allow us to to increase this spectral reuse and spectral efficiency of the whole network.
现在的想法是 So now the idea is,
你知道的 you know,
像这样 like so,
为什么我们现在不这样做呢 why don't we do this to do now?
层次化的联合学习 A hierarchical federated learning,
在一个小单元格内的移动用户实际上可以做更快的渐变所以他们实际上可以不经常更新他们的模型 where mobile users within a small cell can actually do much faster gradient everything so they can actually not update their models more often.
全球利用 global leveraging,
但只是在这个小细胞内 but only within this small cell.
但他们只会偶尔更新宏基站的模型 But they would only occasionally updates with the model with the macro base station.
我们有肋间分级聚集 So we have intercostal graded aggregation.
平均联邦学习的肋间模型是怎么做到的这里的结果显示 What intercostal model averaging the Federated Learning managed so here the results showed that actually,
这些食物 these foods,
你知道玛姬进步很大 you know Maggie was significant improvements.
所以婴儿 So babies,
我们不仅缺乏接近基准的性能当我们全天使用所有的数据以一种集中的方式通过使用这种内部集群G not only we get in a lack of closer to a baseline performs when we do in all day you use all the data to get in a centralized manner by using this kind of intra cluster as G,
D和整个集群的联邦平均 D and entire clusters in a federated averaging.
正如你所看到的 And as you see that you know,
当我们增加较小单元的数量时 在整个宏单元的全局平均之间进行迭代 as we increase the number of less a small cell iterations in between a global averaging across the whole macro cell,
这大大减少了他的40次延迟 so did that significantly reduces the latency associated with his forty times.
减少延迟 reduction latency.
因为我们不知道更多的重量 Because we don't know any more weight for you know,
就像那些远离银河系的用户一样 like the users that are far from the Milky Way Station,
两个将 two to offload,
采用发送他们的更新 这样他们可以做什么 我们仍然可以使用他们 adopted to send their updates so they can do and what we can still use them.
我们不需要做出牺牲 We don't need to kind of sacrifice,
他们对模型的贡献以及模型给我们的待办事项 their contributions to the model and the todos that that gives us.
我们买了更好的机型和更快的培训 our bought a better model and a faster training.
好吧 OK,
到目前为止 我们一直在看 so R so far so we were looking at.
你知道的 You know,
主要是如何提高渠道性能 Like mainly kind of how to improve the channel performance,
你必须选择频道 you have to choose the channels,
还有一种时间表 but also kind of schedule.
你知道 用户的主要想法是我们想安排每一个用户 the users in a way you know kind of the main idea is to we want to schedule every user once in a while,
所以这可能是与典型的区别 so this may be the main difference is compared to the typical.
你知道的 You know,
比如速率最优调度 like some rate optimal scheduling.
当然我们也会这样做 Of course we also do it.
在我们的传统交流中 Do that in our traditional communications,
但是使用一些联邦党人 but using some some federalists,
尴尬的几个小时和公平的日程安排 awkward aspects of it for hours and fair scheduling.
这就是主要思想 So that's the main idea,
但是回答说 but answered,
另一种可能性是我们可以对罕见的皮肤做一些更新因为每次用户在局部训练他们的外观网络时 another possibility is actually we can do some updates a rare skin because every time the user's you know like train their look network locally,
所以他们有一个更新 所以我们实际上可以看看更新 也许我们可以说 so they have an update so we can actually look into the update and maybe we can say,
好的 OK.
因此 如果一个用户有一个更重要的更新 然后另一个 So if one user has a more significant updates and then another,
也许我们想要调度那个用户 我们实际上可以使用他们在外观的北面 and maybe we want to schedule that user and for that we can actually use may be the they are true north of the look.
什么都没有 Nothing,
所以如果你 so if you,
如果你真的 if you actually.
本地只有一个设备 Locally with one device,
我们需要改变整个模型的参数 you know things that we have to really shift the overall model parameters.
也许最好还是用那个 Maybe it's best better to use that one.
因为如果其他用户认为这实际上降低了顺序 Because if some other user thinks that actually degrading order,
对于它的本地数据集来说已经足够好了 it is good enough for for its local data set.
也许真的没有意义 maybe there's no point to really,
我发送 I transmit that,
然后你知道 and then you know,
我用它来更新手机因为它想在最后改变 I use that for the mobile update because it wants to be the changed in the end,
我喜欢全球模型 I much the global model,
所以在这项工作中 so in this work,
我们会根据最好的渠道或者你知道的 and we compare scheduling users based on the best channel or the just you know,
看看这个模型你喜欢的顶级ElNorte用户 looking at the model you like the top El Norte users.
或者你知道 Or you know,
像是两首歌的组合 like some combination of the two songs,
例如 for example,
回忆 recollection,
选择一定数量的唯一性 choose a certain number of uniqueness.
这里你假设计划密匙用户 So here you're assuming that scheduled key users,
每次迭代 这样我们就能用C用的 every every iteration so that we can use case C uses,
我们可以查看用户在渠道方面的最佳情况 从这些渠道中 我们可以选择具有最高algernon的最佳K we can look into the best case he users with in terms of the channels and from those we can choose the best K that have the best in the highest algernon and every time.
或者你知道 Or you know,
就像我们不能 like we can't.
在这些情况下 一旦你安排了用户 So in these cases there is also so once you scheduled the users,
我们需要分配带宽 we need to allocate the bandwidth.
所以这些用户的资源 so somehow the resources across these users,
通常我们所做的是根据下午的厄尔尼诺值来分配带宽 and typically what we do is that we allocate the bandwidth according to the El Norte proportional to the afternoon.
所以我们想给用户更多的资源 So we want to give more resources to the user,
其中有最大的细胞隧道包含在一些日期是更准确的 which has the largest cell tunnels which contained in some dates are more accurately.
所以他的鞋子 And so the one his shoes that so,
当然 of course,
你知道在他们定位了资源之后 现在的用户被隔离 you know after they are located the resources users nowadays to quarantines,
他们的更新 their updates.
就像最后一种方法一样 And so maybe you know like it in this last approach,
也许更好的方法是观察量子尺度的范数 maybe it's better actually to look at the norm of the quantum sized up.
这些谎言所以而不是因为也许也许想用 These lies so rather than because maybe maybe wants to use.
她的论文有一个火热的高正常值 它在资源分配 Her thesis has a hot high normal at once it's allocated at the resources.
也许他的语气不好 Maybe his tone is not good,
所以它最终不能发送尽可能好的质量 so it ends up not being able to send that acted as good quality as possible.
所以我们对更新质量做了这样的研究 So we do this kind of a look into the the update quality.
啊 Ah,
好的 OK!
所以如果我们现在把这个和侵蚀计划比较 So if we now compare this to erode like schemes,
如果你有ID数据 so if you have ID data,
所以数据在这些用户中是相同分布的 so the data is in like identically distributed across these users,
我的意思是首先 it turns out that I mean first of all,
知道这个更新 know this update.
有意识的安排更好 Aware scheduling is much better,
显然和你知道的相比 obviously compared to you know,
就像看露珠一样 like just looking at either the dew.
只有渠道和规范 And only the channels or the only the norms,
就像你知道的那样 他们就能看到法定人数是多少 and and the sooner like the one that you know like they'd be able to see which looks at the quorum.
珍视的更新质量是最好的最好的一个 Prized update qualities is the best best one.
但这里另一个有趣的观察是 在这种情况下 在每次迭代中安排一个单一的设备会更好 But another interesting observation here is that in this case is sexually better best to schedule a single device at every iteration.
所以我应该选择用户如果你知道这样可以给我最好的更新最好的相对渠道 so I should choose the user if you know like that can give me the best update with the best kind of relative channel,
我每次都把它放进去 就像时间表一样 and I just get that one in every time you know like schedule.
用户需要这样的用户 Users want such user,
这主要是因为数据是眼科疾病 and this is mainly because the data is eye disorders.
在某种程度上 In a way less,
我可以通过安排道德用户和it来获益 I can gain by scheduling moral users and it.
基本上 basically,
因为我安排了更多的用户 since I schedule more users,
我做到了 I did.
他们输了 They lose.
他们需要分享数据 They need to share the data,
增加了光谱 added the spectrum,
所以他们用来建立更新质量的资源减少了所以我没有一个用户 只能得到最好的 and so the resources so they can set up the update quality decreases so rather than that I scarce with one user and get at the best,
质量更好的更新 better quality updates.
这就是结果 so that's what the result says,
然而结果略有不同 whereas the result is slightly different.
如果我不考虑蒜泥蛋黄酱的数据 If I consider none aioli data.
在这个例子中我们可以再次看到 So in this case we can see that again.
所以我现在不比较他们所有的技能 so I'm not now comparing all their skills,
因为这个更新 because this updated,
调度肯定比其他的好 where scheduling is definitely better than the others.
但我们观察到的是 现在所取得的最佳品质 But what we observe is that now the best qualities achieved actually,
如果在40种设备中 if out of forty devices,
ice在红色曲线上安排了10个设备 并安排了更多的设备 ice scheduled ten devices to the red curve and scheduling more devices,
两者都给了我更稳定的学习性能 也增加了周期性的更高的准确性比调度发烧或更多的用户 both gives me a much more kind of stable learning performance and also adds a periodic higher accuracy than scheduling either a fever or a larger number of users.
当然I一般是 Of course I in general is.
这是一个悬而未决的问题 it's an open question.
基本上 Basically,
我如何决定计划多少用户 how do I decide how many users to schedule?
也许我们可以改变他的日程 and maybe you know like we can actually changed his schedule?
你知道的 You know,
比如在训练的开始和结束的时候到目前为止 还有很多有待解决的问题 like at the beginning or towards the end of the training and so so far so there are lots of open questions remain.
好吧 Okay,
所以现在我 so now I.
最后 Finally.
我想劝他放弃另一种方法 I would like to talk him out of a different type of approach.
一种完全不同的交流方式在联邦学习环境中更新这个模型 A completely different way of communicating this model updates in a federated learning setting,
这一点和我讲的分布推断是一致的 this is a little bits are aligned with what I talked about for the distributed inference,
我们说过 基本上我们可以把它们看成是源和信道编码的联合问题而不是先压缩信息然后再进行信道编码 where we said Basically we can treat those as a joint source and channel coding problems or rather than first compressing the information and then channel coding.
实际上 我们可以通过同时做这两个任务来获益这里的情况也是这样 We can actually gained by doing the two tasks together and this is also the case here,
因为现在让我们从你们的信息来看这个问题 because now let's look at this problem in a more from your information.
从理论上看 我们所拥有的是每一个设备都有的 Theoretic perspective to what we have is at every time each device has there up.
模型更新 Model updates,
的梯度 gradients,
不管它们是什么 whatever they are,
他们想要传输到服务器的外围和外围 that they want to transmit out to the perimeter of server and the perimeter.
所有的参数都只是取它们的平均值然后更新全局模型 再把它们发送回来 All the parameters are really wants to do is to just take the average of them and update the global model and send them back.
所以现在基本上在这里 So now are basically here.
这是这是一个关于多访问的分布式计算的问题 This is this is a problem is actually distributed computation over multiple access,
一般的问题 general problem.
正确的 Right.
所以这些参数是 so so deeply the parameters are,
它并不会同时关心这些个体 It doesn't actually care about individual of this all at once.
他们的平均水平是多少 我们能做的是用数字方法代替维也纳 Is there some of their average and what we can do is instead of Vienna in the digital approach.
我们所做的是为了分析渠道资源 What we are doing is that we're kind of trying to order to analyze the channel resources,
因此 我们希望尽量减少此设备与分配的每个用户之间的干扰 so we want to minimize interference between this device and each user is allocated.
一些资源被压缩了 Some resources be compressed,
爆发并发送了这个正交的资源 erupted and sent over this orthogonal resource,
所以 so instead,
我 I,
如果我不做任何压缩 if I don't do any compression at all,
如果我可以通过信道以编码信息发送这个如果信道只是把它们加起来 if I could just send this in an encoded message over the channel and if the Channel is just add them up,
这正是我想要的 this is exactly what I want.
这就叫做空中计算 So now this is called over the air computation and,
我们会看到访问可以带来显著的改善 and we will see visits it can be can bring significant improvement,
但有一个挑战 but there is one challenge that,
而不是 and not,
如果我 if I,
如果我这样做 并称之为传输 if I do this and called the transmission,
所以这些向量通常都很大 so these these vectors are typically huge,
写在我们谈到的地方 writes where we talked about.
你知道的 you know,
大型神经网络有数百万个参数 millions of parameters for large neural networks,
这就意味着 so that that would mean because,
因为我需要 since I need,
每个维度都有一个符号 you know one symbol per dimension of this.
这些都是更新 Of these are updates,
这意味着最终可能需要大量的频谱资源 so that would spell that might end up requiring a huge amount of spectrum resources.
例如 For example,
就像这样 you know like this,
所以我说她有数百万的承诺比英国更大的网络 so I said there are millions of her promises than England are much bigger networks.
所以 so,
就像数码产品一样 as in the case of digital,
我可以压缩 I can compress right.
所以有这些爱的方法 So there are these methods of love,
例如 for example,
像科学家一样 like scientists,
我是否可以把它缩减为每维1位 should I can reduce it into one bit per dimension,
然后我就有了这么多比特 and then now I have those many bits.
通过使用一些信道编码 By using some channel coding,
我可以把它们发送到更小的光谱资源上 I can send them over a much smaller spectral resource,
但是如果我忽略了 whereas if I do overlook,
这似乎是你的生活 it seems your life.
我不清楚我该怎么做 It's not clear how I would do so,
但有一个解决办法 but there is a solution.
所以我们所看到的这是通过规范技术实现的 而不是通过对话 So what we saw and this is achieved by this specification technique rather than the conversation,
现在好了 so now OK,
所以我要传递的这些向量 so the idea is that these vectors that I want to transmit,
正如我们看到的 我们实际上可以把它们具体化 as we have seen that we can actually specify them quite a bit,
我们现在就可以 and we can we can just now.
我们的目标是发送这些载体 Our goal is to send this parcel vectors such that you know,
总理 the Prime Minister,
再来看看这些稀疏向量的和 again against the sum of those asked sparse vectors.
所以我们要做的是一旦我们指定了它们 So what we will do is that once we specify them,
我们不能把它们投影到更低的维度上 we can't be will actually project them onto a much lower dimension.
发送那些低得多的维数 也就是向量 Send those much lower dimensions which are vectors.
它们会相加 在参数服务器上 And they will add up and at the parameter server,
我会用其中的一些 I will use some of these,
你知道的 you know,
比如利用重建工具的稀疏性 like sparsity exploiting reconstruction tools like you know,
在某种程度上类似于压缩 然后说这就是我做的对的 in a way similar to compress and saying this is what I'm doing right,
一些暗的投影 根据已知这些向量是稀疏的 some dark projecting and then from that knowing that these vectors are sparse,
我强加的支持编码的 which I imposed that supported the ENCODED.
现在我可以找回他们了 Now I will be able to recover them,
即使我有你知道 even though I have you know,
就像一个理想的条件 like an ideal conditions,
有重建的问题 have reconstruction problem,
多亏了辣味 and thanks to the spicy,
我将能够重建这个知识分子 I will be able to reconstruct so for this intellectual,
这条该死的线 this damned to a line kind of properly,
我将用同样的方法 I will use the same.
如何通过投影矩阵的所有发射机 How projection matrix across all the transmitters.
这就得到 So this gives ah,
我有点不正常 I quite a bit off,
非常显著的改善 quite significant improvement.
实际上 actually,
这里我们看左手边 so here we are looking at on the left hand side,
ID数据会发生什么这是一个相对简单的分类问题 what happens with ID data so here so it's a relatively simple and its classification problem,
但是我们可以看到 与所有这些不同的数字传输相比 but we can see that compared to all these kind of different digital transmission,
以X压缩技术为例 take the X compression techniques and within this in all,
这些你知道 these you know,
像这样 like this,
三行是三 three lines are three.
汽车正在使用数字传输 cars are using digital transmissions.
我们和我安排好了 We scheduled it with me.
在三个用户之间划分多路复用信道资源 Divide the kind of to the multiplexes channel resources across three users,
每个人都挤在楼梯上 and each one compressed the stairs.
更新 Updates to that.
这里的资源所允许的速率与这个投影相反 Rate that is allowed by the the resources there are located now as opposed to this in a projection.
不像Mrs无涂层变速器 通过他们的竞争 我们几乎达到了基准性能 And unlike Mrs Uncoated transmission and over their competition gets us almost to the baseline performance,
这是通过无误差实现的 which is achieved by an error free,
共享的泄漏 shared leak.
有趣的是 And interestingly,
如果没有 改进仍然存在 the improvement is is still there if you have none,
ID数据 因为如果数据不是一个想法 ID data because one would expect that you know maybe because if the data is not an idea,
不同用户之间可能会有不同的辣度模式 当使用次数增加时 you might have different spicy patterns across users and when use add up,
经过这些补丁之后 你最终得到的可能不是稀疏的 difference past the patches you might end up getting something not sparse,
但事实并非如此 因为他不知道这些 but it turns out that's not the case because he wouldn't know this.
没有其他的 它们不是完全不同的数据 There are none else that they are not completely different data.
所以这里有某种对齐和那边的场景传输帮助巩固了这些对齐 so there's some kind of alignment and the scene on over there and transmission kind of helps to consolidate those alignments.
你知道我们仍然可以学习额外的那些 And you know we can still learn the extra ones,
它仍然是 and it's still,
你知道的 you know,
喜欢它 like it,
它是 it's,
当然 与完美的和她的免费的穿梭包相比 现在有一点保留了 of course there's a little bit kept now compared to the perfect and her free shuttling case.
但它比数字替代品要好得多 But it's much much better again Compared to the digital alternatives.
好吧 OK,
所以你知道 so this you know,
就像之前一样 like in the previous one,
我以为你知道 I assumed somehow you know,
比如高斯麦克斯韦 like a gaussian Maxwell,
无论我们传送什么木头 whatever we transmit wood,
只要我们同步传输它们就能很好地在圣西罗就位 as long as we synchronized the transmissions that they would align nicely in the San Siro.
就像所有的原料一样 Like all the ingredients,
更新只会增加 updates would just add up,
但是我们知道在无线领域不是这样的 在无线领域 but we know that that's not the case in wireless right soared in the wireless.
这些来自不同用户的传输将通过不同的信道门进行缩放 Now these transmissions from different users will be scaled with different Channel gates,
这就意味着 and that would mean that,
例如 for example,
如果用户有更好的渠道增益 if a user has a better channel gain.
如果我什么都不做 if I don't do anything,
这将意味着来自于你知道的设备更新的竞争将会更大规模然后我们将会影响最终的模型 it will mean that the competition from you know the update from that device will be scaled more and then we'll end up affecting the the final model.
苔藓 Moss.
我不希望这样的事情发生 I don't want that to happen.
在这方面也是如此 So in the way too.
绕开这个问题就是对新能源的控制 Circumvent this problem is to new power control.
所以如果他假设他的信道增益是我们估计的在发射机上他们实际上可以应用信道反演 So if he assumed that did his channel gains are our estimated known at the transmitters they can actually apply channel inversion.
他们可以抵消这些挑战的影响 and they can cancel out the effect of these challenges.
当然 And of course,
这就意味着 即使信道由于功率的限制 this will mean that even if the channel due to the power constraint,
如果这个频道太糟糕 if the channel is too too bad,
我们可能真的做到了 we might actually did.
那个设备可能选择不发送 that device might choose not to transmit,
但我们知道 but we know,
渗透到学习 infiltrated learning.
它是 It's,
没关系 it's OK.
所以我们可以在每次迭代中只安排一部分用户 So we can sound can schedule only a subset of the users at each iteration,
因为我们仍将收敛 and because we will still converge,
我们可以从音乐上看出来 and here we can see that musically.
做这个周末 Doing this weekend,
有很多不同的方法 so there are different ways of doing this,
所以你知道 so it's just you know,
比较两篇不同论文中的不同方法 comparing different methods in the two different papers.
但是总的来说 But in general again,
与数字方法相比 你的表现越好 the sooner like on a look like you're performs quite good compared to this digital approach,
下面的方式 that's way below.
就能达到的训练精度而言 In terms of the training accuracy it can achieve.
这就是说 So this this basically says,
你知道的 you know,
喜欢周末 like weekend,
我们仍然可以实现空中计算 we can still achieve over the air computation.
只要我们能 As long as we can,
我们可以确保这些更新是对齐在同一功率水平在接收器 we can make sure that these updates are aligned at the same power level at the receiver.
我还想评论一下同步 I would like to also comment about the synchronization is sure,
如你所知 as you know,
所以我们假设这些传输在某种程度上同步了联盟 so here we are assuming that these transmissions are somehow synchronize an alliance in terms of you know,
让我们把每个符号都收到 let's place each symbol are received.
一个明确的 A clear.
在许可证服务器上的相同资源元素上 Over the same resource element at the permits to server,
但你知道我们可以做到 but you know we can enough.
这是通过使用一些标准的同步工具来实现的 我们在今天的标准中有 That achieved is using some of the standard synchronization tools we have in the standards today,
因为即使是 because even,
例如 for example,
这些符号可以和这些符号一起传输 在当前设置中我们已经有一些事情需要同步 these symbols can be transmitted over with the symbols and we already have some are matters to synchronize in the current setting.
我们想通过交流来分析用户 每个用户都以一种同步的方式在不同的资源之间进行传输 We want to talk to analyze users and each has a kind of transmit across different resources in a synchronized manner,
但现在我们也可以用同样的方法来确保 but now we can also use the same techniques to make sure that we,
基本上 我们分配相同的资源 basically we allocate the same resource told users,
它们在相同的资源元素上发送相同的和组件 and they send the same and components on the same resource elements.
但这在技术上是可能的 But this is technologically possible.
好吧 Okay,
另外一点 so another points now,
在上一篇的最后 in the end of this previous,
或者他们的竞争方式 or their competition approach,
我们假设的是 what we were assuming,
我们计算这些梯度 然后把它们按原样发送出去 that we compute these gradients and we send them as they are,
所以我们不用任何星座 so we don't use any constellation.
我们不使用任何数字基础设施 We don't use any kind of digital infrastructure so,
因此 无论它们传输的实数是多少 and so whatever their value in the kind of real number they are transmitted,
但这可能不太可能 but this might not be too.
让我们看看适合系统中可用的电流是我们拥有的设备所以人们可以质疑 Let's see fitting for what is available in the system in the currents are devices we have so one can question that you know,
因为这些设备在它们的星座上有一个硬编码 because they typically the devices have a hard coded in their constellations,
然后他们从这些星座传送你的点 and then they transmit your points from those constellations,
这将不可能传达他们可能不可能传递这个真实的值 which would not be possible to convey that they may not be possible to transmit this real values.
在这项工作中 我们说 So in this work we said,
好吧 OK,
所以 如果我们做到了 so so what can be achieved if we are,
例如 for example,
约束沙子 constraint to sand,
二进制或二进制符号和设备中的符号 binary symbols or two and symbols from the devices,
在这种情况下 and in this case,
我们能做的 what we can do,
而不是发送傻瓜梯度更新 因为真正的数字现在是周末 rather than sending the fool gradient update as the real numbers are now weekend?
我们可以量化烹饪 We can cook quantifies it,
例如 for example,
GDP成为科学家 became scientists GDP.
现在每个用户都得到了每个元素的符号我们可以将每个元素的符号映射到Bps框或者在复杂通道情况下的丰富增益而不是他们会做什么 Now each each user gets the sign of each element and we can map this sign of each element to a Bps casing or copious gain in the case of complex channel and not what they will do,
他们只会 they will just,
你知道的 you know,
就像缩放这种一比特的信息 like scale this kind of one bit of information.
他们都是发射机和基站他会收到这周的总和吗 they all transmitter and the base station is he'll just receives the summation of these are this week?
我们可以使用 We can use,
当然 of course,
智利的版本 the Chilean version,
但一旦这些旧的成为Bps同性恋的象征 but once these old is kind of Bps gay symbols,
经过夏天的逃避 塞缪尔伤心地说加上一名桑德森 elided through summer saw Samuel sadly say plus one Sanderson,
负一 服务器许可会做的是做多数投票实际上现在这允许我们做空中多数豆瓣菜 minus one and what the the permit to server will do is to do a majority votes and actually now this allows us to do over the air majority watercress,
所以它们自然是无论什么所以所有基站要做的就是看接收信号的符号和向量 so they naturally is whatever so all the base station is to do is just to look at the sign of the receipt and vectors,
然后告诉哈姆雷特更新全局模型和应用程序 and then tells Hamlet that updates at the global model and an app.
这给了我们一个相当好的性能 如果与数字或除尘 this this gives us as a quite good performance or if compared to digital or dusting down.
因为在你知道之前 Because before you know,
提议的只是一个比特和传输变得非常相似 the proposed just a single bit and transmission gets very similar.
的性能 Performance.
而且 And,
然后你也知道 and then also you know,
在这里我们可以量化绩效工作 like we get you know so here we can quantify the performance work.
在aiddeJean的情况下 Pin的喂养甚至是不完美的 In the case of aid de Jean Pin feeding are even imperfect.
OSI The OSI it,
你当然知道 of course you know,
好像失去了什么 like there's some loss,
但它并不重要仍然能够学习能够收敛 but it's not significant is still able to learn able to converge.
即使是在这种一般性损伤下 Even under this kind of various general impairments.
好吧 Okay,
所以 so,
但是都在这些模型上 but are all on all these models.
我们假设发射机有完美的信道状态信息 What we assumed was that the transmitters have perfect channel State information,
他们能够以某种方式应用一般版本 and they were able to somehow apply generally version.
但在实践中 这可能很难实现 But in practice that might be hard to realize.
在没有信道状态信息的情况下 我们还能通过它们的计算 So can we still do this over their computation without in the absence of Channel State information,
结果是 and it turns out,
答案是肯定的 the answer is yes,
但是你知道 but for that you know,
我们可能需要利用发射基地的多个天线 like we might need to exploit multiple antennas at the base they shoot.
这里 So here,
例如 for example,
我们正在研究20种设备 we are looking at twenty devices.
在空中使用这个应该有一个竞争训练 Using this over the air should have a competition for a training,
如果你观察它当我们增加基站天线的数量几乎所有的外部机会信息只有在基站我们仍然可以 你知道 and if you look at it in elected as we increase the number of our base station antennas and all nearly all external chance that information only at the base station we can still you know,
我们可以通过单个天线来学习 like we can learn as they may even with actually single antenna.
基本上不需要进行任何权力控制 Without basically doing any power control,
我们仍然可以学习 we can still learn,
但是他们的船员最终的准确性是你知道它会 but their crews to the final accuracy is you know that it's gonna.
要花更多的时间去学习和谎言 今天的商业最终不会有同样的表现 Take much more time to learn and lies will not commerce today the same performance eventually.
但随着天线数量的增加 But as you increase the number of antennas,
多亏了这样的频道 thanks to this kind of channel,
突出显示效果 highlighting effect,
我们会接近误差阈值 就像基线性能一样 we will actually get close to the error threshold looked like there the baseline performance,
这说明 so this this says that you know,
我们也可以使用物质资源来弥补一般状态信息的不足 like we can also use physical resources to compensate for the lack of general State information.
这主要是因为我们用迭代的方式进行实际计算 So this is mainly because his we are doing actual computation in an iterative manner.
我们实际上并不需要每一次计算都是完美的 We don't actually need each computation to be perfect.
只要你知道 As long as you know,
就像每一次迭代都引导我们朝着正确的方向翱翔 like each each iteration steers our soared towards the right direction.
那就足够了 That will be sufficient.
好吧 OK,
我想在此结束我的演讲 so I would like to ask to conclude here.
教程的这一部分是 This part of the tutorial is,
他们没有谈到我们面临的各种挑战以及我们通过无线网络进行分布式学习的不同方法 they haven't talked about various our challenges and our different approaches to distributed learning across wireless networks,
我想强调的是 and I think that what I would like to highlight,
作为这篇文章的主要信息 as as the main message of this,
本教程的这一部分是这些非常 this part of the tutorial is that so these are very,
问题非常重要 very important are problems,
当我们想在无线网络中应用机器学习分布式学习时 and when we want to actually to apply machine learning distributed learning across wireless networks,
有许多许多开放的挑战 there are many many open challenges.
实际上 actually,
这些技术都还没有在实践中找到应用 许多实际的挑战和我们需要解决的是理解客户证明 none of these are techniques have yet to find applications in practice and many practical challenges and our need to be resolved are understanding customer prove.
所以 So,
我认为 听这个教程的人要传达的主要信息是 I think the main message for our people listening to this tutorial,
特别是对于学生 especially for students,
现在是进入这个领域的好时机很多活动正在进行 this is this is a very good time to get into this area lots of activities going on,
但我认为 至少在未来几年 这仍将是一个非常相关和热门的研究领域 but I think it will remain a very relevant and hot research area for several years to come at least,
我还想指出一些其他的资源 and I would like to also point to something that others are resources.
所以 So,
当然 of course,
在我们的网页上 一个小时后 你就可以查看我的小组关于最近锻炼的爱单 For a sheet of love from my group argues for a more recent workout can be checked an hour on our web page,
但我也想强调和即将来临 but I would also like to highlight and upcoming.
啊 Ah,
有一个关于机械 通信和网络的J信号特别系列 there's a J Signal special series on machinery and communications and networks,
在这个系列中 and in that series,
该地区的冲浪者是分布式联合学习 which ran the area that surfers are distributed federated learning.
因此 第一个问题将在明年1月很快到来 届时各方将同意任何一个方案 So the first issue is coming soon in January to agree to any one.
但有些人是我们的产品线 but some people's are the available our line.
关于如何使用这张卡应用我们的联邦学习有很多论文 And so there are lots of papers on how to use this card to apply our federated learning,
一个人的网络是很多非常好的高质量的报纸我在本教程中无法触及和其他相关的特别是鞋子远离了通讯杂志很快会再次出现 a person that networks are many very good quality papers that I wasn't able to touch upon in this tutorial and other relevant are especially shoe is far from the communications magazine to be out there soon again,
确切地说 你的话题沟通技术的有效边缘学习 exactly on the topic of your communication technologies for efficient edge learning.
我看到了房子 I saw the House.
我想指出我们的一篇论文 I would like to point to one of our papers that is going to achieve,
它出现在特刊上 and it appear in that special issue,
它涵盖了我在这里谈到的许多方面和其他方面 which covers many of the aspects that I talked about here and some others,
这是另一种更普遍的论文从另一艘相邻的船中学习无线技术 And also this another kind of paper that a little bit more general high into the Commission learning in wireless from another's recent adjacent specially ship.
所以我想感谢大家的聆听 我也很乐意在他们的培训课程中回答关于这些部分的问题 So I would like to thank everyone for listening and I will be very happy to answer questions during their training session about these parts of the tutorial.
谢谢你 Thank you.
大家好 欢迎来到大礼堂的第四部分 我要讲的是分布式学习和你们的网络压缩 Hello and welcome to the fourth part of an auditorium I will talk about in distributed learning and your network compression in this part.
所以 So.
你知道今天的神经网络的复杂性正在增长 You know that today's the complexity of today's neural networks is growing.
我们有模型 We have models which.
它有成千上万个参数需要对singer和推断过去的傻笑操作 Which have like millions of hundreds of millions of parameters and require giggle operations for a singer and inference past.
比如我们刚刚交换的模型 Such as such as models such as we just exchange,
例如 for example,
表观遗传标签或摔跤152 epigenetic tagging or wrestling one hundred fifty two.
如果我们 And if we,
如果你能在我们的应用中使用这些模型 if you would use these kinds of models in our applications on the edge,
例如 for example,
哈里森 Harrison.
文章中有不安全驾驶的汽车 Article where an unsafe driving cars,
所以你需要大量的能量来使用这些火星来冷却它们 so you would need an incredible amount of power to to use these kind of Mars and for cooling for using them.
所以在很多应用中 and so in many applications,
使用这种强大的模型是不可能的 it wouldn't be possible to two to make use of this powerful models.
但我们必须降低它们的复杂性 But we have to reduce their complexity.
特别是在分布式学习中 Especially in distributed learning,
分布设置 distributed settings,
你的资源有限 you have limited resources.
我们需要减少这些模型的复杂性 使其在有限资源的路径上运行 We need to reduce the complexity of these models in order to one to two to make the executable on on a pathway of limited resources,
所以有不同的方法来减少癌症问题不同方面的复杂性 so there are different different ways to kind of reduce complexity of the different dimensions of the problem of cancerous.
整体来看 The whole,
喜欢模型本身的大小 like the size of the model itself.
但也像它执行的复杂性 But also like the complexity related to its execution,
这取决于硬件 this depends on the hardware,
当然 of course,
还有如何实现 also how things are implemented.
所以它就像一个复合物 So it's like a complex,
不同维度的复杂问题 complex problem with different dimensions,
但首先让我们来关注压缩的问题 but an let us first and stick with the problem of compression.
所以影响 So impact.
目前正在制定一项新的冲击标准 并在 多媒体新网络压缩十七项标准 中包含七个部分 is currently developing a new impact standards and packed seven parts in seventeen standard on compression of new networks for multimedia,
棉花描述和分析 cotton description and analysis.
所以这个想法也是 So the idea is too.
这就是指定 That is to specify.
技术或开发一个工具箱 并提出一些独立的编码方法 广泛适用于神经网络 并提供一个有效和可靠的压缩 Techniques or to develop a toolbox and propose some independent coding methods which are widely applicable to neural networks and which give an efficient and reliable compression.
所以我们的想法是让这两本书适用于外部框架和格式 So the idea is to to make these two books of applicable with external frameworks and formats,
如管涌般观察脖颈的流动并开发编码方法 such as piping watched as the flow of necks and to develop coding methods,
例如 for example,
对于压缩重量 for compression of weights,
偏差或压缩能力 biases or compression power.
强度的参数 parameter of tensile.
我们需要包含某种形式的情况 Class of cases we need to include some form,
一个特定的拓扑图和格式转换成一个小时 a specific topology and into the format into an hour.
这条小溪 The stream.
但如果我们是女人 But if we are women,
我们可以证明一下 we could show to get that.
目前我们可以在不降低分类质量的情况下获得高达甚至超过95%的压缩效率 Currently we can get compression efficiency of up to or even over ninety five percent without any degradation in classification quality.
这是一种外向的主动 That's an outgoing initiative.
在那里 Where,
就像你在你的网络上有一些命令 你有不同选择的想法 like you have some ordered on your network and you have the idea of different options.
我们有三步合一这是一种预处理服装参数减少的步骤你可以应用 So we have like three steps in one step is kind of type of a preprocessing apparel parameter reduction step where you can apply,
修剪到一个规范的网络 pruning to a network of specification.
我看到了一些其他的技巧让你的母亲对压缩更友好 I saw some other kinds of techniques to make your mother more friendly for compression.
第二步就像是对话的步骤 The second step is like a conversation step.
提出了不同的方法 where a different approaches were proposed,
统一的组成叫做优先排序依赖的对话 uniform composition called prioritization dependent conversation,
如果你 and if you,
如果你想通过一个通信通道来传输你的网络 if you want to transmit your network over a communication channels for channel,
例如 for example,
你也可以在上面应用熵编码就像deepcover you can also apply entropy coding on top and here like one approach as deep covered,
我一会儿会告诉你们更多 I will tell you a bit more about that in a minute.
但你可以结合 but the idea is that you can combine.
我将使用这些技术中的不同部分 and I'll use different streams of this of these techniques,
这些都是由标准规定的 which are specified by the standard.
因此 峰会将在明年4月准备就绪 And so that the summit will be ready next year in April.
由于我的团队和我们的机构的年龄 And by my group and our institutions ages,
我已经提议 I has proposed.
Oil已经成为Ref有建议的参考软件和cityam您的网络压缩测试模型 也喜欢作为一个标准 发挥在HIV开发的技术 Oil became the Ref has had the proposal for the reference software and city am your network compression test model and also like to serve as a standard exerts technology developed at HIV,
如制服 such as uniform,
他们独立的补偿努力 their independent compensation efforts,
深切作为一种熵编码方法 deep cut back as an entropy coding methods.
如何通过定义语法或位的语法 How does syntax or syntax for bit through definition?
所以我们从一开始就参与了这个开发 有点 So we have been involved from the beginning of this development and are kind of?
他们提出了减少神经网络规模和复杂性的技术 They are proposing the technology for reducing the size and complexity of neural networks.
在第一次出版之后 And after the first time that will be published,
显然有一个呼叫基金 there is apparently a call fund.
疯子是 A madman is kind of.
它被称为关于神经网络增量压缩的建议 it's called a call for proposals on incremental compression of neural networks.
所以我们的目标是开发一种技术 So the aim is to develop a technology for.
压缩传输网络更新的主要互操作表示的使用案例 如联邦学习 Compressing the principal inter operable representations of updates of transmission networks saw use cases such as federated learning,
但不是你的网络需要被压缩 but not your network need to be compressed.
但是苔藓的更新被堵塞了 But the update of a lichen plugged up,
一个配偶 a spouse.
更新香料 Update Spice,
说服矩阵 persuade Matrix,
我需要被压缩 I need to be compressed,
或者你也可以考虑用例 or you can also think of use cases where.
在精炼或者增加更多的训练数据之后或者在转移学习之后你想要一些基本的网络你想让它适应特定的数据 There is an network update after refining or adding more training data or after transfer learning where you want to have some basic network and you want to adapt it to specific data.
或者你有一些 Or you have some,
也许有些更新 因为你的新网络有更高的精度 maybe some updates because your new network has higher precision.
所以有这些不同的融合类我们的目标是为它的下一个发展技术作为一个额外的史诗文明 So there are these different fuse classes which will be and which we aimed to develop the technology for its next as an extra amount of epic civilization.
好吧 Okay,
我简单介绍一下这项技术 so a few words about this technology about deep cutbacks.
我们有一些权重分布 So the idea is that you have some some weight distributions.
所以如果你有一列火车去一个网络如果你等待 So if you have a train to a network where if you wait,
你可以从一些分布中看到压缩的样本 you can see them as samples from some distribution and in this compression.
你有一些光彩 Assistance you have some luster,
陈述和无损 它点燃了一个对话步骤 其中包含的权重 删除的信息的这辆车的信息和sun statement and lossless that fires up a conversation step which comprises the weights which removes information of this car's information and sun.
编码更稳定的熵编码 The coding stabler entropy coding,
可以改变你的a比赛 which can converts your a contest,
将值转换成比特流 values into a bit stream,
这通常是解码器的大小 and that's usually the decoder size.
您必须撤销这些操作 You have to undo these operations.
因为这些系统以音频视频压缩而闻名 And because these systems are well known for audio video compression,
通常他们所做的就是喜欢 and usually what they do is to like.
为了衡量你的原创和数量之间的扭曲 以及原创输入加倍你的快乐和你的重建输入 你已经唱了一个右转 To measure a distortion between your original and the quantity and the original input to double your joy and your reconstructed input and you have sung a right turn,
的措施 which that measures,
需要传输多少位 how many bits need to be transferred.
传送到达科他 Transmitted to the Dakota.
你的优化排列的失真目标 And your optimizes arrayed distortion objective.
当你想到你的网络压缩时情况就有点不同了你不会觉得这是我们第一次对权值矩阵的重构不感兴趣 When you think about your network compression is the situation is slightly different here you are not so much as if it's the first time we are not interested in reconstructing the weight matrix.
但你感兴趣的是重建神经网络的功能属性 But you are interested in reconstructing the functional properties of the neural network,
所以你想测量新网络的输出和预测的失真 so you want to measure a distortion in the output of the new network Anza and the predictions,
不一定是它的参数 not necessarily in the way its parameters.
因此 Therefore,
这是一个稍微不同的提法 it's a slightly different formulation,
但是我们的论文可以证明这一点 but our paper we could show that.
As是一种形式 As is one form.
对变形的另一种变形 最后一种变形可以翻译成 Of distortions of the distortion and another on the last kind of can be translated into.
变成加权的右扭曲 Into a weighted right distortion,
累了 就像 tired where you have like,
我们都有像通常的畸变之间的参数WI和Q眼 we all have like the usual distortion between the parameters W I and Q eye.
但是有一些项 如果我 But you have some term and if I,
sweet参数对A预测输出的影响 which kind of measure the impact of the sweet parameter to the A prediction output,
所以你想通过测量参数的失真来最小化预测输出的影响 so you kind of you want to minimize the impact of the output of the prediction output by measuring the distortion of the parameters?
但是如果有一个从官方信息矩阵中得到的术语呢 But by having like a term which comes back from the official information matrix?
用什么来衡量对预测的影响 Which measures impact on the prediction?
这就是为什么 So this is what how,
我们所做的就像一个办公室算法抓住了魁北克 what was we did like an office algorithm gripped Quebec.
我试图找到一些量化的方法来最小化姐妹 I've tried to find some quantitation which minimizes sisters.
这个比率失真项 This rate distortion term?
我只是在等待注册期 I just waited registration term,
它运行得很好 and it worked very well,
但是你必须确定它的权重参数 but you had to still like determines its weight parameters,
就好像我为每一个权重进行了参数计算然后我们也进行了试验并计算出了结果就像有统一的对话和对话步骤一样 as if I parameter for every weight and what were then also tried out and figured out to us to work quite well as just having like uniform conversation and for the conversation step.
所以在这里 So here.
24个对话和调整涵盖 Twentyfour conversation and adjusted covered for the?
同事是做算术编码的 For the colleague is for the arithmetic coding stuff.
所以你有一个非常简单的对话 And so you have a very simple conversation.
顺便说一下 你们有一些高级算术编码 just buy by the way that you have some advanced arithmetic coding.
cutkabak是指上下文自适应二进制算法 So cut kabak means and context adaptive binary arithmetic,
编码器 coder.
这种技术在今天的视频压缩标准中被混淆了 It's the technology which is confused in today's video compression standards,
就像一个非常高级的算术编码器 close like a very advanced arithmetic coder.
它有一个令人满意的性质 and it has a desirable properties.
如通用 such as universal,
t就是他得到的代码 that t so the code which he gets.
这些特性是我们想要的 these are properties which are desirable and come back for,
满足他们 fulfils them.
所以这个是通用的 So this guy's universal.
普遍性意味着代码应该有一种机制允许它适应 Universality means that the code should have a mechanism that allows it to adapt.
来翻译这个模型 to translate this model.
它的概率模型是在样本有效的方式下对各种不同类型的输入分布 It's probability model to a wide range of different types of input distributions in the sample efficient manner,
这意味着视频你也有一个机制 因此 典型的适应你 which means that videos you also have a mechanism so that and typically have adapts to thee.
非常有效地分布 Very efficiently to the distribution it sees.
所以满足于这种变化效果很好 So it works very well to content with this kind of changing,
这是非常好的视频 which is very good for videos.
对于神经网络也是如此 And also for neural networks.
这是一个非常有用的性质 This is a property which is very useful.
通过使用所谓的上下文 And come back achieves this by using socalled contexts,
模型 models.
它应该有最小的冗余和高效率和橱柜 It should have minimum redundancy and also high efficiency and cupboard.
这些属性 Has these properties.
这只是一个很简单的直觉 And it's just like a very brief intuition how it works,
所以它 so it.
它首先将其唯一的输入值表示为二进制决策序列 it first represents its unique input value as a sequence of binary decisions.
我假设你和参数有冲突 So I assume you have a contest with parameter.
作为它的内容和价值之一 你想要表示吗 As its content and its value one would you want to represent?
你在柜子里怎么表现 How would you represented in cupboard,
你需要做一系列的决定 you would have a set of decisions to make.
第一个决定是你想要编码的数字是否只是一个叫做不等于零的东西 The first decision is whether it's the number you want to encode and simply one thing called different from zero or not.
如果它不等于0 If it's a different from zero,
然后你应该告诉它 第一个比特应该是一个它和另一个是不同的 then you should give it the fact that the first bit should be one it is the one is different from the other.
因为第一个比特是1 For the first bit is one.
然后你就有了一个征兆 Then you have a sign.
它告诉你一个你想要编码的数字 Which tells you is a number you want to encode.
一个负数或正数 讨论它的正数 所以它应该得到0 A negative or positive and discuss its positive so it should get the zero,
如果这是你想要的数字 而且有很好的理由 and if it's the number you want and got absolutely great reasons,
一个 one,
没有人比完美更伟大的了 no one is not absolute greater than wonderful it.
这里应该是0 It should have a zero here,
所以你把这个符号编码成二进制位 and so you encode the symbol into a binary bit.
二进制流为100 Binary stream of one zero zero.
我将用类似的方式来画其他符号 I'll do what I called like other symbols in a similar fashion,
你有一些我曾经做过的事情 and you have some contexts which I was.
说到背景 Talking about the context,
妈妈 Mother.
每个决定都有一个概率模型 There's a probability model for each decision.
并根据有关日期的本地统计资料即时更新 Which is updated on the fly by the local statistics of the date,
根据你们看到的数据 as the depending on the data that you see,
编码器 the encoder,
还有解码器 and also the decoder,
一种对概率的适应可以非常有效地适应你想要编码的数据中的特定统计数据磁编码是一种非常有效的呼叫 a kind of adapts the probabilities and can adapt very efficiently to a particular statistics in the data that you want to encode and magnetic coding is a very efficient calling,
这个计划也很高效 a scheme which is also highly efficient,
具有更多的冗余和高效率 which has many more redundancy and high efficiency.
所以碗柜可以接近 So cupboard can kind of approach.
0附近的任何分布形状 any distribution shape around zero here.
所以我们所做的 So what we did,
我们研究过 we kind of studied,
我们如何看待竞赛的分布 how we looked at the distribution of contests,
神经网络 neural networks,
我们看到大多数的分布都在0左右 并且以不同的形状返回 and we saw that most of the distributions were around zero and head back different shapes.
但是橱柜可以非常有效地适应不同的阴影通过使用上下文模型只有当你远离零时 But a cupboard can came a very efficiently adapt to different shades by using the context model and only like if you are far away from from zero,
你的歌 you get song,
像步进近似一样 由于指数急速冷却 so step like approximations due to the exponential galloped cooling with you,
剩下的人呢 would you have for the remainders?
这是一个非常高级的编码方案 So it's a quite advanced coding schemes.
关于这个有很多文献 there is whole literature about that,
但我们可以采用它并优化它来编码你的网络 我们可以这样做 but we can have adopted it and optimize it for coding your networks and could do that.
我们可以 We could.
你得到了很好的结果 You get very good results.
例如 For example,
我们可以编码视频基因 we could encode video gene.
从五百五十兆字节到八兆字节 甚至更少然后从一百兆字节推进到下落兆字节所以通过使用不同的组合 From five hundred and fifty megabyte eight megabyte or even less and thrust it from one hundred to fall megabyte so by using the different combinations.
就像他们中的一些人还结合到一个规格的谈话柜 可以减少月 Like some of them were also combined to a specification of the conversation cupboard and could reduce the months.
覆盖网络的规模有很大的影响 The covering the size of the networks by a significant amount.
好吧 Okay,
这是技术的一部分可以用来编码你的网络也可以用来编码联邦学习中的更新 so this is one one parts of technology which can be used for coding your networks and which also can be used for encoding updates in federated learning.
我们也做过 We also did that.
在我们申请的一篇论文中 And in one paper where we applied,
卡巴克力编码适用 我们将进一步研究它 kabak force encoding applies and we will study it more.
的时候 When,
在研究这种新风格时 while working on this new style,
新的号召号召对神经网络的增量压缩提出建议 new kind of call the call for proposals on incremental compression of neural networks,
然后我们试着调整它 then we will try to adapt it come back for exactly this,
一个应用程序 an application.
但另一件事 But another thing,
这是很重要的 which is an importance in.
在分布式应用程序中 In distributed applications is,
当然 of course,
比如网络的规模和内存的使用 the army like the size of the network and the memory usage,
但也喜欢 but also like.
高效的表示法 Efficient representation and also like efficient,
你使用它 you use it.
所以 So,
例如 for example,
你可能想用 you may want to use.
空气中 Air.
这些模型 These models,
但你知道我们可能想用什么形式来表示它 but you know what we might want to represent it in the format.
这在存储方面是有效的 也是有效的 Which is efficient with respect to storage and also efficient.
我们忘了推断 We forgot to inference to,
例如 for example,
你想为神经网络找到一种格式 you want to find a format for neural networks,
这很好 which is very good.
那五个数字是什么 What were the five numbers?
修改的老鼠数量很多 Number of modifications is many mice.
所以如果我们有像这样的权值矩阵我们开始密集的格式我们存储每一项 So if we have like like weight matrix like here and we start the dense format we store like every entry,
然后你需要60个条目 then you would need sixty entries,
六个数字要存储如果你要用如果你要用这个权值矩阵计算一些矩阵向量乘积 six numbers to store and if you if you would use if you would compute some matrix vector product with this weight matrix,
你会有很多我的申请 you would have a lot of my applications,
这是不必要的 which are unnecessary.
例如 For example,
因为这里乘以0反正是0 because multiplication by zero there zero anyway,
所以你不需要 so you don't need to you,
您可以保存所有的加载操作和乘法操作 you can save all the loading operations and the multiplication operations.
在这个角色中我们也有力量 and also here we have all force in the role.
如果是我看到的而不是运气呢 So what if I kind of saw that instead of luck?
同一个数乘以某个数 Multiplying the same number with some,
我在这里找到的 I find it up here.
到目前为止成功了吗 Is it did so far?
每次都有一些激活量你也可以把数字拿出来然后把这些激活量加起来然后在这个例子中相乘 We have like some activation every time you could kind of also take out there the number and just just sum up these activations and just multiplies in this case.
第四是一个总结的激活 The fourth was a summed up activations.
如果我们这样做 So if we do that,
我还想告诉大家的是他的故事 what I'd also like saves us a lot of and not on his stories,
而且还像Scala产品 but also like Scala products.
所以如果我们 So if we,
如果我们那样做 if we do that,
我们只需要在这个案子上努力 we just need to work in this case.
只需再做一个应用程序和AUD just do one more application and AUD.
为了将这一行与某个向量相乘 In order to multiply this row with some vector,
因为我们有聪明的等待 and because we have like the smart wait,
股份和财产会利用它 share and property will make use of it.
看看我们能看到塔的纸在哪里 And see where the paper where we can have looked at the tower,
它如何影响事实的存储 how it can affect the storage of the facts,
精力 时间和操作 energy and time and operations.
这里是密集格式的对比 And you see here a comparison of the dense format.
WithasCSR是文献中常用的一种特定格式 另外两种是我们提出的 With as CSR is a specific format which is usually used in literature and the two other ones are the ones proposed by us.
而且比过去的模式更有效率 水果也更多 And there are more efficient than their usual past format and also more fruits,
并由此产生一种格式 因为我们使用将利用这种方式来共享属性 and thence a format because we use will make use of this way of sharing property.
我们也做了一些功 We also did some work,
它进一步优化了 which goes one step further and optimized,
像这种格式的其他硬件一样 like other hardware for this kind of format,
但我不会在这里详述 but I will not go into detail here.
所以我想读一读写下来 So I would like to read write up.
因此 我们看到分布式环境带来了新的挑战 So we saw the distributed environments come with new challenges.
我们看到 当它在训练中的效率 We saw that when its efficiency in training,
交流和推断 communications and inference so,
我在第二部分中介绍了减少交流的技巧 I presented in part two techniques for reducing communication for reducing,
提高培训效率 making training more efficient,
这里我们看到 它们也是更有效地进行推理的方法 and here we saw that they are also approaches to make an inference in more efficient.
我们看到了压缩和更新神经网络的不同技术 We saw a different techniques for compressing neural networks and updates.
还有很多其他的事情我们没有谈到 There are many other things we didn't talk about it.
本教程中 This tutorial,
比如网络的结构变化 like what about structural change changes to the network.
你可以用fire-up来获得更小的网络然后翻译一个小的 you could use fires up a disturbing to to get the smaller networks and then just translate a small,
一个活跃的网络 an active network,
或者您可以使用newarchitecturesearch来优化网络的结构 or you could use new architecture search for optimizing the structure of the network.
还有像这样的硬件组件 There is also like this hardware components,
我还没有讲过 which I am not really have haven't really talked about.
所以这对于提高效率是非常重要的史诗文明带来了一些压缩技术 So which is extremely important to get efficiency on the edge and the epic civilization kind of brings some compression technology on the table,
这不仅可以用来为他的屁股压缩单位在未来 which can be used not only for compressing unit for his butt in future.
也用于压缩更新和 also for compressing updates and.
对神经网络进行增量更新 然后等待更新 Incremental updates of neural networks and then wait updates.
我的演讲就到此结束 And with that I would like to close.
这里有一些关于我们在网络压缩方面的工作的参考资料 Here are some references on our work on your network compression.
联合学习 federated learning.
我还选了一个网站 在那里你可以找到所有的幻灯片 并在软件中为结果付费 As the realization and I've also pick out a website where you can find all the slides and pay for results in the software.
非常感谢 Thank you very much.
很高兴提供这篇教程 It was a pleasure to give this tutorial.
谢谢你 Thank you.