大家好 我是来自民族城镇大学语音处理实验室的蒋晓明 Hello, everyone, I'm Jiang Xiao Ming from the speech processing laboratory of National Town University.
今天 我将介绍我们为2021年ICAST所做的研究 以及关于挑战的研究 Today I'm going to introduce our research for the ICAST, 2021 and about challenge.
他的作品与张林 张浩 庞翔宇合作 她买了杜松子酒和房子 E His work is coworked with Lin Zhang Hao Pong Xiang Yu. She bought gin and the home, E.
我演讲的题目是研究如何在多说话人 多风格的文本语音中结合预趋势和可学习的说话人表示 The title of my presentation is investigating on incorporating pre trend and learnable speaker representations for multi speaker, multi style text speech.
我的报告第一次在线是在哪里 Where is first is the online of my report?
我将从税务描述开始 并给出一些启发我们工作的背景知识 I'll start from the tax description, and give some background knowledge which inspires our work.
然后是一个方法论和实验 然后是结论 部分出生是一个测试描述 And then is a methodology and experiment followed by the conclusion, part birth is a test description.
在山地 高音喇叭 多风格语音 提示任务中 TT系统以文本输入和少量参考样本作为输入 In a mountain, high speaker, multi style voice, proming task, a TT system takes a text input and a few references, these samples as input.
敏感语音样本要求在说话人身份 说话人和说话风格等方面与参考样本相似 The sensify speech samples are required to be similar to the reference samples in terms of the speaker identity and speaker and speaking style.
这项任务有两个主要挑战 There are two major challenges of this task.
首先 繁琐的模型必须有效地从少数参考样本中获取说话人和风格信息 First, the tedious model has to efficiently get the speaker and style information from only a few reference samples.
而且 这个模型还必须能够适用于许多不同的演讲者和风格 And also, the model also has to be able to generalize to many different speakers and styles.
还能保持合成音质 然后是启发我们工作的背景知识 And still preserve the synthesized audio quality. And then is the background knowledge which inspires our work.
它是一个较高级的讲话者的一般框架系统 It is a general framework of a more high speaker, pedious system.
除了税额输入外 还向TT系统输入说话人表示 以控制合成语音的说话人身份 In addition to the tax input, a speaker representation is fed into the TT system to control the speaker identity of the synthesized speech.
这个演讲者代表来自 This speaker representation come from.
无论是操作扬声器在较冷的模式 或它可以来自一个可学习的扬声器表示 Either operation speaker in colder model, or it can come from a learnable speaker representation.
例如 在床上桌子上的扬声器 或可训练的扬声器编码器可以作为可学习的扬声器表示 For example, a speaker in bedding table, or a trainable speaker encoder can be used as a learnable speaker Representation.
如果使用床表中的扬声器 则ID是每个扬声器和扬声器代表的符号 If a speaker in bed table is used, an ID is a sign to each speaker and speaker representation.
然后就可以被说话者从床头柜上抬起 ID Then can then be looked up from the beding table by the speaker. ID.
然而 这种方法只能用于 However, this approach can only be used to.
生成训练演讲人的演讲 而且它不能推广到演讲者身上 To generate the speech of the training speakers. And it cannot generalize to speakers.
哦 哪个是在闪光的时候看的 Oh, which are on seeing during the shining time?
如果我们希望演讲人的代表能够应付演讲人 那些在培训公园的演讲人 And if we hope the speaker representation should be able to handle speakers, speakers that are on seeing the training park.
我们可以使用可训练的扬声器录音机 We may use a trainable speaker recorder.
让我们听 持有税 输入后的一些引用 Let's be hearing, hold a tax. A few references, after the input.
并可与整个繁琐系统进行联合优化 在婴儿时期 任意扬声器的声音都可以输入到扬声器编码器中 And it can be jointly optimized with the whole tedious system. And at infant time, the speech of an arbitrary speaker can be fed into the speaker encoder.
获取演讲者的发言代表 另一方面 扬声器编码器可以打印在另一个任务上 To get a speaker representation of the speaker. On the other hand, the speaker encoders can be printed on another task.
它把这个图像编码器带入了彼得S模型 并伪造了它的重量 It brings this picture encoder into the Peter S model and fake its weight.
当以这种方式训练心理系统时 包括说话者在内的人仍然可以概括为看不见的说话者 When training the mental system in this way, the speaker include can still generalize to unseen speakers.
在之前的工作中 不同之处在于 因为表示已经应用到TPS状态 例如 董事和部门 In previous works, difference, because representations have been applied to a TPS state. For example, the directors and sectors.
它们都是由度假网络的一个speakercase生成的 并且它们已经被证实在我的医院全科医生中很有用 They are both generated by a speaker case of vacation network, and they have been verified to be useful in my hospital GPS.
基于这些背景 我们得出了我们工作的动机 Based on the backgrounds, we got to motivation of our work.
就像我刚才提到的 For a first one, as I just mentioned.
以往的工作中 通过PI年 学习型说话人表征或欣赏型说话人表征来控制教师的说话人身份模型 Previous work, a PI year, learnable speaker representation, or appreciate speaker representation to the teacher s model to control the speaker identity.
开放的合成语音 Open synthesized speech.
那么 为什么我们不把它们全部结合起来呢 如果我们使用多个说话人表示 撕裂模型会变得更好吗 Then why don't we just combine all of them Can the tear model become better if we use multiple speaker representations.
第二个动机是 几乎所有经过训练的假表象都没有被习惯 And the second motivation is that almost all of the pre trained fake representations are not being used to.
就拿导演和期待者来说 他们都是假装的 It is, for example, director and expector, they are all pretend on a.
演讲者ChrisAffect可以讲一个案例 假期任务 或歧视任务 Speaker Chris Affect can speak a case, vacation task, or a discriminated task, and.
用其他类型的训练前任务怎么样 例如 生成测试 How about using other kinds of pretraining tasks For example, a generative test.
这两个想法启发了我们的研究 我们将在以下部分讨论和探索它们 These two ideas inspire our research, and we will discuss and explore them in the following sections.
然后是我们的工作方法 Then is the methodology of our work.
这是我们研究地图的工作流程 我们的工作流程有三个阶段 This is the workflow of our map of our research. There are three stages of our workflow.
他们是发言者的代表 假装Pps训练 并采取他的影响 我们将在接下来的几张幻灯片中介绍今天的主持人 They are speaker representations, pretending Pps training, and take his influence. We will go for the day host in the next several slides.
首先是讲话者病人 讲话者代表 讲话者部分 First is a speaker patient in speaker representation, presenting part.
使用了假导演和检查员 两者都是由扬声器分类器生成的 The pretend director and inspector are used, both of which are generated by a speaker classifier.
以往的工作表明 分类广告的中间特征包含一些信息 Previous works suggest that the intermediate features of the classifieds contain some information.
这是区分不同说话人所必需的 所以D因子和效应都是这样产生的 That is required to discriminate between different speakers. So both the D factor and effect are generated in this way.
对于发电机 我们使用的假装任务 和一个镜头 声音转换模型 被称为其他DC And for the generator, pretending task we use, and one shot, voice conversion model, which is called other DC.
在该模型中 使用说话人编码器和记帐码来捕获语音中的不同信息 In this model, a speaker encoder and accounting code are used to capture different information in speech.
解码器用于从扬声器编码器和公司记录器的输出重构语音输入 And a decoder is used to recontruct the speech input from output of the speaker encoder and a company recorder.
因为扬声器编码器的设计和试图解开 Because the speaker encoder is designed and trying to disentangle.
演讲者信息来源于真实的演讲 所以我们认为它可以应用于我们的市场 市场 演讲者 繁琐的系统 Speaker information from real speech, so we think it can be applied to our markets, market, speaker, tedious system.
第二阶段是票券模型的训练 And the second stage is the training of the ticket model.
蒂娜的多说话者模型与不同的说话者表示相结合 在我们的实验中 我们也尝试用技术和面部语言 The multi speaker, tina s model is combined with different speaker representations. We try both technical to and face speech, too, in our experiment,
结果和差异将在稍后公布 And the result and difference will be reported later on.
伪说话人表示包括专家DeVector和其他说话人表示 The pretended speaker representations include the De Vector, an expert, and other in speaker representations.
对于可训练扬声器编码器 我们采用了多数编码器 这是GT技术首次提出的 For the trainable speaker encoder, we used majority encoder, which was first proposed in GT Tech.
我们还使用了北京喇叭表 与遗憾模型联合优化 And we also use a speaker in Beijing table, which is jointly optimized with the pity model.
图中虚线表示可选路径 And we can see that the dotted lines in the figures stand for optional paths.
这是因为我们在实验中尝试了不同的说话人表征组合 This is because that we try different combination of speaker representations in our experiments.
每次实验中 我们只是用其中一些来训练一个乏味的模型 So the speaker representations are not always used In each experiment, we just use some of them to train a tedious model.
从整体上看 智慧和谨慎的框架正在试图重构广州演讲 And as a whole, the intelligence and heedous framework is trying to recontruct the Guangzhou speech.
它也被输入到说话人编码器中 并提取说话人的信息 Which is also fed into the speaker encoders, and to extract the speaker information.
下一个阶段 或者说影响时间 演讲者和演讲者ID And in the next stage, or the influence time, the speaker and speaker, ID,
而目标说话人的言语可以用来得到说话人的表示和 And the speech of the targets speaker can be used to get a speaker representations and.
根据条件 对于说话人表示Pity的模型能够合成目标说话人的语音 By condition, on the speaker represents that Pity's model is able to synthesize the speech of a target speaker.
我们的实验呢 And is our experiment?
我们用了230位演讲者数小时的普通话演讲来训练模型 We used hours of Mandarin speech by 230 speakers to train models.
目标扬声器是六个未来的扬声器 其中三个有一百段录音 The target speakers are six future speakers, and three of we have one hundred recordings.
ER 去找一个扬声器 ER, to extract one speakers.
另外三个扬声器只有五段录音 我们称之为音轨扬声器 And the other three speakers have only five recordings, which we refer them as the trackal speakers.
并且我们使用所有的演讲者 包括这些社会演讲者 到中国门票模型和训练代表模型 And we use all speakers, including these social speakers, to China ticket model and to train the representation models.
我们使用所有的扬声器 We use all speakers.
对于电话2和面部语音2 说话者表示直接添加到两者的遭遇输出中 For both telephone two and face speech two, the speaker representations are directly added to the encounter output of the two.
启用更高的扬声器 并使用宽水平冷转换城市模型的男性特殊图输出到铁路形式 Enable the more high speakers it is, and a wide level colder is used to convert the male specialgram outputs of the city s model to the railway form.
为了评估我们的模型 我们使用了第三方说话人复制系统和合成语音 And to evaluate our model, we use a third party speaker replication system and a synthesized utterance.
他说 这个验证系统 如果系统认为它是由目标说话者说的 He says this verification system, if the system considers it to be spoken by the target speaker,
我们报告合成想法的百分比 And we report the percentage of the synthesized idea to.
通过这个验证系统 我们称这个百分比为说话人验证的准确率 Pass this verification system, and we call this percentage the speaker verification accuracy.
在下表中 我们可以看到其他的英语表示都执行任何其他表示 In the table below, we can see that the other English representations all perform any other representation.
无论如何 带上无人机或面对鱼使用它 所以我们可以得出一个小结论 生成图像任务 No matter, take a drone to or face fish to its use. So we can make a small conclusion here that generating pictureing tasks.
事实上 它比区别的patriming任务和其他说话者表征表现得更好 Indeed, outperforms discriminated patriming tasks and other speaker representations.
让我们来听一些音频样本 And let's listen to some audio samples.
首先是 A first is.
这些音频样本是从曲目到只有五个参考话语给出 These audio samples are from the track to where only five reference utterances are given.
首先是目标说话者 一个目标 So first is the target speaker, a target.
然后是由一个导演 说话人代表的模型产生的话语 And then is the utterance generated by the model with a director, speaker representation.
拉丁语 很好 李静 Latin, good, li Jing.
一个演员 An actor.
于是他买了一个南昌灯笼 So that he buy one Nanchang lantern.
另一个英国的名声 据说是表中最好的一个 The other English reputation, which is reported to the best one in the table.
落地可能会伤及孩子 而不是嵌入 查找 Landing could lay to the children and not embedding. Look up.
着陆 Landing.
局长和巡视员的调查结果在北京查一下 Uh, the results of the director and inspector And look up in Beijing.
在某种程度上缺少变化 音调真的很痛苦 这是VC的结果 Are somehow lack of variation and the tone is really pain. And both the result of the VC,
在北京 非常棒 In Beijing and just are great.
我们猜测 生成性假装和歧视性提问之所以更好是因为需要一些信息 And we guess the reason why generative pretending is better and discriminated questioning is that some information that is required to.
要求使用讲话句子 但说话没有帮助 分类可以不丢弃 也可以丢弃 Require for speech sentences, but it's not helpful to speak. Classification may not be discarded and may be discarded.
在一个受歧视的假定任务中 Uh, in a discriminated presuming task.
例如 在滑冰部分 我们可以看到BC表示的拉丁分布更连续 与 For example, in a skater part, we can see that the Latin distribution of the BC representations is more continuous, compared with.
主任和专家 Director and expert.
因为一个主管和督察在一个有区别的任务中 唯一的 Since a director and inspector are present on a discriminated task, the only thing.
范例是区分不同说话人的话语 The model case is to separate the utterances of different speakers.
并且尽可能地证明他们 这样他们就可以在演讲中讨论一些重要的信息 例如 连续性 And prove them as far as possible, so they may discuss some important information in speech. For example, the continuity.
并通过在一个pedious模型中使用多个说话人表示的有效性来跟踪 And to follow track by effectiveness of using multiple speaker representations in one pedious model.
我们将其他BC表示形式与其他表示形式相结合 见下表 在检查中没有明显的区别 We combine the other in BC representation with other representations, see the table below. There is no obvious difference in check, one.
Erences 我们看到的 Erences are given. We see.
在某些情况下 多重说话人表征的结合提高了说话人 反射 准确性 The combining multiple speaker representations improve a speaker, reflection, accuracy, in some cases,
我们进一步进行了主观评价 以判断质量 We further conducted subjective evaluation to judge the quality.
演讲者 句子的相似度 话语 听众被要求为每个合成的话语打分1到5分 And the speaker, similarity of sentences, utterances, listeners are asked to score between one to five points to each synthesized utterance.
在说话人相似度方面 具有多个说话人表示的单调模型优于其他模型 In terms of speaker similarity, the tedious model with multiple speaker representations is better than other models.
而对于音频质量 多重行为模式 注意力比使用稍差 And for the audio quality, the model of multiple behavior, attention is slightly worse than using.
只有VC代表还是在北京往上看 但差别很小 Only the VC representation or look up in Beijing, but the difference is very small.
我们也可以在这里听一些音频中心 We can also listen to some audio center here.
也适用于有吸引力的演讲者 Also also for attractive speaker.
突然听到一个可怕的演讲者 Burst with a terrible speaker.
我在北京了解到这次访问 I learned the visit in Beijing.
Sichuanda 广州 Sichuanda, guangzhou.
然后将该模型结合VC表示方法在北京进行查找 Then the model combining the VC representation and look up in Beijing.
所以我们要去四川泉州 So for Tripping Gala to the Sichuan Quanz.
因为我们之前在北京比较过X因子和VC Since we have compared X factor and VC And in Beijing before.
所以我们只比较模型和VC扬声器表示 模型和VC+ So we only compare the model with the VC speaker representation, and model with VC plus.
看看这里 结果 结合扬声器的名声听起来更自然 Inviting look up here. The result, with the combined speaker reputation sounds more natural.
总的来说 我们的方法在拖拉机的反工作挑战中取得了很好的效果 Overall, our method achieved very good result in tractor of the anti work challenge.
滑冰选手的部分成绩是官方主观评价的结果 在赛道上达到了a级 The skater parts are the result of the official subjective evaluation, in track to a.
当不允许外部培训数据时 我们排名第二 虽然我们没有 When external training data is not allowed, we rank the second place. And although we did not.
使用任何外部培训数据 我们仍然可以在第三阶段得到外部培训数据 然后是我们的结论 Use any external training data, we still get the third phase in track to be where external training data is permitted. And then is our conclusion.
根据实验结果 我们认为将多个说话人表示应用于多个说话人是有用的 Based on the result of experiment, we conclude that it is useful to apply multiple speaker representations to a multiple.
到一个更高的扬声器 乏味的模式 To a more high speaker, tedious model.
尽管如此 几乎所有之前的研究都使用了说话人代表 And despite that, almost all previous research use speaker representation.
在一个有区别的任务上 到多个怜悯系统 我们发现使用生成性的 描绘性的任务可能是一个更好的选择 On a discriminative task, to multispecle pity systems. We find that using a generative, portraying task may be a better choice.
我的报告到此结束 And this is the end of my report.