1
00:00:00,000 --> 00:00:29,980
大家好,今天给大家介绍一篇比较新的paper,也是基于Visper作语音识别的。这篇论文的题目叫Visper CD,专门解决长音频的语音识别问题。如果用一句话来总结的话,这篇paper它的核心亮点就在于不改变Visper模型的参数,也无需训练。只在Visper解码时给它加一个反向的揪片器,这样专门来压制长音频的语音识别。

2
00:00:30,000 --> 00:00:31,740
音频转写里面的幻觉

3
00:00:31,740 --> 00:00:34,580
重复循环和漏字等问题

4
00:00:34,580 --> 00:00:36,500
这就是这篇论文

5
00:00:36,500 --> 00:00:37,920
它核心要解决的问题

6
00:00:37,920 --> 00:00:39,360
这里面CD代表什么意思

7
00:00:39,360 --> 00:00:40,420
就是一个对比解码

8
00:00:40,420 --> 00:00:41,540
那我们下面来看一下

9
00:00:41,540 --> 00:00:43,360
这篇PayPal它主要解决什么问题

10
00:00:43,360 --> 00:00:45,040
它核心要解决的

11
00:00:45,040 --> 00:00:47,320
就是可能比较长的一些语音

12
00:00:47,320 --> 00:00:48,060
然后呢

13
00:00:48,060 --> 00:00:50,680
以Visper为代表的一些语音识别模型

14
00:00:50,680 --> 00:00:52,280
它面临的幻觉

15
00:00:52,280 --> 00:00:55,220
重复循环和内容遗漏

16
00:00:55,220 --> 00:00:56,500
等错误累积的问题

17
00:00:56,500 --> 00:00:58,080
因为上海文太长了嘛

18
00:00:58,080 --> 00:00:59,340
而且这种Visper

19
00:00:59,340 --> 00:01:02,300
它是自回归的一个模型结构

20
00:01:02,300 --> 00:01:04,540
所以上文的错误会累积到下文

21
00:01:04,540 --> 00:01:08,780
这三种错误基本上可以看成分成以下三类

22
00:01:08,780 --> 00:01:11,100
第一种就是静音区的幻觉

23
00:01:11,100 --> 00:01:14,320
例如在非语音的间隔期间

24
00:01:14,320 --> 00:01:15,400
但是模型呢

25
00:01:15,400 --> 00:01:17,200
它还是错误的生成了文本

26
00:01:17,200 --> 00:01:20,800
第二个就是跨段的重复的循环

27
00:01:20,800 --> 00:01:23,260
就是模型陷入自我强化的重复模式

28
00:01:23,260 --> 00:01:26,320
它会错误的不停的输出相同的字

29
00:01:26,320 --> 00:01:27,760
然后一直循环一直循环

30
00:01:27,760 --> 00:01:29,100
然后永远不停

31
00:01:29,100 --> 00:01:30,740
第三个就是跳过

32
00:01:30,740 --> 00:01:33,960
那就是遗漏了实际的语音内容

33
00:01:33,960 --> 00:01:35,300
导致转路不完整

34
00:01:35,300 --> 00:01:38,360
实际上就是我们语音识别里常见的三类错误

35
00:01:38,360 --> 00:01:41,000
插入删除替换这三类错误

36
00:01:41,000 --> 00:01:42,920
此外这个论文也指出

37
00:01:42,920 --> 00:01:46,380
当前因为是自回归的语音识别模型

38
00:01:46,380 --> 00:01:50,380
它会利用前一段转路的结果作为解码的上下文

39
00:01:50,380 --> 00:01:52,780
那么这样就会带来一个问题

40
00:01:52,780 --> 00:01:54,660
就是上面的错误会被累积

41
00:01:54,660 --> 00:01:57,140
引起后面解码带来更多的错误

42
00:01:57,140 --> 00:01:59,180
这就要形成了新的累积错误

43
00:01:59,180 --> 00:02:01,680
这也是自卫规模型它常见的问题

44
00:02:01,680 --> 00:02:04,760
那这篇paper它是如何解决这个问题的呢

45
00:02:04,760 --> 00:02:06,660
它就提出了叫Visper CD

46
00:02:06,660 --> 00:02:09,000
这种方式啊

47
00:02:09,000 --> 00:02:10,960
不需要训练这个Visper模型的

48
00:02:10,960 --> 00:02:16,460
在推理时对比干净的音频和声学扰动后的音频的一个输出

49
00:02:16,460 --> 00:02:17,420
一个logis

50
00:02:17,420 --> 00:02:18,560
然后从而呢

51
00:02:18,560 --> 00:02:23,260
抑制长形式语音转换中的一些插入删除和替换错误

52
00:02:23,260 --> 00:02:26,000
那我们就来看一下它具体是怎么做的

53
00:02:26,000 --> 00:02:28,380
我们先看右面这张图吧

54
00:02:28,380 --> 00:02:29,820
我把这张图看懂了

55
00:02:29,820 --> 00:02:33,080
那这篇论文最精华最核心的部分也就懂了

56
00:02:33,080 --> 00:02:36,440
首先白色这一部分就是输入的语音

57
00:02:36,440 --> 00:02:37,860
然后这一部分呢

58
00:02:37,860 --> 00:02:39,980
就是粉色的这部分就是Whisper

59
00:02:39,980 --> 00:02:41,560
Whisper我们知道

60
00:02:41,560 --> 00:02:44,140
它就整体过程中就不需要动

61
00:02:44,140 --> 00:02:45,860
那过程是怎么样的呢

62
00:02:45,860 --> 00:02:47,360
我们先看虚线这部分啊

63
00:02:47,360 --> 00:02:50,340
就是上一个30秒已经识别完

64
00:02:50,340 --> 00:02:52,500
识别之后的一个语音片段

65
00:02:52,500 --> 00:02:53,520
那现在呢

66
00:02:53,520 --> 00:02:54,860
我们要识别最新的

67
00:02:54,860 --> 00:02:56,900
这个新的这30秒的语音片段

68
00:02:56,900 --> 00:03:00,820
需要把它给到我们Whisper的encoder

69
00:03:00,820 --> 00:03:02,740
那你看这就是黑色这部分

70
00:03:02,740 --> 00:03:04,280
就是原始的这段

71
00:03:04,280 --> 00:03:05,560
此外呢

72
00:03:05,560 --> 00:03:07,620
我们会对原始的音频

73
00:03:07,620 --> 00:03:09,240
做一些相关的扰动

74
00:03:09,240 --> 00:03:10,580
把它当做复理

75
00:03:10,580 --> 00:03:12,420
原来这段就当做正理

76
00:03:12,420 --> 00:03:14,260
复理第一个种复理呢

77
00:03:14,260 --> 00:03:16,180
就是我们加一些高斯噪声

78
00:03:16,180 --> 00:03:18,280
就是在原始的这段干净音频上

79
00:03:18,280 --> 00:03:19,440
加一些高斯噪声

80
00:03:19,440 --> 00:03:21,020
把它变成第一种复理

81
00:03:21,020 --> 00:03:22,600
也就是这个蓝色这部分

82
00:03:22,600 --> 00:03:24,000
第二种复利呢

83
00:03:24,000 --> 00:03:26,140
就是我们把原始的音频啊

84
00:03:26,140 --> 00:03:28,180
向右稍微偏移一小段

85
00:03:28,180 --> 00:03:30,200
这个就变成shift

86
00:03:30,200 --> 00:03:33,100
就是偏移的这段扰动的音频

87
00:03:33,100 --> 00:03:34,500
第三种音频啊

88
00:03:34,500 --> 00:03:35,760
就是silence

89
00:03:35,760 --> 00:03:37,520
就是完全静音的这段音频

90
00:03:37,520 --> 00:03:38,460
然后呢

91
00:03:38,460 --> 00:03:39,960
我们把这四种音频

92
00:03:39,960 --> 00:03:41,320
同时送给Whisper

93
00:03:41,320 --> 00:03:43,760
它的encoder和decoder去解码

94
00:03:43,760 --> 00:03:45,620
解码之后呢

95
00:03:45,620 --> 00:03:48,860
我们就得到了对应的四个输出

96
00:03:48,860 --> 00:03:51,260
那黑色这部分就是原始的输出

97
00:03:51,260 --> 00:03:55,600
蓝色绿色和红色分别也对应了三个输出

98
00:03:55,600 --> 00:03:56,420
然后呢

99
00:03:56,420 --> 00:03:59,960
我们就把正力的得分输出和负力的得分输出

100
00:03:59,960 --> 00:04:01,140
进行一个整合

101
00:04:01,140 --> 00:04:03,340
以叫我们叫对比的一个解码

102
00:04:03,340 --> 00:04:07,420
解码之后就可以帮助我们得到

103
00:04:07,420 --> 00:04:09,980
更准确的语音识别的结果

104
00:04:09,980 --> 00:04:13,120
那具体它是怎么去做这个对比解码的呢

105
00:04:13,120 --> 00:04:16,060
我们就来看一下论文中的原始

106
00:04:16,060 --> 00:04:17,040
它是怎么说的

107
00:04:17,040 --> 00:04:18,820
其实方式也很简单

108
00:04:18,820 --> 00:04:22,480
正力得到的这个X就是干净的

109
00:04:22,480 --> 00:04:24,020
原始干净的语音

110
00:04:24,020 --> 00:04:26,380
那自回归之后

111
00:04:26,380 --> 00:04:28,360
得到我们正力的一个结果

112
00:04:28,360 --> 00:04:29,320
复利的呢

113
00:04:29,320 --> 00:04:31,520
这个X上面加一个波浪

114
00:04:31,520 --> 00:04:33,780
就是我们经过三种变换

115
00:04:33,780 --> 00:04:34,920
加高斯啊

116
00:04:34,920 --> 00:04:36,340
加偏移啊

117
00:04:36,340 --> 00:04:39,240
和静音的变换之后的这种音频

118
00:04:39,240 --> 00:04:41,060
也经过我们F

119
00:04:41,060 --> 00:04:43,480
也就是Whisper之后

120
00:04:43,480 --> 00:04:45,040
得到我们复利的输出

121
00:04:45,040 --> 00:04:46,080
然后呢

122
00:04:46,080 --> 00:04:48,860
我就通过一个α的一个超三

123
00:04:48,860 --> 00:04:51,260
把这两种得分输出

124
00:04:51,260 --> 00:04:52,720
把它整合在一起

125
00:04:52,720 --> 00:04:56,620
然后就变成我们最终的一个CD的一个输出

126
00:04:56,620 --> 00:04:58,620
那具体是怎么整合的呢

127
00:04:58,620 --> 00:05:01,080
实际上就是1加上α

128
00:05:01,080 --> 00:05:03,120
然后乘以我们正力的输出

129
00:05:03,120 --> 00:05:05,400
减去α乘以我们负力的输出

130
00:05:05,400 --> 00:05:08,020
实际上你如果把这个式子变换一下

131
00:05:08,020 --> 00:05:10,760
其实就是把这个括号拆开啊

132
00:05:10,760 --> 00:05:12,140
就变成正力的输出

133
00:05:12,140 --> 00:05:14,260
加上正力的输出

134
00:05:14,260 --> 00:05:15,480
减去负力的输出

135
00:05:15,480 --> 00:05:17,160
然后再乘一个alpha的参数

136
00:05:17,160 --> 00:05:17,600
对吧

137
00:05:17,600 --> 00:05:21,000
所以这它代表的是什么含义呢

138
00:05:21,000 --> 00:05:24,800
实际上就是我在保持正力输出不变的前提下

139
00:05:24,800 --> 00:05:27,100
我把负力的那些扰动

140
00:05:27,100 --> 00:05:28,440
希望都把它剪掉

141
00:05:28,440 --> 00:05:30,400
然后把这些干扰项去掉

142
00:05:30,400 --> 00:05:32,020
然后让它有一个对比的结果

143
00:05:32,020 --> 00:05:33,980
从而帮助我更好的识别

144
00:05:33,980 --> 00:05:36,060
最终的语音识别结果

145
00:05:36,060 --> 00:05:39,480
这就是这篇paper最重要的

146
00:05:39,480 --> 00:05:41,200
也最核心的一个机制

147
00:05:41,200 --> 00:05:42,020
叫对比解码

148
00:05:42,020 --> 00:05:43,800
那下面我们就来看一下

149
00:05:43,800 --> 00:05:46,220
它声学扰动具体是怎么做的

150
00:05:46,220 --> 00:05:47,340
基本上就是三种

151
00:05:47,340 --> 00:05:49,000
一个是高斯造声的注入

152
00:05:49,000 --> 00:05:50,440
就是我们随机生成

153
00:05:50,440 --> 00:05:52,440
符合正态分布的高斯造声

154
00:05:52,440 --> 00:05:53,740
加到原始信号上

155
00:05:53,740 --> 00:05:55,480
第二个呢就是静音

156
00:05:55,480 --> 00:05:56,820
我就把输入图谱

157
00:05:56,820 --> 00:05:58,180
全都变成零

158
00:05:58,180 --> 00:05:59,180
全零的波形

159
00:05:59,180 --> 00:06:00,160
就是静音

160
00:06:00,160 --> 00:06:02,560
第三个呢就是音频的时序偏移

161
00:06:02,560 --> 00:06:04,680
我就把原始的30秒

162
00:06:04,680 --> 00:06:07,180
向左偏移一些

163
00:06:07,180 --> 00:06:09,080
然后左侧的丢弃

164
00:06:09,080 --> 00:06:10,760
右侧的用零来填充

165
00:06:10,760 --> 00:06:11,620
这就好了

166
00:06:11,620 --> 00:06:14,260
这就是三种升学扰动的策略

167
00:06:14,260 --> 00:06:15,540
然后呢

168
00:06:15,540 --> 00:06:17,760
如果我们有了升学扰动的策略

169
00:06:17,760 --> 00:06:18,980
那这三种策略

170
00:06:18,980 --> 00:06:21,260
它是怎么结合在一起的呢

171
00:06:21,260 --> 00:06:22,680
结合其实也非常简单

172
00:06:22,680 --> 00:06:24,720
大家还记得我原始的诗子吧

173
00:06:24,720 --> 00:06:27,000
就是1加上α乘以正力

174
00:06:27,000 --> 00:06:28,480
减去α乘以负力

175
00:06:28,480 --> 00:06:30,360
这里只不过负力有很多啊

176
00:06:30,360 --> 00:06:32,740
就是有三种扰动策略嘛

177
00:06:32,740 --> 00:06:36,360
所以负力产生的输出结果就有三种

178
00:06:36,360 --> 00:06:38,520
那这三种我怎么整合到一起呢

179
00:06:38,520 --> 00:06:40,940
我就把它在e的指数上

180
00:06:40,940 --> 00:06:41,960
进行一个整合

181
00:06:41,960 --> 00:06:43,760
先求和再求个平均

182
00:06:43,760 --> 00:06:46,280
无非是这里加了一个超次参数

183
00:06:46,280 --> 00:06:47,280
来控制一下

184
00:06:47,280 --> 00:06:47,700
对吧

185
00:06:47,700 --> 00:06:49,200
所以这个超次参数呢

186
00:06:49,200 --> 00:06:50,200
一般也就设为1

187
00:06:50,200 --> 00:06:53,000
那这个1其实就等于没有吗

188
00:06:53,000 --> 00:06:54,100
你除它就等于没有了

189
00:06:54,100 --> 00:06:56,080
那其实在原始的例子上

190
00:06:56,080 --> 00:06:59,600
无非就是把三个负利的输出

191
00:06:59,600 --> 00:07:01,560
进行求和再取平均

192
00:07:01,560 --> 00:07:02,700
就非常简单吧

193
00:07:03,927 --> 00:07:18,207
然后呢,我们再看在推理的时候。推理的时候,我们知道,呃,其实是可以并行推理的嘛,对吧?就是尽管输入的是K条额外的解码路径,但是我们可以通过P处理来计算控制的开销,也就是一次处理把它

194
00:07:18,207 --> 00:07:18,927
都做完

195
00:07:18,927 --> 00:07:19,947
然后整体的

196
00:07:19,947 --> 00:07:20,887
它最大的优势

197
00:07:20,887 --> 00:07:21,867
就是无需训练

198
00:07:21,867 --> 00:07:24,047
然后我只需要在后面

199
00:07:24,047 --> 00:07:26,467
只需要在解码的时候

200
00:07:26,467 --> 00:07:27,487
稍加一些

201
00:07:27,487 --> 00:07:30,427
所谓的工程化的处理就够了

202
00:07:30,427 --> 00:07:32,687
那我们可以在这里

203
00:07:32,687 --> 00:07:33,867
做一个形象的比喻

204
00:07:33,867 --> 00:07:34,927
就是如果你上面

205
00:07:34,927 --> 00:07:36,827
听的还不是特别懂的话

206
00:07:36,827 --> 00:07:39,307
如果我们把原始的Whisper

207
00:07:39,307 --> 00:07:40,647
它正常转型的时候

208
00:07:40,647 --> 00:07:41,567
它就是像一个

209
00:07:41,567 --> 00:07:43,007
边听边写的速记员

210
00:07:43,007 --> 00:07:44,867
那这篇paper

211
00:07:44,867 --> 00:07:45,987
它这个Whisper CD

212
00:07:45,987 --> 00:07:47,667
它实际上做的就是

213
00:07:47,667 --> 00:07:50,727
在旁边放了三个捣乱版的监听器

214
00:07:50,727 --> 00:07:51,767
那一个呢

215
00:07:51,767 --> 00:07:52,787
让它听起来很吵

216
00:07:52,787 --> 00:07:54,307
也就是加噪音的

217
00:07:54,307 --> 00:07:55,607
加高斯噪声的

218
00:07:55,607 --> 00:07:57,647
一个让它几乎什么都听不到

219
00:07:57,647 --> 00:08:00,147
然后就是静音的

220
00:08:00,147 --> 00:08:01,027
还有一个呢

221
00:08:01,027 --> 00:08:02,847
是让它错位的

222
00:08:02,847 --> 00:08:04,107
就是这个偏移的

223
00:08:04,107 --> 00:08:05,467
然后如果某个词

224
00:08:05,467 --> 00:08:07,247
在这三种不利的条件下

225
00:08:07,247 --> 00:08:09,107
它仍然特别想写出来

226
00:08:09,107 --> 00:08:12,927
那这个词多半就不是真的想听到的

227
00:08:12,927 --> 00:08:14,467
而是它习惯性脑补的

228
00:08:14,467 --> 00:08:17,007
于是系统就把这些词的分数扣掉

229
00:08:17,007 --> 00:08:18,707
然后这个解释啊

230
00:08:18,707 --> 00:08:19,987
基本上或者这个比喻啊

231
00:08:19,987 --> 00:08:22,127
就是整篇论文方法的本质

232
00:08:22,127 --> 00:08:22,867
嗯

233
00:08:22,867 --> 00:08:25,287
最后我们来看一下它的效果啊

234
00:08:25,287 --> 00:08:25,947
整个论文呢

235
00:08:25,947 --> 00:08:29,707
是在这五个长音频的英文测试机上来做的

236
00:08:29,707 --> 00:08:30,647
然后我们看一下

237
00:08:30,647 --> 00:08:33,407
它这个效果还是很不错的

238
00:08:33,407 --> 00:08:36,347
就是加CD是这次改编之后的吧

239
00:08:36,347 --> 00:08:38,667
然后它在两个VSPR的版本上做

240
00:08:38,667 --> 00:08:40,787
我们看到原始的VSPR

241
00:08:40,787 --> 00:08:42,507
像V3那种

242
00:08:42,507 --> 00:08:43,887
它在这些测试机上

243
00:08:43,887 --> 00:08:45,707
这个凑率都非常高

244
00:08:45,707 --> 00:08:47,427
这都超过100%了

245
00:08:47,427 --> 00:08:48,907
有200多500多

246
00:08:48,907 --> 00:08:53,847
但是呢我们在Visper V3上加了解码上的改进之后

247
00:08:53,847 --> 00:08:54,947
下降非常多

248
00:08:54,947 --> 00:08:57,027
就都从200多变成45

249
00:08:57,027 --> 00:08:58,647
从44变成19

250
00:08:58,647 --> 00:08:59,827
从500多变成57

251
00:08:59,827 --> 00:09:00,267
对吧

252
00:09:00,267 --> 00:09:01,607
第二种方式呢

253
00:09:01,607 --> 00:09:02,927
就是我用更强的版本

254
00:09:02,927 --> 00:09:04,007
用Turbo这个版本

255
00:09:04,007 --> 00:09:05,827
虽然原始的版本

256
00:09:05,827 --> 00:09:07,647
直接用Visper

257
00:09:07,647 --> 00:09:11,587
识别这个长文效果比V3会好一些啊

258
00:09:11,587 --> 00:09:14,067
但是也不尽如人意吧

259
00:09:14,067 --> 00:09:14,967
但是呢

260
00:09:14,967 --> 00:09:16,467
我们加了这种CD之后

261
00:09:16,467 --> 00:09:19,027
你会看到它效果像样的也是非常明显

262
00:09:19,027 --> 00:09:21,747
所以这就是我觉得这篇paper非常

263
00:09:21,747 --> 00:09:24,507
值得分享的吧

264
00:09:24,507 --> 00:09:26,167
就是它用非常小的改动

265
00:09:26,167 --> 00:09:29,607
然后实现了对常文的一个效果非常大的提升

266
00:09:29,607 --> 00:09:32,047
我们最后来总结一下这篇paper

267
00:09:32,047 --> 00:09:34,007
就是长音频的识别

268
00:09:34,007 --> 00:09:36,527
Whisper的话会有很多严重的错误

269
00:09:36,527 --> 00:09:39,987
但这些错误它大概率不是模型不会识别

270
00:09:39,987 --> 00:09:41,647
而是解码的过程中

271
00:09:41,647 --> 00:09:44,847
被之前的识别结果和一些上下文绑架了

272
00:09:44,847 --> 00:09:47,127
导致它越识别越错

273
00:09:47,127 --> 00:09:47,947
越识别越错

274
00:09:47,947 --> 00:09:51,127
所以作者他不是去强化原始模型的能力

275
00:09:51,127 --> 00:09:52,627
也就是没有训练原始的模型

276
00:09:52,627 --> 00:09:56,827
而是直接在token选择的阶段做干预

277
00:09:56,827 --> 00:09:59,267
这个角度就是在工程测作

278
00:09:59,267 --> 00:10:00,387
然后也很聪明

279
00:10:00,387 --> 00:10:03,007
好这期视频就给大家介绍到这