WEBVTT

00:00:00.000 --> 00:00:16.320
開會更討厭什麼?除了開會本身,大概是會後整理錄音。一段一個小時的多人討論,導出錄音之後,你要把它變成會議記要。

00:00:16.860 --> 00:00:25.800
傳統的語音轉寫工具,確實能幫你把話變成字,但馬上你就會撞上一個更頭疼的問題,誰在什麼時候說了什麼?

00:00:25.800 --> 00:00:28.600
张三里四四个人轮流发言

00:00:28.600 --> 00:00:30.000
还有插话强话

00:00:30.000 --> 00:00:32.340
转写出来的文本是一锅粥

00:00:32.340 --> 00:00:35.400
根本分不清哪句是谁说的

00:00:35.400 --> 00:00:38.360
这就是所谓说话人分离难题

00:00:38.360 --> 00:00:42.060
最近复旦的团队放出了一个开源模型

00:00:42.060 --> 00:00:44.300
叫Mouse Transcribe Diaries

00:00:44.300 --> 00:00:48.520
他把转写和分离这两件事捏成了一个模型

00:00:48.520 --> 00:00:50.320
一次推理全部搞定

00:00:50.440 --> 00:00:52.140
而且准确率把豆包

00:00:52.140 --> 00:00:53.040
GPT

00:00:53.040 --> 00:00:53.680
4O

00:00:53.680 --> 00:00:54.400
Gemini

00:00:54.400 --> 00:00:55.620
都甩在后面

00:00:55.800 --> 00:00:57.800
这期就来拆一拆它

00:00:57.800 --> 00:00:59.900
先把问题说清楚

00:00:59.900 --> 00:01:01.920
语音转写这件事

00:01:01.920 --> 00:01:03.840
技术上叫ASR

00:01:03.840 --> 00:01:05.220
自动语音识别

00:01:05.220 --> 00:01:07.640
单独看其实已经很成熟了

00:01:07.640 --> 00:01:10.540
你手机输入法的语音转文字就是

00:01:10.540 --> 00:01:14.800
但真实场景远比一个人对着话筒念要复杂

00:01:14.800 --> 00:01:18.920
会议、播客、访谈、电话、讲座

00:01:18.920 --> 00:01:21.300
这些录音有三个共同特征

00:01:21.300 --> 00:01:22.860
第一、长

00:01:22.860 --> 00:01:25.240
动不动几十分钟到一个小时

00:01:25.240 --> 00:01:29.940
第二,人多,两个到十几个人不等,还互相打断

00:01:29.940 --> 00:01:36.360
第三,杂,有背景噪音,有关门声笑声,还有大段沉默的停顿

00:01:36.360 --> 00:01:40.000
对这种录音,光把话转成字远远不够

00:01:40.000 --> 00:01:44.200
你还想知道每句话是谁说的,发生在哪一秒

00:01:44.200 --> 00:01:49.440
后者,有个专门的名字,叫Dialization,说话人分离

00:01:49.940 --> 00:01:52.620
传统做法怎么解决这个问题的?

00:01:53.240 --> 00:01:54.140
拼流水线

00:01:54.140 --> 00:01:55.360
第一步

00:01:55.360 --> 00:01:56.340
用VAD

00:01:56.340 --> 00:01:58.320
也就是语音活动检测

00:01:58.320 --> 00:02:01.720
把录音切成一小段一小段有人声的片段

00:02:01.720 --> 00:02:02.900
第二步

00:02:02.900 --> 00:02:04.520
用ASR模型

00:02:04.520 --> 00:02:06.080
把每段转成文字

00:02:06.080 --> 00:02:07.520
第三步

00:02:07.520 --> 00:02:10.140
再上独立的说话人分离模型

00:02:10.140 --> 00:02:12.300
给每段贴上说话人标签

00:02:12.300 --> 00:02:13.600
第四步

00:02:13.600 --> 00:02:15.040
做时间戳对齐

00:02:15.040 --> 00:02:16.000
把文字

00:02:16.000 --> 00:02:16.920
说话人

00:02:16.920 --> 00:02:18.880
时间这三样东西拼回去

00:02:18.880 --> 00:02:20.740
听起来挺合理

00:02:20.740 --> 00:02:21.820
问题在于

00:02:21.820 --> 00:02:23.840
每一步都是独立的模型

00:02:23.840 --> 00:02:25.360
每一步都会犯错

00:02:25.360 --> 00:02:27.480
而且错误会一路累积

00:02:27.480 --> 00:02:29.320
VAD切错了

00:02:29.320 --> 00:02:30.720
后面全跟着错

00:02:30.720 --> 00:02:33.100
ASR听差了一个词

00:02:33.100 --> 00:02:34.920
分离模型也跟着乱

00:02:34.920 --> 00:02:36.500
更要命的是

00:02:36.500 --> 00:02:38.420
这些模型各自为正

00:02:38.420 --> 00:02:40.000
谁也看不到全局

00:02:40.000 --> 00:02:44.220
比如同一个人的两段画被切到了不同片段

00:02:44.220 --> 00:02:47.500
分离模型很可能把它们当成两个人

00:02:47.500 --> 00:02:50.160
这就是模块化方案的死学

00:02:50.160 --> 00:02:51.880
局部每一步都最优

00:02:51.880 --> 00:02:53.620
拼起来全局崩盘

00:02:53.620 --> 00:02:57.380
这次放模型的团队叫OpenMOSS

00:02:57.380 --> 00:02:59.420
中文名模斯智能

00:02:59.420 --> 00:03:03.360
是从复旦大学邱熙鹏教授团队孵化出来的

00:03:03.360 --> 00:03:06.340
邱熙鹏这个名字你可能不熟

00:03:06.340 --> 00:03:09.720
但他写的那本神经网络与深度学习

00:03:09.720 --> 00:03:11.880
是国内很多高校的教材

00:03:11.880 --> 00:03:15.940
他带的团队之前做过一个很有名的对话大模型

00:03:15.940 --> 00:03:16.940
也叫MOS

00:03:16.940 --> 00:03:21.300
号称中国第一个LayChatGPT的开源模型

00:03:21.300 --> 00:03:25.160
所以这支队伍在开源圈是有传统的

00:03:25.160 --> 00:03:28.480
这次他们做的Mouse Transcribed Diaries

00:03:28.480 --> 00:03:30.240
参数量0.9B

00:03:30.240 --> 00:03:32.880
大约9亿完全开源

00:03:32.880 --> 00:03:35.100
权重放在Hugging Face上

00:03:35.100 --> 00:03:37.300
协议是Apache 2.0

00:03:37.300 --> 00:03:39.540
意思是你可以拿去商用

00:03:39.540 --> 00:03:40.920
不用交一分钱

00:03:40.920 --> 00:03:44.060
这个体量加上这个开放程度

00:03:44.060 --> 00:03:47.480
对个人开发者和中小团队特别友好

00:03:47.480 --> 00:03:51.500
那它凭什么能把转写和分离捏在一起呢

00:03:51.500 --> 00:03:52.760
看架构

00:03:52.760 --> 00:03:56.800
它其实是把两个现成的墙模型做了架接

00:03:56.800 --> 00:03:58.300
音频那一侧

00:03:58.300 --> 00:04:00.980
用的是Whisper Medium的编码器

00:04:00.980 --> 00:04:04.920
就是OpenAI那个Whisper的中等版本

00:04:04.920 --> 00:04:07.320
负责把声音变成声学特征

00:04:07.320 --> 00:04:08.780
文本那一侧

00:04:08.780 --> 00:04:11.980
用的是QAM3 0.6B的解码器

00:04:11.980 --> 00:04:13.720
通一千问的小模型

00:04:13.720 --> 00:04:15.760
负责生成文字

00:04:15.760 --> 00:04:19.200
中间用一个适配器把两者连起来

00:04:19.200 --> 00:04:21.940
把音频特征做四倍的时间压缩

00:04:21.940 --> 00:04:24.040
再塞进文本模型里去

00:04:24.040 --> 00:04:26.740
这样做有两个直接的好处

00:04:26.740 --> 00:04:27.780
第一

00:04:27.780 --> 00:04:30.440
它能一口气吃下超长音频

00:04:30.440 --> 00:04:32.860
上下文窗口12万8千

00:04:32.860 --> 00:04:34.000
换算下来

00:04:34.000 --> 00:04:36.520
大约能一次处理90分钟的录音

00:04:36.520 --> 00:04:37.960
完全不用切片

00:04:37.960 --> 00:04:39.020
第二

00:04:39.020 --> 00:04:43.020
转写和分离是在同一个模型里联合训练的

00:04:43.020 --> 00:04:45.840
模型天生就知道这段话是谁的

00:04:45.840 --> 00:04:48.240
根本不存在事后拼接的问题

00:04:48.240 --> 00:04:51.760
输入是16000赫兹的单声道音频

00:04:51.760 --> 00:04:55.940
输出直接就是带说话人标签和时间戳的文本

00:04:55.940 --> 00:04:58.400
输出的格式非常紧凑

00:04:58.400 --> 00:04:59.800
一眼就能看懂

00:04:59.800 --> 00:05:03.100
他把整段录音编码成一个时间流

00:05:03.100 --> 00:05:05.420
每句话前面标两样东西

00:05:05.420 --> 00:05:07.600
一个是方括号里的时间戳

00:05:07.600 --> 00:05:10.260
一个是方括号里的说话人编号

00:05:10.260 --> 00:05:11.700
举个例子

00:05:11.700 --> 00:05:12.720
开头几秒

00:05:12.720 --> 00:05:14.300
方括号0.48

00:05:14.300 --> 00:05:15.920
方括号S01

00:05:15.920 --> 00:05:18.560
意思是0.48秒开始

00:05:18.560 --> 00:05:20.280
说话人1号发言

00:05:20.280 --> 00:05:21.920
内容是欢迎各位

00:05:21.920 --> 00:05:23.820
到1.66秒结束

00:05:23.820 --> 00:05:26.880
紧接着跳到12.26秒

00:05:26.880 --> 00:05:28.340
方括号S02

00:05:28.340 --> 00:05:30.440
说话人2号接过话茬

00:05:30.440 --> 00:05:34.040
这个格式可以直接为给后处理脚本

00:05:34.040 --> 00:05:36.020
解析成结构化的段落

00:05:36.020 --> 00:05:38.220
每段都带起始时间

00:05:38.220 --> 00:05:39.360
结束时间

00:05:39.360 --> 00:05:40.800
说话人

00:05:40.800 --> 00:05:42.260
文本这四个字段

00:05:42.260 --> 00:05:46.260
想导出城JSON还是字幕格式都行

00:05:46.260 --> 00:05:48.220
对开发者很友好

00:05:48.220 --> 00:05:51.560
你不用再自己写证则去硬抠时间戳了

00:05:51.560 --> 00:05:53.660
到了最关键的环节

00:05:53.660 --> 00:05:55.060
它到底有多准

00:05:55.060 --> 00:05:57.940
团队在四个基准上做了评测

00:05:57.940 --> 00:06:04.860
全是真实的长音频多人场景

00:06:04.860 --> 00:06:08.340
横跨中文会议和英文播客电影

00:06:08.340 --> 00:06:11.100
核心指标是字符错误率

00:06:11.100 --> 00:06:12.200
越低越好

00:06:12.200 --> 00:06:15.160
我挑几个最能说明问题的数字

00:06:15.160 --> 00:06:18.840
在Azheil 4这个中文会议基准上

00:06:18.840 --> 00:06:22.160
它的错误率是14.84%

00:06:22.160 --> 00:06:22.940
作为对比

00:06:22.940 --> 00:06:24.820
豆包是18.18%

00:06:24.820 --> 00:06:28.240
Gemini 3 Pro是22.75%

00:06:28.240 --> 00:06:29.020
也就是说

00:06:29.020 --> 00:06:31.560
它一个0.9B的开源模型

00:06:31.560 --> 00:06:35.280
把字节和谷歌的商业模型都压在了身下

00:06:35.280 --> 00:06:39.040
在Movies这个英文电影基准上更夸张

00:06:39.040 --> 00:06:42.100
它的错误率只有6.36%

00:06:42.200 --> 00:06:45.600
而GPT4O是14.37

00:06:45.600 --> 00:06:47.320
豆包是9.94

00:06:47.320 --> 00:06:50.000
电影对白里有背景音

00:06:50.000 --> 00:06:50.900
有口音

00:06:50.900 --> 00:06:51.780
有重叠

00:06:51.780 --> 00:06:53.360
是最难啃的骨头

00:06:53.360 --> 00:06:54.560
它照样通吃

00:06:54.560 --> 00:06:57.320
这里还要特别说一个指标

00:06:57.320 --> 00:06:58.680
叫德尔塔CP

00:06:58.680 --> 00:07:02.080
它衡量的是说话人分离的额外代价

00:07:02.080 --> 00:07:03.300
通俗讲就是

00:07:03.300 --> 00:07:05.160
为了分清是谁说的

00:07:05.160 --> 00:07:06.780
你额外多错了多少

00:07:06.780 --> 00:07:09.520
它的这个值只有0.99

00:07:09.520 --> 00:07:10.720
几乎为0

00:07:10.720 --> 00:07:15.140
意味着分清说话人这件事几乎不拖累转写准确率

00:07:15.140 --> 00:07:18.760
这在传统流水线里是想都不敢想的

00:07:18.760 --> 00:07:23.180
那它跟现在最流行的开源方案比到底差在哪

00:07:23.180 --> 00:07:27.980
先说Whisper,OpenAI的老牌选手,能力很强

00:07:27.980 --> 00:07:29.440
但他只管转写

00:07:29.440 --> 00:07:34.500
不管分离,你想分清谁说的,得自己再外挂一个模型

00:07:34.860 --> 00:07:39.620
再说WhisperX,这个是社区里用的最多的组合方案

00:07:39.620 --> 00:07:44.860
他在Whisper基础上接了VAD和PiNote的分离模型

00:07:44.860 --> 00:07:47.000
确实能做对齐和分离

00:07:47.000 --> 00:07:49.500
但本质上还是拼接流水线

00:07:49.500 --> 00:07:51.560
前面说的错误累积问题

00:07:51.560 --> 00:07:53.040
他一个都没躲掉

00:08:08.426 --> 00:08:13.826
你像个搬砖工一样把它们拼起来,每块砖之间都有缝隙。

00:08:13.826 --> 00:08:21.826
Moth这套做法的价值就在于,他把这些砖直接烧成了一整块磁砖,从根上消除了缝隙。

00:08:21.826 --> 00:08:24.426
这么能打,部署难不难?

00:08:24.426 --> 00:08:26.126
他给了三条路。

00:08:26.126 --> 00:08:32.526
第一条,研究用的,直接拿原生Transformers库加载,几行,Pytho。

00:08:32.526 --> 00:08:38.326
N代码就能跑,适合你想看内部细节,做二次开发的场景。

00:08:38.326 --> 00:08:41.426
第二条生产环境要高吞吐

00:08:41.426 --> 00:08:43.606
用SGLine Omni部署

00:08:43.606 --> 00:08:46.206
这是一个专门的服务框架

00:08:46.206 --> 00:08:49.306
单张H100显卡就能跑起来

00:08:49.306 --> 00:08:50.606
并发16路的时候

00:08:50.606 --> 00:08:53.066
每秒能处理82秒的音频

00:08:53.066 --> 00:08:55.986
换算成时时率是0.092

00:08:55.986 --> 00:09:00.246
意味着处理速度是音频播放速度的10倍以上

00:09:00.246 --> 00:09:02.406
如果走长音频单路

00:09:02.406 --> 00:09:05.526
时时率最低能压到0.0197

00:09:05.526 --> 00:09:08.926
也就是处理速度是音频时长的50倍

00:09:08.926 --> 00:09:12.206
一个小时的录音一分多钟就处理完了

00:09:12.206 --> 00:09:13.706
第三条

00:09:13.706 --> 00:09:15.086
用VLLM

00:09:15.086 --> 00:09:17.346
适合你的环境是Cuda12

00:09:17.346 --> 00:09:21.166
因为SGLAM Omni目前要求Cuda13

00:09:21.166 --> 00:09:23.806
VLLM的兼容性更好

00:09:23.806 --> 00:09:25.246
三条路

00:09:25.246 --> 00:09:26.606
从研究到生产

00:09:26.606 --> 00:09:29.046
基本把各种需求都覆盖了

00:09:29.046 --> 00:09:32.406
但不是所有人都愿意敲命令型

00:09:32.406 --> 00:09:36.006
所以团队还做了一个字幕网页应用

00:09:36.006 --> 00:09:38.066
叫MPD Subtitle Web

00:09:38.066 --> 00:09:40.246
它就是个浏览器界面

00:09:40.246 --> 00:09:42.606
你上传一段音频或视频

00:09:42.606 --> 00:09:44.126
它自动跑模型

00:09:44.126 --> 00:09:47.926
跑完你能在网页上直接审阅转写结果

00:09:47.926 --> 00:09:50.106
谁说了什么一目了然

00:09:50.106 --> 00:09:52.086
审完点一下导出

00:09:52.086 --> 00:09:53.386
格式随你挑

00:09:53.386 --> 00:09:55.406
JSON给程序用

00:09:55.406 --> 00:09:58.466
SRT和S是标准字幕格式

00:09:58.466 --> 00:10:00.546
直接位给播放器就行

00:10:00.546 --> 00:10:04.946
甚至还能用Fpack把字幕稍录进视频

00:10:04.946 --> 00:10:07.526
一键导出一个带字幕的MP4

00:10:07.526 --> 00:10:09.966
界面支持中文和英文

00:10:09.966 --> 00:10:12.626
会自动跟着你浏览器的语言走

00:10:12.626 --> 00:10:15.526
还有个很实用的小功能叫热词

00:10:15.526 --> 00:10:17.926
如果你录音里有专有名词

00:10:17.926 --> 00:10:18.706
人名

00:10:18.706 --> 00:10:19.846
行业术语

00:10:19.846 --> 00:10:21.386
模型老式识别错

00:10:21.386 --> 00:10:24.246
你可以在提示词里加几个热词

00:10:24.246 --> 00:10:26.566
引导它往正确的方向认

00:10:26.566 --> 00:10:29.846
这对行业内部的录音特别管用

00:10:29.846 --> 00:10:32.626
这种能力到底能用在哪

00:10:32.626 --> 00:10:35.246
我自己最看好的几个场景

00:10:35.246 --> 00:10:37.326
第一会议纪要

00:10:37.326 --> 00:10:39.146
一个小时的多人会议

00:10:39.146 --> 00:10:40.086
扔进去

00:10:40.086 --> 00:10:43.826
几分钟出来一份带说话人和时间戳的全文

00:10:43.826 --> 00:10:46.706
秘书或者你自己再做摘要就行

00:10:46.706 --> 00:10:49.726
省下来的时间是实打实的

00:10:49.726 --> 00:10:52.446
第二播客和视频字幕

00:10:52.446 --> 00:10:56.226
做内容的人最痛苦的就是给节目上字幕

00:10:56.226 --> 00:10:58.706
这个工具能直接出SRT

00:10:58.706 --> 00:11:00.866
连谁说的都给你雕好了

00:11:00.866 --> 00:11:02.006
第三

00:11:02.006 --> 00:11:03.206
访谈整理

00:11:03.206 --> 00:11:03.986
记者

00:11:03.986 --> 00:11:05.906
研究员做智性访谈

00:11:05.906 --> 00:11:08.386
几十段录音要转写要编码

00:11:08.386 --> 00:11:10.986
这是真正能解放生产力的工具

00:11:10.986 --> 00:11:12.086
第四

00:11:12.086 --> 00:11:13.526
更专业的领域

00:11:13.526 --> 00:11:14.606
法庭记录

00:11:14.606 --> 00:11:15.686
医疗口述

00:11:15.686 --> 00:11:16.786
客服质检

00:11:16.786 --> 00:11:21.546
这些对准确率和说话人识别要求都很高的场景

00:11:21.546 --> 00:11:25.366
他的杀手检就是那90分钟的长上下文

00:11:25.366 --> 00:11:27.746
意味着绝大部分真实录音

00:11:27.746 --> 00:11:29.226
它能一口气吃完

00:11:29.226 --> 00:11:30.846
不用你手动切块

00:11:30.846 --> 00:11:32.326
想试的话

00:11:32.326 --> 00:11:33.706
装起来不复杂

00:11:33.706 --> 00:11:36.366
前提是Python 3.12

00:11:36.366 --> 00:11:38.346
用UV这个包管理器

00:11:38.346 --> 00:11:39.706
建一个虚拟环境

00:11:39.706 --> 00:11:42.626
然后从GitHub客龙仓库

00:11:42.626 --> 00:11:45.306
一行UVPIP Install装好依赖

00:11:45.306 --> 00:11:49.006
它会自动帮你处理PyTorch的GPU后端

00:11:49.006 --> 00:11:50.946
装完有两条路

00:11:50.946 --> 00:11:52.986
想看代码细节的

00:11:52.986 --> 00:11:56.386
直接用Transformers加在模型跑推里

00:11:56.386 --> 00:11:58.186
想跑成服务的

00:11:58.186 --> 00:12:03.826
用SGLang Omni或者VLLM启动一个API服务

00:12:03.826 --> 00:12:07.406
调用方式兼容OpenAI那套音频接口

00:12:07.406 --> 00:12:10.366
你原来用Whisper API的代码

00:12:10.366 --> 00:12:12.986
改个地址就能直接迁移过来

00:12:12.986 --> 00:12:15.646
模型权重从Hugging Face拉

00:12:15.646 --> 00:12:17.166
第一次跑会下载

00:12:17.166 --> 00:12:18.466
大约两个G

00:12:18.466 --> 00:12:22.366
最后说一个让这个项目分量很重的头衔

00:12:22.366 --> 00:12:25.186
这个模型7月9号才开源

00:12:25.186 --> 00:12:27.046
到现在不到20天

00:12:27.046 --> 00:12:29.946
GitHub已经冲到1300多star

00:12:29.946 --> 00:12:34.646
还登上了Trendshift当日Python仓库的第11名

00:12:34.646 --> 00:12:36.326
更硬的成绩是

00:12:36.326 --> 00:12:39.806
2026年语音领域的鼎会Interspeech

00:12:39.806 --> 00:12:43.626
办了第二届多语言低资源语音挑战赛

00:12:43.626 --> 00:12:46.526
他拿了第一名覆盖14种语言

00:12:46.526 --> 00:12:49.166
一个久易参数的开源模型

00:12:49.166 --> 00:12:53.206
在长音频多说话人这个最难的真实场景上

00:12:53.206 --> 00:12:56.806
同时打赢了准确率和效率两场仗

00:12:56.806 --> 00:12:59.126
还把代码和权重都敞开了

00:12:59.126 --> 00:13:00.726
代码在GitHub

00:13:00.726 --> 00:13:02.406
搜OpenMOSS

00:13:02.406 --> 00:13:03.486
斜杠

00:13:03.486 --> 00:13:06.586
MouseTranscribedDiaries就能找到

00:13:06.586 --> 00:13:08.146
权重在HuggingFace

00:13:08.146 --> 00:13:11.226
如果你平时要跟长录音打交道

00:13:11.226 --> 00:13:12.206
做字幕

00:13:12.206 --> 00:13:13.126
做记要

00:13:13.126 --> 00:13:14.446
做访谈整理

00:13:14.446 --> 00:13:17.606
这个工具真的值得装一个试试

00:13:17.606 --> 00:13:19.126
这期到这里

00:13:19.126 --> 00:13:20.326
我们下期见