逐字稿文字預覽
Can the open AI whisper model dostreaming ASR? My name is Bai. I'm a chine learning engineer and a PhD inatural language processing. And today Iwill answer this question. So if you haven't seen the whisper model, it is aspeechtoext model. It is trained onabout 100 languages on 680,000 hours of data. The model architecture is aencoder decoder transformer model and itcomes in five different sizes and it is growing in popularity recently becauseit is robust to noise and accents. Sothe question of the day is can this model do streaming speech recognition.So first of all what do we mean bystreaming ASR? Well, automated speech recognition can come in two forms batchand streaming. Batch means that themodel needs to take a bunch of speech and produce a bunch of text. Butstreaming ASR is when the model needs toproduce an output as the speaker isaying things with a delay of not more than a few seconds. So for example, ifyou're listening to a live broadcast,you don't want to first record the ntire broadcast and then produce thesubtitles, you want it to appear at mosta few seconds after it is said. And generally you'll expect streaming ASR tobe a few percentage points loweraccuracy than batch ASR because the model does not have access to all thecontext only the point up until thecurrent word. The question of how to do streaming ASR came up when I wasbuilding the voice writer. This is an AIwriting assistant that works in twosteps. In the first step, you just speak your thoughts without worrying too muchabout the grammar. And in the secondstep, the AI corrects the grammar for you. I've been finding it super helpfulfor writing all kinds of things,including emails, blog posts, and Slack messages. You can try it out for free inthe link here. And I'll also post a linkin the description of the video. Now, back to the video. First of all, why isthis even difficult? Why is it not so easy to use the Whisper model forstreaming ASR? The issue is Whisper is trained to process input audio of 30seconds long. If the audio is less than30 seconds, you can pad it to be longer. So, this is not too much of a problem.But what if your audio is longer than 30seconds? There is no way to feed into Whisper anything longer than 30 seconds.So, the first thing you might think ofis what if you just split it up into chunks of 30 seconds at a time andprocess each chunk one at a time. Well,the first problem is you might split it in the middle of the word. Like if yousplit it while in the middle of theword, this word will probably not berecognized correctly. The second problem is the latency will be really high. Soimagine if you're processing audio,you're waiting for audio to fill up these 30 seconds and only then will youprocess this entire chunk and then thelatency um for for the first word will be up to 30 seconds. Fortunately, thereis an open source report called WhisperStreaming that does this for you and it urns the Whisper model into a streamingASR system. There are instructions onhow to set this up. So, let's try it out. So, I run this command to start amodel running um it's using a whispersmall model and then I go to this tab and run another command and I just startspeaking. So you can pretty quickly seethat it's um picking it up and it's pretty quick and responsive and it evengives timestamps. So um it's pretty cooldemo. Um now let's talk about how this works. So we start with this audio fileand what we do is we feed chunks ofincreasing size into whisper. The size of these chunks is configured by theminimum chunk size parameter which bydefault is 1 second. So in eachiteration, we increase the size of this buffer by 1 second and feed all of itinto whisper. And this process continuesuntil we hit a end of sentence marker like a period or a question mark. Thenwe move the buffer forward and start theprocess again. So each piece of audio is processed by whisper multiple times asmany times as it takes until we hit thend of the sentence. The reasons for this is whisper is trained on sentences.So, it gives the best result when thestart of the audio aligns with the start of a sentence. And that's why the bufferonly moves forward when the previousentence is complete and we're beginning a new sentence. This way, Whisper neverneeds to start transcribing from themiddle of a sentence, which would give suboptimal results. In manyapplications, it is useful to have someresults available as soon as possible, even if they're not completely accurate.In the voice writer, I am showing theincomplete results in gray whereas the confirmed results are in black. Thedifference is the gray part of thesentence is unconfirmed. So it maychange as the model gets new information. But the black part is theconfirmed results and they arepermanent. How this works is using analgorithm called local agreement with n equals 2. This means that for a token tobe confirmed, it needs to be generatedin two consecutive audio buffers. Let's give an example. Let's say in the firstep, the worst per model outputs thethree tokens if you like and nothing is confirmed at this step. In the secondstep, the model produces more tokens,but only the first two tokens agree with e previous step. So those two areconfirmed. In the third and the fourthsteps, more tokens are generated. But at each step, it does not confirm anytokens until it is at least generated intwo consecutive chunks. So anything in the gray part is possible to change andgiven new information. For example, theword view might change to video once the model hears the rest of the sentence.But anything in the black part ispermanent. So even if the model wants to change it to something else after moreiterations it cannot be changed anymore.One last thing that this model does when it generates a new sentence is it feedsthe previous sentence into the model asprompt tokens. So this is something that you can do in the whisper format is youcan give it a bunch of prompt tokensbefore you start generating and this tends to improve th
本次執行流程與問題處理
下達指令
預期流程
- 自動字幕優先，Whisper 作為備援
- 來源選擇與品質檢查
- 文字校正與句子單元
- 雙語翻譯
- 雙語時間軸與學習單元
- 筆記與發布
- 公開網頁發布
1. 下載與語音辨識
工具：videonote-transcribe-url（主辨識管線：下載、Whisper 辨識與選稿）
處理：下載音訊、以 Whisper 辨識並選出最佳逐字稿
產出：yt_dlp_meta.json（影片元資料）、audio_16k_mono.wav（16kHz 單聲道音訊）、transcript.whisper.json（Whisper 辨識結果）、transcript.json（正式逐字稿）、transcript_selection_report.json（來源選擇報告）
耗時：3.8 秒
關係：依序執行
結果：完成
2. 全文快速校讀
工具：videonote_rapid_transcript_review.py
處理：先速讀完整逐字稿，再修正高可信突兀詞句並列出待確認項目
產出：rapid_transcript_review_report.json（產出檔案）
耗時：39.5 秒
關係：依序執行
結果：完成
3. 建立句子單元
工具：videonote-build-sentence-units
處理：將逐字稿切成帶時間軸的句子單元
產出：sentence_units.json（句子單元）
耗時：0.6 秒
關係：依序執行
結果：完成
4. 時間軸稽核
工具：videonote_timeline_audit.py
處理：檢查時間軸與音訊對齊
產出：timeline_audit_report.json（時間軸稽核）
耗時：0.6 秒
關係：依序執行
結果：完成
5. 品質檢查
工具：videonote_controlled_quality.py（檢查時間軸與品質）
處理：檢查語言、時間軸與可讀性品質
產出：controlled_quality_report.json（品質報告）、language_sanity_report.json（語言檢查）
耗時：0.6 秒
關係：依序執行
結果：完成
6. 播放時間軸回填
工具：videonote_rebase_display_timeline.py
處理：將網頁播放時間軸回填為校正字幕時間
產出：display_timeline_rebase_report.json（時間軸回填報告）
耗時：0.7 秒
關係：依序執行
結果：完成
7. 雙語翻譯
工具：vnt-sentence（句子雙語翻譯）
處理：將句子翻譯成繁體中文
產出：sentence_bilingual_transcript.json（雙語逐字稿）
耗時：47.8 秒
關係：依序執行
結果：完成
8. 雙語時間軸與學習單元
工具：videonote_learning_timeline_v2.py（建立雙語時間軸與學習單元）
處理：建立雙語時間軸與學習內容
產出：sentence_bilingual_timeline.html（雙語時間軸頁）、learning.json（學習單元）、japanese_learning.json（日文學習單元）
耗時：110.4 秒
關係：依序執行
結果：完成
9. 生成筆記摘要
工具：videonote-generate-note（生成筆記摘要）
處理：生成筆記摘要與重點
產出：note.md（筆記 Markdown）、note.html（筆記 HTML）
耗時：114.6 秒
關係：依序執行
結果：完成
10. 建立筆記頁面
工具：videonote-render-html（產生筆記 HTML）
處理：產生可閱讀的筆記 HTML 頁面
產出：note.html（筆記 HTML）
耗時：0.7 秒
關係：依序執行
結果：完成
11. 發布公開頁面
工具：videonote-publish（發布公開頁面）
處理：發布到公開網站與每日列表
產出：公開筆記網址、result.json（管線結果）
耗時：0.8 秒
關係：依序執行
結果：完成
手動來源字幕
自動來源字幕
Whisper 字幕
- 尚未產生
Qwen 字幕
- 尚未產生
Parakeet 字幕
- 尚未產生
MOSS 字幕
- 尚未產生
聲學時間證據（PyAnnote / WhisperX）
- 尚未產生
校正前後逐字稿
- 校正前 transcript.source.json 純文字 JSON
- 校正後 transcript.json 純文字 JSON
目前待解決問題與初步改善方向
- 逐字稿全文快速校讀未完成
已保留自動字幕/原始逐字稿並繼續生成筆記。；未完成的全文速讀項目可在精進模式回聽確認。；RAPID_REVIEW_STATUS=REVIEW_REQUIRED_INVALID_OUTPUT
- Qwen3-ASR 未完成，需確認 qwen_asr_lab 環境與音訊後重試精進模式。
- 音訊下載不穩定，需更新 cookies / PO token 後重試精進模式。