逐字稿文字預覽
hey everyone welcome to nning IO I'm JDand today we're going to be looking atreal time in browser speech recognition using web GPU and Transformer JS toactually use speech in the browser sowith that let's go ahead and getstarted all right the first thing that we're going to do everyone is we'regoing to go to this hugging face spaceand basically what we're going to do is look at this demo for the whisper webGPO uh it's real time in browserecognition it's using the whisper basewhich is a 73 million parameter speech recognition mod model and using onx andTransformer JS so you can see that theaudio audio visualizer is already picking up my voice from the microphoneand uh able to do some kind of uh visualwhat we're going to do is actually load this model and then we should be able tosee real time uh transcription but whatwe want to showis also that if you go and look at your network uh right now I have I'm on thiswasm tab and we want to pay attention toboth Fetch and wasm so we're going to go ahead and look at the model and you cansee it pulled it in really quickly uhcould be uh because it's cached but uh this was incredibly fast and it'salready doing the the transcription inreal time you can see that it's trying addpunctuation and uh the token count downhere if we go over to the wasm tab thisall that was loaded it's going through a worker to establish this andthe size was 4 megabytes so again thiscrazy fast for doing real time uh transcription in the browser superhelpful if you were trying to do uh likechat based systems where you actually wanted to have voice in with your chatas without having to go out to open AIor some otherum closed Source or even open source model that uh requires a backend callright again we didn't fetch anythingfrom our back end this was just loaded through the browser so now what we'regoing to do is we're going to take thisand we're going to actually look at the repo and and dig into the code so if youcome over here to Transformers Jsthe thing you want to note is you wanto be on uh the V3 Branch this is the the new Branchit's still experimental but it's goingto be the next version of Transformer JS a lot of great examples we've beentrying to go through some of them itfeels like every day they're coming out with a new demo a new model that they'reable to implement um definitely reallycool stuff so if we look at this example so we go into Transformer JS examplesand Whisper then we're able to actuallyD dive into this code so what we're going to do is we're just going to pullthis down get this running on our localmachine and then we'll start digging into the code so I'm going to use cursorI'm already I already have it pulledown but you can do uh get clone andthen come over here and just dothis uh we are so I'm in the root sowhat we need to do is go toexamples and then we're going to do web GPU and go towhisper and we'll do an npminstall and then we'll go ahead and doan npm runDev and what we're going to do is we have our Local Host here so we can gethis up and running we'll watch reallyquickly if we can gethis uh loaded again super quick you saw how quickly the the load time was forthat again some of it sometimes it'scashed and the first one's going to be slower but still reallyimpressive so if we look at some of thecore files in here we have our app whichis the main main file and what we're going to be looking at is how we'reactually pulling in this worker so thisworker. JS is where we're actually going to do our Logic for Transformer Js we'repulling in the model we're actuallysending messages back and forth where you can see right here that as messagesare coming being received we're actuallygoing to be looking for the type of event that is being pass and then sendthat uh data as wellso you can see most of this is for likeloading there's uh some update which is where you're updating the textwe're actually going to look and seewhere this is uh is working so if wecontinue down and we are looking at the recorder this is all the informationabout starting and stopping therecording and then what's Happening Here is we are actually looking at how thisinformation is being sent with this codeso righthere it's saying as the recording is coming in we're going to take a blob andchunk it and so then we're going to makea file reader and read those and then send that on when the file is beingread get that bit information that'sdecoded slice it and then send thoseslices over to our worker so we're posting the message to generate ourtranscription on the Fly then this isjust our uh our HTML that we're going to be pullingin so what we'll do is we'll jump overto the worker and then we'll take a lookat how this is actually functioning real quick everyone if you haven't alreadyplease remember to like And subscribe ithelps more than you know if you have any questions please leave them in thecomments and with that let's get back toit all right so now that we're in our worker file we're going to startlooking at how this actually functionsfirst thing is we're loading inTransformers we're going to pull in our Autokenizer our Auto processor and then uhwhisper and text stream as well as fullso if we look at what we're first doing we have this class which is our Pipelineand then we're actually using ourinstance where we're getting our model so we're saying this is our model ID wehave our tokenizer our processor we'reactually getting the model based on our ID at this point uh we're telling itwhat our dtype is and then specificallywe're using device webGPU and once all of these have processed uh then our git instance has beenestablished if we scroll down to thebottom we can kind of look at like someof the loader functions so you can see here there's the load this happens likewhere we're actually posting backmessages to actually see see theprogress bar things like that we're stablishing some events based on ourmessage and then once the the browser uhor once the model is actually loaded we're going to b
本次執行流程與問題處理
下達指令
預期流程
- 自動字幕優先，Whisper 作為備援
- 來源選擇與品質檢查
- 文字校正與句子單元
- 雙語翻譯
- 雙語時間軸與學習單元
- 筆記與發布
- 公開網頁發布
1. 下載與語音辨識
工具：videonote-transcribe-url（主辨識管線：下載、Whisper 辨識與選稿）
處理：下載音訊、以 Whisper 辨識並選出最佳逐字稿
產出：yt_dlp_meta.json（影片元資料）、audio_16k_mono.wav（16kHz 單聲道音訊）、transcript.whisper.json（Whisper 辨識結果）、transcript.json（正式逐字稿）、transcript_selection_report.json（來源選擇報告）
耗時：4.4 秒
關係：依序執行
結果：完成
2. 全文快速校讀
工具：videonote_rapid_transcript_review.py
處理：先速讀完整逐字稿，再修正高可信突兀詞句並列出待確認項目
產出：rapid_transcript_review_report.json（產出檔案）
耗時：39.7 秒
關係：依序執行
結果：完成
3. 建立句子單元
工具：videonote-build-sentence-units
處理：將逐字稿切成帶時間軸的句子單元
產出：sentence_units.json（句子單元）
耗時：0.7 秒
關係：依序執行
結果：完成
4. 時間軸稽核
工具：videonote_timeline_audit.py
處理：檢查時間軸與音訊對齊
產出：timeline_audit_report.json（時間軸稽核）
耗時：0.8 秒
關係：依序執行
結果：完成
5. 品質檢查
工具：videonote_controlled_quality.py（檢查時間軸與品質）
處理：檢查語言、時間軸與可讀性品質
產出：controlled_quality_report.json（品質報告）、language_sanity_report.json（語言檢查）
耗時：0.8 秒
關係：依序執行
結果：完成
6. 播放時間軸回填
工具：videonote_rebase_display_timeline.py
處理：將網頁播放時間軸回填為校正字幕時間
產出：display_timeline_rebase_report.json（時間軸回填報告）
耗時：0.8 秒
關係：依序執行
結果：完成
7. 雙語翻譯
工具：vnt-sentence（句子雙語翻譯）
處理：將句子翻譯成繁體中文
產出：sentence_bilingual_transcript.json（雙語逐字稿）
耗時：53.4 秒
關係：依序執行
結果：完成
8. 雙語時間軸與學習單元
工具：videonote_learning_timeline_v2.py（建立雙語時間軸與學習單元）
處理：建立雙語時間軸與學習內容
產出：sentence_bilingual_timeline.html（雙語時間軸頁）、learning.json（學習單元）、japanese_learning.json（日文學習單元）
耗時：115.1 秒
關係：依序執行
結果：完成
9. 生成筆記摘要
工具：videonote-generate-note（生成筆記摘要）
處理：生成筆記摘要與重點
產出：note.md（筆記 Markdown）、note.html（筆記 HTML）
耗時：96.6 秒
關係：依序執行
結果：完成
10. 建立筆記頁面
工具：videonote-render-html（產生筆記 HTML）
處理：產生可閱讀的筆記 HTML 頁面
產出：note.html（筆記 HTML）
耗時：0.7 秒
關係：依序執行
結果：完成
11. 發布公開頁面
工具：videonote-publish（發布公開頁面）
處理：發布到公開網站與每日列表
產出：公開筆記網址、result.json（管線結果）
耗時：0.7 秒
關係：依序執行
結果：完成
手動來源字幕
自動來源字幕
Whisper 字幕
- 尚未產生
Qwen 字幕
- 尚未產生
Parakeet 字幕
- 尚未產生
MOSS 字幕
- 尚未產生
聲學時間證據（PyAnnote / WhisperX）
- 尚未產生
校正前後逐字稿
- 校正前 transcript.source.json 純文字 JSON
- 校正後 transcript.json 純文字 JSON
目前待解決問題與初步改善方向
- 逐字稿全文快速校讀未完成
已保留自動字幕/原始逐字稿並繼續生成筆記。；未完成的全文速讀項目可在精進模式回聽確認。；RAPID_REVIEW_STATUS=REVIEW_REQUIRED_INVALID_OUTPUT
- Qwen3-ASR 未完成，需確認 qwen_asr_lab 環境與音訊後重試精進模式。
- 音訊下載不穩定，需更新 cookies / PO token 後重試精進模式。