工作清單｜How I Ran Qwen3.8-27B GSQ-RCO on a 24GB Mac Mini !

逐字稿文字預覽

Hi guys, welcome back.
In this video, we'll be talking about how you can run a QN 3.8 27 billion parameter model in an Apple M4 Mac Mini machine with just 24 GB of unified memory.
We know that QN 3.8 27 billion parameter model will be of at least 15 GB of size.
and loading the entire model in the unified memory will not only load the model but also
loads the kvcache runtime and the mac operating system all together and while you load all of
them in the vram it is definitely going to hit the memory ceiling for you and it is going to be
getting you the out of memory exception so how can we even handle the situation of these kind of
memory problem and can we even run a QUEN 3.8 27 billion parameter model in this machine and is it
just the problem with the model itself or is it the problem with the quantization that we have
but what if we have a quantization of the same exact model with just two-bit quantization instead
of the four-bit quantization can we run that at all well welcome to the QUEN model quantized to
2-bit called as the QEN3.827BGSQRCOIQ2QS model and this model can load in the 24GB of unified
machine and it can support a full GPU offloading as well as it has multi-token prediction enabled
as well and this model can easily fit in an Apple M4 machine of 24GB of unified memory
which includes the model kv cache runtime and mac operating system without hitting the memory
ceiling as well so it can easily load everything for you within your apple machine which is quite
amazing so the memory requirement is going to be so much less because of the two-bit quantization
that you have got in this particular model so how exactly is this possible will the quen 3.8 right
now will lose its intelligence ability or the output will be of less quality or how exactly
is this happening what exactly is this new model of a two-bit quantization well this particular
quen model is a standard quen model but instead of applying the same quantization everywhere the
technique will determine which tensor deserves more precision and which can tolerate the aggressive
compression that's exactly what this particular technique does and what exactly is this technique
that we're talking about the gsq rco or otherwise called as the gumbel softmax quantization
which intelligently optimizes the quantization value while RCO intelligently decide where to
allocate more or fewer bits across the models or tens are so essentially even if the model sizes
is going to be reduced from the 16 bpw which is a 16-bit original model to a 2-bit quantization
model it is going to be of almost 6.4 times of a size lower but still the model is going to give
you an amazing precision over here that is the power of this particular model itself and that's
the reason why even though the size of the model is so much smaller the quality of the output is
still going to be way more same like how you can expect with a quantization 4 model of the same
QN model that you have got that's exactly what this particular model is and we are going to see
in this particular video how we can run the same exact QN 2-bit model in the Mac Mini M4 machine
and we're also going to see how fast it is going to be doing the inferencing for us and what it
means with different sizes of the KV cache adjustment that I'm going to be doing in this
particular video and i will show you how it is going to be affecting the the speed as well as
the number of tokens per second as well so in this particular video we are going to see which kv size
or which kv cache is going to be faster like 16 bit or 4 bit or 8 bit quantization and which is
going to take more time and what is the total number of tokens per second that you are going
to be getting uh with different kinds of settings that you are going to be doing in the lm studio
and one more time this particular model is actually not available in the mlx format i
unfortunately have got only the ggf format and i am going to be running this particular model
in the lm studio and i will show you how the tps is going to be for the different kv cache that we
are going to be setting and running it but you know what this particular model is not really
going to hit the out of memory exception based on the different kv cache setting you can actually
see that this model is actually going to be performant as well and this is the good news for
this particular model well as that said let's start doing the testing of the model with different
quantization so you can see that i have got my mac mini m4 machine total number of physical
that you can see over here is 24 gig and i'm going to be loading the uh the quen model uh in the lm
studio and i'm going to set the context length over here uh to at least like a 30 31 or 32 000
uh over here and you can see that this is the maximum concurrent prediction or instead of four
i'm going to be setting it to one because it's going to be reducing the total lump compute as
well and i'm also going to change the kv cache quantization type there so instead of using the
default i'm going to be setting the fp16 this time and i'm going to be loading this particular model
and the moment i'm going to load this particular model you can see that it is going to be consuming
around 20 gig or 21 gig of the total memory uh in my uh in my ram there and now i'm going to be
running the the questions over there inferencing it without the thinking enabled so the moment i'm
going to run let's say hi over here i'm getting around 4.57 tokens per second and now if i'm
going to say again like hi again it is going to give me around 4.9 tokens per second this is a
dense model that we're talking about guys look at that and now if i'm going to pass a long prompt
to write a selenium c sharp dot net code for google search operation with page object model
and dependency injection and if i'm going to run it i'm going to start the timer as well just in
case and if i'm going to show you the memory spike there you see that the memory pressure is no

來源：transcript.txt

本次執行流程與問題處理

下達指令
指令：videonote https://www.youtube.com/watch?v=rfcf3HDB5f8 --language zh --bilingual --learning --auto-source-subtitles

來源：https://www.youtube.com/watch?v=rfcf3HDB5f8

辨識語言：zh ｜參數：--language zh --bilingual --learning --auto-source-subtitles
預期流程
依指令產生的處理順序：
1. 自動字幕優先，Whisper 作為備援
2. 來源選擇與品質檢查
3. 文字校正與句子單元
4. 雙語翻譯
5. 雙語時間軸與學習單元
6. 筆記與發布
7. 公開網頁發布
1. 下載與語音辨識
工具：videonote-transcribe-url（主辨識管線：下載、Whisper 辨識與選稿）
處理：下載音訊、以 Whisper 辨識並選出最佳逐字稿
產出：yt_dlp_meta.json（影片元資料）、audio_16k_mono.wav（16kHz 單聲道音訊）、transcript.whisper.json（Whisper 辨識結果）、transcript.json（正式逐字稿）、transcript_selection_report.json（來源選擇報告）
耗時：74.2 秒
關係：依序執行
結果：完成
2. 全文快速校讀
工具：videonote_rapid_transcript_review.py
處理：先速讀完整逐字稿，再修正高可信突兀詞句並列出待確認項目
產出：rapid_transcript_review_report.json（產出檔案）
耗時：41.9 秒
關係：依序執行
結果：完成
3. 建立句子單元
工具：videonote-build-sentence-units
處理：將逐字稿切成帶時間軸的句子單元
產出：sentence_units.json（句子單元）
耗時：0.9 秒
關係：依序執行
結果：完成
4. 句子密度平衡
工具：videonote_balance_sentence_units.py
處理：平衡句子長度與密度
產出：sentence_units.json（句子單元）
耗時：1.0 秒
關係：依序執行
結果：完成
5. 時間軸稽核
工具：videonote_timeline_audit.py
處理：檢查時間軸與音訊對齊
產出：timeline_audit_report.json（時間軸稽核）
耗時：1.0 秒
關係：依序執行
結果：完成
6. 品質檢查
工具：videonote_controlled_quality.py（檢查時間軸與品質）
處理：檢查語言、時間軸與可讀性品質
產出：controlled_quality_report.json（品質報告）、language_sanity_report.json（語言檢查）
耗時：0.1 秒
關係：依序執行
結果：完成
7. 播放時間軸回填
工具：videonote_rebase_display_timeline.py
處理：將網頁播放時間軸回填為校正字幕時間
產出：display_timeline_rebase_report.json（時間軸回填報告）
耗時：0.1 秒
關係：依序執行
結果：完成
8. 逐字稿交叉檢查
工具：videonote_cross_check_transcript.py
處理：交叉檢查逐字稿與來源字幕
產出：transcript_crosscheck_report.json（交叉檢查）
耗時：0.1 秒
關係：依序執行
結果：完成
9. 來源句整理
工具：videonote_prepare_source_entries.py
處理：整理來源句子供時間軸與頁面使用
產出：sentence_units.json（句子單元）
耗時：0.1 秒
關係：依序執行
結果：完成
10. 雙語時間軸與學習單元
工具：videonote_learning_timeline_v2.py（建立雙語時間軸與學習單元）
處理：建立雙語時間軸與學習內容
產出：sentence_bilingual_timeline.html（雙語時間軸頁）、learning.json（學習單元）、japanese_learning.json（日文學習單元）
耗時：110.0 秒
關係：依序執行
結果：完成
11. 生成筆記摘要
工具：videonote-generate-note（生成筆記摘要）
處理：生成筆記摘要與重點
產出：note.md（筆記 Markdown）、note.html（筆記 HTML）
耗時：109.6 秒
關係：依序執行
結果：完成
12. 建立筆記頁面
工具：videonote-render-html（產生筆記 HTML）
處理：產生可閱讀的筆記 HTML 頁面
產出：note.html（筆記 HTML）
耗時：0.7 秒
關係：依序執行
結果：完成
13. 發布公開頁面
工具：videonote-publish（發布公開頁面）
處理：發布到公開網站與每日列表
產出：公開筆記網址、result.json（管線結果）
耗時：0.8 秒
關係：依序執行
結果：完成

手動來源字幕

尚未產生

自動來源字幕

subtitle_acquisition.json 下載143 bytes

Whisper 字幕

transcript.whisper.raw.json 下載59,880 bytes

Qwen 字幕

尚未產生

Parakeet 字幕

尚未產生

MOSS 字幕

尚未產生

聲學時間證據（PyAnnote / WhisperX）

尚未產生

校正前後逐字稿

校正前 transcript.json 純文字 JSON 11,054 / 56,025 bytes
校正後 transcript.json 純文字 JSON 11,054 / 56,025 bytes

目前待解決問題與初步改善方向

Whisper 幻覺段落已切除待修正
已移除 1 段幻覺內容，其餘逐字稿保留使用。；原始 Whisper 證據保留在 transcript.whisper.raw.json。
診斷筆記：目前沒有可可靠理解的逐字稿待修正
asr_language_failure；wrong_language_after_segmented_asr；宣告語言與辨識結果不一致；原始證據已保留。；筆記仍已建立並可進入精進模式；未捏造影片摘要或翻譯。
逐字稿品質檢核未完全通過待修正
requested_language_repair_inconclusive；保留 133 段 Whisper 逐字稿，已先行發布。
逐字稿語言與宣告語言不一致待修正
已保留來源字幕/Whisper 內容並繼續產生筆記；請確認影片語言。；若語音辨識輸出英文，也會被語言檢查標記，不會覆蓋日文底稿。
逐字稿全文快速校讀未完成待修正
已保留自動字幕/原始逐字稿並繼續生成筆記。；未完成的全文速讀項目可在精進模式回聽確認。；RAPID_REVIEW_STATUS=REVIEW_REQUIRED_INVALID_OUTPUT
逐字稿可先使用，但有待校正清單待修正
segment=305 adjacent_duplicate: 與前一段完全重複，可能是滾動字幕或 ASR 重複
逐字稿品質檢核未完全通過待修正
{'stage': 'canonical_transcript_quality', 'action': 'published_with_warning', 'signals': ['transcript_selection_not_resolved', 'asr_language_mismatch:zh']}
Qwen3-ASR 未完成，需確認 qwen_asr_lab 環境與音訊後重試精進模式。

進一步精進字幕與筆記
使用 Workbench 上傳校正字幕、中文參考稿或回饋 SRT，重新產生雙語與學習單元。
開啟 Workbench

逐字稿文字預覽

本次執行流程與問題處理

下達指令

預期流程

1. 下載與語音辨識

2. 全文快速校讀

3. 建立句子單元

4. 句子密度平衡

5. 時間軸稽核

6. 品質檢查

7. 播放時間軸回填

8. 逐字稿交叉檢查

9. 來源句整理

10. 雙語時間軸與學習單元

11. 生成筆記摘要

12. 建立筆記頁面

13. 發布公開頁面

手動來源字幕

自動來源字幕

Whisper 字幕

Qwen 字幕

Parakeet 字幕

MOSS 字幕

聲學時間證據（PyAnnote / WhisperX）

校正前後逐字稿

目前待解決問題與初步改善方向