工作清單

How I Ran Qwen3.8-27B GSQ-RCO on a 24GB Mac Mini !

逐字稿文字預覽

Hi guys, welcome back. In this video,we'll be talking about how you can run aQwen in an AppleM4 Mac mini machine with just 24 GB of
unified memory. We know that Qwen 3.8 27billion parameter model will be of atleast 15 GB of size. And loading the
ntire model in the unified memory willnot only load the model, but also loadsthe KV cache, runtime, and the Mac
operating system all together. And whileyou load all of them in the VRAM, it isdefinitely going to hit the memory
ceiling for you, and it is going to begetting you the out of memory exception.So, how can we even handle the situation
of these kind of memory problem? And can
we even run a Qwen 3.8 27 billionparameter model in this machine? And is
it just the problem with the modelitself, or is it the problem with thequantization that we have? But what if
we have a quantization of the same exactmodel with just two-bit quantizationinstead of the four-bit quantization?
Can we run that at all? Well, welcome to
the Qwen model quantized to two-bitcalled as the Qwen 3.8 27 billion GSQRCO IQ2 QS model. And this model canload in the 24 GB of unified machine,
and it can support a full GPUoffloading, as well as it hasmulti-token prediction enabled as well.And this model can easily fit in an
Apple M4 machine of 24 GB of unifiedmemory, which includes the model, KVcache, runtime, and Mac operating system
without hitting the memory ceiling aswell. So, it can easily load everythingfor you within your Apple machine, which
is quite amazing. So, the memoryrequirement is going to be so much lessbecause of the two-bit quantization that
you have got in this particular model.So, how exactly is this possible? Will
the Qwen 3.8 right now will lose itsintelligence ability or the output willbe of less quality or how exactly is
this happening? What exactly is this new
model of a two-bit quantization? Well,
this particular Qwen model is a standardQwen model, but instead of applying thesame quantization everywhere, the
technique will determine which tensorsdeserves more precision and which cantolerate the aggressive compression.
That's exactly what this particulartechnique does. And what exactly is thistechnique that we're talking about? The
GSQ RCO or otherwise called as theGumbel Softmax quantization, whichintelligently optimizes the quantization
value while RCO intelligently decidewhere to allocate more or fewer bitsacross the model's uh tensors. So,
essentially, even if the model sizes isgoing to be reduced from the 16 BPW,which is a 16-bit original model, to a
two-bit quantization model, it is goingto be off almost 6.4 time of a sizelower, but still the model is going to
give you an amazing precision over here.That is the power of this particularmodel itself. And that's the reason why
even though the size of the model is somuch smaller, the quality of the outputis still going to be way more same like
how you can expect with theirquantization four model of the same Qwenmodel that you have got. That's exactly
what this particular model is, and weare going to see in this particularvideo how we can run the same exact Qwen
two-bit model in the Mac mini M4machine, and we're also going to see howfast it is going to be doing theinferencing for us and what it means
with different sizes of the KV cacheadjustment that I'm going to be doing inthis particular video. Now, I will show
you how it is going to be affecting thespeed as well as the number oftokens per second as well. So, in thisparticular video, we are going to see
which KV size or which KV cache is goingto be faster like 16-bit or 4-bit or8-bit quantization and which is going to
take more time and what is the totalnumber of tokens per second that you aregoing to be gettingwith different kinds of settings that
you are going to be doing in the LMStudio. And one more time, thisparticular model is actually notavailable in the MLX format. I
unfortunately have got only the GGMLformat and I'm going to be running thisparticular model in the LM Studio and I
will show you how the TPS is going to befor the different KV cache that we aregoing to be setting and running it. But,
you know what? This particular model is
not really going to hit the out ofmemory exception. Based on the differentKV cache setting, you can actually see
that this model is actually going to beperformant as well. And this is a goodnews for this particular model. Well, as
that said, let's start doing the testingof the model with a differentquantization. So, you can see that Ihave got my Mac mini M4 machine. The
total number of physical memory that youcan see over here is 24 gig and I'mgoing to be loading theQwen model in the LM Studio and I'm
going to set the context length overhere to at least like a 30 31 or 32,000over here. And you can see that this is
the maximum concurrent prediction.Instead of four, I'm going to be settingit to one cuz it's going to be reducing
the total number of compute as well. AndI'm also going to change the KV cachequantizationtype there. So, instead of using the
default, I'm going to be setting theFP16 this time. And I'm going to beloading this particular model. And the
moment I'm going to load this particularmodel, you can see that it is going tobe consuming around 20 gig or 21 gig of
the total memory in myRAM there. And now I'm going to berunning the the questions over there,inferencing it without the thinking
enabled. So the moment I'm going to run,let's say hi over here, I'm gettingaround 4.57 tokens per second. And now
if I'm going to say again like hi again,it is going to give me around 4.9 tokensper second. This is a dense model that
we're talking about, guys. Look at that.And now if I'm going to pass a longprompt to write a Selenium C# .NET code
for Google search operation with pageobject model and dependency injection.And if I'm going to run it, I'm going to
start the timer as well just in case.And if I'm going to show you the memoryspike there, you see that the memory
pressure is not increasing because it'sa two-bit q

來源：transcript.txt

本次執行流程與問題處理

  1. 下達指令

    指令：videonote https://www.youtube.com/watch?v=rfcf3HDB5f8 --language en --bilingual --learning --auto-source-subtitles

    來源：https://www.youtube.com/watch?v=rfcf3HDB5f8

    辨識語言：en ｜ 參數：--language en --bilingual --learning --auto-source-subtitles

  2. 預期流程

    依指令產生的處理順序：

    1. 自動字幕優先，Whisper 作為備援
    2. 來源選擇與品質檢查
    3. 文字校正與句子單元
    4. 雙語翻譯
    5. 雙語時間軸與學習單元
    6. 筆記與發布
    7. 公開網頁發布

  3. 1. 下載與語音辨識

    工具：videonote-transcribe-url（主辨識管線：下載、Whisper 辨識與選稿）

    處理：下載音訊、以 Whisper 辨識並選出最佳逐字稿

    產出：yt_dlp_meta.json（影片元資料）、audio_16k_mono.wav（16kHz 單聲道音訊）、transcript.whisper.json（Whisper 辨識結果）、transcript.json（正式逐字稿）、transcript_selection_report.json（來源選擇報告）

    耗時：5.6 秒

    關係：依序執行

    結果：完成

  4. 2. 全文快速校讀

    工具：videonote_rapid_transcript_review.py

    處理：先速讀完整逐字稿，再修正高可信突兀詞句並列出待確認項目

    產出：rapid_transcript_review_report.json（產出檔案）

    耗時：42.4 秒

    關係：依序執行

    結果：完成

  5. 3. 建立句子單元

    工具：videonote-build-sentence-units

    處理：將逐字稿切成帶時間軸的句子單元

    產出：sentence_units.json（句子單元）

    耗時：0.5 秒

    關係：依序執行

    結果：完成

  6. 4. 時間軸稽核

    工具：videonote_timeline_audit.py

    處理：檢查時間軸與音訊對齊

    產出：timeline_audit_report.json（時間軸稽核）

    耗時：0.5 秒

    關係：依序執行

    結果：完成

  7. 5. 品質檢查

    工具：videonote_controlled_quality.py（檢查時間軸與品質）

    處理：檢查語言、時間軸與可讀性品質

    產出：controlled_quality_report.json（品質報告）、language_sanity_report.json（語言檢查）

    耗時：0.5 秒

    關係：依序執行

    結果：完成

  8. 6. 播放時間軸回填

    工具：videonote_rebase_display_timeline.py

    處理：將網頁播放時間軸回填為校正字幕時間

    產出：display_timeline_rebase_report.json（時間軸回填報告）

    耗時：0.6 秒

    關係：依序執行

    結果：完成

  9. 7. 雙語翻譯

    工具：vnt-sentence（句子雙語翻譯）

    處理：將句子翻譯成繁體中文

    產出：sentence_bilingual_transcript.json（雙語逐字稿）

    耗時：71.0 秒

    關係：依序執行

    結果：完成

  10. 8. 雙語時間軸與學習單元

    工具：videonote_learning_timeline_v2.py（建立雙語時間軸與學習單元）

    處理：建立雙語時間軸與學習內容

    產出：sentence_bilingual_timeline.html（雙語時間軸頁）、learning.json（學習單元）、japanese_learning.json（日文學習單元）

    耗時：129.3 秒

    關係：依序執行

    結果：完成

  11. 9. 生成筆記摘要

    工具：videonote-generate-note（生成筆記摘要）

    處理：生成筆記摘要與重點

    產出：note.md（筆記 Markdown）、note.html（筆記 HTML）

    耗時：102.1 秒

    關係：依序執行

    結果：完成

  12. 10. 建立筆記頁面

    工具：videonote-render-html（產生筆記 HTML）

    處理：產生可閱讀的筆記 HTML 頁面

    產出：note.html（筆記 HTML）

    耗時：0.1 秒

    關係：依序執行

    結果：完成

  13. 11. 發布公開頁面

    工具：videonote-publish（發布公開頁面）

    處理：發布到公開網站與每日列表

    產出：公開筆記網址、result.json（管線結果）

    耗時：0.2 秒

    關係：依序執行

    結果：完成

手動來源字幕

自動來源字幕

Whisper 字幕

Qwen 字幕

Parakeet 字幕

MOSS 字幕

聲學時間證據（PyAnnote / WhisperX）

校正前後逐字稿

目前待解決問題與初步改善方向

進一步精進字幕與筆記
使用 Workbench 上傳校正字幕、中文參考稿或回饋 SRT，重新產生雙語與學習單元。
開啟 Workbench