0:00.080–0:06.720
Quen's 27 billion parameter model reads images and text. And the lab behind most local model
0:06.720–0:13.760
compression just shrank it into a single 11.8 GB file. A model that size normally needs more memory
0:13.760–0:19.680
than any gaming computer has. This one is smaller than a video game install. Needless to say,
0:19.680–0:25.120
this is a seriously impressive demonstration of model compression. But the question must be asked,
0:25.120–0:31.120
can a 12 GBTE graphics card actually run Quen's 27 billion parameter model? To find out,
0:31.120–0:35.440
we have to look at the repository where we see that the download page doesn't actually offer
0:35.440–0:40.800
a single file, but a menu of four different sizes shipped two ways for a total of eight
0:40.800–0:47.280
model files. That 11.8 GB model from our title is the largest of the plane options, which the
0:47.280–0:53.280
lab recommends as its task lossless build. But sitting right beside those models is a ninth file,
0:53.280–1:00.320
a 0.93 GBTE vision projector called MM Proj. None of the eight main checkpoints actually contain
1:00.320–1:04.400
this vision projector, which means you have to get it separately if you want to use the model's
1:04.400–1:12.560
multimodal capabilities. On paper, the 11.8 GB of the GSQ RCO build and the 12 GB of your graphics
1:12.560–1:17.520
card sit right next to each other. But those two numbers are actually measured in entirely
1:17.520–1:23.360
different units. When a card is sold as having 12 GB of memory, the operating system actually
1:23.360–1:29.120
counts that space in binary gigabytes, which is what your hardware actually has to work with. On
1:29.120–1:37.120
the other side of the ledger, that recommended GSQ RCO build is listed on HuggingFace at 11.77 GB.
1:37.120–1:44.320
But when you inspect the actual weights of that GSQ RCO build, they only take up 10.96 GB of space
1:44.320–1:52.000
because the download page is using decimal GB. If you subtract 10.96 from your 12 GB of capacity,
1:52.000–1:58.960
you are left with 1.04 GB of free space, that means the actual headroom on your card is larger
1:58.960–2:03.760
than the 200 megabytes the headline number suggested. But that extra headroom is about
2:03.760–2:08.720
to vanish because the model's ability to read images is stored in a completely separate file
2:08.720–2:18.320
called mm praj, which weighs 931 million bytes or 0.87 GB. And in a release designed entirely around
2:18.320–2:24.320
model compression, this vision projector is the only file that wasn't compressed at all. Instead,
2:24.320–2:29.840
the lab left it at its original 16-bit format. And because one copy serves every single build
2:29.840–2:34.560
on the menu, this memory penalty is exactly the same whether you pick the largest budget
2:34.560–2:41.040
or the smallest. In llama.cpp, you load this projector using a separate command line flag,
2:41.040–2:48.480
which makes it easy to forget that it's even there. But once it loads, 0.8 87 GB of the 1.04 we
2:48.480–2:54.240
just found is instantly gone. This means the part of the model the title sells, the system that lets
2:54.240–2:59.280
it read images, is the one part the compression never touched, and it eats almost all of the
2:59.280–3:04.400
headroom your card has left. The quantization paper behind this build splits the field of
3:04.400–3:11.680
model compression into simple scalar methods like GPTQ and AWQ that everyone deploys and vector or
3:11.680–3:18.800
trellis methods like Q-tip and AQLM that score better. In the paper's own words, scalar methods
3:18.800–3:24.000
plateau in accuracy at 3 to four bits, while the more accurate vector family is notoriously
3:24.000–3:30.640
hard to implement and scale. So GSQ closes most of that accuracy gap while staying on a scalar grid,
3:30.640–3:34.880
ensuring the compressed model remains fully compatible with the existing scalar inference
3:34.880–3:41.280
kernels we already run on our cards. It turns out the same lab published GPTQ, which is one of the
3:41.280–3:47.520
standard methods GSQ measures itself against. On top of that, GSQ can be applied directly to an
3:47.520–3:53.200
existing GGUF release you already have, improving its accuracy and exporting it right back into the
3:53.200–4:00.480
exact same format. This file runs in your everyday apps like Llama.cpp, Olama, and LM Studio without
4:00.480–4:05.840
modifying any of them. And that's not luck. It happens because the lab deliberately chose the
4:05.840–4:11.200
family of methods that scores worse in order to stay inside a format the existing software already
4:11.200–4:15.920
reads. Compressing a single tensor doesn't build a whole file because you still have to
4:15.920–4:21.600
decide how much compression to apply to each part of the model to fit your hardware. To solve that,
4:21.600–4:27.280
Michael Hellig and Dan Allistar published a second paper titled Model Compression with Exact Budget
4:27.280–4:32.400
Constraints, framing the problem as assigning one of several options to each group under a
4:32.400–4:38.240
total cost budget. While penalty-based approaches only enforce a budget approximately and require
4:38.240–4:44.800
extensive hyperparameter tuning, this method meets it exactly without any constraintspecific tuning.
4:44.800–4:50.160
Relaxing the budget constraint describes a smooth curved surface called a remmanion manifold in
4:50.160–4:55.760
lojit space allowing a gradient method to walk along it without ever leaving it. The procedure
4:55.760–5:01.440
takes three steps. You quantize every tensor into every candidate format, search for the assignment
5:01.440–5:06.640
that meets your budget and stitch those winners into one ordinary file. And that means you can
5:06.640–5:11.360
spend your memory budget down to the last bite, which is exactly the problem you face when trying
5:11.360–5:17.920
to squeeze a 27 billion parameter model onto a 12 GB card. You don't even have to download the model
5:17.920–5:23.120
to see what that search algorithm decided because the lab published the entire decision list as a
5:23.120–5:30.240
plain text file. Inside the recommended build, the search algorithm ended up slicing 851 individual
5:30.240–5:37.040
tensors across 11 different storage formats. It left 96 tensors at full 16-bit precision
5:37.040–5:42.240
while dropping the rest into a staircase of smaller quantization formats. For instance,
5:42.240–5:48.880
the search assigned 144 tensors to the IQ3S format. And the measured average lands right
5:48.880–5:55.200
on the lab's 3.5 target, but non-uniform is not just a marketing word, and the search isn't afraid
5:55.200–6:00.160
to starve a single component if it means saving the rest of the network. In the published dump,
6:00.160–6:07.120
block 13's feed forward gate is squeezed down to a starved format called IQ1M, running on just 1 and
6:07.120–6:13.280
3/4 bits. That single tensor isn't surrounded by full precision. It sits inside a block whose other
6:13.280–6:19.200
matrices were all given 2, 3, and 4bit formats. It's simply the cheapest thing in a neighborhood
6:19.200–6:27.040
of ordinary compromises. More than a tenth of the 11.77 GB file is just vocabulary. Specifically,
6:27.040–6:33.280
two tensors that do nothing but turn tokens into numbers and back account for 1.12 GB of
6:33.280–6:38.720
the file size together, split between the output head and the embedding table. Against that,
6:38.720–6:44.960
every uncompressed tensor in the file adds up to 58 megabytes, under half a percent. Inside a
6:44.960–6:50.160
single attention block, the search gave the query projection the cheapest format in its block, while
6:50.160–6:55.520
the key and value neighbors beside it kept more expensive ones. If you watch that same starved
6:55.520–7:00.720
gate across the four size budgets, it starts at just one bit, goes up to four bits at the
7:00.720–7:06.160
intermediate three-bit budget and drops back under two bits at the largest. This happens because a
7:06.160–7:12.000
global budget optimizer isn't ranking tensors by importance. It's trading bytes and it'll happily
7:12.000–7:16.560
starve a tensor at a larger budget than it fed it at a smaller one because the only thing it's
7:16.560–7:20.960
protecting is the total. Every performance score for these compressed builds comes from the lab
7:20.960–7:25.920
that performed the compression, published on its own model card, and nobody outside that lab has
7:25.920–7:32.480
reproduced these comparisons. According to their results, the recommended 11.8 GB file matches the
7:32.480–7:41.040
uncompressed 53.8 GB original exactly on AIM25 and Live Codebench, representing a 4.6 time size
7:41.040–7:46.960
reduction. On GPQA Diamond, this recommended build trails the original by only half a point,
7:46.960–7:51.760
and the scores remain high even as you step down to the threebit build, which is a gigabyte and
7:51.760–7:59.200
a half smaller. At 10.09 GB, this 3bit version still hits a perfect 100 on that main reasoning
7:59.200–8:04.640
benchmark, while trailing the recommended build by only half a point on GPQA Diamond. And near
8:04.640–8:11.760
the bottom of the size scale, their 2.75 bit build actually recovers 101.8% 8% of the original
8:11.760–8:18.800
zeroshot average, scoring above the uncompressed baseline. At a matched 8.4 GB, their build beats
8:18.800–8:24.960
Unslaught's dynamic quantizations by 10 points on Aim25. Because the gap between the recommended
8:24.960–8:30.080
build and the one a gigabyte and a half smaller is only half a point on one benchmark, picking a
8:30.080–8:36.400
build is not really a quality decision. The base model's configuration lists 64 transformer layers
8:36.400–8:42.240
in a repeating pattern of three linear attention layers followed by one full attention layer. That
8:42.240–8:49.040
leaves 48 gated delta layers and 16 full attention layers that the files own tensor names confirm. A
8:49.040–8:54.640
llama.cpp bug report describes the running model in those exact terms, which matters because the
8:54.640–8:59.840
way these layers process information changes how memory behaves. While a full attention
8:59.840–9:04.880
layer has to keep every token of the conversation available, so its memory grows with every word,
9:04.880–9:10.400
a recurrent layer keeps one fixed size summary that is the same size after 10 words as after
9:10.400–9:17.280
10,000. That fixed size summary is the reason 48 of its 64 layers never accumulate conversation
9:17.280–9:23.040
history. And that is the reason a 12 GB graphics card is in this conversation at all. We can work
9:23.040–9:28.560
this out directly from the base configuration. The model has 16 attention layers and each layer
9:28.560–9:36.320
holds two key value tensors containing four heads of 256 numbers stored at two bytes a piece. That
9:36.320–9:42.800
multiplies out to 64 kilob for every single token in the conversation. Compressing that cache brings
9:42.800–9:49.040
the price down. Drop it to 8 bits and you spend 34 kilob a token. Squeeze it to four bits and
9:49.040–9:56.400
you only spend 18. Meanwhile, those 48 recurrent layers cost a flat 150 megbytes at any context
9:56.400–10:03.600
length. So when Alibaba advertises a native 262,000 token context, that window has a very
10:03.600–10:12.080
real price. At full precision, the cache alone needs 16 GB. That drops to 8 1/2 GB at 8 bit and
10:12.080–10:18.080
4 1/2 GB at 4bit. We can check that arithmetic against a machine known to work. A developer
10:18.080–10:24.480
running this model at full context on a 16 GB card has to use a 4-bit cache, which the math puts at
10:24.480–10:31.920
4 1/2 GB. If all 64 layers were attention, that cache would require 18 GB and it wouldn't fit on
10:31.920–10:37.600
their card at all. So, what does that actually leave for our 12 GB card? But what happens at
10:37.600–10:42.560
the long context this model advertises on the software your card would actually be running,
10:42.560–10:49.040
like llama.cpp. In late August, a developer filed a bug report showing that at long context,
10:49.040–10:54.560
the model occasionally generates an end of text marker as its very first token with absolutely no
10:54.560–11:00.240
error. The failure is stubborn, reproducing on the graphics card, on the CPU with the same build,
11:00.240–11:05.280
and on an independent fork of the codebase. But the reporter's differential test points to an
11:05.280–11:11.840
engine defect rather than the weights themselves since the exact same prompt passes cleanly on VLM.
11:11.840–11:19.280
When testing context capacity, around 113,000 tokens passes cleanly, but around 130,000 fails
11:19.280–11:24.240
with that instant end of text marker, while a slightly deeper prompt works perfectly fine.
11:24.240–11:29.440
Again, keep in mind this was measured on another lab's quantizations rather than this build. And
11:29.440–11:35.360
a sibling model with only 30 recurrent layers survives much deeper. The reporter eventually
11:35.360–11:40.240
withdrew his own proposed threshold because of these random holes of failure, advising that
11:40.240–11:45.600
there is no safe limit to quote, "and you simply have to validate the exact depths you plan to
11:45.600–11:52.000
serve." Olama pulls this exact build straight from hugging face with a single command. And although a
11:52.000–11:57.040
maintainer demonstrated it working, actual usage breaks on the chat template rather than your
11:57.040–12:03.040
graphics card. The built-in template refuses any system message that isn't first. So if you send
12:03.040–12:09.280
one mid conversation, you get an empty string, a finished status, and absolutely no error,
12:09.280–12:14.640
fixing it takes a threeline patch in a model file. And since this exact fault was filed twice
12:14.640–12:21.520
in August against the plain weights, it isn't this build's doing. Switching to llama.cpp's multimodal
12:21.520–12:26.800
tool is even worse because failing to reserve memory while encoding an image crashes the process
12:26.800–12:32.400
with a segmentation fault instead of reporting an error. Its crash log shows the tool dying on
12:32.400–12:39.840
a failed attempt to allocate just 247 megabytes while reading a single picture. LM Studio avoids
12:39.840–12:45.360
crashing entirely, but its offload slider will quietly push excess weights into ordinary system
12:45.360–12:51.120
memory rather than refuse to load them. The lab's entire LM Studio guidance is a single
12:51.120–12:56.400
sentence telling you to search the repository, and nobody has published a test of this build on a 12
12:56.400–13:02.320
GB card. None of the three applications tells you when the model won't fit, which is why the only
13:02.320–13:08.320
way to know if it runs is to count your bites before you click download. In early September,
13:08.320–13:14.640
a developer running a 16 GB Blackwell card with a specific toolkit version found that Llama.cpp's
13:14.640–13:20.720
own back-end test failed matrix multiplication for three formats, including the exact format this
13:20.720–13:26.720
build is named after. The reported errors reached up to 1.1, which is large enough to mean you just
13:26.720–13:32.320
get random garbage output, but the neighboring formats in that same run passed, making the error
13:32.320–13:38.880
highly specific. Then a second user reported the same garbage output on another Blackwell card,
13:38.880–13:45.040
fixing it locally with a small change to the bite handling code. But a third user on the same chip
13:45.040–13:49.680
generation couldn't reproduce it, arguing the fault lies in the build chain rather than the
13:49.680–13:55.600
hardware since there's no chip specific path for these formats. Anyway, while this debate is still
13:55.600–14:01.120
unresolved, choosing a file assembled from 11 formats means you're depending on 11 distinct
14:01.120–14:07.280
code paths. And those least used paths are exactly where the arguments are still happening. The
14:07.280–14:17.120
recommended model file itself is 11.77 GB, but on your card, those weights consume 10.96 GB of VRAM.
14:17.120–14:23.600
Adding the separate vision projector eats another 0.87 GB of memory, which brings the combined total
14:23.600–14:33.680
to 11.83 GB and leaves you with a tiny pocket of just 174 MGB. Subtracting the flat 150 megabytes
14:33.680–14:40.240
demanded by the recurrent state layers leaves 24 megabytes of free headroom. At 64 kilobytes
14:40.240–14:48.000
per token, that remaining space can hold at most about 386 tokens and fewer in practice. Of course,
14:48.000–14:53.040
that is a theoretical ceiling that completely ignores whatever graphics memory your desktop
14:53.040–14:57.840
operating system is already using, not to mention the working memory that the inference software
14:57.840–15:04.240
itself reserves before it allocates a single token of cash. If you try to step up to the largest
15:04.240–15:12.080
12.12 GB build, adding the projector overshoots your card's memory by 308 MGB, meaning it cannot
15:12.080–15:18.320
hold its own eyes at zero context. But if you go text only and work out the math without that
15:18.320–15:25.280
vision projector, the recommended build leaves 912 megabytes of free space, which translates to about
15:25.280–15:34.320
14 12,000 tokens of context at full precision or up to 27,000 tokens at 8 bit. Your 12 GB graphics
15:34.320–15:40.640
card is fully capable of running this 27 billion parameter model, but it simply cannot handle the
15:40.640–15:46.800
specific 11.8 8 GB build that the laboratory recommends once that build can see. The file
15:46.800–15:54.640
you actually want is the 3bit build which comes in at 10.09 GB on the download page or about 9.40
15:54.640–15:59.840
in your card's own units. Once you load those weights together with the vision projector,
15:59.840–16:07.200
you're left with 1,624 megabytes of headroom. That leaves a ceiling of about 26,000 tokens of active
16:07.200–16:13.040
conversation with images enabled, which jumps to 40,000 tokens if you're running text only. If you
16:13.040–16:18.800
optimize your cache or step down to the even smaller 2.75 bit build, those numbers stretch
16:18.800–16:24.000
even further. Keep in mind, these are absolute mathematical ceilings that ignore the runtime's
16:24.000–16:29.040
own working buffers, so your real capacity is lower. According to the laboratory's own
16:29.040–16:35.040
evaluation, that downgrade costs half a point of GPQA diamond and about a point of live codebench
16:35.040–16:41.600
with the perfect score of 100 on AIM25 completely unchanged. And that tiny reduction in accuracy
16:41.600–16:47.600
buys you roughly 26,000 tokens of conversation. Two separate teams have filed the same request
16:47.600–16:53.280
on the lab's repository, but with no reply so far, outside developers can only recover the final map
16:53.280–16:58.960
of which format went to which tensor without any way to actually run the search themselves. They've
16:58.960–17:03.440
asked for the missing pieces because several critical steps like the candidate generation
17:03.440–17:09.040
are completely absent from the public code. Specifically, they're missing the per tensor cost
17:09.040–17:13.920
model, the settings for two of the budgets, and the export code that assembles the mixed format
17:13.920–17:19.280
file. One of those teams wants the code because their server fleet is limited by memory bandwidth
17:19.280–17:24.720
rather than capacity, making a smaller file a direct speed win, and this mixed format is their
17:24.720–17:31.360
only real deployment path. The budget paper names mixed precision quantization, non-uniform pruning,
17:31.360–17:36.400
and choosing which experts to keep as the same problem of assigning one of K options to each of
17:36.400–17:42.320
N groups, which means this is a method for fitting almost anything into a fixed memory footprint.
17:42.320–17:47.120
So, while the lab's release gives us a finished answer to one specific model's budget, the one
17:47.120–17:52.320
thing they haven't shipped yet is the method for anyone else's. Anyways, that's all, folks.

0:00.080–0:06.720
Quen 的 270 億參數模型可以讀取影像和文字。而負責大多數本地模型
0:06.720–0:13.760
壓縮的實驗室，只是將其壓縮成單一 11.8 GB 的檔案。這種體積的模型通常需要比任何遊戲電腦更多的記憶體
0:13.760–0:19.680
比任何遊戲電腦的記憶體都大。這個檔案比一個遊戲安裝檔還小。不用說，
0:19.680–0:25.120
這確實是一個令人印象深刻的模型壓縮示範。但必須提出一個問題，
0:25.120–0:31.120
12 GB 的顯示卡真的能運行 Quen 的 270 億參數模型嗎？為了找出答案，
0:31.120–0:35.440
我們必須查看該儲存庫，我們發現下載頁面實際上並未提供
0:35.440–0:40.800
單一檔案，而是提供四種不同大小的選單，並以兩種方式提供，總共八個
0:40.800–0:47.280
模型檔案。標題中提到的 11.8 GB 模型是平面選項中最大的，實驗室
0:47.280–0:53.280
建議將其作為任務無損構建。但在這些模型旁邊，還有一個第九個檔案，
0:53.280–1:00.320
一個名為 MM Proj 的 0.93 GB 視覺投影器。八個主要檢查點中實際上並沒有包含
1:00.320–1:04.400
這個視覺投影器，這意味著如果你想使用模型的
1:04.400–1:12.560
多模態功能，你必須單獨獲取它。從紙面上看，GSQ RCO 構建的 11.8 GB 和你的顯示卡的 12 GB
1:12.560–1:17.520
記憶體緊鄰在一起。但實際上，這兩個數字的測量單位完全不同。
1:17.520–1:23.360
當顯示卡被銷售為擁有 12 GB 記憶體時，作業系統實際上是以二進位千兆位元組來計算該空間，這才是你的硬體實際可用的空間。在
1:23.360–1:29.120
帳簿的另一端，該建議的 GSQ RCO 構建在 HuggingFace 上列為 11.77 GB。
1:29.120–1:37.120
但當你檢查該 GSQ RCO 構建的實際權重時，它們只佔用 10.96 GB 的空間，
1:37.120–1:44.320
因為下載頁面使用的是十進位 GB。如果你從 12 GB 的容量中減去 10.96，
1:44.320–1:52.000
你將剩下 1.04 GB 的可用空間，這意味著你的顯示卡實際上的剩餘空間比標題數字所暗示的
1:52.000–1:58.960
200 MB 還要大。但這些額外的剩餘空間即將消失，因為模型讀取影像的能力存儲在一個完全不同的檔案中
1:58.960–2:03.760
，名為 mm praj，其重量為 9.31 億位元組或 0.87 GB。而在一個完全圍繞
2:03.760–2:08.720
模型壓縮設計的發布中，這個視覺投影器是唯一完全沒有壓縮的檔案。相反地，
2:08.720–2:18.320
實驗室將其保留在原始的 16 位元格式。由於一份副本服務於選單上的每一個構建
2:18.320–2:24.320
模型壓縮，這個視覺投影機是唯一完全沒有壓縮的檔案。相反地，
2:24.320–2:29.840
還是最小的，這個記憶體懲罰都是完全相同的。在 llama.cpp 中，你使用單獨的命令列旗標加載這個投影器，
2:29.840–2:34.560
這使得很容易忘記它的存在。但一旦加載，我們剛才找到的 1.04 GB 中的 0.87 GB 就會瞬間消失。這意味著標題所銷售的模型部分，讓它
2:34.560–2:41.040
讀取影像的系統，是壓縮從未觸及的部分，它幾乎耗盡了
2:41.040–2:48.480
你的顯示卡所剩餘的所有空間。支持此構建的量化論文將模型壓縮領域分為簡單的標量方法，如 GPTQ 和 AWQ，這些是每個人都會部署的方法，以及向量或
2:48.480–2:54.240
柵格方法，如 Q-tip 和 AQLM，這些方法得分更高。在論文的原文中，標量方法
2:54.240–2:59.280
在 3 到 4 位元時準確率達到平台期，而更準確的向量家族以難以
2:59.280–3:04.400
實現和擴展而聞名。因此，GSQ 在保持標量網格的同時，縮小了大部分準確率差距，
3:04.400–3:11.680
確保壓縮後的模型與我們已在顯示卡上運行的現有標量推理
3:11.680–3:18.800
內核完全兼容。結果顯示，同一個實驗室發布了 GPTQ，這是 GSQ 用來衡量自身的標準方法之一。此外，GSQ 可以直接應用於
3:18.800–3:24.000
你已經擁有的現有 GGUF 發布，提高其準確率並將其導出回
3:24.000–3:30.640
完全相同的格式。這個檔案可以在你日常的應用程式中運行，如 Llama.cpp、Olama 和 LM Studio，而無需
3:30.640–3:34.880
修改任何一個。這並非運氣。這是因為實驗室刻意選擇了
3:34.880–3:41.280
我們已在顯示卡上執行的核心。結果顯示，同一個實驗室發表了 GPTQ，這是其中一種
3:41.280–3:47.520
方法家族，以便留在現有軟體已經讀取的格式內。壓縮單一張量並不會構建整個檔案，因為你仍然必須
3:47.520–3:53.200
決定對模型的每個部分應用多少壓縮以適應你的硬體。為了解決這個問題，
3:53.200–4:00.480
完全相同的格式。這個檔案可以在你日常使用的應用程式中運行，例如 Llama.cpp、Olama 和 LM Studio，而無需
4:00.480–4:05.840
修改任何一個。這並非運氣使然。這是因為實驗室刻意選擇了
4:05.840–4:11.200
為了符合現有軟體已經能讀取的格式，選擇了評分較低的編碼方法家族。
4:11.200–4:15.920
壓縮單一張量並不會建立整個檔案，因為你仍然必須
4:15.920–4:21.600
決定對模型的每個部分應用多少壓縮，以適應你的硬體。為了解決這個問題，
4:21.600–4:27.280
Michael Hellig 和 Dan Allistar 發表了第二篇論文，題為《具有精確預算限制的模型壓縮》，
4:27.280–4:32.400
將問題框架為在總成本預算下，為每個群組分配多個選項之一。雖然基於懲罰的方法僅近似地執行預算，且需要
4:32.400–4:38.240
大量的超參數調整，但此方法則能精確地滿足預算，無需任何特定於約束的調整。
4:38.240–4:44.800
放寬預算約束描述了一個稱為殘留流形的平滑曲面，位於
4:44.800–4:50.160
邏輯空間中，允許梯度方法沿著它行走而不會離開。該程序
4:50.160–4:55.760
分為三個步驟。你將每個張量量化為每個候選格式，搜尋符合你預算的分配，並將這些勝出者縫合成一個普通檔案。這意味著你可以
4:55.760–5:01.440
將你的記憶體預算精確用到最後一丁點，這正是你在嘗試
5:01.440–5:06.640
將 270 億參數的模型塞進 12 GB 的顯示卡時所面臨的問題。你甚至不需要下載模型
5:06.640–5:11.360
就能知道搜尋演算法做了什麼決定，因為該實驗室將整個決策列表發布為
5:11.360–5:17.920
純文字檔案。在推薦的建構中，搜尋演算法最終將 851 個獨立
5:17.920–5:23.120
張量分割到 11 種不同的儲存格式中。它將 96 個張量保留在完整的 16 位元精度
5:23.120–5:30.240
，而將其餘的張量放入較小的量化格式階梯中。例如，
5:30.240–5:37.040
搜尋將 144 個張量分配給 IQ3S 格式。而測量的平均值正好落在
5:37.040–5:42.240
實驗室的 3.5 目標上，但非均勻不僅是一個行銷用語，搜尋也不害怕
5:42.240–5:48.880
如果意味著節省網路的其餘部分，就讓單一組件挨餓。在發布的轉儲中，
5:48.880–5:55.200
區塊 13 的前饋閘門被壓縮到稱為 IQ1M 的挨餓格式，僅運行在 1 和
5:55.200–6:00.160
3/4 位元上。該單一張量並未周圍環繞著完整精度。它位於一個區塊內，該區塊的其他
6:00.160–6:07.120
矩陣都被賦予了 2、3 和 4 位元的格式。它僅僅是普通妥協區域中
6:07.120–6:13.280
最便宜的東西。超過 11.77 GB 檔案的十分之一僅僅是詞彙。具體來說，
6:13.280–6:19.200
兩個僅負責將詞元轉換為數字並轉回來的張量，合計佔據了
6:19.200–6:27.040
檔案大小的 1.12 GB，分別分配給輸出頭部和嵌入表。與此相對，
6:27.040–6:33.280
檔案中每個未壓縮的張量總計僅為 58 MB，不到百分之零点五。在單一
6:33.280–6:38.720
注意力區塊內，搜尋將查詢投影賦予了其區塊中最便宜的格式，而
6:38.720–6:44.960
旁邊的鍵和值鄰居則保留了更昂貴的格式。如果你觀察同一個挨餓的
6:44.960–6:50.160
單一注意力區塊，搜尋結果為查詢投影提供了該區塊中最便宜的格式，同時
6:50.160–6:55.520
中間的三位元預算時上升到 4 位元，並在最大預算時回落至 2 位元以下。這是因為
6:55.520–7:00.720
全域預算優化器並非根據重要性對張量進行排名。它是在交換位元組，並且它會樂意
7:00.720–7:06.160
在較大的預算下讓張量挨餓，而不是在較小的預算下餵養它，因為它唯一
7:06.160–7:12.000
保護的是總量。這些壓縮建構的每個效能評分都來自執行
7:12.000–7:16.560
壓縮的實驗室，發布在其自己的模型卡上，且該實驗室之外的人員尚未
7:16.560–7:20.960
重現這些比較。根據他們的結果，推薦的 11.8 GB 檔案在 AIM25 和 Live Codebench 上與
7:20.960–7:25.920
未壓縮的 53.8 GB 原始檔案完全匹配，實現了 4.6 倍的體積
7:25.920–7:32.480
縮減。在 GPQA Diamond 上，此推薦建構僅落後原始檔案 0.5 分，
7:32.480–7:41.040
並且即使你退回到小 1.5 GB 的三位元建構，分數仍然很高。在 10.09 GB 時，這個 3 位元版本在那個主要推理
7:41.040–7:46.960
基準測試上仍然達到完美的 100 分，而在 GPQA Diamond 上僅落後推薦建構 0.5 分。並且在
7:46.960–7:51.760
體積尺度的底部，他們的 2.75 位元建構實際上恢復了原始
7:51.760–7:59.200
零樣本平均值的 101.8%，得分高於未壓縮的基準。在匹配的 8.4 GB 時，他們的建構擊敗了
7:59.200–8:04.640
基準測試中，僅在 GPQA Diamond 上比推薦構建低半點。而在
8:04.640–8:11.760
體積尺度的底部，他們的 2.75 位元構建實際上恢復了原始
8:11.760–8:18.800
零樣本平均值的 101.8%（8%），得分高於未壓縮基準。在匹配的 8.4 GB 下，他們的構建在 Aim25 上比
8:18.800–8:24.960
Unslaught 的動態量化高出 10 分。因為推薦構建與小 1.5 GB 構建之間
8:24.960–8:30.080
的差距僅在一個基準測試中為半點，選擇構建並非真正的質量決策。基礎模型的配置列出 64 個 transformer 層
8:30.080–8:36.400
，以三個線性注意力層後接一個完整注意力層的重複模式排列。這
8:36.400–8:42.240
留下了 48 個閾值 delta 層和 16 個完整注意力層，檔案擁有的張量名稱證實了這一點。一
8:42.240–8:49.040
個 llama.cpp 的錯誤報告以完全相同的術語描述了運行中的模型，這很重要，因為
8:49.040–8:54.640
這些層處理資訊的方式會改變記憶體行為。雖然完整注意力
8:54.640–8:59.840
層必須保留對話中的每個 token，因此其記憶體隨每個單詞增長，
8:59.840–9:04.880
循環層則保持一個固定大小的摘要，在 10 個單詞後與
9:04.880–9:10.400
10,000 個單詞後的大小相同。這個固定大小的摘要是 64 個層中的 48 個層從不累積對話
9:10.400–9:17.280
歷史的原因。這也是為什麼 12 GB 顯示卡會出現在這段對話中的原因。我們可以
9:17.280–9:23.040
直接從基礎配置中計算出來。該模型有 16 個注意力層，每個層
9:23.040–9:28.560
包含兩個鍵值張量，包含四個頭部、每個 256 個數字、每兩個位元組存儲一個。這
9:28.560–9:36.320
使得對話中的每個 token 乘以 64 KB。壓縮該緩存會
9:36.320–9:42.800
降低價格。降至 8 位元，每個 token 花費 34 KB。壓縮至 4 位元，
9:42.800–9:49.040
你只花費 18 KB。同時，這 48 個循環層在任何上下文
9:49.040–9:56.400
長度下都固定花費 150 MB。因此，當阿里巴巴宣傳原生 262,000 個 token 上下文時，該窗口有非常
9:56.400–10:03.600
實際的價格。在全精度下，僅緩存就需要 16 GB。這在 8 位元時降至 8.5 GB，
10:03.600–10:12.080
在 4 位元時降至 4.5 GB。我們可以針對已知能運行的機器檢查該算術。開發者
10:12.080–10:18.080
在 16 GB 顯示卡上以完整上下文運行此模型時，必須使用 4 位元緩存，根據數學計算為
10:18.080–10:24.480
4.5 GB。如果所有 64 個層都是注意力層，該緩存將需要 18 GB，並且根本無法裝入
10:24.480–10:31.920
他們的顯示卡。那麼，這實際上為我們的 12 GB 顯示卡留下了什麼？但在
10:31.920–10:37.600
該模型宣傳的長上下文情況下，在你顯示卡實際運行的軟體（如
10:37.600–10:42.560
llama.cpp）中會發生什麼？8 月底，一名開發者提交了一份錯誤報告，顯示在長上下文下，
10:42.560–10:49.040
該模型偶爾會將其第一個 token 生成為文本結束標記，且完全沒有
10:49.040–10:54.560
錯誤。該故障頑固，在顯示卡上、在同一構建的 CPU 上以及
10:54.560–11:00.240
在代碼庫的獨立分支上都能重現。但報告者的差異測試指向
11:00.240–11:05.280
引擎缺陷而非權重本身，因為完全相同的提示在 VLM 上順利通過。
11:05.280–11:11.840
在測試上下文容量時，約 113,000 個 token 順利通過，但約 130,000 個 token 會
11:11.840–11:19.280
因該即時文本結束標記而失敗，而稍深層的提示則完全正常。
11:19.280–11:24.240
再次提醒，這是基於另一個實驗室的量化版本測量的，而非此構建。並且
11:24.240–11:29.440
一個僅有 30 個循環層的兄弟模型能存活得更深。報告者最終
11:29.440–11:35.360
撤回了他自己提出的閾值，因為這些隨機的失敗空洞，建議
11:35.360–11:40.240
「沒有可引用的安全限制，你必須驗證你計劃服務的
11:40.240–11:45.600
確切深度。」Olama 透過單一命令直接從 Hugging Face 拉取此構建。雖然一名
11:45.600–11:52.000
維護者演示了其運作，但實際使用會在聊天模板上崩潰，而非你的
11:52.000–11:57.040
顯示卡。內建模板拒絕任何非第一個出現的系統訊息。因此，如果你發送
11:57.040–12:03.040
顯示卡。內建的範本會拒絕任何非第一條的系統訊息。因此，如果你傳送
12:03.040–12:09.280
對話中途發送時，你只會得到空字串、完成狀態，以及完全沒有錯誤訊息，
12:09.280–12:14.640
修復它需要在模型檔案中進行三行程式碼的修改。而且因為這個確切的錯誤在八月被回報了兩次
12:14.640–12:21.520
針對純權重版本，所以這不是這個構建版本的問題。切換到 llama.cpp 的多模態
12:21.520–12:26.800
工具甚至更糟，因為在編碼圖片時未能預留記憶體會導致程式崩潰
12:26.800–12:32.400
出現分段錯誤（segmentation fault），而不是回報錯誤。它的崩潰日誌顯示該工具在
12:32.400–12:39.840
嘗試分配僅 247 MB 記憶體以讀取單張圖片時失敗而死亡。LM Studio 完全避免了
12:39.840–12:45.360
崩潰，但它的卸載滑塊會靜默地將多餘的權重推入普通系統
12:45.360–12:51.120
記憶體，而不是拒絕載入它們。該實驗室對 LM Studio 的全部指導僅有一句話
12:51.120–12:56.400
一句話告訴你搜尋儲存庫，而且沒有人發表過在 12
12:56.400–13:02.320
GB 顯卡上測試此構建版本的結果。這三款應用程式都沒有在模型無法載入時通知你，這就是為什麼唯一
13:02.320–13:08.320
知道它是否能運作的方法是在點擊下載前計算你的「bite」（位元組）。九月初，
13:08.320–13:14.640
一位使用 16 GB Blackwell 顯卡並搭配特定工具包版本的開發者發現，Llama.cpp 的
13:14.640–13:20.720
後端測試對三種格式進行了矩陣乘法失敗，包括這個構建版本名稱所依據的那種格式。回報的錯誤率高達 1.1，這大到意味著你只會得到
13:20.720–13:26.720
隨機的垃圾輸出，但同一運行中相鄰的格式卻通過了測試，使得這個錯誤
13:26.720–13:32.320
非常特定。接著，另一位用戶在同一款 Blackwell 顯卡上報告了相同的垃圾輸出，
13:32.320–13:38.880
並透過對 bite 處理程式碼進行小幅修改來在本地修復它。但第三位使用相同晶片
13:38.880–13:45.040
世代的用戶無法重現此問題，認為問題出在構建鏈而非
13:45.040–13:49.680
硬體上，因為這些格式沒有晶片特定的路徑。總之，雖然這場辯論仍未解決，
13:49.680–13:55.600
選擇由 11 種格式組成的檔案意味著你依賴著 11 條不同的
13:55.600–14:01.120
程式碼路徑。而那些最少使用的路徑正是爭議仍在發生的地方。
14:01.120–14:07.280
推薦的模型檔案本身為 11.77 GB，但在你的顯卡上，這些權重佔用了 10.96 GB 的 VRAM。
14:07.280–14:17.120
加上獨立的視覺投影器會再消耗 0.87 GB 的記憶體，這使得總計
14:17.120–14:23.600
達到 11.83 GB，並只留下僅 174 MB 的微小剩餘空間。減去循環狀態層
14:23.600–14:33.680
所需的固定 150 MB，剩餘 24 MB 的可用緩衝空間。以每
14:33.680–14:40.240
個 token 64 KB 計算，剩餘空間最多只能容納約 386 個 token，實際情況更少。當然，
14:40.240–14:48.000
每個 token 的話，剩餘的空間最多只能容納約 386 個 token，實際情況可能更少。當然，
14:48.000–14:53.040
作業系統已經使用的任何圖形記憶體，更不用說推論軟體
14:53.040–14:57.840
本身在分配任何 token 之前預留的工作記憶體。如果你嘗試升級到最大的
14:57.840–15:04.240
12.12 GB 構建版本，加上投影器會超出你的顯卡記憶體 308 MB，這意味著它無法
15:04.240–15:12.080
在零上下文的情況下保留它自己的「眼睛」（視覺資訊）。但如果你只進行文字處理，並在不考慮該
15:12.080–15:18.320
視覺投影器的情況下計算數學，推薦的構建版本會留下 912 MB 的剩餘空間，這相當於
15:18.320–15:25.280
全精度下約 14,12,000 個 token 的上下文，或在 8 位元下高達 27,000 個 token。你的 12 GB 圖形
15:25.280–15:34.320
顯卡完全有能力運行這個 270 億參數的模型，但它 simply 無法處理
15:34.320–15:40.640
該實驗室推薦的特定 11.88 GB 構建版本，一旦該構建版本具備視覺能力。你真正需要的檔案
15:40.640–15:46.800
是 3bit 構建版本，在下載頁面上為 10.09 GB，或在你的顯卡單位中約為 9.40
15:46.800–15:54.640
GB。一旦你載入這些權重並結合視覺投影器，
15:54.640–15:59.840
你將剩餘 1,624 MB 的緩衝空間。這意味著啟用圖像時的活躍
15:59.840–16:07.200
對話上限約為 26,000 個 token，如果你只運行文字模式，則可達 40,000 個 token。如果你
16:07.200–16:13.040
最佳化你的快取或退回到更小的 2.75 bit 構建版本，這些數字會進一步延伸
16:13.040–16:18.800
最佳化你的快取或退回到更小的 2.75 位元版本，這些數字會進一步擴大
16:18.800–16:24.000
更遠。請記住，這些是絕對的數學上限，未考慮執行時的
16:24.000–16:29.040
自身工作緩衝區，因此你的實際容量較低。根據該實驗室自身的
16:29.040–16:35.040
評估，這種降級會使 GPQA Diamond 的得分損失半點，Live Codebench 損失約一點，而 AIM25 的完美分數 100 則完全未變。
16:35.040–16:41.600
而精確度的微小降低，換來了大約 26,000 個 token 的對話容量。
16:41.600–16:47.600
兩個獨立的團隊已在該實驗室的儲存庫上提交了相同的請求，但截至目前為止尚未收到回覆，
16:47.600–16:53.280
外部開發者只能恢復最終的映射，了解哪種格式對應哪個張量，卻無法自行執行搜尋。他們
16:53.280–16:58.960
要求補齊遺漏的部分，因為公共程式碼中完全缺少幾個關鍵步驟，例如候選生成
16:58.960–17:03.440
要求補齊遺漏的部分，因為幾個關鍵步驟，例如候選生成
17:03.440–17:09.040
完全沒有包含在公開程式碼中。具體來說，他們缺少每個張量的成本
17:09.040–17:13.920
檔案的匯出程式碼。其中一個團隊需要該程式碼，因為他們的伺服器機隊受限於記憶體頻寬
17:13.920–17:19.280
而非容量，因此較小的檔案能直接提升速度，而這種混合格式是他們
17:19.280–17:24.720
唯一可行的部署途徑。預算論文將混合精度量化、非均勻剪枝
17:24.720–17:31.360
，以及選擇保留哪些專家，視為將 K 個選項之一分配給 N 個群組的同一問題，
17:31.360–17:36.400
這意味著這是一種將幾乎任何內容適應固定記憶體佔用量的方法。
17:36.400–17:42.320
因此，雖然該實驗室的發布為特定模型的預算提供了一個完整的答案，但
17:42.320–17:47.120
他們尚未推出的東西，是適用於其他任何人的方法。好了，各位，就是這些。
17:47.120–17:52.320
他們尚未推出的東西是其他人的方法。總之，就這些了，各位。

0:00.080–0:06.720
Quen's 27 billion parameter model reads images and text. And the lab behind most local model
Quen 的 270 億參數模型可以讀取影像和文字。而負責大多數本地模型
0:06.720–0:13.760
compression just shrank it into a single 11.8 GB file. A model that size normally needs more memory
壓縮的實驗室，只是將其壓縮成單一 11.8 GB 的檔案。這種體積的模型通常需要比任何遊戲電腦更多的記憶體
0:13.760–0:19.680
than any gaming computer has. This one is smaller than a video game install. Needless to say,
比任何遊戲電腦的記憶體都大。這個檔案比一個遊戲安裝檔還小。不用說，
0:19.680–0:25.120
this is a seriously impressive demonstration of model compression. But the question must be asked,
這確實是一個令人印象深刻的模型壓縮示範。但必須提出一個問題，
0:25.120–0:31.120
can a 12 GBTE graphics card actually run Quen's 27 billion parameter model? To find out,
12 GB 的顯示卡真的能運行 Quen 的 270 億參數模型嗎？為了找出答案，
0:31.120–0:35.440
we have to look at the repository where we see that the download page doesn't actually offer
我們必須查看該儲存庫，我們發現下載頁面實際上並未提供
0:35.440–0:40.800
a single file, but a menu of four different sizes shipped two ways for a total of eight
單一檔案，而是提供四種不同大小的選單，並以兩種方式提供，總共八個
0:40.800–0:47.280
model files. That 11.8 GB model from our title is the largest of the plane options, which the
模型檔案。標題中提到的 11.8 GB 模型是平面選項中最大的，實驗室
0:47.280–0:53.280
lab recommends as its task lossless build. But sitting right beside those models is a ninth file,
建議將其作為任務無損構建。但在這些模型旁邊，還有一個第九個檔案，
0:53.280–1:00.320
a 0.93 GBTE vision projector called MM Proj. None of the eight main checkpoints actually contain
一個名為 MM Proj 的 0.93 GB 視覺投影器。八個主要檢查點中實際上並沒有包含
1:00.320–1:04.400
this vision projector, which means you have to get it separately if you want to use the model's
這個視覺投影器，這意味著如果你想使用模型的
1:04.400–1:12.560
multimodal capabilities. On paper, the 11.8 GB of the GSQ RCO build and the 12 GB of your graphics
多模態功能，你必須單獨獲取它。從紙面上看，GSQ RCO 構建的 11.8 GB 和你的顯示卡的 12 GB
1:12.560–1:17.520
card sit right next to each other. But those two numbers are actually measured in entirely
記憶體緊鄰在一起。但實際上，這兩個數字的測量單位完全不同。
1:17.520–1:23.360
different units. When a card is sold as having 12 GB of memory, the operating system actually
當顯示卡被銷售為擁有 12 GB 記憶體時，作業系統實際上是以二進位千兆位元組來計算該空間，這才是你的硬體實際可用的空間。在
1:23.360–1:29.120
counts that space in binary gigabytes, which is what your hardware actually has to work with. On
帳簿的另一端，該建議的 GSQ RCO 構建在 HuggingFace 上列為 11.77 GB。
1:29.120–1:37.120
the other side of the ledger, that recommended GSQ RCO build is listed on HuggingFace at 11.77 GB.
但當你檢查該 GSQ RCO 構建的實際權重時，它們只佔用 10.96 GB 的空間，
1:37.120–1:44.320
But when you inspect the actual weights of that GSQ RCO build, they only take up 10.96 GB of space
因為下載頁面使用的是十進位 GB。如果你從 12 GB 的容量中減去 10.96，
1:44.320–1:52.000
because the download page is using decimal GB. If you subtract 10.96 from your 12 GB of capacity,
你將剩下 1.04 GB 的可用空間，這意味著你的顯示卡實際上的剩餘空間比標題數字所暗示的
1:52.000–1:58.960
you are left with 1.04 GB of free space, that means the actual headroom on your card is larger
200 MB 還要大。但這些額外的剩餘空間即將消失，因為模型讀取影像的能力存儲在一個完全不同的檔案中
1:58.960–2:03.760
than the 200 megabytes the headline number suggested. But that extra headroom is about
，名為 mm praj，其重量為 9.31 億位元組或 0.87 GB。而在一個完全圍繞
2:03.760–2:08.720
to vanish because the model's ability to read images is stored in a completely separate file
模型壓縮設計的發布中，這個視覺投影器是唯一完全沒有壓縮的檔案。相反地，
2:08.720–2:18.320
called mm praj, which weighs 931 million bytes or 0.87 GB. And in a release designed entirely around
實驗室將其保留在原始的 16 位元格式。由於一份副本服務於選單上的每一個構建
2:18.320–2:24.320
model compression, this vision projector is the only file that wasn't compressed at all. Instead,
模型壓縮，這個視覺投影機是唯一完全沒有壓縮的檔案。相反地，
2:24.320–2:29.840
the lab left it at its original 16-bit format. And because one copy serves every single build
還是最小的，這個記憶體懲罰都是完全相同的。在 llama.cpp 中，你使用單獨的命令列旗標加載這個投影器，
2:29.840–2:34.560
on the menu, this memory penalty is exactly the same whether you pick the largest budget
這使得很容易忘記它的存在。但一旦加載，我們剛才找到的 1.04 GB 中的 0.87 GB 就會瞬間消失。這意味著標題所銷售的模型部分，讓它
2:34.560–2:41.040
or the smallest. In llama.cpp, you load this projector using a separate command line flag,
讀取影像的系統，是壓縮從未觸及的部分，它幾乎耗盡了
2:41.040–2:48.480
which makes it easy to forget that it's even there. But once it loads, 0.8 87 GB of the 1.04 we
你的顯示卡所剩餘的所有空間。支持此構建的量化論文將模型壓縮領域分為簡單的標量方法，如 GPTQ 和 AWQ，這些是每個人都會部署的方法，以及向量或
2:48.480–2:54.240
just found is instantly gone. This means the part of the model the title sells, the system that lets
柵格方法，如 Q-tip 和 AQLM，這些方法得分更高。在論文的原文中，標量方法
2:54.240–2:59.280
it read images, is the one part the compression never touched, and it eats almost all of the
在 3 到 4 位元時準確率達到平台期，而更準確的向量家族以難以
2:59.280–3:04.400
headroom your card has left. The quantization paper behind this build splits the field of
實現和擴展而聞名。因此，GSQ 在保持標量網格的同時，縮小了大部分準確率差距，
3:04.400–3:11.680
model compression into simple scalar methods like GPTQ and AWQ that everyone deploys and vector or
確保壓縮後的模型與我們已在顯示卡上運行的現有標量推理
3:11.680–3:18.800
trellis methods like Q-tip and AQLM that score better. In the paper's own words, scalar methods
內核完全兼容。結果顯示，同一個實驗室發布了 GPTQ，這是 GSQ 用來衡量自身的標準方法之一。此外，GSQ 可以直接應用於
3:18.800–3:24.000
plateau in accuracy at 3 to four bits, while the more accurate vector family is notoriously
你已經擁有的現有 GGUF 發布，提高其準確率並將其導出回
3:24.000–3:30.640
hard to implement and scale. So GSQ closes most of that accuracy gap while staying on a scalar grid,
完全相同的格式。這個檔案可以在你日常的應用程式中運行，如 Llama.cpp、Olama 和 LM Studio，而無需
3:30.640–3:34.880
ensuring the compressed model remains fully compatible with the existing scalar inference
修改任何一個。這並非運氣。這是因為實驗室刻意選擇了
3:34.880–3:41.280
kernels we already run on our cards. It turns out the same lab published GPTQ, which is one of the
我們已在顯示卡上執行的核心。結果顯示，同一個實驗室發表了 GPTQ，這是其中一種
3:41.280–3:47.520
standard methods GSQ measures itself against. On top of that, GSQ can be applied directly to an
方法家族，以便留在現有軟體已經讀取的格式內。壓縮單一張量並不會構建整個檔案，因為你仍然必須
3:47.520–3:53.200
existing GGUF release you already have, improving its accuracy and exporting it right back into the
決定對模型的每個部分應用多少壓縮以適應你的硬體。為了解決這個問題，
3:53.200–4:00.480
exact same format. This file runs in your everyday apps like Llama.cpp, Olama, and LM Studio without
完全相同的格式。這個檔案可以在你日常使用的應用程式中運行，例如 Llama.cpp、Olama 和 LM Studio，而無需
4:00.480–4:05.840
modifying any of them. And that's not luck. It happens because the lab deliberately chose the
修改任何一個。這並非運氣使然。這是因為實驗室刻意選擇了
4:05.840–4:11.200
family of methods that scores worse in order to stay inside a format the existing software already
為了符合現有軟體已經能讀取的格式，選擇了評分較低的編碼方法家族。
4:11.200–4:15.920
reads. Compressing a single tensor doesn't build a whole file because you still have to
壓縮單一張量並不會建立整個檔案，因為你仍然必須
4:15.920–4:21.600
decide how much compression to apply to each part of the model to fit your hardware. To solve that,
決定對模型的每個部分應用多少壓縮，以適應你的硬體。為了解決這個問題，
4:21.600–4:27.280
Michael Hellig and Dan Allistar published a second paper titled Model Compression with Exact Budget
Michael Hellig 和 Dan Allistar 發表了第二篇論文，題為《具有精確預算限制的模型壓縮》，
4:27.280–4:32.400
Constraints, framing the problem as assigning one of several options to each group under a
將問題框架為在總成本預算下，為每個群組分配多個選項之一。雖然基於懲罰的方法僅近似地執行預算，且需要
4:32.400–4:38.240
total cost budget. While penalty-based approaches only enforce a budget approximately and require
大量的超參數調整，但此方法則能精確地滿足預算，無需任何特定於約束的調整。
4:38.240–4:44.800
extensive hyperparameter tuning, this method meets it exactly without any constraintspecific tuning.
放寬預算約束描述了一個稱為殘留流形的平滑曲面，位於
4:44.800–4:50.160
Relaxing the budget constraint describes a smooth curved surface called a remmanion manifold in
邏輯空間中，允許梯度方法沿著它行走而不會離開。該程序
4:50.160–4:55.760
lojit space allowing a gradient method to walk along it without ever leaving it. The procedure
分為三個步驟。你將每個張量量化為每個候選格式，搜尋符合你預算的分配，並將這些勝出者縫合成一個普通檔案。這意味著你可以
4:55.760–5:01.440
takes three steps. You quantize every tensor into every candidate format, search for the assignment
將你的記憶體預算精確用到最後一丁點，這正是你在嘗試
5:01.440–5:06.640
that meets your budget and stitch those winners into one ordinary file. And that means you can
將 270 億參數的模型塞進 12 GB 的顯示卡時所面臨的問題。你甚至不需要下載模型
5:06.640–5:11.360
spend your memory budget down to the last bite, which is exactly the problem you face when trying
就能知道搜尋演算法做了什麼決定，因為該實驗室將整個決策列表發布為
5:11.360–5:17.920
to squeeze a 27 billion parameter model onto a 12 GB card. You don't even have to download the model
純文字檔案。在推薦的建構中，搜尋演算法最終將 851 個獨立
5:17.920–5:23.120
to see what that search algorithm decided because the lab published the entire decision list as a
張量分割到 11 種不同的儲存格式中。它將 96 個張量保留在完整的 16 位元精度
5:23.120–5:30.240
plain text file. Inside the recommended build, the search algorithm ended up slicing 851 individual
，而將其餘的張量放入較小的量化格式階梯中。例如，
5:30.240–5:37.040
tensors across 11 different storage formats. It left 96 tensors at full 16-bit precision
搜尋將 144 個張量分配給 IQ3S 格式。而測量的平均值正好落在
5:37.040–5:42.240
while dropping the rest into a staircase of smaller quantization formats. For instance,
實驗室的 3.5 目標上，但非均勻不僅是一個行銷用語，搜尋也不害怕
5:42.240–5:48.880
the search assigned 144 tensors to the IQ3S format. And the measured average lands right
如果意味著節省網路的其餘部分，就讓單一組件挨餓。在發布的轉儲中，
5:48.880–5:55.200
on the lab's 3.5 target, but non-uniform is not just a marketing word, and the search isn't afraid
區塊 13 的前饋閘門被壓縮到稱為 IQ1M 的挨餓格式，僅運行在 1 和
5:55.200–6:00.160
to starve a single component if it means saving the rest of the network. In the published dump,
3/4 位元上。該單一張量並未周圍環繞著完整精度。它位於一個區塊內，該區塊的其他
6:00.160–6:07.120
block 13's feed forward gate is squeezed down to a starved format called IQ1M, running on just 1 and
矩陣都被賦予了 2、3 和 4 位元的格式。它僅僅是普通妥協區域中
6:07.120–6:13.280
3/4 bits. That single tensor isn't surrounded by full precision. It sits inside a block whose other
最便宜的東西。超過 11.77 GB 檔案的十分之一僅僅是詞彙。具體來說，
6:13.280–6:19.200
matrices were all given 2, 3, and 4bit formats. It's simply the cheapest thing in a neighborhood
兩個僅負責將詞元轉換為數字並轉回來的張量，合計佔據了
6:19.200–6:27.040
of ordinary compromises. More than a tenth of the 11.77 GB file is just vocabulary. Specifically,
檔案大小的 1.12 GB，分別分配給輸出頭部和嵌入表。與此相對，
6:27.040–6:33.280
two tensors that do nothing but turn tokens into numbers and back account for 1.12 GB of
檔案中每個未壓縮的張量總計僅為 58 MB，不到百分之零点五。在單一
6:33.280–6:38.720
the file size together, split between the output head and the embedding table. Against that,
注意力區塊內，搜尋將查詢投影賦予了其區塊中最便宜的格式，而
6:38.720–6:44.960
every uncompressed tensor in the file adds up to 58 megabytes, under half a percent. Inside a
旁邊的鍵和值鄰居則保留了更昂貴的格式。如果你觀察同一個挨餓的
6:44.960–6:50.160
single attention block, the search gave the query projection the cheapest format in its block, while
單一注意力區塊，搜尋結果為查詢投影提供了該區塊中最便宜的格式，同時
6:50.160–6:55.520
the key and value neighbors beside it kept more expensive ones. If you watch that same starved
中間的三位元預算時上升到 4 位元，並在最大預算時回落至 2 位元以下。這是因為
6:55.520–7:00.720
gate across the four size budgets, it starts at just one bit, goes up to four bits at the
全域預算優化器並非根據重要性對張量進行排名。它是在交換位元組，並且它會樂意
7:00.720–7:06.160
intermediate three-bit budget and drops back under two bits at the largest. This happens because a
在較大的預算下讓張量挨餓，而不是在較小的預算下餵養它，因為它唯一
7:06.160–7:12.000
global budget optimizer isn't ranking tensors by importance. It's trading bytes and it'll happily
保護的是總量。這些壓縮建構的每個效能評分都來自執行
7:12.000–7:16.560
starve a tensor at a larger budget than it fed it at a smaller one because the only thing it's
壓縮的實驗室，發布在其自己的模型卡上，且該實驗室之外的人員尚未
7:16.560–7:20.960
protecting is the total. Every performance score for these compressed builds comes from the lab
重現這些比較。根據他們的結果，推薦的 11.8 GB 檔案在 AIM25 和 Live Codebench 上與
7:20.960–7:25.920
that performed the compression, published on its own model card, and nobody outside that lab has
未壓縮的 53.8 GB 原始檔案完全匹配，實現了 4.6 倍的體積
7:25.920–7:32.480
reproduced these comparisons. According to their results, the recommended 11.8 GB file matches the
縮減。在 GPQA Diamond 上，此推薦建構僅落後原始檔案 0.5 分，
7:32.480–7:41.040
uncompressed 53.8 GB original exactly on AIM25 and Live Codebench, representing a 4.6 time size
並且即使你退回到小 1.5 GB 的三位元建構，分數仍然很高。在 10.09 GB 時，這個 3 位元版本在那個主要推理
7:41.040–7:46.960
reduction. On GPQA Diamond, this recommended build trails the original by only half a point,
基準測試上仍然達到完美的 100 分，而在 GPQA Diamond 上僅落後推薦建構 0.5 分。並且在
7:46.960–7:51.760
and the scores remain high even as you step down to the threebit build, which is a gigabyte and
體積尺度的底部，他們的 2.75 位元建構實際上恢復了原始
7:51.760–7:59.200
a half smaller. At 10.09 GB, this 3bit version still hits a perfect 100 on that main reasoning
零樣本平均值的 101.8%，得分高於未壓縮的基準。在匹配的 8.4 GB 時，他們的建構擊敗了
7:59.200–8:04.640
benchmark, while trailing the recommended build by only half a point on GPQA Diamond. And near
基準測試中，僅在 GPQA Diamond 上比推薦構建低半點。而在
8:04.640–8:11.760
the bottom of the size scale, their 2.75 bit build actually recovers 101.8% 8% of the original
體積尺度的底部，他們的 2.75 位元構建實際上恢復了原始
8:11.760–8:18.800
zeroshot average, scoring above the uncompressed baseline. At a matched 8.4 GB, their build beats
零樣本平均值的 101.8%（8%），得分高於未壓縮基準。在匹配的 8.4 GB 下，他們的構建在 Aim25 上比
8:18.800–8:24.960
Unslaught's dynamic quantizations by 10 points on Aim25. Because the gap between the recommended
Unslaught 的動態量化高出 10 分。因為推薦構建與小 1.5 GB 構建之間
8:24.960–8:30.080
build and the one a gigabyte and a half smaller is only half a point on one benchmark, picking a
的差距僅在一個基準測試中為半點，選擇構建並非真正的質量決策。基礎模型的配置列出 64 個 transformer 層
8:30.080–8:36.400
build is not really a quality decision. The base model's configuration lists 64 transformer layers
，以三個線性注意力層後接一個完整注意力層的重複模式排列。這
8:36.400–8:42.240
in a repeating pattern of three linear attention layers followed by one full attention layer. That
留下了 48 個閾值 delta 層和 16 個完整注意力層，檔案擁有的張量名稱證實了這一點。一
8:42.240–8:49.040
leaves 48 gated delta layers and 16 full attention layers that the files own tensor names confirm. A
個 llama.cpp 的錯誤報告以完全相同的術語描述了運行中的模型，這很重要，因為
8:49.040–8:54.640
llama.cpp bug report describes the running model in those exact terms, which matters because the
這些層處理資訊的方式會改變記憶體行為。雖然完整注意力
8:54.640–8:59.840
way these layers process information changes how memory behaves. While a full attention
層必須保留對話中的每個 token，因此其記憶體隨每個單詞增長，
8:59.840–9:04.880
layer has to keep every token of the conversation available, so its memory grows with every word,
循環層則保持一個固定大小的摘要，在 10 個單詞後與
9:04.880–9:10.400
a recurrent layer keeps one fixed size summary that is the same size after 10 words as after
10,000 個單詞後的大小相同。這個固定大小的摘要是 64 個層中的 48 個層從不累積對話
9:10.400–9:17.280
10,000. That fixed size summary is the reason 48 of its 64 layers never accumulate conversation
歷史的原因。這也是為什麼 12 GB 顯示卡會出現在這段對話中的原因。我們可以
9:17.280–9:23.040
history. And that is the reason a 12 GB graphics card is in this conversation at all. We can work
直接從基礎配置中計算出來。該模型有 16 個注意力層，每個層
9:23.040–9:28.560
this out directly from the base configuration. The model has 16 attention layers and each layer
包含兩個鍵值張量，包含四個頭部、每個 256 個數字、每兩個位元組存儲一個。這
9:28.560–9:36.320
holds two key value tensors containing four heads of 256 numbers stored at two bytes a piece. That
使得對話中的每個 token 乘以 64 KB。壓縮該緩存會
9:36.320–9:42.800
multiplies out to 64 kilob for every single token in the conversation. Compressing that cache brings
降低價格。降至 8 位元，每個 token 花費 34 KB。壓縮至 4 位元，
9:42.800–9:49.040
the price down. Drop it to 8 bits and you spend 34 kilob a token. Squeeze it to four bits and
你只花費 18 KB。同時，這 48 個循環層在任何上下文
9:49.040–9:56.400
you only spend 18. Meanwhile, those 48 recurrent layers cost a flat 150 megbytes at any context
長度下都固定花費 150 MB。因此，當阿里巴巴宣傳原生 262,000 個 token 上下文時，該窗口有非常
9:56.400–10:03.600
length. So when Alibaba advertises a native 262,000 token context, that window has a very
實際的價格。在全精度下，僅緩存就需要 16 GB。這在 8 位元時降至 8.5 GB，
10:03.600–10:12.080
real price. At full precision, the cache alone needs 16 GB. That drops to 8 1/2 GB at 8 bit and
在 4 位元時降至 4.5 GB。我們可以針對已知能運行的機器檢查該算術。開發者
10:12.080–10:18.080
4 1/2 GB at 4bit. We can check that arithmetic against a machine known to work. A developer
在 16 GB 顯示卡上以完整上下文運行此模型時，必須使用 4 位元緩存，根據數學計算為
10:18.080–10:24.480
running this model at full context on a 16 GB card has to use a 4-bit cache, which the math puts at
4.5 GB。如果所有 64 個層都是注意力層，該緩存將需要 18 GB，並且根本無法裝入
10:24.480–10:31.920
4 1/2 GB. If all 64 layers were attention, that cache would require 18 GB and it wouldn't fit on
他們的顯示卡。那麼，這實際上為我們的 12 GB 顯示卡留下了什麼？但在
10:31.920–10:37.600
their card at all. So, what does that actually leave for our 12 GB card? But what happens at
該模型宣傳的長上下文情況下，在你顯示卡實際運行的軟體（如
10:37.600–10:42.560
the long context this model advertises on the software your card would actually be running,
llama.cpp）中會發生什麼？8 月底，一名開發者提交了一份錯誤報告，顯示在長上下文下，
10:42.560–10:49.040
like llama.cpp. In late August, a developer filed a bug report showing that at long context,
該模型偶爾會將其第一個 token 生成為文本結束標記，且完全沒有
10:49.040–10:54.560
the model occasionally generates an end of text marker as its very first token with absolutely no
錯誤。該故障頑固，在顯示卡上、在同一構建的 CPU 上以及
10:54.560–11:00.240
error. The failure is stubborn, reproducing on the graphics card, on the CPU with the same build,
在代碼庫的獨立分支上都能重現。但報告者的差異測試指向
11:00.240–11:05.280
and on an independent fork of the codebase. But the reporter's differential test points to an
引擎缺陷而非權重本身，因為完全相同的提示在 VLM 上順利通過。
11:05.280–11:11.840
engine defect rather than the weights themselves since the exact same prompt passes cleanly on VLM.
在測試上下文容量時，約 113,000 個 token 順利通過，但約 130,000 個 token 會
11:11.840–11:19.280
When testing context capacity, around 113,000 tokens passes cleanly, but around 130,000 fails
因該即時文本結束標記而失敗，而稍深層的提示則完全正常。
11:19.280–11:24.240
with that instant end of text marker, while a slightly deeper prompt works perfectly fine.
再次提醒，這是基於另一個實驗室的量化版本測量的，而非此構建。並且
11:24.240–11:29.440
Again, keep in mind this was measured on another lab's quantizations rather than this build. And
一個僅有 30 個循環層的兄弟模型能存活得更深。報告者最終
11:29.440–11:35.360
a sibling model with only 30 recurrent layers survives much deeper. The reporter eventually
撤回了他自己提出的閾值，因為這些隨機的失敗空洞，建議
11:35.360–11:40.240
withdrew his own proposed threshold because of these random holes of failure, advising that
「沒有可引用的安全限制，你必須驗證你計劃服務的
11:40.240–11:45.600
there is no safe limit to quote, "and you simply have to validate the exact depths you plan to
確切深度。」Olama 透過單一命令直接從 Hugging Face 拉取此構建。雖然一名
11:45.600–11:52.000
serve." Olama pulls this exact build straight from hugging face with a single command. And although a
維護者演示了其運作，但實際使用會在聊天模板上崩潰，而非你的
11:52.000–11:57.040
maintainer demonstrated it working, actual usage breaks on the chat template rather than your
顯示卡。內建模板拒絕任何非第一個出現的系統訊息。因此，如果你發送
11:57.040–12:03.040
graphics card. The built-in template refuses any system message that isn't first. So if you send
顯示卡。內建的範本會拒絕任何非第一條的系統訊息。因此，如果你傳送
12:03.040–12:09.280
one mid conversation, you get an empty string, a finished status, and absolutely no error,
對話中途發送時，你只會得到空字串、完成狀態，以及完全沒有錯誤訊息，
12:09.280–12:14.640
fixing it takes a threeline patch in a model file. And since this exact fault was filed twice
修復它需要在模型檔案中進行三行程式碼的修改。而且因為這個確切的錯誤在八月被回報了兩次
12:14.640–12:21.520
in August against the plain weights, it isn't this build's doing. Switching to llama.cpp's multimodal
針對純權重版本，所以這不是這個構建版本的問題。切換到 llama.cpp 的多模態
12:21.520–12:26.800
tool is even worse because failing to reserve memory while encoding an image crashes the process
工具甚至更糟，因為在編碼圖片時未能預留記憶體會導致程式崩潰
12:26.800–12:32.400
with a segmentation fault instead of reporting an error. Its crash log shows the tool dying on
出現分段錯誤（segmentation fault），而不是回報錯誤。它的崩潰日誌顯示該工具在
12:32.400–12:39.840
a failed attempt to allocate just 247 megabytes while reading a single picture. LM Studio avoids
嘗試分配僅 247 MB 記憶體以讀取單張圖片時失敗而死亡。LM Studio 完全避免了
12:39.840–12:45.360
crashing entirely, but its offload slider will quietly push excess weights into ordinary system
崩潰，但它的卸載滑塊會靜默地將多餘的權重推入普通系統
12:45.360–12:51.120
memory rather than refuse to load them. The lab's entire LM Studio guidance is a single
記憶體，而不是拒絕載入它們。該實驗室對 LM Studio 的全部指導僅有一句話
12:51.120–12:56.400
sentence telling you to search the repository, and nobody has published a test of this build on a 12
一句話告訴你搜尋儲存庫，而且沒有人發表過在 12
12:56.400–13:02.320
GB card. None of the three applications tells you when the model won't fit, which is why the only
GB 顯卡上測試此構建版本的結果。這三款應用程式都沒有在模型無法載入時通知你，這就是為什麼唯一
13:02.320–13:08.320
way to know if it runs is to count your bites before you click download. In early September,
知道它是否能運作的方法是在點擊下載前計算你的「bite」（位元組）。九月初，
13:08.320–13:14.640
a developer running a 16 GB Blackwell card with a specific toolkit version found that Llama.cpp's
一位使用 16 GB Blackwell 顯卡並搭配特定工具包版本的開發者發現，Llama.cpp 的
13:14.640–13:20.720
own back-end test failed matrix multiplication for three formats, including the exact format this
後端測試對三種格式進行了矩陣乘法失敗，包括這個構建版本名稱所依據的那種格式。回報的錯誤率高達 1.1，這大到意味著你只會得到
13:20.720–13:26.720
build is named after. The reported errors reached up to 1.1, which is large enough to mean you just
隨機的垃圾輸出，但同一運行中相鄰的格式卻通過了測試，使得這個錯誤
13:26.720–13:32.320
get random garbage output, but the neighboring formats in that same run passed, making the error
非常特定。接著，另一位用戶在同一款 Blackwell 顯卡上報告了相同的垃圾輸出，
13:32.320–13:38.880
highly specific. Then a second user reported the same garbage output on another Blackwell card,
並透過對 bite 處理程式碼進行小幅修改來在本地修復它。但第三位使用相同晶片
13:38.880–13:45.040
fixing it locally with a small change to the bite handling code. But a third user on the same chip
世代的用戶無法重現此問題，認為問題出在構建鏈而非
13:45.040–13:49.680
generation couldn't reproduce it, arguing the fault lies in the build chain rather than the
硬體上，因為這些格式沒有晶片特定的路徑。總之，雖然這場辯論仍未解決，
13:49.680–13:55.600
hardware since there's no chip specific path for these formats. Anyway, while this debate is still
選擇由 11 種格式組成的檔案意味著你依賴著 11 條不同的
13:55.600–14:01.120
unresolved, choosing a file assembled from 11 formats means you're depending on 11 distinct
程式碼路徑。而那些最少使用的路徑正是爭議仍在發生的地方。
14:01.120–14:07.280
code paths. And those least used paths are exactly where the arguments are still happening. The
推薦的模型檔案本身為 11.77 GB，但在你的顯卡上，這些權重佔用了 10.96 GB 的 VRAM。
14:07.280–14:17.120
recommended model file itself is 11.77 GB, but on your card, those weights consume 10.96 GB of VRAM.
加上獨立的視覺投影器會再消耗 0.87 GB 的記憶體，這使得總計
14:17.120–14:23.600
Adding the separate vision projector eats another 0.87 GB of memory, which brings the combined total
達到 11.83 GB，並只留下僅 174 MB 的微小剩餘空間。減去循環狀態層
14:23.600–14:33.680
to 11.83 GB and leaves you with a tiny pocket of just 174 MGB. Subtracting the flat 150 megabytes
所需的固定 150 MB，剩餘 24 MB 的可用緩衝空間。以每
14:33.680–14:40.240
demanded by the recurrent state layers leaves 24 megabytes of free headroom. At 64 kilobytes
個 token 64 KB 計算，剩餘空間最多只能容納約 386 個 token，實際情況更少。當然，
14:40.240–14:48.000
per token, that remaining space can hold at most about 386 tokens and fewer in practice. Of course,
每個 token 的話，剩餘的空間最多只能容納約 386 個 token，實際情況可能更少。當然，
14:48.000–14:53.040
that is a theoretical ceiling that completely ignores whatever graphics memory your desktop
作業系統已經使用的任何圖形記憶體，更不用說推論軟體
14:53.040–14:57.840
operating system is already using, not to mention the working memory that the inference software
本身在分配任何 token 之前預留的工作記憶體。如果你嘗試升級到最大的
14:57.840–15:04.240
itself reserves before it allocates a single token of cash. If you try to step up to the largest
12.12 GB 構建版本，加上投影器會超出你的顯卡記憶體 308 MB，這意味著它無法
15:04.240–15:12.080
12.12 GB build, adding the projector overshoots your card's memory by 308 MGB, meaning it cannot
在零上下文的情況下保留它自己的「眼睛」（視覺資訊）。但如果你只進行文字處理，並在不考慮該
15:12.080–15:18.320
hold its own eyes at zero context. But if you go text only and work out the math without that
視覺投影器的情況下計算數學，推薦的構建版本會留下 912 MB 的剩餘空間，這相當於
15:18.320–15:25.280
vision projector, the recommended build leaves 912 megabytes of free space, which translates to about
全精度下約 14,12,000 個 token 的上下文，或在 8 位元下高達 27,000 個 token。你的 12 GB 圖形
15:25.280–15:34.320
14 12,000 tokens of context at full precision or up to 27,000 tokens at 8 bit. Your 12 GB graphics
顯卡完全有能力運行這個 270 億參數的模型，但它 simply 無法處理
15:34.320–15:40.640
card is fully capable of running this 27 billion parameter model, but it simply cannot handle the
該實驗室推薦的特定 11.88 GB 構建版本，一旦該構建版本具備視覺能力。你真正需要的檔案
15:40.640–15:46.800
specific 11.8 8 GB build that the laboratory recommends once that build can see. The file
是 3bit 構建版本，在下載頁面上為 10.09 GB，或在你的顯卡單位中約為 9.40
15:46.800–15:54.640
you actually want is the 3bit build which comes in at 10.09 GB on the download page or about 9.40
GB。一旦你載入這些權重並結合視覺投影器，
15:54.640–15:59.840
in your card's own units. Once you load those weights together with the vision projector,
你將剩餘 1,624 MB 的緩衝空間。這意味著啟用圖像時的活躍
15:59.840–16:07.200
you're left with 1,624 megabytes of headroom. That leaves a ceiling of about 26,000 tokens of active
對話上限約為 26,000 個 token，如果你只運行文字模式，則可達 40,000 個 token。如果你
16:07.200–16:13.040
conversation with images enabled, which jumps to 40,000 tokens if you're running text only. If you
最佳化你的快取或退回到更小的 2.75 bit 構建版本，這些數字會進一步延伸
16:13.040–16:18.800
optimize your cache or step down to the even smaller 2.75 bit build, those numbers stretch
最佳化你的快取或退回到更小的 2.75 位元版本，這些數字會進一步擴大
16:18.800–16:24.000
even further. Keep in mind, these are absolute mathematical ceilings that ignore the runtime's
更遠。請記住，這些是絕對的數學上限，未考慮執行時的
16:24.000–16:29.040
own working buffers, so your real capacity is lower. According to the laboratory's own
自身工作緩衝區，因此你的實際容量較低。根據該實驗室自身的
16:29.040–16:35.040
evaluation, that downgrade costs half a point of GPQA diamond and about a point of live codebench
評估，這種降級會使 GPQA Diamond 的得分損失半點，Live Codebench 損失約一點，而 AIM25 的完美分數 100 則完全未變。
16:35.040–16:41.600
with the perfect score of 100 on AIM25 completely unchanged. And that tiny reduction in accuracy
而精確度的微小降低，換來了大約 26,000 個 token 的對話容量。
16:41.600–16:47.600
buys you roughly 26,000 tokens of conversation. Two separate teams have filed the same request
兩個獨立的團隊已在該實驗室的儲存庫上提交了相同的請求，但截至目前為止尚未收到回覆，
16:47.600–16:53.280
on the lab's repository, but with no reply so far, outside developers can only recover the final map
外部開發者只能恢復最終的映射，了解哪種格式對應哪個張量，卻無法自行執行搜尋。他們
16:53.280–16:58.960
of which format went to which tensor without any way to actually run the search themselves. They've
要求補齊遺漏的部分，因為公共程式碼中完全缺少幾個關鍵步驟，例如候選生成
16:58.960–17:03.440
asked for the missing pieces because several critical steps like the candidate generation
要求補齊遺漏的部分，因為幾個關鍵步驟，例如候選生成
17:03.440–17:09.040
are completely absent from the public code. Specifically, they're missing the per tensor cost
完全沒有包含在公開程式碼中。具體來說，他們缺少每個張量的成本
17:09.040–17:13.920
model, the settings for two of the budgets, and the export code that assembles the mixed format
檔案的匯出程式碼。其中一個團隊需要該程式碼，因為他們的伺服器機隊受限於記憶體頻寬
17:13.920–17:19.280
file. One of those teams wants the code because their server fleet is limited by memory bandwidth
而非容量，因此較小的檔案能直接提升速度，而這種混合格式是他們
17:19.280–17:24.720
rather than capacity, making a smaller file a direct speed win, and this mixed format is their
唯一可行的部署途徑。預算論文將混合精度量化、非均勻剪枝
17:24.720–17:31.360
only real deployment path. The budget paper names mixed precision quantization, non-uniform pruning,
，以及選擇保留哪些專家，視為將 K 個選項之一分配給 N 個群組的同一問題，
17:31.360–17:36.400
and choosing which experts to keep as the same problem of assigning one of K options to each of
這意味著這是一種將幾乎任何內容適應固定記憶體佔用量的方法。
17:36.400–17:42.320
N groups, which means this is a method for fitting almost anything into a fixed memory footprint.
因此，雖然該實驗室的發布為特定模型的預算提供了一個完整的答案，但
17:42.320–17:47.120
So, while the lab's release gives us a finished answer to one specific model's budget, the one
他們尚未推出的東西，是適用於其他任何人的方法。好了，各位，就是這些。
17:47.120–17:52.320
thing they haven't shipped yet is the method for anyone else's. Anyways, that's all, folks.
他們尚未推出的東西是其他人的方法。總之，就這些了，各位。

影片筆記：Qwen's 27B Model Is Now An 11.8GB Download

⚠️ 可先使用，仍有待修正項目

本筆記已完成可閱讀的逐字稿與時間跳轉；下列為待處理問題，建議播放原片確認。

待修正：逐字稿全文快速校讀未完成（已保留自動字幕/原始逐字稿並繼續生成筆記。；未完成的全文速讀項目可在精進模式回聽確認。）

一句話總結

影片詳細分析了將 Qwen 27B 模型壓縮至 11.8GB 並在 12GB 顯存硬體上運行的技術細節，指出雖然透過 GSQ 量化技術實現了壓縮，但視覺投影器（MM Proj）未壓縮導致記憶體極度緊繃，且存在長上下文 Bug、軟體相容性問題及黑盒量化帶來的穩定性風險。

核心重點

壓縮技術與架構：使用 GSQ（一種標量量化方法）結合「精確預算約束」（Exact Budget Constraints）算法，將 270 億參數模型壓縮至 11.8GB。模型採用非均勻量化，將 851 個張量分佈在 11 種不同的存儲格式中，部分組件（如 Feed forward gate）被壓縮至極低比特（低至 1.75 bits）。
記憶體陷阱與視覺投影器：標題中的 11.8GB 為十進位 GB，而 12GB 顯存為二進位 GB。實際模型權重約佔 10.96GB，但獨立的視覺投影器（MM Proj，約 0.87-0.93GB）未壓縮且必須額外載入，導致 12GB 顯卡在載入多模態能力後幾乎無剩餘空間用於上下文（KV Cache）。
架構與長上下文問題：模型由 48 個循環層（Recurrent layers）和 16 個完整注意力層組成。循環層固定記憶體佔用，使得 12GB 顯存成為可能，但長上下文測試中出現「立即結束文字標記」（End of text marker）的 Bug，模型偶爾會將結束標記作為第一個令牌生成。
軟體相容性與 Bug：
Llama.cpp：多模態工具在編碼圖像時因未預留記憶體導致分段錯誤（Segmentation fault）。
Olama：聊天模板拒絕非第一個出現的系統訊息，導致返回空字串。
LM Studio：若顯存不足，會將過剩權重靜默推入系統記憶體，而非報錯。
效能與黑盒風險：實驗室發布的基準測試顯示壓縮模型與原始模型效能接近，但由於採用 11 種混合格式，依賴 11 條不同的程式碼路徑，其中未被廣泛測試的路徑可能導致矩陣乘法錯誤或隨機輸出。外部開發者缺乏關鍵的組裝程式碼，無法自行驗證或最佳化。

詳細大綱

I. 模型壓縮概覽與檔案結構

核心成就：將 27 billion 參數模型壓縮為單一 11.8 GB 檔案，適合 12 GB 顯存的硬體。
檔案組成：
下載頁面提供 8 個主要檢查點（Checkpoints），分為兩種方式。
標題中的 11.8 GB 模型是平面選項中最大的，被實驗室推薦為「任務無損構建」（task lossless build）。
關鍵分離元件：存在第 9 個檔案，0.93 GB 的視覺投影器（MM Proj）。
8 個主要檢查點不包含視覺投影器，需單獨獲取以使用多模態能力。

II. 記憶體單位與實際可用空間計算

單位差異：
顯存標示的 12 GB 為二進位 GB（Binary GB），硬體實際可用空間。
HuggingFace 下載頁面標示的 11.77 GB 為十進位 GB（Decimal GB）。
實際空間分析：
GSQ RCO 構建的實際權重僅佔 10.96 GB。
12 GB 減去 10.96 GB，剩餘 1.04 GB 自由空間。
標題暗示的 200 MB 餘量被低估，實際餘量更大。

III. 視覺投影器（MM Proj）的記憶體懲罰

未壓縮的元件：MM Proj 是唯一未壓縮的檔案，保持原始 16-bit 格式。
記憶體佔用：
檔案大小為 931 million bytes 或 0.87 GB。
所有構建共用此單一副本，記憶體懲罰固定。
載入方式：在 Llama.cpp 中需使用單獨的命令列旗標（command line flag）載入，易被忽略。
影響：載入後，0.87 GB 從 1.04 GB 餘量中消失，幾乎耗盡剩餘空間。

IV. 量化方法論：GSQ 與精確預算約束

量化方法分類：
標量方法（Scalar methods）：如 GPTQ、AWQ，普及但準確率在 3-4 bits 時達到瓶頸。
向量/網格方法（Vector/Trellis methods）：如 Q-tip、AQLM，準確度高但難以實作與擴展。
GSQ 的定位：
在保持標量網格（scalar grid）的同時縮小準確率差距。
確保壓縮模型與現有標量推理內核（inference kernels）完全兼容。
可直接應用於現有的 GGUF 發布版，提升準確率並導回相同格式。
兼容日常應用程式：Llama.cpp、Olama、LM Studio。
精確預算約束（Exact Budget Constraints）：
參考論文：《Model Compression with Exact Budget Constraints》（作者：Michael Hellig 和 Dan Allistar）。
問題定義：在總成本預算下，為每個組群分配一種壓縮選項。
優勢：相比基於懲罰的方法，此方法無需超參數調整即可精確滿足預算。
技術原理：通過在邏輯空間（logit space）中沿著平滑曲面（Remmanion manifold）行走的梯度方法。
程序：將每個張量量化為候選格式 -> 搜尋符合預算的分配 -> 拼接成單一檔案。

V. 模型內部量化細節與張量分佈

張量分佈：
搜尋算法將 851 個獨立張量分佈在 11 種不同的存儲格式中。
96 個張量保持完整 16-bit 精度。
其餘張量進入較小的量化格式階梯。
具體案例：
144 個張量分配至 IQ3S 格式，平均落在實驗室設定的 3.5 bits 目標。
飢餓格式（Starved format）：Block 13 的 Feed forward gate 被壓縮至 IQ1M 格式，僅 1 和 3/4 bits。
該張量周圍的矩陣採用 2、3、4 bits 格式，該張量是該區域最便宜的選擇。
詞彙表（Vocabulary）佔比：
超過 11.77 GB 檔案的十分之一為詞彙。
兩個張量（輸出頭 Output head 和嵌入表 Embedding table）共佔 1.12 GB。
所有未壓縮張量合計僅 58 MB（不到 0.5%）。
注意力機制中的非均勻量化：
單一注意力區塊中，查詢投影（Query projection）獲得最便宜格式，而鍵（Key）和值（Value）保留較貴格式。
跨四個大小預算觀察：飢餓門控在較大預算時僅 1 bit，中間 3-bit 預算時升至 4 bits，最大預算時又降至 2 bits以下。
原因：全域預算優化器不根據重要性排名張量，而是進行位元組交易，僅保護總體預算。

VI. 效能評估與模型架構

效能數據來源：僅由進行壓縮的實驗室發布於其模型卡片（Model card），無外部複現。
基準測試結果：
AIM25 與 Live Codebench：推薦的 11.8 GB 檔案與未壓縮的 53.8 GB 原始模型完全匹配（4.6 倍體積縮減）。
GPQA Diamond：推薦構建僅落後原始模型 0.5 分。
3-bit 構建（10.09 GB）：在主推理基準測試中獲得完美 100 分，在 GPQA Diamond 上僅落後推薦構建 0.5 分。
2.75-bit 構建：恢復原始零樣本平均值的 101.8% 8%（註：原文如此），得分高於未壓縮基準。
8.4 GB 構建：在 AIM25 上比 Unslaught 的動態量化高出 10 分。
模型架構配置：
基礎模型配置包含 64 個 Transformer 層。
重複模式：3 個線性注意力層（Linear attention layers）後接 1 個完整注意力層（Full attention layer）。
總計：48 個閾值 Delta 層（Gated delta layers）和 16 個完整注意力層。
確認方式：Llama.cpp 的 Bug 報告與檔案張量名稱。

VII. 記憶體行為與長上下文問題

層類型與記憶體行為：
完整注意力層：需保留對話中的所有令牌（Token），記憶體隨字數增長。
循環層（Recurrent layers）：保持固定大小的摘要，與上下文長度無關（10 字與 10,000 字相同）。
48 個循環層不累積對話歷史，這是 12 GB 顯存能運行此模型的原因。
KV Cache 計算：
16 個注意力層，每層包含 2 個鍵值張量，4 個頭，每頭 256 個數字，每數字 2 位元組。
每令牌消耗：全精度 64 KB -> 8-bit 34 KB -> 4-bit 18 KB。
48 個循環層：固定消耗 150 MB。
上下文窗口成本：
阿里巴達宣稱原生 262,000 令牌上下文。
全精度緩存需 16 GB -> 8-bit 需 8.5 GB -> 4-bit 需 4.5 GB。
若 64 層全為注意力層，緩存需 18 GB，無法放入 16 GB 顯存。
開發者在 16 GB 顯存上運行全上下文需使用 4-bit 緩存。

VIII. 軟體相容性與已知 Bug

長上下文 Bug：
8 月底開發者提交 Bug 報告：長上下文下，模型偶爾將「結束文字標記」（End of text marker）作為第一個令牌生成，無錯誤提示。
復現情況：在顯存、CPU（相同構建）及獨立代碼庫分支上均可復現。
測試結果：約 113,000 令牌通過，約 130,000 令牌失敗（立即結束），稍深提示則正常。
原因推測：引擎缺陷而非權重問題（相同提示在 VLM 上通過）。
建議：無安全限制，需驗證具體深度。
Olama 相容性問題：
可直接從 Hugging Face 拉取構建。
問題：聊天模板（Chat template）拒絕非第一個出現的系統訊息（System message）。
後果：發送中途系統訊息會返回空字串和完成狀態，無錯誤。
修復：需對模型檔案進行三行修補。
註：此故障在 8 月針對純權重時被報告兩次，非此構建特有。
Llama.cpp 多模態工具問題：
問題：編碼圖像時未預留記憶體導致崩潰。
結果：分段錯誤（Segmentation fault），而非報告錯誤。
日誌顯示工具在...（內容截斷）。

IX. 記憶體溢出與軟體行為比較（分段 2 補充）

嘗試分配 247 MB 失敗的案例。
LM Studio 的行為：不崩潰，但將過剩權重靜默推入系統記憶體。
實驗室指南僅建議搜尋儲存庫，缺乏明確的「模型無法載入」提示。
用戶需自行計算記憶體需求（「點擊下載前計算位元數」）。

X. Blackwell 顯卡與工具包的相容性爭議

早期 9 月案例：16 GB Blackwell 顯卡搭配特定工具包版本。
Llama.cpp 後端測試失敗：矩陣乘法在三種格式上失敗，包含該構建命名的格式。
錯誤率高達 1.1，導致隨機垃圾輸出，但相鄰格式通過測試。
用戶反饋分歧：
用戶 A：報告垃圾輸出，通過修改位元處理程式碼本地修復。
用戶 B：在同一晶片世代無法重現，認為問題在於構建鏈而非硬體，因為這些格式沒有晶片特定路徑。
結論：選擇包含 11 種格式的檔案意味著依賴 11 條不同的程式碼路徑，爭議集中在最少使用的那些路徑上。

XI. 最佳化構建與效能取捨

推薦的 3-bit 構建：
下載頁面大小：10.09 GB。
顯卡單位大小：約 9.40 GB。
剩餘緩衝：1,624 MB。
帶圖像上下文上限：約 26,000 個 token。
純文字上下文上限：40,000 個 token。
更低比特構建（2.75 bit）：
進一步拉伸上述數字。
準確度代價：
GPQA Diamond：下降 0.5 分。
Live Codebench：下降 1 分。
AIM25：保持完美分數 100 不變。
代價換取：約 26,000 個 token 的對話空間。
注意：這些是絕對數學上限，未包含運行時工作緩衝區，實際容量更低。

XII. 外部開發者的困境與技術黑盒

兩個團隊向實驗室儲存庫提交請求，要求公開關鍵程式碼，目前無回覆。
缺失的關鍵組件：
每張量成本模型（per tensor cost model）。
兩個預算的設定（settings for two of the budgets）。
組裝混合格式檔案的匯出程式碼（export code）。
候選生成（candidate generation）完全缺失。
開發者需求原因：伺服器叢集受限於記憶體頻寬而非容量，較小檔案帶來直接速度優勢，混合格式是唯一的實際部署路徑。
實驗室方法論：混合精度量化、非均勻剪枝、選擇保留哪些專家，本質上是將 K 個選項分配給 N 個組的問題，用於將幾乎任何內容適應固定記憶體佔用。
最終結論：實驗室發布了單一特定模型預算的完成答案，但未發布適用於其他模型的通用方法。

工具 / 模型 / 名詞整理

模型/技術名稱：
Quen's 27 billion parameter model
GSQ RCO build
MM Proj (Vision projector)
GPTQ
AWQ
Q-tip
AQLM
GGUF
IQ3S
IQ1M
AIM25
Live Codebench
GPQA Diamond
Unslaught's dynamic quantizations
VLM
Llama.cpp
Olama
LM Studio
HuggingFace
Blackwell (顯卡晶片世代)

論文/作者：
《Model Compression with Exact Budget Constraints》
Michael Hellig
Dan Allistar

其他專有名詞：
Binary gigabytes
Decimal GB
Scalar inference kernels
Logit space
Remmanion manifold
Transformer layers
Linear attention layers
Full attention layers
Gated delta layers
KV Cache
End of text marker
Segmentation fault
Recurrent State Layers
bite handling code (疑為 bit handling code

尚未產生學習筆記

請在 Telegram 指令最後加上「學習」，例如：videonote 網址英文雙語學習