WEBVTT
Kind: captions
Language: en

00:00:00.080 --> 00:00:06.720
Quen's 27 billion parameter model reads images 
and text. And the lab behind most local model  

00:00:06.720 --> 00:00:13.760
compression just shrank it into a single 11.8 GB 
file. A model that size normally needs more memory  

00:00:13.760 --> 00:00:19.680
than any gaming computer has. This one is smaller 
than a video game install. Needless to say,  

00:00:19.680 --> 00:00:25.120
this is a seriously impressive demonstration of 
model compression. But the question must be asked,  

00:00:25.120 --> 00:00:31.120
can a 12 GBTE graphics card actually run 
Quen's 27 billion parameter model? To find out,  

00:00:31.120 --> 00:00:35.440
we have to look at the repository where we see 
that the download page doesn't actually offer  

00:00:35.440 --> 00:00:40.800
a single file, but a menu of four different 
sizes shipped two ways for a total of eight  

00:00:40.800 --> 00:00:47.280
model files. That 11.8 GB model from our title 
is the largest of the plane options, which the  

00:00:47.280 --> 00:00:53.280
lab recommends as its task lossless build. But 
sitting right beside those models is a ninth file,  

00:00:53.280 --> 00:01:00.320
a 0.93 GBTE vision projector called MM Proj. None 
of the eight main checkpoints actually contain  

00:01:00.320 --> 00:01:04.400
this vision projector, which means you have to 
get it separately if you want to use the model's  

00:01:04.400 --> 00:01:12.560
multimodal capabilities. On paper, the 11.8 GB of 
the GSQ RCO build and the 12 GB of your graphics  

00:01:12.560 --> 00:01:17.520
card sit right next to each other. But those 
two numbers are actually measured in entirely  

00:01:17.520 --> 00:01:23.360
different units. When a card is sold as having 
12 GB of memory, the operating system actually  

00:01:23.360 --> 00:01:29.120
counts that space in binary gigabytes, which is 
what your hardware actually has to work with. On  

00:01:29.120 --> 00:01:37.120
the other side of the ledger, that recommended GSQ 
RCO build is listed on HuggingFace at 11.77 GB.  

00:01:37.120 --> 00:01:44.320
But when you inspect the actual weights of that 
GSQ RCO build, they only take up 10.96 GB of space  

00:01:44.320 --> 00:01:52.000
because the download page is using decimal GB. If 
you subtract 10.96 from your 12 GB of capacity,  

00:01:52.000 --> 00:01:58.960
you are left with 1.04 GB of free space, that 
means the actual headroom on your card is larger  

00:01:58.960 --> 00:02:03.760
than the 200 megabytes the headline number 
suggested. But that extra headroom is about  

00:02:03.760 --> 00:02:08.720
to vanish because the model's ability to read 
images is stored in a completely separate file  

00:02:08.720 --> 00:02:18.320
called mm praj, which weighs 931 million bytes or 
0.87 GB. And in a release designed entirely around  

00:02:18.320 --> 00:02:24.320
model compression, this vision projector is the 
only file that wasn't compressed at all. Instead,  

00:02:24.320 --> 00:02:29.840
the lab left it at its original 16-bit format. 
And because one copy serves every single build  

00:02:29.840 --> 00:02:34.560
on the menu, this memory penalty is exactly 
the same whether you pick the largest budget  

00:02:34.560 --> 00:02:41.040
or the smallest. In llama.cpp, you load this 
projector using a separate command line flag,  

00:02:41.040 --> 00:02:48.480
which makes it easy to forget that it's even 
there. But once it loads, 0.8 87 GB of the 1.04 we  

00:02:48.480 --> 00:02:54.240
just found is instantly gone. This means the part 
of the model the title sells, the system that lets  

00:02:54.240 --> 00:02:59.280
it read images, is the one part the compression 
never touched, and it eats almost all of the  

00:02:59.280 --> 00:03:04.400
headroom your card has left. The quantization 
paper behind this build splits the field of  

00:03:04.400 --> 00:03:11.680
model compression into simple scalar methods like 
GPTQ and AWQ that everyone deploys and vector or  

00:03:11.680 --> 00:03:18.800
trellis methods like Q-tip and AQLM that score 
better. In the paper's own words, scalar methods  

00:03:18.800 --> 00:03:24.000
plateau in accuracy at 3 to four bits, while 
the more accurate vector family is notoriously  

00:03:24.000 --> 00:03:30.640
hard to implement and scale. So GSQ closes most of 
that accuracy gap while staying on a scalar grid,  

00:03:30.640 --> 00:03:34.880
ensuring the compressed model remains fully 
compatible with the existing scalar inference  

00:03:34.880 --> 00:03:41.280
kernels we already run on our cards. It turns out 
the same lab published GPTQ, which is one of the  

00:03:41.280 --> 00:03:47.520
standard methods GSQ measures itself against. On 
top of that, GSQ can be applied directly to an  

00:03:47.520 --> 00:03:53.200
existing GGUF release you already have, improving 
its accuracy and exporting it right back into the  

00:03:53.200 --> 00:04:00.480
exact same format. This file runs in your everyday 
apps like Llama.cpp, Olama, and LM Studio without  

00:04:00.480 --> 00:04:05.840
modifying any of them. And that's not luck. It 
happens because the lab deliberately chose the  

00:04:05.840 --> 00:04:11.200
family of methods that scores worse in order to 
stay inside a format the existing software already  

00:04:11.200 --> 00:04:15.920
reads. Compressing a single tensor doesn't 
build a whole file because you still have to  

00:04:15.920 --> 00:04:21.600
decide how much compression to apply to each part 
of the model to fit your hardware. To solve that,  

00:04:21.600 --> 00:04:27.280
Michael Hellig and Dan Allistar published a second 
paper titled Model Compression with Exact Budget  

00:04:27.280 --> 00:04:32.400
Constraints, framing the problem as assigning 
one of several options to each group under a  

00:04:32.400 --> 00:04:38.240
total cost budget. While penalty-based approaches 
only enforce a budget approximately and require  

00:04:38.240 --> 00:04:44.800
extensive hyperparameter tuning, this method meets 
it exactly without any constraintspecific tuning.  

00:04:44.800 --> 00:04:50.160
Relaxing the budget constraint describes a smooth 
curved surface called a remmanion manifold in  

00:04:50.160 --> 00:04:55.760
lojit space allowing a gradient method to walk 
along it without ever leaving it. The procedure  

00:04:55.760 --> 00:05:01.440
takes three steps. You quantize every tensor into 
every candidate format, search for the assignment  

00:05:01.440 --> 00:05:06.640
that meets your budget and stitch those winners 
into one ordinary file. And that means you can  

00:05:06.640 --> 00:05:11.360
spend your memory budget down to the last bite, 
which is exactly the problem you face when trying  

00:05:11.360 --> 00:05:17.920
to squeeze a 27 billion parameter model onto a 12 
GB card. You don't even have to download the model  

00:05:17.920 --> 00:05:23.120
to see what that search algorithm decided because 
the lab published the entire decision list as a  

00:05:23.120 --> 00:05:30.240
plain text file. Inside the recommended build, the 
search algorithm ended up slicing 851 individual  

00:05:30.240 --> 00:05:37.040
tensors across 11 different storage formats. 
It left 96 tensors at full 16-bit precision  

00:05:37.040 --> 00:05:42.240
while dropping the rest into a staircase of 
smaller quantization formats. For instance,  

00:05:42.240 --> 00:05:48.880
the search assigned 144 tensors to the IQ3S 
format. And the measured average lands right  

00:05:48.880 --> 00:05:55.200
on the lab's 3.5 target, but non-uniform is not 
just a marketing word, and the search isn't afraid  

00:05:55.200 --> 00:06:00.160
to starve a single component if it means saving 
the rest of the network. In the published dump,  

00:06:00.160 --> 00:06:07.120
block 13's feed forward gate is squeezed down to a 
starved format called IQ1M, running on just 1 and  

00:06:07.120 --> 00:06:13.280
3/4 bits. That single tensor isn't surrounded by 
full precision. It sits inside a block whose other  

00:06:13.280 --> 00:06:19.200
matrices were all given 2, 3, and 4bit formats. 
It's simply the cheapest thing in a neighborhood  

00:06:19.200 --> 00:06:27.040
of ordinary compromises. More than a tenth of the 
11.77 GB file is just vocabulary. Specifically,  

00:06:27.040 --> 00:06:33.280
two tensors that do nothing but turn tokens 
into numbers and back account for 1.12 GB of  

00:06:33.280 --> 00:06:38.720
the file size together, split between the output 
head and the embedding table. Against that,  

00:06:38.720 --> 00:06:44.960
every uncompressed tensor in the file adds up 
to 58 megabytes, under half a percent. Inside a  

00:06:44.960 --> 00:06:50.160
single attention block, the search gave the query 
projection the cheapest format in its block, while  

00:06:50.160 --> 00:06:55.520
the key and value neighbors beside it kept more 
expensive ones. If you watch that same starved  

00:06:55.520 --> 00:07:00.720
gate across the four size budgets, it starts 
at just one bit, goes up to four bits at the  

00:07:00.720 --> 00:07:06.160
intermediate three-bit budget and drops back under 
two bits at the largest. This happens because a  

00:07:06.160 --> 00:07:12.000
global budget optimizer isn't ranking tensors by 
importance. It's trading bytes and it'll happily  

00:07:12.000 --> 00:07:16.560
starve a tensor at a larger budget than it fed 
it at a smaller one because the only thing it's  

00:07:16.560 --> 00:07:20.960
protecting is the total. Every performance score 
for these compressed builds comes from the lab  

00:07:20.960 --> 00:07:25.920
that performed the compression, published on its 
own model card, and nobody outside that lab has  

00:07:25.920 --> 00:07:32.480
reproduced these comparisons. According to their 
results, the recommended 11.8 GB file matches the  

00:07:32.480 --> 00:07:41.040
uncompressed 53.8 GB original exactly on AIM25 
and Live Codebench, representing a 4.6 time size  

00:07:41.040 --> 00:07:46.960
reduction. On GPQA Diamond, this recommended 
build trails the original by only half a point,  

00:07:46.960 --> 00:07:51.760
and the scores remain high even as you step down 
to the threebit build, which is a gigabyte and  

00:07:51.760 --> 00:07:59.200
a half smaller. At 10.09 GB, this 3bit version 
still hits a perfect 100 on that main reasoning  

00:07:59.200 --> 00:08:04.640
benchmark, while trailing the recommended build 
by only half a point on GPQA Diamond. And near  

00:08:04.640 --> 00:08:11.760
the bottom of the size scale, their 2.75 bit 
build actually recovers 101.8% 8% of the original  

00:08:11.760 --> 00:08:18.800
zeroshot average, scoring above the uncompressed 
baseline. At a matched 8.4 GB, their build beats  

00:08:18.800 --> 00:08:24.960
Unslaught's dynamic quantizations by 10 points 
on Aim25. Because the gap between the recommended  

00:08:24.960 --> 00:08:30.080
build and the one a gigabyte and a half smaller 
is only half a point on one benchmark, picking a  

00:08:30.080 --> 00:08:36.400
build is not really a quality decision. The base 
model's configuration lists 64 transformer layers  

00:08:36.400 --> 00:08:42.240
in a repeating pattern of three linear attention 
layers followed by one full attention layer. That  

00:08:42.240 --> 00:08:49.040
leaves 48 gated delta layers and 16 full attention 
layers that the files own tensor names confirm. A  

00:08:49.040 --> 00:08:54.640
llama.cpp bug report describes the running model 
in those exact terms, which matters because the  

00:08:54.640 --> 00:08:59.840
way these layers process information changes 
how memory behaves. While a full attention  

00:08:59.840 --> 00:09:04.880
layer has to keep every token of the conversation 
available, so its memory grows with every word,  

00:09:04.880 --> 00:09:10.400
a recurrent layer keeps one fixed size summary 
that is the same size after 10 words as after  

00:09:10.400 --> 00:09:17.280
10,000. That fixed size summary is the reason 48 
of its 64 layers never accumulate conversation  

00:09:17.280 --> 00:09:23.040
history. And that is the reason a 12 GB graphics 
card is in this conversation at all. We can work  

00:09:23.040 --> 00:09:28.560
this out directly from the base configuration. 
The model has 16 attention layers and each layer  

00:09:28.560 --> 00:09:36.320
holds two key value tensors containing four heads 
of 256 numbers stored at two bytes a piece. That  

00:09:36.320 --> 00:09:42.800
multiplies out to 64 kilob for every single token 
in the conversation. Compressing that cache brings  

00:09:42.800 --> 00:09:49.040
the price down. Drop it to 8 bits and you spend 
34 kilob a token. Squeeze it to four bits and  

00:09:49.040 --> 00:09:56.400
you only spend 18. Meanwhile, those 48 recurrent 
layers cost a flat 150 megbytes at any context  

00:09:56.400 --> 00:10:03.600
length. So when Alibaba advertises a native 
262,000 token context, that window has a very  

00:10:03.600 --> 00:10:12.080
real price. At full precision, the cache alone 
needs 16 GB. That drops to 8 1/2 GB at 8 bit and  

00:10:12.080 --> 00:10:18.080
4 1/2 GB at 4bit. We can check that arithmetic 
against a machine known to work. A developer  

00:10:18.080 --> 00:10:24.480
running this model at full context on a 16 GB card 
has to use a 4-bit cache, which the math puts at  

00:10:24.480 --> 00:10:31.920
4 1/2 GB. If all 64 layers were attention, that 
cache would require 18 GB and it wouldn't fit on  

00:10:31.920 --> 00:10:37.600
their card at all. So, what does that actually 
leave for our 12 GB card? But what happens at  

00:10:37.600 --> 00:10:42.560
the long context this model advertises on the 
software your card would actually be running,  

00:10:42.560 --> 00:10:49.040
like llama.cpp. In late August, a developer 
filed a bug report showing that at long context,  

00:10:49.040 --> 00:10:54.560
the model occasionally generates an end of text 
marker as its very first token with absolutely no  

00:10:54.560 --> 00:11:00.240
error. The failure is stubborn, reproducing on 
the graphics card, on the CPU with the same build,  

00:11:00.240 --> 00:11:05.280
and on an independent fork of the codebase. But 
the reporter's differential test points to an  

00:11:05.280 --> 00:11:11.840
engine defect rather than the weights themselves 
since the exact same prompt passes cleanly on VLM.  

00:11:11.840 --> 00:11:19.280
When testing context capacity, around 113,000 
tokens passes cleanly, but around 130,000 fails  

00:11:19.280 --> 00:11:24.240
with that instant end of text marker, while a 
slightly deeper prompt works perfectly fine.  

00:11:24.240 --> 00:11:29.440
Again, keep in mind this was measured on another 
lab's quantizations rather than this build. And  

00:11:29.440 --> 00:11:35.360
a sibling model with only 30 recurrent layers 
survives much deeper. The reporter eventually  

00:11:35.360 --> 00:11:40.240
withdrew his own proposed threshold because of 
these random holes of failure, advising that  

00:11:40.240 --> 00:11:45.600
there is no safe limit to quote, "and you simply 
have to validate the exact depths you plan to  

00:11:45.600 --> 00:11:52.000
serve." Olama pulls this exact build straight from 
hugging face with a single command. And although a  

00:11:52.000 --> 00:11:57.040
maintainer demonstrated it working, actual usage 
breaks on the chat template rather than your  

00:11:57.040 --> 00:12:03.040
graphics card. The built-in template refuses any 
system message that isn't first. So if you send  

00:12:03.040 --> 00:12:09.280
one mid conversation, you get an empty string, 
a finished status, and absolutely no error,  

00:12:09.280 --> 00:12:14.640
fixing it takes a threeline patch in a model 
file. And since this exact fault was filed twice  

00:12:14.640 --> 00:12:21.520
in August against the plain weights, it isn't this 
build's doing. Switching to llama.cpp's multimodal  

00:12:21.520 --> 00:12:26.800
tool is even worse because failing to reserve 
memory while encoding an image crashes the process  

00:12:26.800 --> 00:12:32.400
with a segmentation fault instead of reporting 
an error. Its crash log shows the tool dying on  

00:12:32.400 --> 00:12:39.840
a failed attempt to allocate just 247 megabytes 
while reading a single picture. LM Studio avoids  

00:12:39.840 --> 00:12:45.360
crashing entirely, but its offload slider will 
quietly push excess weights into ordinary system  

00:12:45.360 --> 00:12:51.120
memory rather than refuse to load them. The 
lab's entire LM Studio guidance is a single  

00:12:51.120 --> 00:12:56.400
sentence telling you to search the repository, and 
nobody has published a test of this build on a 12  

00:12:56.400 --> 00:13:02.320
GB card. None of the three applications tells you 
when the model won't fit, which is why the only  

00:13:02.320 --> 00:13:08.320
way to know if it runs is to count your bites 
before you click download. In early September,  

00:13:08.320 --> 00:13:14.640
a developer running a 16 GB Blackwell card with 
a specific toolkit version found that Llama.cpp's  

00:13:14.640 --> 00:13:20.720
own back-end test failed matrix multiplication 
for three formats, including the exact format this  

00:13:20.720 --> 00:13:26.720
build is named after. The reported errors reached 
up to 1.1, which is large enough to mean you just  

00:13:26.720 --> 00:13:32.320
get random garbage output, but the neighboring 
formats in that same run passed, making the error  

00:13:32.320 --> 00:13:38.880
highly specific. Then a second user reported the 
same garbage output on another Blackwell card,  

00:13:38.880 --> 00:13:45.040
fixing it locally with a small change to the bite 
handling code. But a third user on the same chip  

00:13:45.040 --> 00:13:49.680
generation couldn't reproduce it, arguing the 
fault lies in the build chain rather than the  

00:13:49.680 --> 00:13:55.600
hardware since there's no chip specific path for 
these formats. Anyway, while this debate is still  

00:13:55.600 --> 00:14:01.120
unresolved, choosing a file assembled from 11 
formats means you're depending on 11 distinct  

00:14:01.120 --> 00:14:07.280
code paths. And those least used paths are exactly 
where the arguments are still happening. The  

00:14:07.280 --> 00:14:17.120
recommended model file itself is 11.77 GB, but on 
your card, those weights consume 10.96 GB of VRAM.  

00:14:17.120 --> 00:14:23.600
Adding the separate vision projector eats another 
0.87 GB of memory, which brings the combined total  

00:14:23.600 --> 00:14:33.680
to 11.83 GB and leaves you with a tiny pocket of 
just 174 MGB. Subtracting the flat 150 megabytes  

00:14:33.680 --> 00:14:40.240
demanded by the recurrent state layers leaves 
24 megabytes of free headroom. At 64 kilobytes  

00:14:40.240 --> 00:14:48.000
per token, that remaining space can hold at most 
about 386 tokens and fewer in practice. Of course,  

00:14:48.000 --> 00:14:53.040
that is a theoretical ceiling that completely 
ignores whatever graphics memory your desktop  

00:14:53.040 --> 00:14:57.840
operating system is already using, not to mention 
the working memory that the inference software  

00:14:57.840 --> 00:15:04.240
itself reserves before it allocates a single token 
of cash. If you try to step up to the largest  

00:15:04.240 --> 00:15:12.080
12.12 GB build, adding the projector overshoots 
your card's memory by 308 MGB, meaning it cannot  

00:15:12.080 --> 00:15:18.320
hold its own eyes at zero context. But if you 
go text only and work out the math without that  

00:15:18.320 --> 00:15:25.280
vision projector, the recommended build leaves 912 
megabytes of free space, which translates to about  

00:15:25.280 --> 00:15:34.320
14 12,000 tokens of context at full precision or 
up to 27,000 tokens at 8 bit. Your 12 GB graphics  

00:15:34.320 --> 00:15:40.640
card is fully capable of running this 27 billion 
parameter model, but it simply cannot handle the  

00:15:40.640 --> 00:15:46.800
specific 11.8 8 GB build that the laboratory 
recommends once that build can see. The file  

00:15:46.800 --> 00:15:54.640
you actually want is the 3bit build which comes 
in at 10.09 GB on the download page or about 9.40  

00:15:54.640 --> 00:15:59.840
in your card's own units. Once you load those 
weights together with the vision projector,  

00:15:59.840 --> 00:16:07.200
you're left with 1,624 megabytes of headroom. That 
leaves a ceiling of about 26,000 tokens of active  

00:16:07.200 --> 00:16:13.040
conversation with images enabled, which jumps to 
40,000 tokens if you're running text only. If you  

00:16:13.040 --> 00:16:18.800
optimize your cache or step down to the even 
smaller 2.75 bit build, those numbers stretch  

00:16:18.800 --> 00:16:24.000
even further. Keep in mind, these are absolute 
mathematical ceilings that ignore the runtime's  

00:16:24.000 --> 00:16:29.040
own working buffers, so your real capacity 
is lower. According to the laboratory's own  

00:16:29.040 --> 00:16:35.040
evaluation, that downgrade costs half a point of 
GPQA diamond and about a point of live codebench  

00:16:35.040 --> 00:16:41.600
with the perfect score of 100 on AIM25 completely 
unchanged. And that tiny reduction in accuracy  

00:16:41.600 --> 00:16:47.600
buys you roughly 26,000 tokens of conversation. 
Two separate teams have filed the same request  

00:16:47.600 --> 00:16:53.280
on the lab's repository, but with no reply so far, 
outside developers can only recover the final map  

00:16:53.280 --> 00:16:58.960
of which format went to which tensor without any 
way to actually run the search themselves. They've  

00:16:58.960 --> 00:17:03.440
asked for the missing pieces because several 
critical steps like the candidate generation  

00:17:03.440 --> 00:17:09.040
are completely absent from the public code. 
Specifically, they're missing the per tensor cost  

00:17:09.040 --> 00:17:13.920
model, the settings for two of the budgets, and 
the export code that assembles the mixed format  

00:17:13.920 --> 00:17:19.280
file. One of those teams wants the code because 
their server fleet is limited by memory bandwidth  

00:17:19.280 --> 00:17:24.720
rather than capacity, making a smaller file a 
direct speed win, and this mixed format is their  

00:17:24.720 --> 00:17:31.360
only real deployment path. The budget paper names 
mixed precision quantization, non-uniform pruning,  

00:17:31.360 --> 00:17:36.400
and choosing which experts to keep as the same 
problem of assigning one of K options to each of  

00:17:36.400 --> 00:17:42.320
N groups, which means this is a method for fitting 
almost anything into a fixed memory footprint.  

00:17:42.320 --> 00:17:47.120
So, while the lab's release gives us a finished 
answer to one specific model's budget, the one  

00:17:47.120 --> 00:17:52.320
thing they haven't shipped yet is the method 
for anyone else's. Anyways, that's all, folks.