Quen's 27 billion parameter model reads images
and text. And the lab behind most local model
compression just shrank it into a single 11.8 GB
file. A model that size normally needs more memory
than any gaming computer has. This one is smaller
than a video game install. Needless to say,
this is a seriously impressive demonstration of
model compression. But the question must be asked,
can a 12 GBTE graphics card actually run
Quen's 27 billion parameter model? To find out,
we have to look at the repository where we see
that the download page doesn't actually offer
a single file, but a menu of four different
sizes shipped two ways for a total of eight
model files. That 11.8 GB model from our title
is the largest of the plane options, which the
lab recommends as its task lossless build. But
sitting right beside those models is a ninth file,
a 0.93 GBTE vision projector called MM Proj. None
of the eight main checkpoints actually contain
this vision projector, which means you have to
get it separately if you want to use the model's
multimodal capabilities. On paper, the 11.8 GB of
the GSQ RCO build and the 12 GB of your graphics
card sit right next to each other. But those
two numbers are actually measured in entirely
different units. When a card is sold as having
12 GB of memory, the operating system actually
counts that space in binary gigabytes, which is
what your hardware actually has to work with. On
the other side of the ledger, that recommended GSQ
RCO build is listed on HuggingFace at 11.77 GB.
But when you inspect the actual weights of that
GSQ RCO build, they only take up 10.96 GB of space
because the download page is using decimal GB. If
you subtract 10.96 from your 12 GB of capacity,
you are left with 1.04 GB of free space, that
means the actual headroom on your card is larger
than the 200 megabytes the headline number
suggested. But that extra headroom is about
to vanish because the model's ability to read
images is stored in a completely separate file
called mm praj, which weighs 931 million bytes or
0.87 GB. And in a release designed entirely around
model compression, this vision projector is the
only file that wasn't compressed at all. Instead,
the lab left it at its original 16-bit format.
And because one copy serves every single build
on the menu, this memory penalty is exactly
the same whether you pick the largest budget
or the smallest. In llama.cpp, you load this
projector using a separate command line flag,
which makes it easy to forget that it's even
there. But once it loads, 0.8 87 GB of the 1.04 we
just found is instantly gone. This means the part
of the model the title sells, the system that lets
it read images, is the one part the compression
never touched, and it eats almost all of the
headroom your card has left. The quantization
paper behind this build splits the field of
model compression into simple scalar methods like
GPTQ and AWQ that everyone deploys and vector or
trellis methods like Q-tip and AQLM that score
better. In the paper's own words, scalar methods
plateau in accuracy at 3 to four bits, while
the more accurate vector family is notoriously
hard to implement and scale. So GSQ closes most of
that accuracy gap while staying on a scalar grid,
ensuring the compressed model remains fully
compatible with the existing scalar inference
kernels we already run on our cards. It turns out
the same lab published GPTQ, which is one of the
standard methods GSQ measures itself against. On
top of that, GSQ can be applied directly to an
existing GGUF release you already have, improving
its accuracy and exporting it right back into the
exact same format. This file runs in your everyday
apps like Llama.cpp, Olama, and LM Studio without
modifying any of them. And that's not luck. It
happens because the lab deliberately chose the
family of methods that scores worse in order to
stay inside a format the existing software already
reads. Compressing a single tensor doesn't
build a whole file because you still have to
decide how much compression to apply to each part
of the model to fit your hardware. To solve that,
Michael Hellig and Dan Allistar published a second
paper titled Model Compression with Exact Budget
Constraints, framing the problem as assigning
one of several options to each group under a
total cost budget. While penalty-based approaches
only enforce a budget approximately and require
extensive hyperparameter tuning, this method meets
it exactly without any constraintspecific tuning.
Relaxing the budget constraint describes a smooth
curved surface called a remmanion manifold in
lojit space allowing a gradient method to walk
along it without ever leaving it. The procedure
takes three steps. You quantize every tensor into
every candidate format, search for the assignment
that meets your budget and stitch those winners
into one ordinary file. And that means you can
spend your memory budget down to the last bite,
which is exactly the problem you face when trying
to squeeze a 27 billion parameter model onto a 12
GB card. You don't even have to download the model
to see what that search algorithm decided because
the lab published the entire decision list as a
plain text file. Inside the recommended build, the
search algorithm ended up slicing 851 individual
tensors across 11 different storage formats.
It left 96 tensors at full 16-bit precision
while dropping the rest into a staircase of
smaller quantization formats. For instance,
the search assigned 144 tensors to the IQ3S
format. And the measured average lands right
on the lab's 3.5 target, but non-uniform is not
just a marketing word, and the search isn't afraid
to starve a single component if it means saving
the rest of the network. In the published dump,
block 13's feed forward gate is squeezed down to a
starved format called IQ1M, running on just 1 and
3/4 bits. That single tensor isn't surrounded by
full precision. It sits inside a block whose other
matrices were all given 2, 3, and 4bit formats.
It's simply the cheapest thing in a neighborhood
of ordinary compromises. More than a tenth of the
11.77 GB file is just vocabulary. Specifically,
two tensors that do nothing but turn tokens
into numbers and back account for 1.12 GB of
the file size together, split between the output
head and the embedding table. Against that,
every uncompressed tensor in the file adds up
to 58 megabytes, under half a percent. Inside a
single attention block, the search gave the query
projection the cheapest format in its block, while
the key and value neighbors beside it kept more
expensive ones. If you watch that same starved
gate across the four size budgets, it starts
at just one bit, goes up to four bits at the
intermediate three-bit budget and drops back under
two bits at the largest. This happens because a
global budget optimizer isn't ranking tensors by
importance. It's trading bytes and it'll happily
starve a tensor at a larger budget than it fed
it at a smaller one because the only thing it's
protecting is the total. Every performance score
for these compressed builds comes from the lab
that performed the compression, published on its
own model card, and nobody outside that lab has
reproduced these comparisons. According to their
results, the recommended 11.8 GB file matches the
uncompressed 53.8 GB original exactly on AIM25
and Live Codebench, representing a 4.6 time size
reduction. On GPQA Diamond, this recommended
build trails the original by only half a point,
and the scores remain high even as you step down
to the threebit build, which is a gigabyte and
a half smaller. At 10.09 GB, this 3bit version
still hits a perfect 100 on that main reasoning
benchmark, while trailing the recommended build
by only half a point on GPQA Diamond. And near
the bottom of the size scale, their 2.75 bit
build actually recovers 101.8% 8% of the original
zeroshot average, scoring above the uncompressed
baseline. At a matched 8.4 GB, their build beats
Unslaught's dynamic quantizations by 10 points
on Aim25. Because the gap between the recommended
build and the one a gigabyte and a half smaller
is only half a point on one benchmark, picking a
build is not really a quality decision. The base
model's configuration lists 64 transformer layers
in a repeating pattern of three linear attention
layers followed by one full attention layer. That
leaves 48 gated delta layers and 16 full attention
layers that the files own tensor names confirm. A
llama.cpp bug report describes the running model
in those exact terms, which matters because the
way these layers process information changes
how memory behaves. While a full attention
layer has to keep every token of the conversation
available, so its memory grows with every word,
a recurrent layer keeps one fixed size summary
that is the same size after 10 words as after
10,000. That fixed size summary is the reason 48
of its 64 layers never accumulate conversation
history. And that is the reason a 12 GB graphics
card is in this conversation at all. We can work
this out directly from the base configuration.
The model has 16 attention layers and each layer
holds two key value tensors containing four heads
of 256 numbers stored at two bytes a piece. That
multiplies out to 64 kilob for every single token
in the conversation. Compressing that cache brings
the price down. Drop it to 8 bits and you spend
34 kilob a token. Squeeze it to four bits and
you only spend 18. Meanwhile, those 48 recurrent
layers cost a flat 150 megbytes at any context
length. So when Alibaba advertises a native
262,000 token context, that window has a very
real price. At full precision, the cache alone
needs 16 GB. That drops to 8 1/2 GB at 8 bit and
4 1/2 GB at 4bit. We can check that arithmetic
against a machine known to work. A developer
running this model at full context on a 16 GB card
has to use a 4-bit cache, which the math puts at
4 1/2 GB. If all 64 layers were attention, that
cache would require 18 GB and it wouldn't fit on
their card at all. So, what does that actually
leave for our 12 GB card? But what happens at
the long context this model advertises on the
software your card would actually be running,
like llama.cpp. In late August, a developer
filed a bug report showing that at long context,
the model occasionally generates an end of text
marker as its very first token with absolutely no
error. The failure is stubborn, reproducing on
the graphics card, on the CPU with the same build,
and on an independent fork of the codebase. But
the reporter's differential test points to an
engine defect rather than the weights themselves
since the exact same prompt passes cleanly on VLM.
When testing context capacity, around 113,000
tokens passes cleanly, but around 130,000 fails
with that instant end of text marker, while a
slightly deeper prompt works perfectly fine.
Again, keep in mind this was measured on another
lab's quantizations rather than this build. And
a sibling model with only 30 recurrent layers
survives much deeper. The reporter eventually
withdrew his own proposed threshold because of
these random holes of failure, advising that
there is no safe limit to quote, "and you simply
have to validate the exact depths you plan to
serve." Olama pulls this exact build straight from
hugging face with a single command. And although a
maintainer demonstrated it working, actual usage
breaks on the chat template rather than your
graphics card. The built-in template refuses any
system message that isn't first. So if you send
one mid conversation, you get an empty string,
a finished status, and absolutely no error,
fixing it takes a threeline patch in a model
file. And since this exact fault was filed twice
in August against the plain weights, it isn't this
build's doing. Switching to llama.cpp's multimodal
tool is even worse because failing to reserve
memory while encoding an image crashes the process
with a segmentation fault instead of reporting
an error. Its crash log shows the tool dying on
a failed attempt to allocate just 247 megabytes
while reading a single picture. LM Studio avoids
crashing entirely, but its offload slider will
quietly push excess weights into ordinary system
memory rather than refuse to load them. The
lab's entire LM Studio guidance is a single
sentence telling you to search the repository, and
nobody has published a test of this build on a 12
GB card. None of the three applications tells you
when the model won't fit, which is why the only
way to know if it runs is to count your bites
before you click download. In early September,
a developer running a 16 GB Blackwell card with
a specific toolkit version found that Llama.cpp's
own back-end test failed matrix multiplication
for three formats, including the exact format this
build is named after. The reported errors reached
up to 1.1, which is large enough to mean you just
get random garbage output, but the neighboring
formats in that same run passed, making the error
highly specific. Then a second user reported the
same garbage output on another Blackwell card,
fixing it locally with a small change to the bite
handling code. But a third user on the same chip
generation couldn't reproduce it, arguing the
fault lies in the build chain rather than the
hardware since there's no chip specific path for
these formats. Anyway, while this debate is still
unresolved, choosing a file assembled from 11
formats means you're depending on 11 distinct
code paths. And those least used paths are exactly
where the arguments are still happening. The
recommended model file itself is 11.77 GB, but on
your card, those weights consume 10.96 GB of VRAM.
Adding the separate vision projector eats another
0.87 GB of memory, which brings the combined total
to 11.83 GB and leaves you with a tiny pocket of
just 174 MGB. Subtracting the flat 150 megabytes
demanded by the recurrent state layers leaves
24 megabytes of free headroom. At 64 kilobytes
per token, that remaining space can hold at most
about 386 tokens and fewer in practice. Of course,
that is a theoretical ceiling that completely
ignores whatever graphics memory your desktop
operating system is already using, not to mention
the working memory that the inference software
itself reserves before it allocates a single token
of cash. If you try to step up to the largest
12.12 GB build, adding the projector overshoots
your card's memory by 308 MGB, meaning it cannot
hold its own eyes at zero context. But if you
go text only and work out the math without that
vision projector, the recommended build leaves 912
megabytes of free space, which translates to about
14 12,000 tokens of context at full precision or
up to 27,000 tokens at 8 bit. Your 12 GB graphics
card is fully capable of running this 27 billion
parameter model, but it simply cannot handle the
specific 11.8 8 GB build that the laboratory
recommends once that build can see. The file
you actually want is the 3bit build which comes
in at 10.09 GB on the download page or about 9.40
in your card's own units. Once you load those
weights together with the vision projector,
you're left with 1,624 megabytes of headroom. That
leaves a ceiling of about 26,000 tokens of active
conversation with images enabled, which jumps to
40,000 tokens if you're running text only. If you
optimize your cache or step down to the even
smaller 2.75 bit build, those numbers stretch
even further. Keep in mind, these are absolute
mathematical ceilings that ignore the runtime's
own working buffers, so your real capacity
is lower. According to the laboratory's own
evaluation, that downgrade costs half a point of
GPQA diamond and about a point of live codebench
with the perfect score of 100 on AIM25 completely
unchanged. And that tiny reduction in accuracy
buys you roughly 26,000 tokens of conversation.
Two separate teams have filed the same request
on the lab's repository, but with no reply so far,
outside developers can only recover the final map
of which format went to which tensor without any
way to actually run the search themselves. They've
asked for the missing pieces because several
critical steps like the candidate generation
are completely absent from the public code.
Specifically, they're missing the per tensor cost
model, the settings for two of the budgets, and
the export code that assembles the mixed format
file. One of those teams wants the code because
their server fleet is limited by memory bandwidth
rather than capacity, making a smaller file a
direct speed win, and this mixed format is their
only real deployment path. The budget paper names
mixed precision quantization, non-uniform pruning,
and choosing which experts to keep as the same
problem of assigning one of K options to each of
N groups, which means this is a method for fitting
almost anything into a fixed memory footprint.
So, while the lab's release gives us a finished
answer to one specific model's budget, the one
thing they haven't shipped yet is the method
for anyone else's. Anyways, that's all, folks.