Hi guys, welcome back. In this video,
we'll be talking about how you can run a
Qwen in an Apple
M4 Mac mini machine with just 24 GB of
unified memory. We know that Qwen 3.8 27
billion parameter model will be of at
least 15 GB of size. And loading the
entire model in the unified memory will
not only load the model, but also loads
the KV cache, runtime, and the Mac
operating system all together. And while
you load all of them in the VRAM, it is
definitely going to hit the memory
ceiling for you, and it is going to be
getting you the out of memory exception.
So, how can we even handle the situation
of these kind of memory problem? And can
we even run a Qwen 3.8 27 billion
parameter model in this machine? And is
it just the problem with the model
itself, or is it the problem with the
quantization that we have? But what if
we have a quantization of the same exact
model with just two-bit quantization
instead of the four-bit quantization?
Can we run that at all? Well, welcome to
the Qwen model quantized to two-bit
called as the Qwen 3.8 27 billion GSQ
RCO IQ2 QS model. And this model can
load in the 24 GB of unified machine,
and it can support a full GPU
offloading, as well as it has
multi-token prediction enabled as well.
And this model can easily fit in an
Apple M4 machine of 24 GB of unified
memory, which includes the model, KV
cache, runtime, and Mac operating system
without hitting the memory ceiling as
well. So, it can easily load everything
for you within your Apple machine, which
is quite amazing. So, the memory
requirement is going to be so much less
because of the two-bit quantization that
you have got in this particular model.
So, how exactly is this possible? Will
the Qwen 3.8 right now will lose its
intelligence ability or the output will
be of less quality or how exactly is
this happening? What exactly is this new
model of a two-bit quantization? Well,
this particular Qwen model is a standard
Qwen model, but instead of applying the
same quantization everywhere, the
technique will determine which tensors
deserves more precision and which can
tolerate the aggressive compression.
That's exactly what this particular
technique does. And what exactly is this
technique that we're talking about? The
GSQ RCO or otherwise called as the
Gumbel Softmax quantization, which
intelligently optimizes the quantization
value while RCO intelligently decide
where to allocate more or fewer bits
across the model's uh tensors. So,
essentially, even if the model sizes is
going to be reduced from the 16 BPW,
which is a 16-bit original model, to a
two-bit quantization model, it is going
to be off almost 6.4 time of a size
lower, but still the model is going to
give you an amazing precision over here.
That is the power of this particular
model itself. And that's the reason why
even though the size of the model is so
much smaller, the quality of the output
is still going to be way more same like
how you can expect with their
quantization four model of the same Qwen
model that you have got. That's exactly
what this particular model is, and we
are going to see in this particular
video how we can run the same exact Qwen
two-bit model in the Mac mini M4
machine, and we're also going to see how
fast it is going to be doing the
inferencing for us and what it means
with different sizes of the KV cache
adjustment that I'm going to be doing in
this particular video. Now, I will show
you how it is going to be affecting the
the speed as well as the number of
tokens per second as well. So, in this
particular video, we are going to see
which KV size or which KV cache is going
to be faster like 16-bit or 4-bit or
8-bit quantization and which is going to
take more time and what is the total
number of tokens per second that you are
going to be getting
with different kinds of settings that
you are going to be doing in the LM
Studio. And one more time, this
particular model is actually not
available in the MLX format. I
unfortunately have got only the GGML
format and I'm going to be running this
particular model in the LM Studio and I
will show you how the TPS is going to be
for the different KV cache that we are
going to be setting and running it. But,
you know what? This particular model is
not really going to hit the out of
memory exception. Based on the different
KV cache setting, you can actually see
that this model is actually going to be
performant as well. And this is a good
news for this particular model. Well, as
that said, let's start doing the testing
of the model with a different
quantization. So, you can see that I
have got my Mac mini M4 machine. The
total number of physical memory that you
can see over here is 24 gig and I'm
going to be loading the
the Qwen model in the LM Studio and I'm
going to set the context length over
here to at least like a 30 31 or 32,000
over here. And you can see that this is
the maximum concurrent prediction.
Instead of four, I'm going to be setting
it to one cuz it's going to be reducing
the total number of compute as well. And
I'm also going to change the KV cache
quantization
type there. So, instead of using the
default, I'm going to be setting the
FP16 this time. And I'm going to be
loading this particular model. And the
moment I'm going to load this particular
model, you can see that it is going to
be consuming around 20 gig or 21 gig of
the total memory in my
in my RAM there. And now I'm going to be
running the the questions over there,
inferencing it without the thinking
enabled. So the moment I'm going to run,
let's say hi over here, I'm getting
around 4.57 tokens per second. And now
if I'm going to say again like hi again,
it is going to give me around 4.9 tokens
per second. This is a dense model that
we're talking about, guys. Look at that.
And now if I'm going to pass a long
prompt to write a Selenium C# .NET code
for Google search operation with page
object model and dependency injection.
And if I'm going to run it, I'm going to
start the timer as well just in case.
And if I'm going to show you the memory
spike there, you see that the memory
pressure is not increasing because it's
a two-bit quantization model. And at the
same time, you can see that I'm actually
getting the output over here. It is not
as fast as you can imagine. It is not
super fast like how you can expect with
a like a M5 Max machine. But you can see
that within 4 minutes, I got the output
with a 5.67 tokens per second. This is
not bad to be honest. This is working.
It's not really hitting that memory
ceiling. And if you try to run a
four-bit quantization, this is never
going to be running for you. And 4
minutes is not that bad at all. Now I'm
going to change the quantization to
eight, which is a quantization eight
there for the KV cache alone. And I'm
going to be loading this particular
model. And I'm going to ask the same
exact prompt there. Sending a hi
message. And going to say hi again. I'm
getting 5.47 token. And I'm going to
send the same exact Selenium code
prompt. And I'm going to start the timer
over there. You can see that this time
the time is going to be a bit higher
than compared to the FP16 there. The
reason is because actually it's not that
even if you try to reduce the
quantization, the things will be faster.
The inferencing will exactly be the same
exact tokens per second because it also
need to dequantize and unpack the
the quantization for you and then it is
going to be doing some compute
operation. That's the reason why
actually the performance goes slower.
the the total number of time it takes
for the same exact prompt is going to be
higher than compared to the FP16 for you
over there. See now, this is going to be
the FP4 that we have got. I'm going to
be asking the same exact question. The
total number of tokens per second is not
changing at all. And now if I'm going to
be running the same exact prompt, the
Selenium WebDriver question over there
with the
quantization four there, instead of the
4.04
minutes which was taking in FP16, it is
4.46 as you can see here. So, based on
the quantization change, it is also
obviously going to change the time over
there. So, this proves the point that
the inferencing actually works even in
LM Studio for us and you can see that
the output has got no compromise as
well. The output looks pretty much
exactly the same and I can see that it
is still working. I mean there is a
change in the output of course in the
WebDriver for the dependence injection.
It's there over here. But in here, you
can see that the dependence injection
setup has got a bit different with like
driver factory class there. That's the
only difference between both of them.
But overall, I can actually see that it
is working pretty much as expected. The
last test which I wanted to do with the
FP16 of the same exact model is by
connecting to a pie coding agent. I want
to really see if this is going to work
or not. So, I have set the FP16 over
here. I'm running it and I'm going to be
opening the pie coding agent over here
and I'm going to choose the model. As
you can see I'm going to be chasing
choosing that GSQRCO onslaught MTP model
there.
And I'm going to be just saying hi over
there. Even for a high message it was
taking a long time. It's not very fast
as you can imagine.
It was taking a long time and the output
you can see that it is going to be
giving you very very slow here. Which is
quite expected because this is a dense
model. And I now have told it to analyze
the code for me this particular
repository and that took me more than 27
minutes just to analyze the code. I mean
it went so deep to be honest like going
through each and every file even start
to figure out what exactly is the
problem in the particular code. But
actually took 27 minutes to give me the
output which is what really happened
over here as you can see. So long story
short, you can run a two-bit
quantization Qwen 3.8 27 billion
parameter model in an Apple M4 Mac mini
machine without hitting the memory
ceiling. But what is the actual
technique which makes the two-bit
quantization which helps you run in a
Mac mini M4 machine is actually the most
important part. That's the reason why I
actually wanted to show you what's
really happening over here. And other
thing is you may ask what happens if you
try to run the same exact model in the
Apple M5 Max machine, right? So I'm
going to be loading the same exact model
as you can see over here and I'm going
to be doing an FP16 here as well. So
FP16 max current prediction is going to
be one as well. I'm going to loading
this particular model and let's try to
ask the same exact question over here
and see how this is actually going to
work in my Apple M5 Max machine. So this
is a 128 gig machine. So you might be
expecting it to be running it faster and
you'll be expecting it to get at least a
100 tokens per second if you're thinking
that way. But, if I'm going to be
running the same exact model in this
machine, you can see that it is
responding with 27 tokens per second. It
is not like 127 tokens per second. So,
it proves the point that it's not just
about the two-bit quantization model
which can run way more faster in an
Apple M5 Max machine because it's even
running in the M4 mini machine. But, the
the thing is even though it is a two-bit
quantization, the quality of the output
still remains, and that's the reason the
the technique of the GSQ RCU sits in.
So, it it safely chooses the best tensor
which can be optimized to do a better
work for better precision instead of
focusing on the low precision
quantizations that is there
on the tensors there. So, [music] that's
it, guys. This is all I wanted to talk
to you about the Quant 3.8 27 billion
parameter model with two-bit
quantization. Once again, thank you so
much for watching this video. Catch you
in the next one.