Hi guys, welcome back. In this video,we'll be talking about how you can run aQwen in an AppleM4 Mac mini machine with just 24 GB of
unified memory. We know that Qwen 3.8 27billion parameter model will be of atleast 15 GB of size. And loading the
ntire model in the unified memory willnot only load the model, but also loadsthe KV cache, runtime, and the Mac
operating system all together. And whileyou load all of them in the VRAM, it isdefinitely going to hit the memory
ceiling for you, and it is going to begetting you the out of memory exception.So, how can we even handle the situation
of these kind of memory problem? And can
we even run a Qwen 3.8 27 billionparameter model in this machine? And is
it just the problem with the modelitself, or is it the problem with thequantization that we have? But what if
we have a quantization of the same exactmodel with just two-bit quantizationinstead of the four-bit quantization?
Can we run that at all? Well, welcome to
the Qwen model quantized to two-bitcalled as the Qwen 3.8 27 billion GSQRCO IQ2 QS model. And this model canload in the 24 GB of unified machine,
and it can support a full GPUoffloading, as well as it hasmulti-token prediction enabled as well.And this model can easily fit in an
Apple M4 machine of 24 GB of unifiedmemory, which includes the model, KVcache, runtime, and Mac operating system
without hitting the memory ceiling aswell. So, it can easily load everythingfor you within your Apple machine, which
is quite amazing. So, the memoryrequirement is going to be so much lessbecause of the two-bit quantization that
you have got in this particular model.So, how exactly is this possible? Will
the Qwen 3.8 right now will lose itsintelligence ability or the output willbe of less quality or how exactly is
this happening? What exactly is this new
model of a two-bit quantization? Well,
this particular Qwen model is a standardQwen model, but instead of applying thesame quantization everywhere, the
technique will determine which tensorsdeserves more precision and which cantolerate the aggressive compression.
That's exactly what this particulartechnique does. And what exactly is thistechnique that we're talking about? The
GSQ RCO or otherwise called as theGumbel Softmax quantization, whichintelligently optimizes the quantization
value while RCO intelligently decidewhere to allocate more or fewer bitsacross the model's uh tensors. So,
essentially, even if the model sizes isgoing to be reduced from the 16 BPW,which is a 16-bit original model, to a
two-bit quantization model, it is goingto be off almost 6.4 time of a sizelower, but still the model is going to
give you an amazing precision over here.That is the power of this particularmodel itself. And that's the reason why
even though the size of the model is somuch smaller, the quality of the outputis still going to be way more same like
how you can expect with theirquantization four model of the same Qwenmodel that you have got. That's exactly
what this particular model is, and weare going to see in this particularvideo how we can run the same exact Qwen
two-bit model in the Mac mini M4machine, and we're also going to see howfast it is going to be doing theinferencing for us and what it means
with different sizes of the KV cacheadjustment that I'm going to be doing inthis particular video. Now, I will show
you how it is going to be affecting thespeed as well as the number oftokens per second as well. So, in thisparticular video, we are going to see
which KV size or which KV cache is goingto be faster like 16-bit or 4-bit or8-bit quantization and which is going to
take more time and what is the totalnumber of tokens per second that you aregoing to be gettingwith different kinds of settings that
you are going to be doing in the LMStudio. And one more time, thisparticular model is actually notavailable in the MLX format. I
unfortunately have got only the GGMLformat and I'm going to be running thisparticular model in the LM Studio and I
will show you how the TPS is going to befor the different KV cache that we aregoing to be setting and running it. But,
you know what? This particular model is
not really going to hit the out ofmemory exception. Based on the differentKV cache setting, you can actually see
that this model is actually going to beperformant as well. And this is a goodnews for this particular model. Well, as
that said, let's start doing the testingof the model with a differentquantization. So, you can see that Ihave got my Mac mini M4 machine. The
total number of physical memory that youcan see over here is 24 gig and I'mgoing to be loading theQwen model in the LM Studio and I'm
going to set the context length overhere to at least like a 30 31 or 32,000over here. And you can see that this is
the maximum concurrent prediction.Instead of four, I'm going to be settingit to one cuz it's going to be reducing
the total number of compute as well. AndI'm also going to change the KV cachequantizationtype there. So, instead of using the
default, I'm going to be setting theFP16 this time. And I'm going to beloading this particular model. And the
moment I'm going to load this particularmodel, you can see that it is going tobe consuming around 20 gig or 21 gig of
the total memory in myRAM there. And now I'm going to berunning the the questions over there,inferencing it without the thinking
enabled. So the moment I'm going to run,let's say hi over here, I'm gettingaround 4.57 tokens per second. And now
if I'm going to say again like hi again,it is going to give me around 4.9 tokensper second. This is a dense model that
we're talking about, guys. Look at that.And now if I'm going to pass a longprompt to write a Selenium C# .NET code
for Google search operation with pageobject model and dependency injection.And if I'm going to run it, I'm going to
start the timer as well just in case.And if I'm going to show you the memoryspike there, you see that the memory
pressure is not increasing because it'sa two-bit quantization model. And at thesame time, you can see that I'm actually
getting the output over here. It is notas fast as you can imagine. It is notsuper fast like how you can expect with
a like a M5 Max machine. But you can seethat within 4 minutes, I got the outputwith a 5.67 tokens per second. This is
not bad to be honest. This is working.It's not really hitting that memoryceiling. And if you try to run afour-bit quantization, this is never
going to be running for you. And 4minutes is not that bad at all. Now I'mgoing to change the quantization toeight, which is a quantization eight
here for the KV cache alone. And I'mgoing to be loading this particularmodel. And I'm going to ask the samexact prompt there. Sending a hi
message. And going to say hi again. I'mgetting 5.47 token. And I'm going tosend the same exact Selenium code
prompt. And I'm going to start the timerover there. You can see that this timethe time is going to be a bit higher
than compared to the FP16 there. Thereason is because actually it's not thateven if you try to reduce thequantization, the things will be faster.
The inferencing will exactly be the samexact tokens per second because it alsoneed to dequantize and unpack the
quantization for you and then it isgoing to be doing some computeoperation. That's the reason whyactually the performance goes slower.
the the total number of time it takesfor the same exact prompt is going to behigher than compared to the FP16 for you
over there. See now, this is going to bethe FP4 that we have got. I'm going tobe asking the same exact question. The
total number of tokens per second is notchanging at all. And now if I'm going tobe running the same exact prompt, the
Selenium WebDriver question over therewith thequantization four there, instead of the4.04minutes which was taking in FP16, it is
4.46 as you can see here. So, based onthe quantization change, it is alsobviously going to change the time over
there. So, this proves the point thathe inferencing actually works even inLM Studio for us and you can see that
he output has got no compromise aswell. The output looks pretty muchexactly the same and I can see that itis still working. I mean there is a
change in the output of course in theWebDriver for the dependence injection.It's there over here. But in here, you
can see that the dependence injectionsetup has got a bit different with likedriver factory class there. That's the
only difference between both of them.But overall, I can actually see that itis working pretty much as expected. The
last test which I wanted to do with theFP16 of the same exact model is byconnecting to a pie coding agent. I want
o really see if this is going to workor not. So, I have set the FP16 overhere. I'm running it and I'm going to beopening the pie coding agent over here
and I'm going to choose the model. Asyou can see I'm going to be chasingchoosing that GSQRCO onslaught MTP model
there.And I'm going to be just saying hi overthere. Even for a high message it wastaking a long time. It's not very fast
as you can imagine.It was taking a long time and the outputyou can see that it is going to begiving you very very slow here. Which is
quite expected because this is a densemodel. And I now have told it to analyzethe code for me this particular
epository and that took me more than 27minutes just to analyze the code. I meanit went so deep to be honest like going
through each and every file even starto figure out what exactly is theproblem in the particular code. Butactually took 27 minutes to give me the
output which is what really happenedover here as you can see. So long storyshort, you can run a two-bitquantization Qwen 3.8 27 billion
parameter model in an Apple M4 Mac minimachine without hitting the memoryceiling. But what is the actualtechnique which makes the two-bit
quantization which helps you run in aMac mini M4 machine is actually the mostimportant part. That's the reason why I
actually wanted to show you what'sreally happening over here. And otherthing is you may ask what happens if you
try to run the same exact model in theApple M5 Max machine, right? So I'm
going to be loading the same exact modelas you can see over here and I'm goingto be doing an FP16 here as well. So
FP16 max current prediction is going tobe one as well. I'm going to loadingthis particular model and let's try to
ask the same exact question over hereand see how this is actually going towork in my Apple M5 Max machine. So this
a 128 gig machine. So you might bexpecting it to be running it faster andyou'll be expecting it to get at least a
100 tokens per second if you're thinkingthat way. But, if I'm going to berunning the same exact model in this
machine, you can see that it isresponding with 27 tokens per second. Itis not like 127 tokens per second. So,
it proves the point that it's not justabout the two-bit quantization modelwhich can run way more faster in an
Apple M5 Max machine because it's evenrunning in the M4 mini machine. But, thething is even though it is a two-bit
quantization, the quality of the outputstill remains, and that's the reason thetechnique of the GSQ RCU sits in.
So, it it safely chooses the best tensorwhich can be optimized to do a betterwork for better precision instead of
ocusing on the low precisionquantizations that is thereon the tensors there. So, [music] that'sit, guys. This is all I wanted to talk
to you about the Quant 3.8 27 billionparameter model with two-bitquantization. Once again, thank you somuch for watching this video. Catch you
in the next one.