{ "text": "Hi guys, welcome back.\nIn this video, we'll be talking about how you can run a QN 3.8 27 billion parameter model in an Apple M4 Mac Mini machine with just 24 GB of unified memory.\nWe know that QN 3.8 27 billion parameter model will be of at least 15 GB of size.\nand loading the entire model in the unified memory will not only load the model but also\nloads the kvcache runtime and the mac operating system all together and while you load all of\nthem in the vram it is definitely going to hit the memory ceiling for you and it is going to be\ngetting you the out of memory exception so how can we even handle the situation of these kind of\nmemory problem and can we even run a QUEN 3.8 27 billion parameter model in this machine and is it\njust the problem with the model itself or is it the problem with the quantization that we have\nbut what if we have a quantization of the same exact model with just two-bit quantization instead\nof the four-bit quantization can we run that at all well welcome to the QUEN model quantized to\n2-bit called as the QEN3.827BGSQRCOIQ2QS model and this model can load in the 24GB of unified\nmachine and it can support a full GPU offloading as well as it has multi-token prediction enabled\nas well and this model can easily fit in an Apple M4 machine of 24GB of unified memory\nwhich includes the model kv cache runtime and mac operating system without hitting the memory\nceiling as well so it can easily load everything for you within your apple machine which is quite\namazing so the memory requirement is going to be so much less because of the two-bit quantization\nthat you have got in this particular model so how exactly is this possible will the quen 3.8 right\nnow will lose its intelligence ability or the output will be of less quality or how exactly\nis this happening what exactly is this new model of a two-bit quantization well this particular\nquen model is a standard quen model but instead of applying the same quantization everywhere the\ntechnique will determine which tensor deserves more precision and which can tolerate the aggressive\ncompression that's exactly what this particular technique does and what exactly is this technique\nthat we're talking about the gsq rco or otherwise called as the gumbel softmax quantization\nwhich intelligently optimizes the quantization value while RCO intelligently decide where to\nallocate more or fewer bits across the models or tens are so essentially even if the model sizes\nis going to be reduced from the 16 bpw which is a 16-bit original model to a 2-bit quantization\nmodel it is going to be of almost 6.4 times of a size lower but still the model is going to give\nyou an amazing precision over here that is the power of this particular model itself and that's\nthe reason why even though the size of the model is so much smaller the quality of the output is\nstill going to be way more same like how you can expect with a quantization 4 model of the same\nQN model that you have got that's exactly what this particular model is and we are going to see\nin this particular video how we can run the same exact QN 2-bit model in the Mac Mini M4 machine\nand we're also going to see how fast it is going to be doing the inferencing for us and what it\nmeans with different sizes of the KV cache adjustment that I'm going to be doing in this\nparticular video and i will show you how it is going to be affecting the the speed as well as\nthe number of tokens per second as well so in this particular video we are going to see which kv size\nor which kv cache is going to be faster like 16 bit or 4 bit or 8 bit quantization and which is\ngoing to take more time and what is the total number of tokens per second that you are going\nto be getting uh with different kinds of settings that you are going to be doing in the lm studio\nand one more time this particular model is actually not available in the mlx format i\nunfortunately have got only the ggf format and i am going to be running this particular model\nin the lm studio and i will show you how the tps is going to be for the different kv cache that we\nare going to be setting and running it but you know what this particular model is not really\ngoing to hit the out of memory exception based on the different kv cache setting you can actually\nsee that this model is actually going to be performant as well and this is the good news for\nthis particular model well as that said let's start doing the testing of the model with different\nquantization so you can see that i have got my mac mini m4 machine total number of physical\nthat you can see over here is 24 gig and i'm going to be loading the uh the quen model uh in the lm\nstudio and i'm going to set the context length over here uh to at least like a 30 31 or 32 000\nuh over here and you can see that this is the maximum concurrent prediction or instead of four\ni'm going to be setting it to one because it's going to be reducing the total lump compute as\nwell and i'm also going to change the kv cache quantization type there so instead of using the\ndefault i'm going to be setting the fp16 this time and i'm going to be loading this particular model\nand the moment i'm going to load this particular model you can see that it is going to be consuming\naround 20 gig or 21 gig of the total memory uh in my uh in my ram there and now i'm going to be\nrunning the the questions over there inferencing it without the thinking enabled so the moment i'm\ngoing to run let's say hi over here i'm getting around 4.57 tokens per second and now if i'm\ngoing to say again like hi again it is going to give me around 4.9 tokens per second this is a\ndense model that we're talking about guys look at that and now if i'm going to pass a long prompt\nto write a selenium c sharp dot net code for google search operation with page object model\nand dependency injection and if i'm going to run it i'm going to start the timer as well just in\ncase and if i'm going to show you the memory spike there you see that the memory pressure is not\nincreasing because it's a two-bit quantization model and at the same time you can see that i'm\nactually getting the output over here it is not as fast as you can imagine it is not super fast\nlike how we can expect with a like an m5 max machine but you can see that within four minutes\ni got the output with a 5.67 tokens per second this is not bad to be honest this is working\nit's not really hitting that memory ceiling and if you try to run a four-bit quantization this is\nnever going to be running for you and four minutes is not that bad at all now i'm going to change the\nquantization to uh eight just the quantization eight there for the kv cache alone and i'm going\nto be loading this particular model and i'm going to ask the same exact prompt there sending a hi\nmessage and i'm going to say hi again i'm getting 5.47 token and i'm going to send the same exact\nSedium code prompt and I'm going to start the timer over there you can see that this time the\ntime is going to be a bit higher than compared to the FP16 there the reason is because actually\nit's not that even if you try to reduce the quantization the things will be faster the\ninferencing will exactly be the same exact tokens per second because it also need to de-quantize\n安安\n然後它會不會有更快。\n所以,它會不會有更快。\n所以,它會不會有更快。\nthan compared to the FP16 for you over there.\nSee now, this is going to be the FP4 that we have got.\nI'm going to be asking the same exact question.\nThe total number of tokens per second is not changing at all.\nAnd now if I'm going to be running the same exact prompt,\nthe Serenium WebDriver question over there with the quantization 4 there,\ninstead of the 4.04 minutes, which was taking in FP16,\nit is 4.46 as you can see here.\nso based on the quantization change it is also obviously going to change the time over there\nso this proves the point that the inferencing actually works even in LM studio for us and you\ncan see that the output has got no compromise as well the output looks pretty much exactly the\nsame and I can see that it is still working I mean there is a change in the output of course\nin the web driver for the dependency injection it's there over here but in here you can see that\nthe dependency injection setup has got a bit different with like driver factory class there\nthat's the only difference between both of them but overall i can actually see that it is working\npretty much as expected the last test which i wanted to do with the fp16 of the same exact\nmodel is by connecting to a pi coding agent i want to really see if this is going to work or not so\ni have set the fp16 over here i'm running it and i'm going to be opening the pi coding agent over\nhere and i'm going to choose the model as you can see i'm going to be chasing choosing that gsq rco\nunsloth mtp model there and i'm going to be just saying hi over there even for a high message it\nwas taking a long time it's not very fast as you can imagine it was taking a long time and the\noutput you can see that it is going to be giving you very very slow here which is quite expected\nbecause this is a dense model and now i have told it to analyze the code for me in this particular\nrepository and that took me more than 27 minutes just to analyze the code i mean it went so deep\nto be honest like going through each and every file even start to figure out what exactly is\nthe problem in the particular code but actually took 27 minutes to give me the output which is\nwhat really happened over here as you can see so long story short you can run a two-bit quantization\nQUEN 3.8 27 billion parameter model in an apple m4 mac mini machine without hitting the memory\nceiling but what is the actual technique which makes the two big quantization which helps you\nrun in a mac mini m4 machine is actually the most important part that's the reason why i actually\nwanted to show you what's really happening over here and other thing is you may ask what happens\nif you try to run the same exact model in the Apple M5 Max machine right so I'm going to be\nloading the same exact model as you can see over here and I'm going to be doing an FP16 here as\nwell so FP16 Max current prediction is going to be one as well I'm going to be loading this particular\nmodel and let's try to ask the same exact question over here and see how this is actually going to\n在我的 Apple M5 Max machine\n所以这个是128GB machine\n所以你可能会被担心\n要是转买它快速\n你会被担心\n要是100兆兆的\n如果我会被担心\n但如果我会被担心\n同样的同样\n在这个机器\n你能看到它是\n连接到27兆兆的\n它是127兆兆的\n所以它试试了\npoint that it's not just about the two-bit quantization model which can run way more\nfaster in an apple m5 max machine because it's even running in the m4 mini machine but the thing\nis even though it is a two-bit quantization the quality of the output still remains and that's\nthe reason the the technique of the gsq rco sits in so it safely chooses the best tensor which can", "language": "zh", "duration": 763.681125, "segments": [ { "start": 0.0, "end": 1.18, "text": "Hi guys, welcome back.", "chunk": 0, "language": "zh", "avg_logprob": -0.21508814493815104, "no_speech_prob": 6.385174056156018e-10, "compression_ratio": 1.434065934065934 }, { "start": 1.54, "end": 13.26, "text": "In this video, we'll be talking about how you can run a QN 3.8 27 billion parameter model in an Apple M4 Mac Mini machine with just 24 GB of unified memory.", "chunk": 0, "language": "zh", "avg_logprob": -0.21508814493815104, "no_speech_prob": 6.385174056156018e-10, "compression_ratio": 1.434065934065934 }, { "start": 13.26, "end": 19.86, "text": "We know that QN 3.8 27 billion parameter model will be of at least 15 GB of size.", "chunk": 0, "language": "zh", "avg_logprob": -0.21508814493815104, "no_speech_prob": 6.385174056156018e-10, "compression_ratio": 1.434065934065934 }, { "start": 19.86, "end": 27.66, "text": "and loading the entire model in the unified memory will not only load the model but also", "chunk": 0, "language": "zh", "avg_logprob": -0.06337051499973644, "no_speech_prob": 1.2638544724663525e-12, "compression_ratio": 1.8028846153846154 }, { "start": 27.66, "end": 35.04, "text": "loads the kvcache runtime and the mac operating system all together and while you load all of", "chunk": 0, "language": "zh", "avg_logprob": -0.06337051499973644, "no_speech_prob": 1.2638544724663525e-12, "compression_ratio": 1.8028846153846154 }, { "start": 35.04, "end": 41.78, "text": "them in the vram it is definitely going to hit the memory ceiling for you and it is going to be", "chunk": 0, "language": "zh", "avg_logprob": -0.06337051499973644, "no_speech_prob": 1.2638544724663525e-12, "compression_ratio": 1.8028846153846154 }, { "start": 41.78, "end": 48.28, "text": "getting you the out of memory exception so how can we even handle the situation of these kind of", "chunk": 0, "language": "zh", "avg_logprob": -0.06337051499973644, "no_speech_prob": 1.2638544724663525e-12, "compression_ratio": 1.8028846153846154 }, { "start": 48.28, "end": 55.16, "text": "memory problem and can we even run a QUEN 3.8 27 billion parameter model in this machine and is it", "chunk": 0, "language": "zh", "avg_logprob": -0.08254692951838176, "no_speech_prob": 1.008106900908523e-12, "compression_ratio": 1.8695652173913044 }, { "start": 55.16, "end": 61.7, "text": "just the problem with the model itself or is it the problem with the quantization that we have", "chunk": 0, "language": "zh", "avg_logprob": -0.08254692951838176, "no_speech_prob": 1.008106900908523e-12, "compression_ratio": 1.8695652173913044 }, { "start": 61.7, "end": 69.18, "text": "but what if we have a quantization of the same exact model with just two-bit quantization instead", "chunk": 0, "language": "zh", "avg_logprob": -0.08254692951838176, "no_speech_prob": 1.008106900908523e-12, "compression_ratio": 1.8695652173913044 }, { "start": 69.18, "end": 76.58, "text": "of the four-bit quantization can we run that at all well welcome to the QUEN model quantized to", "chunk": 0, "language": "zh", "avg_logprob": -0.08254692951838176, "no_speech_prob": 1.008106900908523e-12, "compression_ratio": 1.8695652173913044 }, { "start": 76.58, "end": 89.66, "text": "2-bit called as the QEN3.827BGSQRCOIQ2QS model and this model can load in the 24GB of unified", "chunk": 0, "language": "zh", "avg_logprob": -0.12514291490827287, "no_speech_prob": 1.1281779547026027e-12, "compression_ratio": 1.4787234042553192 }, { "start": 89.66, "end": 97.22, "text": "machine and it can support a full GPU offloading as well as it has multi-token prediction enabled", "chunk": 0, "language": "zh", "avg_logprob": -0.12514291490827287, "no_speech_prob": 1.1281779547026027e-12, "compression_ratio": 1.4787234042553192 }, { "start": 97.22, "end": 105.74, "text": "as well and this model can easily fit in an Apple M4 machine of 24GB of unified memory", "chunk": 0, "language": "zh", "avg_logprob": -0.12514291490827287, "no_speech_prob": 1.1281779547026027e-12, "compression_ratio": 1.4787234042553192 }, { "start": 105.74, "end": 112.04, "text": "which includes the model kv cache runtime and mac operating system without hitting the memory", "chunk": 0, "language": "zh", "avg_logprob": -0.05496616817656017, "no_speech_prob": 1.1511943742020803e-12, "compression_ratio": 1.7204301075268817 }, { "start": 112.04, "end": 118.1, "text": "ceiling as well so it can easily load everything for you within your apple machine which is quite", "chunk": 0, "language": "zh", "avg_logprob": -0.05496616817656017, "no_speech_prob": 1.1511943742020803e-12, "compression_ratio": 1.7204301075268817 }, { "start": 118.1, "end": 122.8, "text": "amazing so the memory requirement is going to be so much less because of the two-bit quantization", "chunk": 0, "language": "zh", "avg_logprob": -0.05496616817656017, "no_speech_prob": 1.1511943742020803e-12, "compression_ratio": 1.7204301075268817 }, { "start": 122.8, "end": 129.08, "text": "that you have got in this particular model so how exactly is this possible will the quen 3.8 right", "chunk": 0, "language": "zh", "avg_logprob": -0.05496616817656017, "no_speech_prob": 1.1511943742020803e-12, "compression_ratio": 1.7204301075268817 }, { "start": 129.08, "end": 135.6, "text": "now will lose its intelligence ability or the output will be of less quality or how exactly", "chunk": 0, "language": "zh", "avg_logprob": -0.05496616817656017, "no_speech_prob": 1.1511943742020803e-12, "compression_ratio": 1.7204301075268817 }, { "start": 135.6, "end": 141.12, "text": "is this happening what exactly is this new model of a two-bit quantization well this particular", "chunk": 0, "language": "zh", "avg_logprob": -0.09496258754356235, "no_speech_prob": 1.4105707704312187e-12, "compression_ratio": 1.9203187250996017 }, { "start": 141.12, "end": 147.76, "text": "quen model is a standard quen model but instead of applying the same quantization everywhere the", "chunk": 0, "language": "zh", "avg_logprob": -0.09496258754356235, "no_speech_prob": 1.4105707704312187e-12, "compression_ratio": 1.9203187250996017 }, { "start": 147.76, "end": 153.6, "text": "technique will determine which tensor deserves more precision and which can tolerate the aggressive", "chunk": 0, "language": "zh", "avg_logprob": -0.09496258754356235, "no_speech_prob": 1.4105707704312187e-12, "compression_ratio": 1.9203187250996017 }, { "start": 153.6, "end": 158.54, "text": "compression that's exactly what this particular technique does and what exactly is this technique", "chunk": 0, "language": "zh", "avg_logprob": -0.09496258754356235, "no_speech_prob": 1.4105707704312187e-12, "compression_ratio": 1.9203187250996017 }, { "start": 158.54, "end": 165.36, "text": "that we're talking about the gsq rco or otherwise called as the gumbel softmax quantization", "chunk": 0, "language": "zh", "avg_logprob": -0.09496258754356235, "no_speech_prob": 1.4105707704312187e-12, "compression_ratio": 1.9203187250996017 }, { "start": 165.36, "end": 172.44, "text": "which intelligently optimizes the quantization value while RCO intelligently decide where to", "chunk": 0, "language": "zh", "avg_logprob": -0.11342027912969174, "no_speech_prob": 1.0986092677939263e-12, "compression_ratio": 1.565934065934066 }, { "start": 172.44, "end": 180.78, "text": "allocate more or fewer bits across the models or tens are so essentially even if the model sizes", "chunk": 0, "language": "zh", "avg_logprob": -0.11342027912969174, "no_speech_prob": 1.0986092677939263e-12, "compression_ratio": 1.565934065934066 }, { "start": 180.78, "end": 189.24, "text": "is going to be reduced from the 16 bpw which is a 16-bit original model to a 2-bit quantization", "chunk": 0, "language": "zh", "avg_logprob": -0.11342027912969174, "no_speech_prob": 1.0986092677939263e-12, "compression_ratio": 1.565934065934066 }, { "start": 189.24, "end": 196.48, "text": "model it is going to be of almost 6.4 times of a size lower but still the model is going to give", "chunk": 0, "language": "zh", "avg_logprob": -0.04080639090589298, "no_speech_prob": 1.4549780651129662e-12, "compression_ratio": 1.75 }, { "start": 196.48, "end": 202.68, "text": "you an amazing precision over here that is the power of this particular model itself and that's", "chunk": 0, "language": "zh", "avg_logprob": -0.04080639090589298, "no_speech_prob": 1.4549780651129662e-12, "compression_ratio": 1.75 }, { "start": 202.68, "end": 208.4, "text": "the reason why even though the size of the model is so much smaller the quality of the output is", "chunk": 0, "language": "zh", "avg_logprob": -0.04080639090589298, "no_speech_prob": 1.4549780651129662e-12, "compression_ratio": 1.75 }, { "start": 208.4, "end": 215.14, "text": "still going to be way more same like how you can expect with a quantization 4 model of the same", "chunk": 0, "language": "zh", "avg_logprob": -0.04080639090589298, "no_speech_prob": 1.4549780651129662e-12, "compression_ratio": 1.75 }, { "start": 215.14, "end": 219.82, "text": "QN model that you have got that's exactly what this particular model is and we are going to see", "chunk": 0, "language": "zh", "avg_logprob": -0.07670187704341928, "no_speech_prob": 1.996763214842656e-12, "compression_ratio": 1.7534883720930232 }, { "start": 219.82, "end": 228.94, "text": "in this particular video how we can run the same exact QN 2-bit model in the Mac Mini M4 machine", "chunk": 0, "language": "zh", "avg_logprob": -0.07670187704341928, "no_speech_prob": 1.996763214842656e-12, "compression_ratio": 1.7534883720930232 }, { "start": 228.94, "end": 235.86, "text": "and we're also going to see how fast it is going to be doing the inferencing for us and what it", "chunk": 0, "language": "zh", "avg_logprob": -0.07670187704341928, "no_speech_prob": 1.996763214842656e-12, "compression_ratio": 1.7534883720930232 }, { "start": 235.86, "end": 241.32, "text": "means with different sizes of the KV cache adjustment that I'm going to be doing in this", "chunk": 0, "language": "zh", "avg_logprob": -0.07670187704341928, "no_speech_prob": 1.996763214842656e-12, "compression_ratio": 1.7534883720930232 }, { "start": 241.32, "end": 247.7, "text": "particular video and i will show you how it is going to be affecting the the speed as well as", "chunk": 0, "language": "zh", "avg_logprob": -0.06532335685471356, "no_speech_prob": 1.69481540347044e-12, "compression_ratio": 2.051063829787234 }, { "start": 247.7, "end": 253.24, "text": "the number of tokens per second as well so in this particular video we are going to see which kv size", "chunk": 0, "language": "zh", "avg_logprob": -0.06532335685471356, "no_speech_prob": 1.69481540347044e-12, "compression_ratio": 2.051063829787234 }, { "start": 253.24, "end": 260.16, "text": "or which kv cache is going to be faster like 16 bit or 4 bit or 8 bit quantization and which is", "chunk": 0, "language": "zh", "avg_logprob": -0.06532335685471356, "no_speech_prob": 1.69481540347044e-12, "compression_ratio": 2.051063829787234 }, { "start": 260.16, "end": 265.5, "text": "going to take more time and what is the total number of tokens per second that you are going", "chunk": 0, "language": "zh", "avg_logprob": -0.06532335685471356, "no_speech_prob": 1.69481540347044e-12, "compression_ratio": 2.051063829787234 }, { "start": 265.5, "end": 270.54, "text": "to be getting uh with different kinds of settings that you are going to be doing in the lm studio", "chunk": 0, "language": "zh", "avg_logprob": -0.06532335685471356, "no_speech_prob": 1.69481540347044e-12, "compression_ratio": 2.051063829787234 }, { "start": 270.54, "end": 275.78, "text": "and one more time this particular model is actually not available in the mlx format i", "chunk": 0, "language": "zh", "avg_logprob": -0.0481704720505723, "no_speech_prob": 2.2188622467167463e-12, "compression_ratio": 2.0480349344978164 }, { "start": 275.78, "end": 280.46, "text": "unfortunately have got only the ggf format and i am going to be running this particular model", "chunk": 0, "language": "zh", "avg_logprob": -0.0481704720505723, "no_speech_prob": 2.2188622467167463e-12, "compression_ratio": 2.0480349344978164 }, { "start": 280.46, "end": 286.52, "text": "in the lm studio and i will show you how the tps is going to be for the different kv cache that we", "chunk": 0, "language": "zh", "avg_logprob": -0.0481704720505723, "no_speech_prob": 2.2188622467167463e-12, "compression_ratio": 2.0480349344978164 }, { "start": 286.52, "end": 291.78, "text": "are going to be setting and running it but you know what this particular model is not really", "chunk": 0, "language": "zh", "avg_logprob": -0.0481704720505723, "no_speech_prob": 2.2188622467167463e-12, "compression_ratio": 2.0480349344978164 }, { "start": 291.78, "end": 297.58, "text": "going to hit the out of memory exception based on the different kv cache setting you can actually", "chunk": 0, "language": "zh", "avg_logprob": -0.0481704720505723, "no_speech_prob": 2.2188622467167463e-12, "compression_ratio": 2.0480349344978164 }, { "start": 297.58, "end": 302.06, "text": "see that this model is actually going to be performant as well and this is the good news for", "chunk": 0, "language": "zh", "avg_logprob": -0.08759559832121197, "no_speech_prob": 1.8833242257376703e-12, "compression_ratio": 1.76036866359447 }, { "start": 302.06, "end": 306.28, "text": "this particular model well as that said let's start doing the testing of the model with different", "chunk": 0, "language": "zh", "avg_logprob": -0.08759559832121197, "no_speech_prob": 1.8833242257376703e-12, "compression_ratio": 1.76036866359447 }, { "start": 306.28, "end": 312.5, "text": "quantization so you can see that i have got my mac mini m4 machine total number of physical", "chunk": 0, "language": "zh", "avg_logprob": -0.08759559832121197, "no_speech_prob": 1.8833242257376703e-12, "compression_ratio": 1.76036866359447 }, { "start": 312.5, "end": 319.5, "text": "that you can see over here is 24 gig and i'm going to be loading the uh the quen model uh in the lm", "chunk": 0, "language": "zh", "avg_logprob": -0.08759559832121197, "no_speech_prob": 1.8833242257376703e-12, "compression_ratio": 1.76036866359447 }, { "start": 319.5, "end": 328.3, "text": "studio and i'm going to set the context length over here uh to at least like a 30 31 or 32 000", "chunk": 0, "language": "zh", "avg_logprob": -0.07327386384369225, "no_speech_prob": 1.2793465288887806e-12, "compression_ratio": 1.7339449541284404 }, { "start": 328.3, "end": 333.42, "text": "uh over here and you can see that this is the maximum concurrent prediction or instead of four", "chunk": 0, "language": "zh", "avg_logprob": -0.07327386384369225, "no_speech_prob": 1.2793465288887806e-12, "compression_ratio": 1.7339449541284404 }, { "start": 333.42, "end": 337.18, "text": "i'm going to be setting it to one because it's going to be reducing the total lump compute as", "chunk": 0, "language": "zh", "avg_logprob": -0.07327386384369225, "no_speech_prob": 1.2793465288887806e-12, "compression_ratio": 1.7339449541284404 }, { "start": 337.18, "end": 345.1, "text": "well and i'm also going to change the kv cache quantization type there so instead of using the", "chunk": 0, "language": "zh", "avg_logprob": -0.07327386384369225, "no_speech_prob": 1.2793465288887806e-12, "compression_ratio": 1.7339449541284404 }, { "start": 345.1, "end": 350.22, "text": "default i'm going to be setting the fp16 this time and i'm going to be loading this particular model", "chunk": 0, "language": "zh", "avg_logprob": -0.05094400386220401, "no_speech_prob": 2.219070196693429e-12, "compression_ratio": 1.975 }, { "start": 350.22, "end": 354.62, "text": "and the moment i'm going to load this particular model you can see that it is going to be consuming", "chunk": 0, "language": "zh", "avg_logprob": -0.05094400386220401, "no_speech_prob": 2.219070196693429e-12, "compression_ratio": 1.975 }, { "start": 354.62, "end": 364.06, "text": "around 20 gig or 21 gig of the total memory uh in my uh in my ram there and now i'm going to be", "chunk": 0, "language": "zh", "avg_logprob": -0.05094400386220401, "no_speech_prob": 2.219070196693429e-12, "compression_ratio": 1.975 }, { "start": 364.06, "end": 370.06, "text": "running the the questions over there inferencing it without the thinking enabled so the moment i'm", "chunk": 0, "language": "zh", "avg_logprob": -0.05094400386220401, "no_speech_prob": 2.219070196693429e-12, "compression_ratio": 1.975 }, { "start": 370.06, "end": 376.22, "text": "going to run let's say hi over here i'm getting around 4.57 tokens per second and now if i'm", "chunk": 0, "language": "zh", "avg_logprob": -0.06334050756986023, "no_speech_prob": 1.5551362281263081e-12, "compression_ratio": 1.8515625 }, { "start": 376.22, "end": 382.7, "text": "going to say again like hi again it is going to give me around 4.9 tokens per second this is a", "chunk": 0, "language": "zh", "avg_logprob": -0.06334050756986023, "no_speech_prob": 1.5551362281263081e-12, "compression_ratio": 1.8515625 }, { "start": 382.7, "end": 387.5, "text": "dense model that we're talking about guys look at that and now if i'm going to pass a long prompt", "chunk": 0, "language": "zh", "avg_logprob": -0.06334050756986023, "no_speech_prob": 1.5551362281263081e-12, "compression_ratio": 1.8515625 }, { "start": 387.5, "end": 393.66, "text": "to write a selenium c sharp dot net code for google search operation with page object model", "chunk": 0, "language": "zh", "avg_logprob": -0.06334050756986023, "no_speech_prob": 1.5551362281263081e-12, "compression_ratio": 1.8515625 }, { "start": 393.66, "end": 398.46, "text": "and dependency injection and if i'm going to run it i'm going to start the timer as well just in", "chunk": 0, "language": "zh", "avg_logprob": -0.06334050756986023, "no_speech_prob": 1.5551362281263081e-12, "compression_ratio": 1.8515625 }, { "start": 398.46, "end": 402.94, "text": "case and if i'm going to show you the memory spike there you see that the memory pressure is not", "chunk": 0, "language": "zh", "avg_logprob": -0.05819187325946355, "no_speech_prob": 2.0846425714121652e-12, "compression_ratio": 1.8262548262548262 }, { "start": 402.94, "end": 408.54, "text": "increasing because it's a two-bit quantization model and at the same time you can see that i'm", "chunk": 0, "language": "zh", "avg_logprob": -0.05819187325946355, "no_speech_prob": 2.0846425714121652e-12, "compression_ratio": 1.8262548262548262 }, { "start": 408.54, "end": 413.9, "text": "actually getting the output over here it is not as fast as you can imagine it is not super fast", "chunk": 0, "language": "zh", "avg_logprob": -0.05819187325946355, "no_speech_prob": 2.0846425714121652e-12, "compression_ratio": 1.8262548262548262 }, { "start": 413.9, "end": 418.94, "text": "like how we can expect with a like an m5 max machine but you can see that within four minutes", "chunk": 0, "language": "zh", "avg_logprob": -0.05819187325946355, "no_speech_prob": 2.0846425714121652e-12, "compression_ratio": 1.8262548262548262 }, { "start": 418.94, "end": 425.98, "text": "i got the output with a 5.67 tokens per second this is not bad to be honest this is working", "chunk": 0, "language": "zh", "avg_logprob": -0.05819187325946355, "no_speech_prob": 2.0846425714121652e-12, "compression_ratio": 1.8262548262548262 }, { "start": 425.98, "end": 430.54, "text": "it's not really hitting that memory ceiling and if you try to run a four-bit quantization this is", "chunk": 0, "language": "zh", "avg_logprob": -0.07153726593265688, "no_speech_prob": 2.0284971619100034e-12, "compression_ratio": 1.9798387096774193 }, { "start": 430.54, "end": 435.58, "text": "never going to be running for you and four minutes is not that bad at all now i'm going to change the", "chunk": 0, "language": "zh", "avg_logprob": -0.07153726593265688, "no_speech_prob": 2.0284971619100034e-12, "compression_ratio": 1.9798387096774193 }, { "start": 435.58, "end": 441.42, "text": "quantization to uh eight just the quantization eight there for the kv cache alone and i'm going", "chunk": 0, "language": "zh", "avg_logprob": -0.07153726593265688, "no_speech_prob": 2.0284971619100034e-12, "compression_ratio": 1.9798387096774193 }, { "start": 441.42, "end": 448.06, "text": "to be loading this particular model and i'm going to ask the same exact prompt there sending a hi", "chunk": 0, "language": "zh", "avg_logprob": -0.07153726593265688, "no_speech_prob": 2.0284971619100034e-12, "compression_ratio": 1.9798387096774193 }, { "start": 448.06, "end": 454.38, "text": "message and i'm going to say hi again i'm getting 5.47 token and i'm going to send the same exact", "chunk": 0, "language": "zh", "avg_logprob": -0.07153726593265688, "no_speech_prob": 2.0284971619100034e-12, "compression_ratio": 1.9798387096774193 }, { "start": 454.38, "end": 459.86, "text": "Sedium code prompt and I'm going to start the timer over there you can see that this time the", "chunk": 0, "language": "zh", "avg_logprob": -0.13657213293987772, "no_speech_prob": 1.3724017582095183e-12, "compression_ratio": 1.6431718061674008 }, { "start": 459.86, "end": 466.36, "text": "time is going to be a bit higher than compared to the FP16 there the reason is because actually", "chunk": 0, "language": "zh", "avg_logprob": -0.13657213293987772, "no_speech_prob": 1.3724017582095183e-12, "compression_ratio": 1.6431718061674008 }, { "start": 466.36, "end": 472.68, "text": "it's not that even if you try to reduce the quantization the things will be faster the", "chunk": 0, "language": "zh", "avg_logprob": -0.13657213293987772, "no_speech_prob": 1.3724017582095183e-12, "compression_ratio": 1.6431718061674008 }, { "start": 472.68, "end": 479.28, "text": "inferencing will exactly be the same exact tokens per second because it also need to de-quantize", "chunk": 0, "language": "zh", "avg_logprob": -0.13657213293987772, "no_speech_prob": 1.3724017582095183e-12, "compression_ratio": 1.6431718061674008 }, { "start": 479.28, "end": 479.84, "text": "安安", "chunk": 0, "language": "zh", "avg_logprob": -1.1274993896484375, "no_speech_prob": 8.530829515018823e-11, "compression_ratio": 0.42857142857142855 }, { "start": 482.4, "end": 485.9, "text": "然後它會不會有更快。", "chunk": 1, "language": "zh", "avg_logprob": -1.0138986375596788, "no_speech_prob": 5.587333928858129e-10, "compression_ratio": 2.2857142857142856 }, { "start": 485.9, "end": 489.7, "text": "所以,它會不會有更快。", "chunk": 1, "language": "zh", "avg_logprob": -1.0138986375596788, "no_speech_prob": 5.587333928858129e-10, "compression_ratio": 2.2857142857142856 }, { "start": 489.7, "end": 494.9, "text": "所以,它會不會有更快。", "chunk": 1, "language": "zh", "avg_logprob": -1.0138986375596788, "no_speech_prob": 5.587333928858129e-10, "compression_ratio": 2.2857142857142856 }, { "start": 494.9, "end": 499.16, "text": "than compared to the FP16 for you over there.", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 499.26, "end": 503.96, "text": "See now, this is going to be the FP4 that we have got.", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 504.06, "end": 505.5, "text": "I'm going to be asking the same exact question.", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 505.64, "end": 507.86, "text": "The total number of tokens per second is not changing at all.", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 508.18, "end": 510.78, "text": "And now if I'm going to be running the same exact prompt,", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 510.92, "end": 516.04, "text": "the Serenium WebDriver question over there with the quantization 4 there,", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 516.38, "end": 520.94, "text": "instead of the 4.04 minutes, which was taking in FP16,", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 521.16, "end": 523.94, "text": "it is 4.46 as you can see here.", "chunk": 1, "language": "zh", "avg_logprob": -0.1441980240836976, "no_speech_prob": 1.2894252722842059e-12, "compression_ratio": 1.6627906976744187 }, { "start": 523.94, "end": 528.84, "text": "so based on the quantization change it is also obviously going to change the time over there", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 528.84, "end": 535.06, "text": "so this proves the point that the inferencing actually works even in LM studio for us and you", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 535.06, "end": 539.54, "text": "can see that the output has got no compromise as well the output looks pretty much exactly the", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 539.54, "end": 543.48, "text": "same and I can see that it is still working I mean there is a change in the output of course", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 543.48, "end": 548.36, "text": "in the web driver for the dependency injection it's there over here but in here you can see that", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 548.36, "end": 552.76, "text": "the dependency injection setup has got a bit different with like driver factory class there", "chunk": 1, "language": "zh", "avg_logprob": -0.05089561080932617, "no_speech_prob": 1.898169555564211e-12, "compression_ratio": 1.9548611111111112 }, { "start": 552.76, "end": 558.46, "text": "that's the only difference between both of them but overall i can actually see that it is working", "chunk": 1, "language": "zh", "avg_logprob": -0.03673605529629454, "no_speech_prob": 1.8981003834656063e-12, "compression_ratio": 1.7545454545454546 }, { "start": 558.46, "end": 564.76, "text": "pretty much as expected the last test which i wanted to do with the fp16 of the same exact", "chunk": 1, "language": "zh", "avg_logprob": -0.03673605529629454, "no_speech_prob": 1.8981003834656063e-12, "compression_ratio": 1.7545454545454546 }, { "start": 564.76, "end": 571.4, "text": "model is by connecting to a pi coding agent i want to really see if this is going to work or not so", "chunk": 1, "language": "zh", "avg_logprob": -0.03673605529629454, "no_speech_prob": 1.8981003834656063e-12, "compression_ratio": 1.7545454545454546 }, { "start": 571.4, "end": 578.5, "text": "i have set the fp16 over here i'm running it and i'm going to be opening the pi coding agent over", "chunk": 1, "language": "zh", "avg_logprob": -0.03673605529629454, "no_speech_prob": 1.8981003834656063e-12, "compression_ratio": 1.7545454545454546 }, { "start": 578.5, "end": 584.1, "text": "here and i'm going to choose the model as you can see i'm going to be chasing choosing that gsq rco", "chunk": 1, "language": "zh", "avg_logprob": -0.06391265091386814, "no_speech_prob": 1.8252159766224407e-12, "compression_ratio": 1.9695431472081217 }, { "start": 584.1, "end": 591.62, "text": "unsloth mtp model there and i'm going to be just saying hi over there even for a high message it", "chunk": 1, "language": "zh", "avg_logprob": -0.06391265091386814, "no_speech_prob": 1.8252159766224407e-12, "compression_ratio": 1.9695431472081217 }, { "start": 591.62, "end": 597.88, "text": "was taking a long time it's not very fast as you can imagine it was taking a long time and the", "chunk": 1, "language": "zh", "avg_logprob": -0.06391265091386814, "no_speech_prob": 1.8252159766224407e-12, "compression_ratio": 1.9695431472081217 }, { "start": 597.88, "end": 603.2, "text": "output you can see that it is going to be giving you very very slow here which is quite expected", "chunk": 1, "language": "zh", "avg_logprob": -0.06391265091386814, "no_speech_prob": 1.8252159766224407e-12, "compression_ratio": 1.9695431472081217 }, { "start": 603.2, "end": 608.6, "text": "because this is a dense model and now i have told it to analyze the code for me in this particular", "chunk": 1, "language": "zh", "avg_logprob": -0.06791354514457085, "no_speech_prob": 1.868732382379057e-12, "compression_ratio": 1.7536231884057971 }, { "start": 608.6, "end": 616.94, "text": "repository and that took me more than 27 minutes just to analyze the code i mean it went so deep", "chunk": 1, "language": "zh", "avg_logprob": -0.06791354514457085, "no_speech_prob": 1.868732382379057e-12, "compression_ratio": 1.7536231884057971 }, { "start": 616.94, "end": 620.82, "text": "to be honest like going through each and every file even start to figure out what exactly is", "chunk": 1, "language": "zh", "avg_logprob": -0.06791354514457085, "no_speech_prob": 1.868732382379057e-12, "compression_ratio": 1.7536231884057971 }, { "start": 620.82, "end": 626.56, "text": "the problem in the particular code but actually took 27 minutes to give me the output which is", "chunk": 1, "language": "zh", "avg_logprob": -0.06791354514457085, "no_speech_prob": 1.868732382379057e-12, "compression_ratio": 1.7536231884057971 }, { "start": 626.56, "end": 632.94, "text": "what really happened over here as you can see so long story short you can run a two-bit quantization", "chunk": 1, "language": "zh", "avg_logprob": -0.06791354514457085, "no_speech_prob": 1.868732382379057e-12, "compression_ratio": 1.7536231884057971 }, { "start": 632.94, "end": 641.32, "text": "QUEN 3.8 27 billion parameter model in an apple m4 mac mini machine without hitting the memory", "chunk": 1, "language": "zh", "avg_logprob": -0.06384852799502286, "no_speech_prob": 1.2690732796236115e-12, "compression_ratio": 1.6297872340425532 }, { "start": 641.32, "end": 648.36, "text": "ceiling but what is the actual technique which makes the two big quantization which helps you", "chunk": 1, "language": "zh", "avg_logprob": -0.06384852799502286, "no_speech_prob": 1.2690732796236115e-12, "compression_ratio": 1.6297872340425532 }, { "start": 648.36, "end": 653.46, "text": "run in a mac mini m4 machine is actually the most important part that's the reason why i actually", "chunk": 1, "language": "zh", "avg_logprob": -0.06384852799502286, "no_speech_prob": 1.2690732796236115e-12, "compression_ratio": 1.6297872340425532 }, { "start": 653.46, "end": 658.46, "text": "wanted to show you what's really happening over here and other thing is you may ask what happens", "chunk": 1, "language": "zh", "avg_logprob": -0.06384852799502286, "no_speech_prob": 1.2690732796236115e-12, "compression_ratio": 1.6297872340425532 }, { "start": 658.46, "end": 665.4, "text": "if you try to run the same exact model in the Apple M5 Max machine right so I'm going to be", "chunk": 1, "language": "zh", "avg_logprob": -0.0671484735276964, "no_speech_prob": 1.2744041932855055e-12, "compression_ratio": 1.9257425742574257 }, { "start": 665.4, "end": 672.32, "text": "loading the same exact model as you can see over here and I'm going to be doing an FP16 here as", "chunk": 1, "language": "zh", "avg_logprob": -0.0671484735276964, "no_speech_prob": 1.2744041932855055e-12, "compression_ratio": 1.9257425742574257 }, { "start": 672.32, "end": 678.6, "text": "well so FP16 Max current prediction is going to be one as well I'm going to be loading this particular", "chunk": 1, "language": "zh", "avg_logprob": -0.0671484735276964, "no_speech_prob": 1.2744041932855055e-12, "compression_ratio": 1.9257425742574257 }, { "start": 678.6, "end": 683.36, "text": "model and let's try to ask the same exact question over here and see how this is actually going to", "chunk": 1, "language": "zh", "avg_logprob": -0.0671484735276964, "no_speech_prob": 1.2744041932855055e-12, "compression_ratio": 1.9257425742574257 }, { "start": 683.36, "end": 685.92, "text": "在我的 Apple M5 Max machine", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 685.92, "end": 688.12, "text": "所以这个是128GB machine", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 688.12, "end": 689.36, "text": "所以你可能会被担心", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 689.36, "end": 691.0, "text": "要是转买它快速", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 691.0, "end": 692.16, "text": "你会被担心", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 692.16, "end": 694.12, "text": "要是100兆兆的", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 694.12, "end": 695.4, "text": "如果我会被担心", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 695.4, "end": 696.72, "text": "但如果我会被担心", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 696.72, "end": 699.0, "text": "同样的同样", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 699.0, "end": 700.32, "text": "在这个机器", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 700.32, "end": 702.6, "text": "你能看到它是", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 702.6, "end": 705.6, "text": "连接到27兆兆的", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 705.6, "end": 708.32, "text": "它是127兆兆的", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 708.32, "end": 709.96, "text": "所以它试试了", "chunk": 1, "language": "zh", "avg_logprob": -1.0316956740159255, "no_speech_prob": 1.7014549491545239e-12, "compression_ratio": 1.4257425742574257 }, { "start": 709.96, "end": 716.34, "text": "point that it's not just about the two-bit quantization model which can run way more", "chunk": 1, "language": "zh", "avg_logprob": -0.045421580264442846, "no_speech_prob": 1.2206396917541218e-12, "compression_ratio": 1.7523364485981308 }, { "start": 716.34, "end": 723.32, "text": "faster in an apple m5 max machine because it's even running in the m4 mini machine but the thing", "chunk": 1, "language": "zh", "avg_logprob": -0.045421580264442846, "no_speech_prob": 1.2206396917541218e-12, "compression_ratio": 1.7523364485981308 }, { "start": 723.32, "end": 729.2, "text": "is even though it is a two-bit quantization the quality of the output still remains and that's", "chunk": 1, "language": "zh", "avg_logprob": -0.045421580264442846, "no_speech_prob": 1.2206396917541218e-12, "compression_ratio": 1.7523364485981308 }, { "start": 729.2, "end": 736.34, "text": "the reason the the technique of the gsq rco sits in so it safely chooses the best tensor which can", "chunk": 1, "language": "zh", "avg_logprob": -0.045421580264442846, "no_speech_prob": 1.2206396917541218e-12, "compression_ratio": 1.7523364485981308 } ], "transcript_selection": { "schema": "videonote.transcript_selection.v1", "status": "SELECTED_WITH_WARNINGS", "selected_source": "whisper_provisional", "decision_reason": "published_with_warning_after_language_guard:requested_language_repair_inconclusive", "publication_allowed": true, "whisper_rejected": true, "publication_mode": "published_with_warnings" } }