WEBVTT Kind: captions Language: en 00:00:00.000 --> 00:00:01.830 align:start position:0% Hi<00:00:00.320> guys,<00:00:00.680> welcome<00:00:00.920> back.<00:00:01.520> In<00:00:01.600> this<00:00:01.720> video, 00:00:01.830 --> 00:00:01.840 align:start position:0% Hi guys, welcome back. In this video, 00:00:01.840 --> 00:00:04.070 align:start position:0% Hi guys, welcome back. In this video, we'll<00:00:01.920> be<00:00:02.000> talking<00:00:02.280> about<00:00:02.720> how<00:00:02.920> you<00:00:03.080> can<00:00:03.360> run<00:00:03.760> a 00:00:04.070 --> 00:00:04.080 align:start position:0% we'll be talking about how you can run a 00:00:04.080 --> 00:00:08.230 align:start position:0% we'll be talking about how you can run a Qwen<00:00:04.680> in<00:00:04.800> an<00:00:05.040> Apple 00:00:08.230 --> 00:00:08.240 align:start position:0% 00:00:08.240 --> 00:00:12.150 align:start position:0% M4<00:00:08.880> Mac<00:00:09.400> mini<00:00:09.800> machine<00:00:10.320> with<00:00:10.600> just<00:00:10.880> 24<00:00:11.440> GB<00:00:11.840> of 00:00:12.150 --> 00:00:12.160 align:start position:0% M4 Mac mini machine with just 24 GB of 00:00:12.160 --> 00:00:16.110 align:start position:0% M4 Mac mini machine with just 24 GB of unified<00:00:12.920> memory.<00:00:13.920> We<00:00:14.120> know<00:00:14.280> that<00:00:14.840> Qwen<00:00:15.120> 3.8<00:00:15.640> 27 00:00:16.110 --> 00:00:16.120 align:start position:0% unified memory. We know that Qwen 3.8 27 00:00:16.120 --> 00:00:17.870 align:start position:0% unified memory. We know that Qwen 3.8 27 billion<00:00:16.400> parameter<00:00:16.880> model<00:00:17.280> will<00:00:17.400> be<00:00:17.560> of<00:00:17.720> at 00:00:17.870 --> 00:00:17.880 align:start position:0% billion parameter model will be of at 00:00:17.880 --> 00:00:22.310 align:start position:0% billion parameter model will be of at least<00:00:18.240> 15<00:00:18.640> GB<00:00:19.120> of<00:00:19.360> size.<00:00:20.400> And<00:00:21.160> loading<00:00:22.000> the 00:00:22.310 --> 00:00:22.320 align:start position:0% least 15 GB of size. And loading the 00:00:22.320 --> 00:00:25.910 align:start position:0% least 15 GB of size. And loading the entire<00:00:22.920> model<00:00:23.960> in<00:00:24.120> the<00:00:24.680> unified<00:00:25.200> memory<00:00:25.760> will 00:00:25.910 --> 00:00:25.920 align:start position:0% entire model in the unified memory will 00:00:25.920 --> 00:00:28.230 align:start position:0% entire model in the unified memory will not<00:00:26.120> only<00:00:26.280> load<00:00:26.560> the<00:00:26.680> model,<00:00:27.160> but<00:00:27.320> also<00:00:27.680> loads 00:00:28.230 --> 00:00:28.240 align:start position:0% not only load the model, but also loads 00:00:28.240 --> 00:00:31.430 align:start position:0% not only load the model, but also loads the<00:00:28.520> KV<00:00:28.880> cache,<00:00:29.480> runtime,<00:00:30.280> and<00:00:30.680> the<00:00:30.960> Mac 00:00:31.430 --> 00:00:31.440 align:start position:0% the KV cache, runtime, and the Mac 00:00:31.440 --> 00:00:33.910 align:start position:0% the KV cache, runtime, and the Mac operating<00:00:31.840> system<00:00:32.360> all<00:00:32.599> together.<00:00:33.560> And<00:00:33.760> while 00:00:33.910 --> 00:00:33.920 align:start position:0% operating system all together. And while 00:00:33.920 --> 00:00:36.950 align:start position:0% operating system all together. And while you<00:00:34.160> load<00:00:34.560> all<00:00:34.800> of<00:00:35.000> them<00:00:35.440> in<00:00:35.560> the<00:00:35.680> VRAM,<00:00:36.560> it<00:00:36.720> is 00:00:36.950 --> 00:00:36.960 align:start position:0% you load all of them in the VRAM, it is 00:00:36.960 --> 00:00:39.310 align:start position:0% you load all of them in the VRAM, it is definitely<00:00:37.760> going<00:00:38.080> to<00:00:38.320> hit<00:00:38.600> the<00:00:38.840> memory 00:00:39.310 --> 00:00:39.320 align:start position:0% definitely going to hit the memory 00:00:39.320 --> 00:00:41.710 align:start position:0% definitely going to hit the memory ceiling<00:00:40.040> for<00:00:40.360> you,<00:00:40.960> and<00:00:41.240> it<00:00:41.320> is<00:00:41.400> going<00:00:41.520> to<00:00:41.600> be 00:00:41.710 --> 00:00:41.720 align:start position:0% ceiling for you, and it is going to be 00:00:41.720 --> 00:00:44.230 align:start position:0% ceiling for you, and it is going to be getting<00:00:42.000> you<00:00:42.440> the<00:00:42.680> out<00:00:42.880> of<00:00:43.080> memory<00:00:43.440> exception. 00:00:44.230 --> 00:00:44.240 align:start position:0% getting you the out of memory exception. 00:00:44.240 --> 00:00:46.950 align:start position:0% getting you the out of memory exception. So,<00:00:44.480> how<00:00:44.720> can<00:00:44.960> we<00:00:45.120> even<00:00:45.480> handle<00:00:46.040> the<00:00:46.160> situation 00:00:46.950 --> 00:00:46.960 align:start position:0% So, how can we even handle the situation 00:00:46.960 --> 00:00:49.790 align:start position:0% So, how can we even handle the situation of<00:00:47.320> these<00:00:47.720> kind<00:00:48.040> of<00:00:48.280> memory<00:00:48.680> problem?<00:00:49.400> And<00:00:49.640> can 00:00:49.790 --> 00:00:49.800 align:start position:0% of these kind of memory problem? And can 00:00:49.800 --> 00:00:52.230 align:start position:0% of these kind of memory problem? And can we<00:00:49.920> even<00:00:50.160> run<00:00:50.440> a<00:00:50.600> Qwen<00:00:50.960> 3.8<00:00:51.520> 27<00:00:51.960> billion 00:00:52.230 --> 00:00:52.240 align:start position:0% we even run a Qwen 3.8 27 billion 00:00:52.240 --> 00:00:54.870 align:start position:0% we even run a Qwen 3.8 27 billion parameter<00:00:52.720> model<00:00:53.440> in<00:00:53.600> this<00:00:53.800> machine?<00:00:54.480> And<00:00:54.720> is 00:00:54.870 --> 00:00:54.880 align:start position:0% parameter model in this machine? And is 00:00:54.880 --> 00:00:57.830 align:start position:0% parameter model in this machine? And is it<00:00:55.360> just<00:00:55.680> the<00:00:55.800> problem<00:00:56.280> with<00:00:56.640> the<00:00:57.400> model 00:00:57.830 --> 00:00:57.840 align:start position:0% it just the problem with the model 00:00:57.840 --> 00:01:00.190 align:start position:0% it just the problem with the model itself,<00:00:58.560> or<00:00:58.840> is<00:00:59.000> it<00:00:59.240> the<00:00:59.360> problem<00:00:59.840> with<00:01:00.000> the 00:01:00.190 --> 00:01:00.200 align:start position:0% itself, or is it the problem with the 00:01:00.200 --> 00:01:02.670 align:start position:0% itself, or is it the problem with the quantization<00:01:01.080> that<00:01:01.280> we<00:01:01.400> have?<00:01:01.960> But<00:01:02.320> what<00:01:02.520> if 00:01:02.670 --> 00:01:02.680 align:start position:0% quantization that we have? But what if 00:01:02.680 --> 00:01:06.310 align:start position:0% quantization that we have? But what if we<00:01:02.840> have<00:01:03.240> a<00:01:03.440> quantization<00:01:04.839> of<00:01:05.519> the<00:01:05.640> same<00:01:05.920> exact 00:01:06.310 --> 00:01:06.320 align:start position:0% we have a quantization of the same exact 00:01:06.320 --> 00:01:08.910 align:start position:0% we have a quantization of the same exact model<00:01:06.920> with<00:01:07.040> just<00:01:07.440> two-bit<00:01:08.120> quantization 00:01:08.910 --> 00:01:08.920 align:start position:0% model with just two-bit quantization 00:01:08.920 --> 00:01:11.150 align:start position:0% model with just two-bit quantization instead<00:01:09.200> of<00:01:09.320> the<00:01:09.520> four-bit<00:01:10.080> quantization? 00:01:11.150 --> 00:01:11.160 align:start position:0% instead of the four-bit quantization? 00:01:11.160 --> 00:01:14.270 align:start position:0% instead of the four-bit quantization? Can<00:01:11.440> we<00:01:11.640> run<00:01:11.880> that<00:01:12.200> at<00:01:12.360> all?<00:01:13.080> Well,<00:01:13.840> welcome<00:01:14.200> to 00:01:14.270 --> 00:01:14.280 align:start position:0% Can we run that at all? Well, welcome to 00:01:14.280 --> 00:01:17.630 align:start position:0% Can we run that at all? Well, welcome to the<00:01:14.480> Qwen<00:01:14.920> model<00:01:15.640> quantized<00:01:16.280> to<00:01:16.800> two-bit 00:01:17.630 --> 00:01:17.640 align:start position:0% the Qwen model quantized to two-bit 00:01:17.640 --> 00:01:22.190 align:start position:0% the Qwen model quantized to two-bit called<00:01:18.080> as<00:01:18.320> the<00:01:18.520> Qwen<00:01:18.840> 3.8<00:01:19.560> 27<00:01:20.120> billion<00:01:21.000> GSQ 00:01:22.190 --> 00:01:22.200 align:start position:0% called as the Qwen 3.8 27 billion GSQ 00:01:22.200 --> 00:01:26.470 align:start position:0% called as the Qwen 3.8 27 billion GSQ RCO<00:01:23.120> IQ2<00:01:24.120> QS<00:01:24.680> model.<00:01:25.400> And<00:01:25.560> this<00:01:25.760> model<00:01:26.120> can 00:01:26.470 --> 00:01:26.480 align:start position:0% RCO IQ2 QS model. And this model can 00:01:26.480 --> 00:01:30.750 align:start position:0% RCO IQ2 QS model. And this model can load<00:01:26.960> in<00:01:27.120> the<00:01:27.640> 24<00:01:28.200> GB<00:01:28.600> of<00:01:29.200> unified<00:01:29.920> machine, 00:01:30.750 --> 00:01:30.760 align:start position:0% load in the 24 GB of unified machine, 00:01:30.760 --> 00:01:33.550 align:start position:0% load in the 24 GB of unified machine, and<00:01:31.160> it<00:01:31.360> can<00:01:31.840> support<00:01:32.400> a<00:01:32.520> full<00:01:33.000> GPU 00:01:33.550 --> 00:01:33.560 align:start position:0% and it can support a full GPU 00:01:33.560 --> 00:01:35.390 align:start position:0% and it can support a full GPU offloading,<00:01:34.320> as<00:01:34.560> well<00:01:34.680> as<00:01:35.000> it<00:01:35.160> has 00:01:35.390 --> 00:01:35.400 align:start position:0% offloading, as well as it has 00:01:35.400 --> 00:01:37.830 align:start position:0% offloading, as well as it has multi-token<00:01:36.160> prediction<00:01:36.720> enabled<00:01:37.320> as<00:01:37.560> well. 00:01:37.830 --> 00:01:37.840 align:start position:0% multi-token prediction enabled as well. 00:01:37.840 --> 00:01:40.950 align:start position:0% multi-token prediction enabled as well. And<00:01:38.040> this<00:01:38.200> model<00:01:38.720> can<00:01:39.000> easily<00:01:39.680> fit<00:01:40.160> in<00:01:40.440> an 00:01:40.950 --> 00:01:40.960 align:start position:0% And this model can easily fit in an 00:01:40.960 --> 00:01:45.350 align:start position:0% And this model can easily fit in an Apple<00:01:41.520> M4<00:01:42.400> machine<00:01:43.040> of<00:01:43.440> 24<00:01:44.040> GB<00:01:44.480> of<00:01:44.760> unified 00:01:45.350 --> 00:01:45.360 align:start position:0% Apple M4 machine of 24 GB of unified 00:01:45.360 --> 00:01:48.630 align:start position:0% Apple M4 machine of 24 GB of unified memory,<00:01:46.120> which<00:01:46.280> includes<00:01:47.200> the<00:01:47.640> model,<00:01:48.280> KV 00:01:48.630 --> 00:01:48.640 align:start position:0% memory, which includes the model, KV 00:01:48.640 --> 00:01:50.790 align:start position:0% memory, which includes the model, KV cache,<00:01:49.080> runtime,<00:01:49.600> and<00:01:49.800> Mac<00:01:50.040> operating<00:01:50.280> system 00:01:50.790 --> 00:01:50.800 align:start position:0% cache, runtime, and Mac operating system 00:01:50.800 --> 00:01:52.870 align:start position:0% cache, runtime, and Mac operating system without<00:01:51.160> hitting<00:01:51.440> the<00:01:51.600> memory<00:01:52.080> ceiling<00:01:52.640> as 00:01:52.870 --> 00:01:52.880 align:start position:0% without hitting the memory ceiling as 00:01:52.880 --> 00:01:54.870 align:start position:0% without hitting the memory ceiling as well.<00:01:53.080> So,<00:01:53.240> it<00:01:53.360> can<00:01:53.680> easily<00:01:54.080> load<00:01:54.400> everything 00:01:54.870 --> 00:01:54.880 align:start position:0% well. So, it can easily load everything 00:01:54.880 --> 00:01:57.510 align:start position:0% well. So, it can easily load everything for<00:01:55.160> you<00:01:55.760> within<00:01:56.160> your<00:01:56.560> Apple<00:01:56.880> machine,<00:01:57.360> which 00:01:57.510 --> 00:01:57.520 align:start position:0% for you within your Apple machine, which 00:01:57.520 --> 00:01:59.150 align:start position:0% for you within your Apple machine, which is<00:01:57.720> quite<00:01:58.160> amazing.<00:01:58.600> So,<00:01:58.760> the<00:01:58.880> memory 00:01:59.150 --> 00:01:59.160 align:start position:0% is quite amazing. So, the memory 00:01:59.160 --> 00:02:00.830 align:start position:0% is quite amazing. So, the memory requirement<00:01:59.720> is<00:01:59.800> going<00:01:59.960> to<00:02:00.040> be<00:02:00.120> so<00:02:00.400> much<00:02:00.600> less 00:02:00.830 --> 00:02:00.840 align:start position:0% requirement is going to be so much less 00:02:00.840 --> 00:02:03.390 align:start position:0% requirement is going to be so much less because<00:02:01.200> of<00:02:01.320> the<00:02:01.840> two-bit<00:02:02.160> quantization<00:02:03.200> that 00:02:03.390 --> 00:02:03.400 align:start position:0% because of the two-bit quantization that 00:02:03.400 --> 00:02:05.870 align:start position:0% because of the two-bit quantization that you<00:02:03.520> have<00:02:03.760> got<00:02:04.160> in<00:02:04.280> this<00:02:04.400> particular<00:02:05.080> model. 00:02:05.870 --> 00:02:05.880 align:start position:0% you have got in this particular model. 00:02:05.880 --> 00:02:07.750 align:start position:0% you have got in this particular model. So,<00:02:06.080> how<00:02:06.280> exactly<00:02:06.760> is<00:02:06.840> this<00:02:07.040> possible?<00:02:07.560> Will 00:02:07.750 --> 00:02:07.760 align:start position:0% So, how exactly is this possible? Will 00:02:07.760 --> 00:02:10.630 align:start position:0% So, how exactly is this possible? Will the<00:02:07.920> Qwen<00:02:08.200> 3.8<00:02:08.840> right<00:02:09.080> now<00:02:09.280> will<00:02:09.520> lose<00:02:09.880> its 00:02:10.630 --> 00:02:10.640 align:start position:0% the Qwen 3.8 right now will lose its 00:02:10.640 --> 00:02:13.430 align:start position:0% the Qwen 3.8 right now will lose its intelligence<00:02:11.280> ability<00:02:12.080> or<00:02:12.480> the<00:02:12.800> output<00:02:13.240> will 00:02:13.430 --> 00:02:13.440 align:start position:0% intelligence ability or the output will 00:02:13.440 --> 00:02:15.670 align:start position:0% intelligence ability or the output will be<00:02:13.600> of<00:02:13.800> less<00:02:14.040> quality<00:02:14.840> or<00:02:14.959> how<00:02:15.120> exactly<00:02:15.600> is 00:02:15.670 --> 00:02:15.680 align:start position:0% be of less quality or how exactly is 00:02:15.680 --> 00:02:17.910 align:start position:0% be of less quality or how exactly is this<00:02:15.920> happening?<00:02:16.280> What<00:02:16.440> exactly<00:02:17.080> is<00:02:17.280> this<00:02:17.560> new 00:02:17.910 --> 00:02:17.920 align:start position:0% this happening? What exactly is this new 00:02:17.920 --> 00:02:20.270 align:start position:0% this happening? What exactly is this new model<00:02:18.240> of<00:02:18.400> a<00:02:18.440> two-bit<00:02:18.840> quantization?<00:02:19.640> Well, 00:02:20.270 --> 00:02:20.280 align:start position:0% model of a two-bit quantization? Well, 00:02:20.280 --> 00:02:23.430 align:start position:0% model of a two-bit quantization? Well, this<00:02:20.560> particular<00:02:21.320> Qwen<00:02:21.880> model<00:02:22.520> is<00:02:22.760> a<00:02:22.840> standard 00:02:23.430 --> 00:02:23.440 align:start position:0% this particular Qwen model is a standard 00:02:23.440 --> 00:02:25.510 align:start position:0% this particular Qwen model is a standard Qwen<00:02:23.760> model,<00:02:24.120> but<00:02:24.360> instead<00:02:24.640> of<00:02:25.000> applying<00:02:25.400> the 00:02:25.510 --> 00:02:25.520 align:start position:0% Qwen model, but instead of applying the 00:02:25.520 --> 00:02:27.710 align:start position:0% Qwen model, but instead of applying the same<00:02:25.840> quantization<00:02:26.720> everywhere,<00:02:27.560> the 00:02:27.710 --> 00:02:27.720 align:start position:0% same quantization everywhere, the 00:02:27.720 --> 00:02:29.990 align:start position:0% same quantization everywhere, the technique<00:02:28.160> will<00:02:28.480> determine<00:02:29.240> which<00:02:29.480> tensors 00:02:29.990 --> 00:02:30.000 align:start position:0% technique will determine which tensors 00:02:30.000 --> 00:02:32.350 align:start position:0% technique will determine which tensors deserves<00:02:30.520> more<00:02:30.880> precision<00:02:31.600> and<00:02:31.840> which<00:02:32.040> can 00:02:32.350 --> 00:02:32.360 align:start position:0% deserves more precision and which can 00:02:32.360 --> 00:02:34.630 align:start position:0% deserves more precision and which can tolerate<00:02:32.880> the<00:02:33.120> aggressive<00:02:33.880> compression. 00:02:34.630 --> 00:02:34.640 align:start position:0% tolerate the aggressive compression. 00:02:34.640 --> 00:02:36.030 align:start position:0% tolerate the aggressive compression. That's<00:02:34.840> exactly<00:02:35.360> what<00:02:35.560> this<00:02:35.680> particular 00:02:36.030 --> 00:02:36.040 align:start position:0% That's exactly what this particular 00:02:36.040 --> 00:02:38.230 align:start position:0% That's exactly what this particular technique<00:02:36.480> does.<00:02:37.240> And<00:02:37.360> what<00:02:37.520> exactly<00:02:37.959> is<00:02:38.040> this 00:02:38.230 --> 00:02:38.240 align:start position:0% technique does. And what exactly is this 00:02:38.240 --> 00:02:39.750 align:start position:0% technique does. And what exactly is this technique<00:02:38.480> that<00:02:38.600> we're<00:02:38.720> talking<00:02:39.000> about?<00:02:39.560> The 00:02:39.750 --> 00:02:39.760 align:start position:0% technique that we're talking about? The 00:02:39.760 --> 00:02:43.430 align:start position:0% technique that we're talking about? The GSQ<00:02:40.680> RCO<00:02:41.800> or<00:02:41.959> otherwise<00:02:42.320> called<00:02:42.720> as<00:02:42.959> the 00:02:43.430 --> 00:02:43.440 align:start position:0% GSQ RCO or otherwise called as the 00:02:43.440 --> 00:02:45.949 align:start position:0% GSQ RCO or otherwise called as the Gumbel<00:02:43.800> Softmax<00:02:44.440> quantization,<00:02:45.640> which 00:02:45.949 --> 00:02:45.959 align:start position:0% Gumbel Softmax quantization, which 00:02:45.959 --> 00:02:48.590 align:start position:0% Gumbel Softmax quantization, which intelligently<00:02:46.680> optimizes<00:02:47.400> the<00:02:47.920> quantization 00:02:48.590 --> 00:02:48.600 align:start position:0% intelligently optimizes the quantization 00:02:48.600 --> 00:02:51.830 align:start position:0% intelligently optimizes the quantization value<00:02:49.320> while<00:02:49.640> RCO<00:02:50.200> intelligently<00:02:50.840> decide 00:02:51.830 --> 00:02:51.840 align:start position:0% value while RCO intelligently decide 00:02:51.840 --> 00:02:54.550 align:start position:0% value while RCO intelligently decide where<00:02:52.200> to<00:02:52.360> allocate<00:02:52.920> more<00:02:53.400> or<00:02:53.640> fewer<00:02:54.040> bits 00:02:54.550 --> 00:02:54.560 align:start position:0% where to allocate more or fewer bits 00:02:54.560 --> 00:02:57.949 align:start position:0% where to allocate more or fewer bits across<00:02:55.440> the<00:02:55.800> model's<00:02:56.760> uh<00:02:56.880> tensors.<00:02:57.800> So, 00:02:57.949 --> 00:02:57.959 align:start position:0% across the model's uh tensors. So, 00:02:57.959 --> 00:03:01.150 align:start position:0% across the model's uh tensors. So, essentially,<00:02:58.800> even<00:02:59.200> if<00:02:59.400> the<00:02:59.560> model<00:03:00.320> sizes<00:03:00.959> is 00:03:01.150 --> 00:03:01.160 align:start position:0% essentially, even if the model sizes is 00:03:01.160 --> 00:03:05.070 align:start position:0% essentially, even if the model sizes is going<00:03:01.440> to<00:03:01.519> be<00:03:01.720> reduced<00:03:02.680> from<00:03:03.480> the<00:03:03.720> 16<00:03:04.200> BPW, 00:03:05.070 --> 00:03:05.080 align:start position:0% going to be reduced from the 16 BPW, 00:03:05.080 --> 00:03:08.070 align:start position:0% going to be reduced from the 16 BPW, which<00:03:05.200> is<00:03:05.320> a<00:03:05.400> 16-bit<00:03:06.000> original<00:03:06.360> model,<00:03:07.040> to<00:03:07.480> a 00:03:08.070 --> 00:03:08.080 align:start position:0% which is a 16-bit original model, to a 00:03:08.080 --> 00:03:10.470 align:start position:0% which is a 16-bit original model, to a two-bit<00:03:08.560> quantization<00:03:09.320> model,<00:03:10.120> it<00:03:10.239> is<00:03:10.360> going 00:03:10.470 --> 00:03:10.480 align:start position:0% two-bit quantization model, it is going 00:03:10.480 --> 00:03:13.710 align:start position:0% two-bit quantization model, it is going to<00:03:10.560> be<00:03:10.680> off<00:03:11.200> almost<00:03:11.760> 6.4<00:03:12.640> time<00:03:12.920> of<00:03:13.120> a<00:03:13.200> size 00:03:13.710 --> 00:03:13.720 align:start position:0% to be off almost 6.4 time of a size 00:03:13.720 --> 00:03:16.150 align:start position:0% to be off almost 6.4 time of a size lower,<00:03:14.320> but<00:03:14.560> still<00:03:15.080> the<00:03:15.200> model<00:03:15.640> is<00:03:15.840> going<00:03:16.040> to 00:03:16.150 --> 00:03:16.160 align:start position:0% lower, but still the model is going to 00:03:16.160 --> 00:03:19.110 align:start position:0% lower, but still the model is going to give<00:03:16.440> you<00:03:16.600> an<00:03:16.880> amazing<00:03:17.600> precision<00:03:18.360> over<00:03:18.519> here. 00:03:19.110 --> 00:03:19.120 align:start position:0% give you an amazing precision over here. 00:03:19.120 --> 00:03:21.310 align:start position:0% give you an amazing precision over here. That<00:03:19.360> is<00:03:19.800> the<00:03:19.959> power<00:03:20.480> of<00:03:20.720> this<00:03:20.920> particular 00:03:21.310 --> 00:03:21.320 align:start position:0% That is the power of this particular 00:03:21.320 --> 00:03:23.509 align:start position:0% That is the power of this particular model<00:03:21.720> itself.<00:03:22.280> And<00:03:22.400> that's<00:03:22.600> the<00:03:22.680> reason<00:03:22.920> why 00:03:23.509 --> 00:03:23.519 align:start position:0% model itself. And that's the reason why 00:03:23.519 --> 00:03:25.830 align:start position:0% model itself. And that's the reason why even<00:03:23.760> though<00:03:23.920> the<00:03:24.040> size<00:03:24.480> of<00:03:24.600> the<00:03:24.760> model<00:03:25.320> is<00:03:25.640> so 00:03:25.830 --> 00:03:25.840 align:start position:0% even though the size of the model is so 00:03:25.840 --> 00:03:28.150 align:start position:0% even though the size of the model is so much<00:03:26.120> smaller,<00:03:26.680> the<00:03:26.840> quality<00:03:27.400> of<00:03:27.640> the<00:03:27.840> output 00:03:28.150 --> 00:03:28.160 align:start position:0% much smaller, the quality of the output 00:03:28.160 --> 00:03:30.990 align:start position:0% much smaller, the quality of the output is<00:03:28.320> still<00:03:28.760> going<00:03:29.000> to<00:03:29.080> be<00:03:29.200> way<00:03:29.400> more<00:03:30.200> same<00:03:30.800> like 00:03:30.990 --> 00:03:31.000 align:start position:0% is still going to be way more same like 00:03:31.000 --> 00:03:32.509 align:start position:0% is still going to be way more same like how<00:03:31.200> you<00:03:31.360> can<00:03:31.600> expect<00:03:32.200> with<00:03:32.320> their 00:03:32.509 --> 00:03:32.519 align:start position:0% how you can expect with their 00:03:32.519 --> 00:03:35.310 align:start position:0% how you can expect with their quantization<00:03:33.239> four<00:03:33.880> model<00:03:34.519> of<00:03:34.640> the<00:03:34.720> same<00:03:35.080> Qwen 00:03:35.310 --> 00:03:35.320 align:start position:0% quantization four model of the same Qwen 00:03:35.320 --> 00:03:37.229 align:start position:0% quantization four model of the same Qwen model<00:03:35.560> that<00:03:35.720> you<00:03:35.840> have<00:03:36.080> got.<00:03:36.680> That's<00:03:36.840> exactly 00:03:37.229 --> 00:03:37.239 align:start position:0% model that you have got. That's exactly 00:03:37.239 --> 00:03:39.030 align:start position:0% model that you have got. That's exactly what<00:03:37.360> this<00:03:37.480> particular<00:03:37.800> model<00:03:38.200> is,<00:03:38.640> and<00:03:38.880> we 00:03:39.030 --> 00:03:39.040 align:start position:0% what this particular model is, and we 00:03:39.040 --> 00:03:40.550 align:start position:0% what this particular model is, and we are<00:03:39.200> going<00:03:39.400> to<00:03:39.519> see<00:03:39.840> in<00:03:40.040> this<00:03:40.200> particular 00:03:40.550 --> 00:03:40.560 align:start position:0% are going to see in this particular 00:03:40.560 --> 00:03:45.070 align:start position:0% are going to see in this particular video<00:03:41.320> how<00:03:41.640> we<00:03:41.880> can<00:03:42.360> run<00:03:42.720> the<00:03:42.880> same<00:03:43.519> exact<00:03:44.480> Qwen 00:03:45.070 --> 00:03:45.080 align:start position:0% video how we can run the same exact Qwen 00:03:45.080 --> 00:03:48.509 align:start position:0% video how we can run the same exact Qwen two-bit<00:03:45.680> model<00:03:46.400> in<00:03:46.560> the<00:03:47.000> Mac<00:03:47.400> mini<00:03:47.800> M4 00:03:48.509 --> 00:03:48.519 align:start position:0% two-bit model in the Mac mini M4 00:03:48.519 --> 00:03:51.310 align:start position:0% two-bit model in the Mac mini M4 machine,<00:03:49.239> and<00:03:49.360> we're<00:03:49.560> also<00:03:50.000> going<00:03:50.239> to<00:03:50.360> see<00:03:50.959> how 00:03:51.310 --> 00:03:51.320 align:start position:0% machine, and we're also going to see how 00:03:51.320 --> 00:03:53.870 align:start position:0% machine, and we're also going to see how fast<00:03:52.200> it<00:03:52.400> is<00:03:52.640> going<00:03:52.960> to<00:03:53.080> be<00:03:53.280> doing<00:03:53.680> the 00:03:53.870 --> 00:03:53.880 align:start position:0% fast it is going to be doing the 00:03:53.880 --> 00:03:56.390 align:start position:0% fast it is going to be doing the inferencing<00:03:54.600> for<00:03:54.840> us<00:03:55.200> and<00:03:55.400> what<00:03:55.680> it<00:03:55.880> means 00:03:56.390 --> 00:03:56.400 align:start position:0% inferencing for us and what it means 00:03:56.400 --> 00:03:59.470 align:start position:0% inferencing for us and what it means with<00:03:56.760> different<00:03:57.160> sizes<00:03:57.760> of<00:03:58.080> the<00:03:58.600> KV<00:03:59.040> cache 00:03:59.470 --> 00:03:59.480 align:start position:0% with different sizes of the KV cache 00:03:59.480 --> 00:04:01.190 align:start position:0% with different sizes of the KV cache adjustment<00:04:00.000> that<00:04:00.200> I'm<00:04:00.360> going<00:04:00.520> to<00:04:00.600> be<00:04:00.720> doing<00:04:01.080> in 00:04:01.190 --> 00:04:01.200 align:start position:0% adjustment that I'm going to be doing in 00:04:01.200 --> 00:04:02.830 align:start position:0% adjustment that I'm going to be doing in this<00:04:01.320> particular<00:04:01.640> video.<00:04:02.320> Now,<00:04:02.440> I<00:04:02.480> will<00:04:02.600> show 00:04:02.830 --> 00:04:02.840 align:start position:0% this particular video. Now, I will show 00:04:02.840 --> 00:04:05.950 align:start position:0% this particular video. Now, I will show you<00:04:03.280> how<00:04:03.680> it<00:04:03.920> is<00:04:04.080> going<00:04:04.320> to<00:04:04.400> be<00:04:04.520> affecting<00:04:05.480> the 00:04:05.950 --> 00:04:05.960 align:start position:0% you how it is going to be affecting the 00:04:05.960 --> 00:04:08.350 align:start position:0% you how it is going to be affecting the the<00:04:06.040> speed<00:04:07.160> as<00:04:07.400> well<00:04:07.560> as<00:04:07.680> the<00:04:07.959> number<00:04:08.200> of 00:04:08.350 --> 00:04:08.360 align:start position:0% the speed as well as the number of 00:04:08.360 --> 00:04:10.430 align:start position:0% the speed as well as the number of tokens<00:04:08.800> per<00:04:08.959> second<00:04:09.520> as<00:04:09.720> well.<00:04:10.120> So,<00:04:10.280> in<00:04:10.360> this 00:04:10.430 --> 00:04:10.440 align:start position:0% tokens per second as well. So, in this 00:04:10.440 --> 00:04:12.110 align:start position:0% tokens per second as well. So, in this particular<00:04:10.720> video,<00:04:11.280> we<00:04:11.400> are<00:04:11.480> going<00:04:11.600> to<00:04:11.680> see 00:04:12.110 --> 00:04:12.120 align:start position:0% particular video, we are going to see 00:04:12.120 --> 00:04:15.270 align:start position:0% particular video, we are going to see which<00:04:12.440> KV<00:04:12.800> size<00:04:13.480> or<00:04:13.640> which<00:04:13.840> KV<00:04:14.160> cache<00:04:14.800> is<00:04:15.040> going 00:04:15.270 --> 00:04:15.280 align:start position:0% which KV size or which KV cache is going 00:04:15.280 --> 00:04:18.190 align:start position:0% which KV size or which KV cache is going to<00:04:15.360> be<00:04:15.520> faster<00:04:15.959> like<00:04:16.120> 16-bit<00:04:16.840> or<00:04:17.000> 4-bit<00:04:17.680> or 00:04:18.190 --> 00:04:18.200 align:start position:0% to be faster like 16-bit or 4-bit or 00:04:18.200 --> 00:04:20.349 align:start position:0% to be faster like 16-bit or 4-bit or 8-bit<00:04:18.519> quantization<00:04:19.640> and<00:04:19.880> which<00:04:20.040> is<00:04:20.160> going<00:04:20.280> to 00:04:20.349 --> 00:04:20.359 align:start position:0% 8-bit quantization and which is going to 00:04:20.359 --> 00:04:23.030 align:start position:0% 8-bit quantization and which is going to take<00:04:20.720> more<00:04:21.120> time<00:04:22.000> and<00:04:22.480> what<00:04:22.640> is<00:04:22.760> the<00:04:22.840> total 00:04:23.030 --> 00:04:23.040 align:start position:0% take more time and what is the total 00:04:23.040 --> 00:04:25.230 align:start position:0% take more time and what is the total number<00:04:23.280> of<00:04:23.400> tokens<00:04:23.800> per<00:04:24.000> second<00:04:24.680> that<00:04:24.840> you<00:04:25.040> are 00:04:25.230 --> 00:04:25.240 align:start position:0% number of tokens per second that you are 00:04:25.240 --> 00:04:26.990 align:start position:0% number of tokens per second that you are going<00:04:25.520> to<00:04:25.600> be<00:04:25.880> getting 00:04:26.990 --> 00:04:27.000 align:start position:0% going to be getting 00:04:27.000 --> 00:04:29.030 align:start position:0% going to be getting with<00:04:27.360> different<00:04:27.880> kinds<00:04:28.360> of<00:04:28.560> settings<00:04:28.919> that 00:04:29.030 --> 00:04:29.040 align:start position:0% with different kinds of settings that 00:04:29.040 --> 00:04:30.150 align:start position:0% with different kinds of settings that you<00:04:29.120> are<00:04:29.200> going<00:04:29.320> to<00:04:29.360> be<00:04:29.480> doing<00:04:29.680> in<00:04:29.760> the<00:04:29.880> LM 00:04:30.150 --> 00:04:30.160 align:start position:0% you are going to be doing in the LM 00:04:30.160 --> 00:04:31.909 align:start position:0% you are going to be doing in the LM Studio.<00:04:30.880> And<00:04:31.000> one<00:04:31.160> more<00:04:31.320> time,<00:04:31.760> this 00:04:31.909 --> 00:04:31.919 align:start position:0% Studio. And one more time, this 00:04:31.919 --> 00:04:33.630 align:start position:0% Studio. And one more time, this particular<00:04:32.280> model<00:04:32.640> is<00:04:32.880> actually<00:04:33.320> not 00:04:33.630 --> 00:04:33.640 align:start position:0% particular model is actually not 00:04:33.640 --> 00:04:35.750 align:start position:0% particular model is actually not available<00:04:34.560> in<00:04:34.680> the<00:04:34.800> MLX<00:04:35.160> format.<00:04:35.600> I 00:04:35.750 --> 00:04:35.760 align:start position:0% available in the MLX format. I 00:04:35.760 --> 00:04:37.990 align:start position:0% available in the MLX format. I unfortunately<00:04:36.440> have<00:04:36.680> got<00:04:36.960> only<00:04:37.320> the<00:04:37.480> GGML 00:04:37.990 --> 00:04:38.000 align:start position:0% unfortunately have got only the GGML 00:04:38.000 --> 00:04:39.590 align:start position:0% unfortunately have got only the GGML format<00:04:38.640> and<00:04:38.760> I'm<00:04:38.960> going<00:04:39.080> to<00:04:39.160> be<00:04:39.240> running<00:04:39.480> this 00:04:39.590 --> 00:04:39.600 align:start position:0% format and I'm going to be running this 00:04:39.600 --> 00:04:41.750 align:start position:0% format and I'm going to be running this particular<00:04:40.080> model<00:04:40.880> in<00:04:41.040> the<00:04:41.160> LM<00:04:41.400> Studio<00:04:41.640> and<00:04:41.720> I 00:04:41.750 --> 00:04:41.760 align:start position:0% particular model in the LM Studio and I 00:04:41.760 --> 00:04:44.190 align:start position:0% particular model in the LM Studio and I will<00:04:41.880> show<00:04:42.080> you<00:04:42.280> how<00:04:42.480> the<00:04:42.640> TPS<00:04:43.160> is<00:04:43.360> going<00:04:43.640> to<00:04:43.720> be 00:04:44.190 --> 00:04:44.200 align:start position:0% will show you how the TPS is going to be 00:04:44.200 --> 00:04:46.590 align:start position:0% will show you how the TPS is going to be for<00:04:44.440> the<00:04:44.560> different<00:04:45.440> KV<00:04:45.800> cache<00:04:46.160> that<00:04:46.360> we<00:04:46.480> are 00:04:46.590 --> 00:04:46.600 align:start position:0% for the different KV cache that we are 00:04:46.600 --> 00:04:48.870 align:start position:0% for the different KV cache that we are going<00:04:46.720> to<00:04:46.800> be<00:04:46.880> setting<00:04:47.360> and<00:04:47.720> running<00:04:48.000> it.<00:04:48.400> But, 00:04:48.870 --> 00:04:48.880 align:start position:0% going to be setting and running it. But, 00:04:48.880 --> 00:04:50.710 align:start position:0% going to be setting and running it. But, you<00:04:48.960> know<00:04:49.080> what?<00:04:49.600> This<00:04:49.720> particular<00:04:50.080> model<00:04:50.480> is 00:04:50.710 --> 00:04:50.720 align:start position:0% you know what? This particular model is 00:04:50.720 --> 00:04:53.950 align:start position:0% you know what? This particular model is not<00:04:51.320> really<00:04:52.000> going<00:04:52.360> to<00:04:52.560> hit<00:04:53.360> the<00:04:53.720> out<00:04:53.840> of 00:04:53.950 --> 00:04:53.960 align:start position:0% not really going to hit the out of 00:04:53.960 --> 00:04:55.909 align:start position:0% not really going to hit the out of memory<00:04:54.240> exception.<00:04:55.200> Based<00:04:55.480> on<00:04:55.560> the<00:04:55.600> different 00:04:55.909 --> 00:04:55.919 align:start position:0% memory exception. Based on the different 00:04:55.919 --> 00:04:57.710 align:start position:0% memory exception. Based on the different KV<00:04:56.200> cache<00:04:56.480> setting,<00:04:57.080> you<00:04:57.160> can<00:04:57.320> actually<00:04:57.600> see 00:04:57.710 --> 00:04:57.720 align:start position:0% KV cache setting, you can actually see 00:04:57.720 --> 00:04:59.510 align:start position:0% KV cache setting, you can actually see that<00:04:57.919> this<00:04:58.080> model<00:04:58.440> is<00:04:58.800> actually<00:04:59.160> going<00:04:59.360> to<00:04:59.400> be 00:04:59.510 --> 00:04:59.520 align:start position:0% that this model is actually going to be 00:04:59.520 --> 00:05:01.430 align:start position:0% that this model is actually going to be performant<00:05:00.200> as<00:05:00.400> well.<00:05:00.600> And<00:05:00.680> this<00:05:00.800> is<00:05:00.960> a<00:05:01.240> good 00:05:01.430 --> 00:05:01.440 align:start position:0% performant as well. And this is a good 00:05:01.440 --> 00:05:03.070 align:start position:0% performant as well. And this is a good news<00:05:01.840> for<00:05:02.000> this<00:05:02.120> particular<00:05:02.400> model.<00:05:02.800> Well,<00:05:02.960> as 00:05:03.070 --> 00:05:03.080 align:start position:0% news for this particular model. Well, as 00:05:03.080 --> 00:05:05.190 align:start position:0% news for this particular model. Well, as that<00:05:03.240> said,<00:05:03.720> let's<00:05:03.880> start<00:05:04.240> doing<00:05:04.520> the<00:05:04.680> testing 00:05:05.190 --> 00:05:05.200 align:start position:0% that said, let's start doing the testing 00:05:05.200 --> 00:05:06.270 align:start position:0% that said, let's start doing the testing of<00:05:05.400> the<00:05:05.520> model<00:05:05.840> with<00:05:05.919> a<00:05:05.960> different 00:05:06.270 --> 00:05:06.280 align:start position:0% of the model with a different 00:05:06.280 --> 00:05:08.550 align:start position:0% of the model with a different quantization.<00:05:07.400> So,<00:05:07.680> you<00:05:07.840> can<00:05:08.040> see<00:05:08.200> that<00:05:08.480> I 00:05:08.550 --> 00:05:08.560 align:start position:0% quantization. So, you can see that I 00:05:08.560 --> 00:05:11.909 align:start position:0% quantization. So, you can see that I have<00:05:08.800> got<00:05:09.160> my<00:05:09.760> Mac<00:05:10.080> mini<00:05:10.480> M4<00:05:11.200> machine.<00:05:11.840> The 00:05:11.909 --> 00:05:11.919 align:start position:0% have got my Mac mini M4 machine. The 00:05:11.919 --> 00:05:12.790 align:start position:0% have got my Mac mini M4 machine. The total<00:05:12.080> number<00:05:12.240> of<00:05:12.320> physical<00:05:12.520> memory<00:05:12.680> that<00:05:12.760> you 00:05:12.790 --> 00:05:12.800 align:start position:0% total number of physical memory that you 00:05:12.800 --> 00:05:15.310 align:start position:0% total number of physical memory that you can<00:05:12.960> see<00:05:13.240> over<00:05:13.400> here<00:05:13.520> is<00:05:13.600> 24<00:05:14.040> gig<00:05:14.720> and<00:05:15.160> I'm 00:05:15.310 --> 00:05:15.320 align:start position:0% can see over here is 24 gig and I'm 00:05:15.320 --> 00:05:17.390 align:start position:0% can see over here is 24 gig and I'm going<00:05:15.480> to<00:05:15.560> be<00:05:15.960> loading<00:05:16.440> the 00:05:17.390 --> 00:05:17.400 align:start position:0% going to be loading the 00:05:17.400 --> 00:05:20.590 align:start position:0% going to be loading the the<00:05:17.520> Qwen<00:05:17.919> model<00:05:18.800> in<00:05:18.919> the<00:05:19.160> LM<00:05:19.440> Studio<00:05:20.320> and<00:05:20.480> I'm 00:05:20.590 --> 00:05:20.600 align:start position:0% the Qwen model in the LM Studio and I'm 00:05:20.600 --> 00:05:22.510 align:start position:0% the Qwen model in the LM Studio and I'm going<00:05:20.720> to<00:05:20.800> set<00:05:21.160> the<00:05:21.360> context<00:05:21.960> length<00:05:22.280> over 00:05:22.510 --> 00:05:22.520 align:start position:0% going to set the context length over 00:05:22.520 --> 00:05:28.630 align:start position:0% going to set the context length over here<00:05:23.480> to<00:05:23.800> at<00:05:23.960> least<00:05:24.400> like<00:05:24.919> a<00:05:24.960> 30<00:05:26.240> 31<00:05:27.200> or<00:05:27.320> 32,000 00:05:28.630 --> 00:05:28.640 align:start position:0% here to at least like a 30 31 or 32,000 00:05:28.640 --> 00:05:30.190 align:start position:0% here to at least like a 30 31 or 32,000 over<00:05:28.880> here.<00:05:29.480> And<00:05:29.600> you<00:05:29.640> can<00:05:29.760> see<00:05:29.840> that<00:05:29.960> this<00:05:30.080> is 00:05:30.190 --> 00:05:30.200 align:start position:0% over here. And you can see that this is 00:05:30.200 --> 00:05:32.270 align:start position:0% over here. And you can see that this is the<00:05:30.680> maximum<00:05:31.160> concurrent<00:05:31.560> prediction. 00:05:32.270 --> 00:05:32.280 align:start position:0% the maximum concurrent prediction. 00:05:32.280 --> 00:05:34.150 align:start position:0% the maximum concurrent prediction. Instead<00:05:32.600> of<00:05:33.000> four,<00:05:33.480> I'm<00:05:33.640> going<00:05:33.760> to<00:05:33.840> be<00:05:33.919> setting 00:05:34.150 --> 00:05:34.160 align:start position:0% Instead of four, I'm going to be setting 00:05:34.160 --> 00:05:35.990 align:start position:0% Instead of four, I'm going to be setting it<00:05:34.280> to<00:05:34.400> one<00:05:35.040> cuz<00:05:35.160> it's<00:05:35.360> going<00:05:35.480> to<00:05:35.560> be<00:05:35.680> reducing 00:05:35.990 --> 00:05:36.000 align:start position:0% it to one cuz it's going to be reducing 00:05:36.000 --> 00:05:38.310 align:start position:0% it to one cuz it's going to be reducing the<00:05:36.080> total<00:05:36.240> number<00:05:36.400> of<00:05:36.480> compute<00:05:36.960> as<00:05:37.160> well.<00:05:37.880> And 00:05:38.310 --> 00:05:38.320 align:start position:0% the total number of compute as well. And 00:05:38.320 --> 00:05:40.790 align:start position:0% the total number of compute as well. And I'm<00:05:38.480> also<00:05:38.800> going<00:05:38.919> to<00:05:39.040> change<00:05:39.520> the<00:05:39.800> KV<00:05:40.280> cache 00:05:40.790 --> 00:05:40.800 align:start position:0% I'm also going to change the KV cache 00:05:40.800 --> 00:05:42.390 align:start position:0% I'm also going to change the KV cache quantization 00:05:42.390 --> 00:05:42.400 align:start position:0% quantization 00:05:42.400 --> 00:05:45.070 align:start position:0% quantization type<00:05:42.760> there.<00:05:43.320> So,<00:05:43.640> instead<00:05:43.960> of<00:05:44.720> using<00:05:44.960> the 00:05:45.070 --> 00:05:45.080 align:start position:0% type there. So, instead of using the 00:05:45.080 --> 00:05:46.470 align:start position:0% type there. So, instead of using the default,<00:05:45.680> I'm<00:05:45.800> going<00:05:45.919> to<00:05:46.000> be<00:05:46.040> setting<00:05:46.320> the 00:05:46.470 --> 00:05:46.480 align:start position:0% default, I'm going to be setting the 00:05:46.480 --> 00:05:48.990 align:start position:0% default, I'm going to be setting the FP16<00:05:47.400> this<00:05:47.560> time.<00:05:48.200> And<00:05:48.560> I'm<00:05:48.680> going<00:05:48.840> to<00:05:48.880> be 00:05:48.990 --> 00:05:49.000 align:start position:0% FP16 this time. And I'm going to be 00:05:49.000 --> 00:05:50.550 align:start position:0% FP16 this time. And I'm going to be loading<00:05:49.320> this<00:05:49.400> particular<00:05:49.680> model.<00:05:50.360> And<00:05:50.480> the 00:05:50.550 --> 00:05:50.560 align:start position:0% loading this particular model. And the 00:05:50.560 --> 00:05:51.830 align:start position:0% loading this particular model. And the moment<00:05:50.920> I'm<00:05:51.040> going<00:05:51.160> to<00:05:51.240> load<00:05:51.440> this<00:05:51.560> particular 00:05:51.830 --> 00:05:51.840 align:start position:0% moment I'm going to load this particular 00:05:51.840 --> 00:05:53.710 align:start position:0% moment I'm going to load this particular model,<00:05:52.440> you<00:05:52.560> can<00:05:52.760> see<00:05:52.840> that<00:05:53.120> it<00:05:53.240> is<00:05:53.360> going<00:05:53.640> to 00:05:53.710 --> 00:05:53.720 align:start position:0% model, you can see that it is going to 00:05:53.720 --> 00:05:57.750 align:start position:0% model, you can see that it is going to be<00:05:53.920> consuming<00:05:54.640> around<00:05:55.200> 20<00:05:55.960> gig<00:05:56.240> or<00:05:56.360> 21<00:05:56.920> gig<00:05:57.520> of 00:05:57.750 --> 00:05:57.760 align:start position:0% be consuming around 20 gig or 21 gig of 00:05:57.760 --> 00:06:00.630 align:start position:0% be consuming around 20 gig or 21 gig of the<00:05:58.120> total<00:05:58.480> memory<00:05:59.480> in<00:05:59.680> my 00:06:00.630 --> 00:06:00.640 align:start position:0% the total memory in my 00:06:00.640 --> 00:06:04.070 align:start position:0% the total memory in my in<00:06:00.840> my<00:06:01.520> RAM<00:06:01.840> there.<00:06:02.440> And<00:06:02.720> now<00:06:03.000> I'm<00:06:03.200> going<00:06:03.440> to<00:06:03.560> be 00:06:04.070 --> 00:06:04.080 align:start position:0% in my RAM there. And now I'm going to be 00:06:04.080 --> 00:06:06.830 align:start position:0% in my RAM there. And now I'm going to be running<00:06:05.000> the<00:06:05.520> the<00:06:05.720> questions<00:06:06.320> over<00:06:06.480> there, 00:06:06.830 --> 00:06:06.840 align:start position:0% running the the questions over there, 00:06:06.840 --> 00:06:08.670 align:start position:0% running the the questions over there, inferencing<00:06:07.440> it<00:06:07.680> without<00:06:08.080> the<00:06:08.240> thinking 00:06:08.670 --> 00:06:08.680 align:start position:0% inferencing it without the thinking 00:06:08.680 --> 00:06:10.630 align:start position:0% inferencing it without the thinking enabled.<00:06:09.480> So<00:06:09.600> the<00:06:09.680> moment<00:06:09.960> I'm<00:06:10.120> going<00:06:10.240> to<00:06:10.320> run, 00:06:10.630 --> 00:06:10.640 align:start position:0% enabled. So the moment I'm going to run, 00:06:10.640 --> 00:06:12.470 align:start position:0% enabled. So the moment I'm going to run, let's<00:06:10.840> say<00:06:11.040> hi<00:06:11.480> over<00:06:11.640> here,<00:06:12.160> I'm<00:06:12.280> getting 00:06:12.470 --> 00:06:12.480 align:start position:0% let's say hi over here, I'm getting 00:06:12.480 --> 00:06:15.870 align:start position:0% let's say hi over here, I'm getting around<00:06:12.800> 4.57<00:06:13.960> tokens<00:06:14.360> per<00:06:14.520> second.<00:06:15.400> And<00:06:15.600> now 00:06:15.870 --> 00:06:15.880 align:start position:0% around 4.57 tokens per second. And now 00:06:15.880 --> 00:06:18.670 align:start position:0% around 4.57 tokens per second. And now if<00:06:16.120> I'm<00:06:16.240> going<00:06:16.360> to<00:06:16.520> say<00:06:16.840> again<00:06:17.560> like<00:06:17.760> hi<00:06:17.919> again, 00:06:18.670 --> 00:06:18.680 align:start position:0% if I'm going to say again like hi again, 00:06:18.680 --> 00:06:21.390 align:start position:0% if I'm going to say again like hi again, it<00:06:18.840> is<00:06:19.000> going<00:06:19.160> to<00:06:19.280> give<00:06:19.480> me<00:06:19.600> around<00:06:20.040> 4.9<00:06:21.000> tokens 00:06:21.390 --> 00:06:21.400 align:start position:0% it is going to give me around 4.9 tokens 00:06:21.400 --> 00:06:23.390 align:start position:0% it is going to give me around 4.9 tokens per<00:06:21.560> second.<00:06:21.960> This<00:06:22.160> is<00:06:22.480> a<00:06:22.600> dense<00:06:23.080> model<00:06:23.280> that 00:06:23.390 --> 00:06:23.400 align:start position:0% per second. This is a dense model that 00:06:23.400 --> 00:06:25.310 align:start position:0% per second. This is a dense model that we're<00:06:23.520> talking<00:06:23.800> about,<00:06:24.080> guys.<00:06:24.400> Look<00:06:24.520> at<00:06:24.640> that. 00:06:25.310 --> 00:06:25.320 align:start position:0% we're talking about, guys. Look at that. 00:06:25.320 --> 00:06:27.150 align:start position:0% we're talking about, guys. Look at that. And<00:06:25.400> now<00:06:25.560> if<00:06:25.680> I'm<00:06:25.800> going<00:06:25.919> to<00:06:26.360> pass<00:06:26.640> a<00:06:26.720> long 00:06:27.150 --> 00:06:27.160 align:start position:0% And now if I'm going to pass a long 00:06:27.160 --> 00:06:30.790 align:start position:0% And now if I'm going to pass a long prompt<00:06:27.520> to<00:06:27.720> write<00:06:28.160> a<00:06:28.360> Selenium<00:06:28.800> C#<00:06:29.760> .NET<00:06:30.160> code 00:06:30.790 --> 00:06:30.800 align:start position:0% prompt to write a Selenium C# .NET code 00:06:30.800 --> 00:06:33.030 align:start position:0% prompt to write a Selenium C# .NET code for<00:06:30.960> Google<00:06:31.400> search<00:06:32.000> operation<00:06:32.520> with<00:06:32.760> page 00:06:33.030 --> 00:06:33.040 align:start position:0% for Google search operation with page 00:06:33.040 --> 00:06:35.510 align:start position:0% for Google search operation with page object<00:06:33.360> model<00:06:33.680> and<00:06:33.800> dependency<00:06:34.240> injection. 00:06:35.510 --> 00:06:35.520 align:start position:0% object model and dependency injection. 00:06:35.520 --> 00:06:36.870 align:start position:0% object model and dependency injection. And<00:06:35.600> if<00:06:35.720> I'm<00:06:35.800> going<00:06:35.919> to<00:06:35.960> run<00:06:36.120> it,<00:06:36.560> I'm<00:06:36.680> going<00:06:36.800> to 00:06:36.870 --> 00:06:36.880 align:start position:0% And if I'm going to run it, I'm going to 00:06:36.880 --> 00:06:39.190 align:start position:0% And if I'm going to run it, I'm going to start<00:06:37.160> the<00:06:37.280> timer<00:06:37.720> as<00:06:37.919> well<00:06:38.160> just<00:06:38.360> in<00:06:38.480> case. 00:06:39.190 --> 00:06:39.200 align:start position:0% start the timer as well just in case. 00:06:39.200 --> 00:06:40.590 align:start position:0% start the timer as well just in case. And<00:06:39.560> if<00:06:39.680> I'm<00:06:39.760> going<00:06:39.880> to<00:06:39.960> show<00:06:40.080> you<00:06:40.160> the<00:06:40.280> memory 00:06:40.590 --> 00:06:40.600 align:start position:0% And if I'm going to show you the memory 00:06:40.600 --> 00:06:42.190 align:start position:0% And if I'm going to show you the memory spike<00:06:40.960> there,<00:06:41.440> you<00:06:41.520> see<00:06:41.600> that<00:06:41.760> the<00:06:41.840> memory 00:06:42.190 --> 00:06:42.200 align:start position:0% spike there, you see that the memory 00:06:42.200 --> 00:06:43.870 align:start position:0% spike there, you see that the memory pressure<00:06:42.560> is<00:06:42.720> not<00:06:42.919> increasing<00:06:43.560> because<00:06:43.760> it's 00:06:43.870 --> 00:06:43.880 align:start position:0% pressure is not increasing because it's 00:06:43.880 --> 00:06:46.950 align:start position:0% pressure is not increasing because it's a<00:06:43.960> two-bit<00:06:44.320> quantization<00:06:44.960> model.<00:06:45.600> And<00:06:46.800> at<00:06:46.880> the 00:06:46.950 --> 00:06:46.960 align:start position:0% a two-bit quantization model. And at the 00:06:46.960 --> 00:06:48.870 align:start position:0% a two-bit quantization model. And at the same<00:06:47.240> time,<00:06:47.880> you<00:06:48.000> can<00:06:48.160> see<00:06:48.280> that<00:06:48.440> I'm<00:06:48.600> actually 00:06:48.870 --> 00:06:48.880 align:start position:0% same time, you can see that I'm actually 00:06:48.880 --> 00:06:50.790 align:start position:0% same time, you can see that I'm actually getting<00:06:49.080> the<00:06:49.280> output<00:06:49.720> over<00:06:49.880> here.<00:06:50.120> It<00:06:50.240> is<00:06:50.400> not 00:06:50.790 --> 00:06:50.800 align:start position:0% getting the output over here. It is not 00:06:50.800 --> 00:06:53.350 align:start position:0% getting the output over here. It is not as<00:06:51.080> fast<00:06:51.440> as<00:06:51.600> you<00:06:51.720> can<00:06:51.919> imagine.<00:06:52.560> It<00:06:52.720> is<00:06:52.880> not 00:06:53.350 --> 00:06:53.360 align:start position:0% as fast as you can imagine. It is not 00:06:53.360 --> 00:06:54.990 align:start position:0% as fast as you can imagine. It is not super<00:06:53.680> fast<00:06:53.919> like<00:06:54.080> how<00:06:54.200> you<00:06:54.280> can<00:06:54.400> expect<00:06:54.840> with 00:06:54.990 --> 00:06:55.000 align:start position:0% super fast like how you can expect with 00:06:55.000 --> 00:06:57.830 align:start position:0% super fast like how you can expect with a<00:06:55.480> like<00:06:55.640> a<00:06:55.720> M5<00:06:56.200> Max<00:06:56.520> machine.<00:06:57.280> But<00:06:57.520> you<00:06:57.600> can<00:06:57.760> see 00:06:57.830 --> 00:06:57.840 align:start position:0% a like a M5 Max machine. But you can see 00:06:57.840 --> 00:07:00.070 align:start position:0% a like a M5 Max machine. But you can see that<00:06:57.960> within<00:06:58.200> 4<00:06:58.480> minutes,<00:06:59.040> I<00:06:59.160> got<00:06:59.400> the<00:06:59.600> output 00:07:00.070 --> 00:07:00.080 align:start position:0% that within 4 minutes, I got the output 00:07:00.080 --> 00:07:03.470 align:start position:0% that within 4 minutes, I got the output with<00:07:00.480> a<00:07:00.600> 5.67<00:07:01.919> tokens<00:07:02.320> per<00:07:02.480> second.<00:07:03.120> This<00:07:03.320> is 00:07:03.470 --> 00:07:03.480 align:start position:0% with a 5.67 tokens per second. This is 00:07:03.480 --> 00:07:06.030 align:start position:0% with a 5.67 tokens per second. This is not<00:07:03.760> bad<00:07:04.160> to<00:07:04.240> be<00:07:04.400> honest.<00:07:04.720> This<00:07:04.840> is<00:07:05.360> working. 00:07:06.030 --> 00:07:06.040 align:start position:0% not bad to be honest. This is working. 00:07:06.040 --> 00:07:07.270 align:start position:0% not bad to be honest. This is working. It's<00:07:06.200> not<00:07:06.360> really<00:07:06.560> hitting<00:07:06.800> that<00:07:06.960> memory 00:07:07.270 --> 00:07:07.280 align:start position:0% It's not really hitting that memory 00:07:07.280 --> 00:07:08.950 align:start position:0% It's not really hitting that memory ceiling.<00:07:08.040> And<00:07:08.240> if<00:07:08.360> you<00:07:08.480> try<00:07:08.640> to<00:07:08.760> run<00:07:08.880> a 00:07:08.950 --> 00:07:08.960 align:start position:0% ceiling. And if you try to run a 00:07:08.960 --> 00:07:11.070 align:start position:0% ceiling. And if you try to run a four-bit<00:07:09.320> quantization,<00:07:10.080> this<00:07:10.280> is<00:07:10.440> never 00:07:11.070 --> 00:07:11.080 align:start position:0% four-bit quantization, this is never 00:07:11.080 --> 00:07:12.909 align:start position:0% four-bit quantization, this is never going<00:07:11.320> to<00:07:11.400> be<00:07:11.480> running<00:07:11.800> for<00:07:12.040> you.<00:07:12.480> And<00:07:12.680> 4 00:07:12.909 --> 00:07:12.919 align:start position:0% going to be running for you. And 4 00:07:12.919 --> 00:07:15.030 align:start position:0% going to be running for you. And 4 minutes<00:07:13.200> is<00:07:13.400> not<00:07:13.600> that<00:07:13.760> bad<00:07:13.960> at<00:07:14.080> all.<00:07:14.760> Now<00:07:14.919> I'm 00:07:15.030 --> 00:07:15.040 align:start position:0% minutes is not that bad at all. Now I'm 00:07:15.040 --> 00:07:17.350 align:start position:0% minutes is not that bad at all. Now I'm going<00:07:15.160> to<00:07:15.240> change<00:07:15.520> the<00:07:15.640> quantization<00:07:16.480> to 00:07:17.350 --> 00:07:17.360 align:start position:0% going to change the quantization to 00:07:17.360 --> 00:07:18.950 align:start position:0% going to change the quantization to eight,<00:07:17.760> which<00:07:17.880> is<00:07:17.960> a<00:07:18.000> quantization<00:07:18.680> eight 00:07:18.950 --> 00:07:18.960 align:start position:0% eight, which is a quantization eight 00:07:18.960 --> 00:07:21.350 align:start position:0% eight, which is a quantization eight there<00:07:19.200> for<00:07:19.320> the<00:07:19.440> KV<00:07:19.800> cache<00:07:20.120> alone.<00:07:20.840> And<00:07:21.240> I'm 00:07:21.350 --> 00:07:21.360 align:start position:0% there for the KV cache alone. And I'm 00:07:21.360 --> 00:07:22.350 align:start position:0% there for the KV cache alone. And I'm going<00:07:21.480> to<00:07:21.560> be<00:07:21.680> loading<00:07:22.000> this<00:07:22.080> particular 00:07:22.350 --> 00:07:22.360 align:start position:0% going to be loading this particular 00:07:22.360 --> 00:07:25.150 align:start position:0% going to be loading this particular model.<00:07:23.040> And<00:07:23.400> I'm<00:07:23.560> going<00:07:23.840> to<00:07:24.040> ask<00:07:24.400> the<00:07:24.520> same 00:07:25.150 --> 00:07:25.160 align:start position:0% model. And I'm going to ask the same 00:07:25.160 --> 00:07:28.070 align:start position:0% model. And I'm going to ask the same exact<00:07:25.919> prompt<00:07:26.360> there.<00:07:27.080> Sending<00:07:27.520> a<00:07:27.720> hi 00:07:28.070 --> 00:07:28.080 align:start position:0% exact prompt there. Sending a hi 00:07:28.080 --> 00:07:31.230 align:start position:0% exact prompt there. Sending a hi message.<00:07:28.840> And<00:07:29.680> going<00:07:29.840> to<00:07:29.919> say<00:07:30.120> hi<00:07:30.440> again.<00:07:31.080> I'm 00:07:31.230 --> 00:07:31.240 align:start position:0% message. And going to say hi again. I'm 00:07:31.240 --> 00:07:33.430 align:start position:0% message. And going to say hi again. I'm getting<00:07:31.520> 5.47<00:07:32.600> token.<00:07:33.080> And<00:07:33.200> I'm<00:07:33.240> going<00:07:33.360> to 00:07:33.430 --> 00:07:33.440 align:start position:0% getting 5.47 token. And I'm going to 00:07:33.440 --> 00:07:35.190 align:start position:0% getting 5.47 token. And I'm going to send<00:07:33.640> the<00:07:33.720> same<00:07:34.000> exact<00:07:34.440> Selenium<00:07:34.880> code 00:07:35.190 --> 00:07:35.200 align:start position:0% send the same exact Selenium code 00:07:35.200 --> 00:07:37.630 align:start position:0% send the same exact Selenium code prompt.<00:07:35.960> And<00:07:36.040> I'm<00:07:36.120> going<00:07:36.240> to<00:07:36.320> start<00:07:36.640> the<00:07:36.800> timer 00:07:37.630 --> 00:07:37.640 align:start position:0% prompt. And I'm going to start the timer 00:07:37.640 --> 00:07:39.630 align:start position:0% prompt. And I'm going to start the timer over<00:07:37.800> there.<00:07:38.400> You<00:07:38.520> can<00:07:38.720> see<00:07:38.880> that<00:07:39.120> this<00:07:39.280> time 00:07:39.630 --> 00:07:39.640 align:start position:0% over there. You can see that this time 00:07:39.640 --> 00:07:42.070 align:start position:0% over there. You can see that this time the<00:07:39.800> time<00:07:40.080> is<00:07:40.200> going<00:07:40.400> to<00:07:40.480> be<00:07:40.600> a<00:07:40.640> bit<00:07:41.360> higher 00:07:42.070 --> 00:07:42.080 align:start position:0% the time is going to be a bit higher 00:07:42.080 --> 00:07:44.310 align:start position:0% the time is going to be a bit higher than<00:07:42.360> compared<00:07:42.680> to<00:07:42.720> the<00:07:42.840> FP16<00:07:43.600> there.<00:07:44.200> The 00:07:44.310 --> 00:07:44.320 align:start position:0% than compared to the FP16 there. The 00:07:44.320 --> 00:07:48.390 align:start position:0% than compared to the FP16 there. The reason<00:07:44.840> is<00:07:45.080> because<00:07:45.960> actually<00:07:46.760> it's<00:07:47.080> not<00:07:47.520> that 00:07:48.390 --> 00:07:48.400 align:start position:0% reason is because actually it's not that 00:07:48.400 --> 00:07:49.750 align:start position:0% reason is because actually it's not that even<00:07:48.720> if<00:07:48.840> you<00:07:49.040> try<00:07:49.280> to<00:07:49.400> reduce<00:07:49.680> the 00:07:49.750 --> 00:07:49.760 align:start position:0% even if you try to reduce the 00:07:49.760 --> 00:07:52.510 align:start position:0% even if you try to reduce the quantization,<00:07:50.960> the<00:07:51.120> things<00:07:51.440> will<00:07:51.600> be<00:07:51.760> faster. 00:07:52.510 --> 00:07:52.520 align:start position:0% quantization, the things will be faster. 00:07:52.520 --> 00:07:54.870 align:start position:0% quantization, the things will be faster. The<00:07:52.680> inferencing<00:07:53.400> will<00:07:53.640> exactly<00:07:54.240> be<00:07:54.360> the<00:07:54.480> same 00:07:54.870 --> 00:07:54.880 align:start position:0% The inferencing will exactly be the same 00:07:54.880 --> 00:07:57.310 align:start position:0% The inferencing will exactly be the same exact<00:07:55.240> tokens<00:07:55.560> per<00:07:55.720> second<00:07:56.360> because<00:07:56.680> it<00:07:56.880> also 00:07:57.310 --> 00:07:57.320 align:start position:0% exact tokens per second because it also 00:07:57.320 --> 00:08:01.470 align:start position:0% exact tokens per second because it also need<00:07:57.680> to<00:07:58.400> dequantize<00:07:59.320> and<00:07:59.560> unpack<00:08:00.360> the 00:08:01.470 --> 00:08:01.480 align:start position:0% need to dequantize and unpack the 00:08:01.480 --> 00:08:04.070 align:start position:0% need to dequantize and unpack the the<00:08:01.600> quantization<00:08:02.360> for<00:08:02.600> you<00:08:03.200> and<00:08:03.480> then<00:08:03.800> it<00:08:03.960> is 00:08:04.070 --> 00:08:04.080 align:start position:0% the quantization for you and then it is 00:08:04.080 --> 00:08:05.350 align:start position:0% the quantization for you and then it is going<00:08:04.360> to<00:08:04.440> be<00:08:04.520> doing<00:08:04.760> some<00:08:04.920> compute 00:08:05.350 --> 00:08:05.360 align:start position:0% going to be doing some compute 00:08:05.360 --> 00:08:07.270 align:start position:0% going to be doing some compute operation.<00:08:05.880> That's<00:08:06.120> the<00:08:06.160> reason<00:08:06.520> why 00:08:07.270 --> 00:08:07.280 align:start position:0% operation. That's the reason why 00:08:07.280 --> 00:08:09.990 align:start position:0% operation. That's the reason why actually<00:08:07.600> the<00:08:07.720> performance<00:08:08.680> goes<00:08:09.120> slower. 00:08:09.990 --> 00:08:10.000 align:start position:0% actually the performance goes slower. 00:08:10.000 --> 00:08:11.830 align:start position:0% actually the performance goes slower. the<00:08:10.280> the<00:08:10.400> total<00:08:10.640> number<00:08:10.880> of<00:08:11.040> time<00:08:11.360> it<00:08:11.560> takes 00:08:11.830 --> 00:08:11.840 align:start position:0% the the total number of time it takes 00:08:11.840 --> 00:08:14.270 align:start position:0% the the total number of time it takes for<00:08:11.960> the<00:08:12.040> same<00:08:12.320> exact<00:08:12.800> prompt<00:08:13.520> is<00:08:13.760> going<00:08:14.000> to<00:08:14.080> be 00:08:14.270 --> 00:08:14.280 align:start position:0% for the same exact prompt is going to be 00:08:14.280 --> 00:08:18.750 align:start position:0% for the same exact prompt is going to be higher<00:08:15.040> than<00:08:15.320> compared<00:08:15.840> to<00:08:16.440> the<00:08:16.960> FP16<00:08:18.320> for<00:08:18.600> you 00:08:18.750 --> 00:08:18.760 align:start position:0% higher than compared to the FP16 for you 00:08:18.760 --> 00:08:21.310 align:start position:0% higher than compared to the FP16 for you over<00:08:18.919> there.<00:08:19.160> See<00:08:19.320> now,<00:08:19.880> this<00:08:20.040> is<00:08:20.280> going<00:08:20.520> to<00:08:20.600> be 00:08:21.310 --> 00:08:21.320 align:start position:0% over there. See now, this is going to be 00:08:21.320 --> 00:08:24.310 align:start position:0% over there. See now, this is going to be the<00:08:21.640> FP4<00:08:23.080> that<00:08:23.280> we<00:08:23.400> have<00:08:23.640> got.<00:08:24.000> I'm<00:08:24.120> going<00:08:24.240> to 00:08:24.310 --> 00:08:24.320 align:start position:0% the FP4 that we have got. I'm going to 00:08:24.320 --> 00:08:25.590 align:start position:0% the FP4 that we have got. I'm going to be<00:08:24.360> asking<00:08:24.600> the<00:08:24.680> same<00:08:24.880> exact<00:08:25.160> question.<00:08:25.520> The 00:08:25.590 --> 00:08:25.600 align:start position:0% be asking the same exact question. The 00:08:25.600 --> 00:08:27.110 align:start position:0% be asking the same exact question. The total<00:08:25.840> number<00:08:26.000> of<00:08:26.120> tokens<00:08:26.400> per<00:08:26.520> second<00:08:26.760> is<00:08:26.880> not 00:08:27.110 --> 00:08:27.120 align:start position:0% total number of tokens per second is not 00:08:27.120 --> 00:08:28.990 align:start position:0% total number of tokens per second is not changing<00:08:27.480> at<00:08:27.640> all.<00:08:28.160> And<00:08:28.320> now<00:08:28.520> if<00:08:28.680> I'm<00:08:28.800> going<00:08:28.919> to 00:08:28.990 --> 00:08:29.000 align:start position:0% changing at all. And now if I'm going to 00:08:29.000 --> 00:08:30.909 align:start position:0% changing at all. And now if I'm going to be<00:08:29.040> running<00:08:29.280> the<00:08:29.400> same<00:08:29.720> exact<00:08:30.360> prompt,<00:08:30.800> the 00:08:30.909 --> 00:08:30.919 align:start position:0% be running the same exact prompt, the 00:08:30.919 --> 00:08:33.350 align:start position:0% be running the same exact prompt, the Selenium<00:08:31.560> WebDriver<00:08:32.159> question<00:08:32.560> over<00:08:32.760> there 00:08:33.350 --> 00:08:33.360 align:start position:0% Selenium WebDriver question over there 00:08:33.360 --> 00:08:34.350 align:start position:0% Selenium WebDriver question over there with<00:08:33.520> the 00:08:34.350 --> 00:08:34.360 align:start position:0% with the 00:08:34.360 --> 00:08:36.830 align:start position:0% with the quantization<00:08:35.080> four<00:08:35.680> there,<00:08:36.320> instead<00:08:36.560> of<00:08:36.680> the 00:08:36.830 --> 00:08:36.840 align:start position:0% quantization four there, instead of the 00:08:36.840 --> 00:08:38.390 align:start position:0% quantization four there, instead of the 4.04 00:08:38.390 --> 00:08:38.400 align:start position:0% 4.04 00:08:38.400 --> 00:08:41.550 align:start position:0% 4.04 minutes<00:08:39.039> which<00:08:39.240> was<00:08:39.400> taking<00:08:39.680> in<00:08:40.200> FP16,<00:08:41.280> it<00:08:41.400> is 00:08:41.550 --> 00:08:41.560 align:start position:0% minutes which was taking in FP16, it is 00:08:41.560 --> 00:08:44.430 align:start position:0% minutes which was taking in FP16, it is 4.46<00:08:42.960> as<00:08:43.159> you<00:08:43.240> can<00:08:43.440> see<00:08:43.640> here.<00:08:43.919> So,<00:08:44.039> based<00:08:44.320> on 00:08:44.430 --> 00:08:44.440 align:start position:0% 4.46 as you can see here. So, based on 00:08:44.440 --> 00:08:46.910 align:start position:0% 4.46 as you can see here. So, based on the<00:08:44.520> quantization<00:08:45.240> change,<00:08:46.160> it<00:08:46.280> is<00:08:46.440> also 00:08:46.910 --> 00:08:46.920 align:start position:0% the quantization change, it is also 00:08:46.920 --> 00:08:48.510 align:start position:0% the quantization change, it is also obviously<00:08:47.360> going<00:08:47.480> to<00:08:47.560> change<00:08:47.880> the<00:08:48.040> time<00:08:48.360> over 00:08:48.510 --> 00:08:48.520 align:start position:0% obviously going to change the time over 00:08:48.520 --> 00:08:51.030 align:start position:0% obviously going to change the time over there.<00:08:49.040> So,<00:08:49.200> this<00:08:49.480> proves<00:08:49.880> the<00:08:50.040> point<00:08:50.520> that 00:08:51.030 --> 00:08:51.040 align:start position:0% there. So, this proves the point that 00:08:51.040 --> 00:08:53.430 align:start position:0% there. So, this proves the point that the<00:08:51.160> inferencing<00:08:51.839> actually<00:08:52.240> works<00:08:52.760> even<00:08:53.120> in 00:08:53.430 --> 00:08:53.440 align:start position:0% the inferencing actually works even in 00:08:53.440 --> 00:08:55.310 align:start position:0% the inferencing actually works even in LM<00:08:53.720> Studio<00:08:54.160> for<00:08:54.360> us<00:08:54.720> and<00:08:54.880> you<00:08:54.960> can<00:08:55.080> see<00:08:55.200> that 00:08:55.310 --> 00:08:55.320 align:start position:0% LM Studio for us and you can see that 00:08:55.320 --> 00:08:57.310 align:start position:0% LM Studio for us and you can see that the<00:08:55.440> output<00:08:55.839> has<00:08:56.080> got<00:08:56.360> no<00:08:56.520> compromise<00:08:57.160> as 00:08:57.310 --> 00:08:57.320 align:start position:0% the output has got no compromise as 00:08:57.320 --> 00:08:58.790 align:start position:0% the output has got no compromise as well.<00:08:57.760> The<00:08:57.880> output<00:08:58.200> looks<00:08:58.440> pretty<00:08:58.640> much 00:08:58.790 --> 00:08:58.800 align:start position:0% well. The output looks pretty much 00:08:58.800 --> 00:09:00.670 align:start position:0% well. The output looks pretty much exactly<00:08:59.280> the<00:08:59.400> same<00:08:59.760> and<00:08:59.880> I<00:08:59.960> can<00:09:00.160> see<00:09:00.280> that<00:09:00.560> it 00:09:00.670 --> 00:09:00.680 align:start position:0% exactly the same and I can see that it 00:09:00.680 --> 00:09:02.070 align:start position:0% exactly the same and I can see that it is<00:09:00.760> still<00:09:01.000> working.<00:09:01.560> I<00:09:01.600> mean<00:09:01.720> there<00:09:01.839> is<00:09:01.960> a 00:09:02.070 --> 00:09:02.080 align:start position:0% is still working. I mean there is a 00:09:02.080 --> 00:09:03.590 align:start position:0% is still working. I mean there is a change<00:09:02.480> in<00:09:02.560> the<00:09:02.680> output<00:09:03.040> of<00:09:03.160> course<00:09:03.440> in<00:09:03.520> the 00:09:03.590 --> 00:09:03.600 align:start position:0% change in the output of course in the 00:09:03.600 --> 00:09:05.630 align:start position:0% change in the output of course in the WebDriver<00:09:04.680> for<00:09:04.800> the<00:09:04.880> dependence<00:09:05.280> injection. 00:09:05.630 --> 00:09:05.640 align:start position:0% WebDriver for the dependence injection. 00:09:05.640 --> 00:09:07.950 align:start position:0% WebDriver for the dependence injection. It's<00:09:05.800> there<00:09:06.120> over<00:09:06.360> here.<00:09:06.960> But<00:09:07.120> in<00:09:07.240> here,<00:09:07.839> you 00:09:07.950 --> 00:09:07.960 align:start position:0% It's there over here. But in here, you 00:09:07.960 --> 00:09:09.230 align:start position:0% It's there over here. But in here, you can<00:09:08.120> see<00:09:08.200> that<00:09:08.320> the<00:09:08.360> dependence<00:09:08.800> injection 00:09:09.230 --> 00:09:09.240 align:start position:0% can see that the dependence injection 00:09:09.240 --> 00:09:11.390 align:start position:0% can see that the dependence injection setup<00:09:09.720> has<00:09:10.000> got<00:09:10.200> a<00:09:10.240> bit<00:09:10.680> different<00:09:11.120> with<00:09:11.240> like 00:09:11.390 --> 00:09:11.400 align:start position:0% setup has got a bit different with like 00:09:11.400 --> 00:09:13.590 align:start position:0% setup has got a bit different with like driver<00:09:11.720> factory<00:09:12.200> class<00:09:12.480> there.<00:09:13.040> That's<00:09:13.240> the 00:09:13.590 --> 00:09:13.600 align:start position:0% driver factory class there. That's the 00:09:13.600 --> 00:09:15.310 align:start position:0% driver factory class there. That's the only<00:09:13.839> difference<00:09:14.160> between<00:09:14.440> both<00:09:14.680> of<00:09:14.800> them. 00:09:15.310 --> 00:09:15.320 align:start position:0% only difference between both of them. 00:09:15.320 --> 00:09:17.870 align:start position:0% only difference between both of them. But<00:09:15.920> overall,<00:09:16.640> I<00:09:16.720> can<00:09:16.960> actually<00:09:17.320> see<00:09:17.520> that<00:09:17.800> it 00:09:17.870 --> 00:09:17.880 align:start position:0% But overall, I can actually see that it 00:09:17.880 --> 00:09:20.270 align:start position:0% But overall, I can actually see that it is<00:09:18.120> working<00:09:18.680> pretty<00:09:18.880> much<00:09:19.160> as<00:09:19.320> expected.<00:09:20.200> The 00:09:20.270 --> 00:09:20.280 align:start position:0% is working pretty much as expected. The 00:09:20.280 --> 00:09:22.750 align:start position:0% is working pretty much as expected. The last<00:09:20.720> test<00:09:20.880> which<00:09:21.040> I<00:09:21.120> wanted<00:09:21.400> to<00:09:21.560> do<00:09:22.080> with<00:09:22.400> the 00:09:22.750 --> 00:09:22.760 align:start position:0% last test which I wanted to do with the 00:09:22.760 --> 00:09:25.710 align:start position:0% last test which I wanted to do with the FP16<00:09:23.920> of<00:09:24.040> the<00:09:24.120> same<00:09:24.360> exact<00:09:24.760> model<00:09:25.320> is<00:09:25.520> by 00:09:25.710 --> 00:09:25.720 align:start position:0% FP16 of the same exact model is by 00:09:25.720 --> 00:09:29.110 align:start position:0% FP16 of the same exact model is by connecting<00:09:26.600> to<00:09:27.080> a<00:09:27.480> pie<00:09:27.760> coding<00:09:28.240> agent.<00:09:28.839> I<00:09:28.920> want 00:09:29.110 --> 00:09:29.120 align:start position:0% connecting to a pie coding agent. I want 00:09:29.120 --> 00:09:30.750 align:start position:0% connecting to a pie coding agent. I want to<00:09:29.200> really<00:09:29.440> see<00:09:29.800> if<00:09:29.920> this<00:09:30.080> is<00:09:30.200> going<00:09:30.320> to<00:09:30.440> work 00:09:30.750 --> 00:09:30.760 align:start position:0% to really see if this is going to work 00:09:30.760 --> 00:09:33.630 align:start position:0% to really see if this is going to work or<00:09:30.880> not.<00:09:31.160> So,<00:09:31.640> I<00:09:31.720> have<00:09:31.880> set<00:09:32.120> the<00:09:32.640> FP16<00:09:33.400> over 00:09:33.630 --> 00:09:33.640 align:start position:0% or not. So, I have set the FP16 over 00:09:33.640 --> 00:09:36.470 align:start position:0% or not. So, I have set the FP16 over here.<00:09:34.280> I'm<00:09:34.520> running<00:09:34.800> it<00:09:35.320> and<00:09:35.760> I'm<00:09:35.920> going<00:09:36.120> to<00:09:36.200> be 00:09:36.470 --> 00:09:36.480 align:start position:0% here. I'm running it and I'm going to be 00:09:36.480 --> 00:09:39.030 align:start position:0% here. I'm running it and I'm going to be opening<00:09:36.880> the<00:09:37.120> pie<00:09:37.360> coding<00:09:37.800> agent<00:09:38.240> over<00:09:38.440> here 00:09:39.030 --> 00:09:39.040 align:start position:0% opening the pie coding agent over here 00:09:39.040 --> 00:09:40.950 align:start position:0% opening the pie coding agent over here and<00:09:39.160> I'm<00:09:39.240> going<00:09:39.360> to<00:09:39.440> choose<00:09:39.840> the<00:09:40.000> model.<00:09:40.800> As 00:09:40.950 --> 00:09:40.960 align:start position:0% and I'm going to choose the model. As 00:09:40.960 --> 00:09:42.190 align:start position:0% and I'm going to choose the model. As you<00:09:41.040> can<00:09:41.240> see<00:09:41.360> I'm<00:09:41.440> going<00:09:41.560> to<00:09:41.640> be<00:09:41.680> chasing 00:09:42.190 --> 00:09:42.200 align:start position:0% you can see I'm going to be chasing 00:09:42.200 --> 00:09:45.830 align:start position:0% you can see I'm going to be chasing choosing<00:09:42.520> that<00:09:42.920> GSQRCO<00:09:44.240> onslaught<00:09:45.000> MTP<00:09:45.520> model 00:09:45.830 --> 00:09:45.840 align:start position:0% choosing that GSQRCO onslaught MTP model 00:09:45.840 --> 00:09:46.950 align:start position:0% choosing that GSQRCO onslaught MTP model there. 00:09:46.950 --> 00:09:46.960 align:start position:0% there. 00:09:46.960 --> 00:09:49.510 align:start position:0% there. And<00:09:47.880> I'm<00:09:48.120> going<00:09:48.320> to<00:09:48.360> be<00:09:48.520> just<00:09:48.720> saying<00:09:49.040> hi<00:09:49.360> over 00:09:49.510 --> 00:09:49.520 align:start position:0% And I'm going to be just saying hi over 00:09:49.520 --> 00:09:51.790 align:start position:0% And I'm going to be just saying hi over there.<00:09:50.160> Even<00:09:50.480> for<00:09:50.640> a<00:09:50.720> high<00:09:51.040> message<00:09:51.440> it<00:09:51.600> was 00:09:51.790 --> 00:09:51.800 align:start position:0% there. Even for a high message it was 00:09:51.800 --> 00:09:54.190 align:start position:0% there. Even for a high message it was taking<00:09:52.200> a<00:09:52.320> long<00:09:52.760> time.<00:09:53.080> It's<00:09:53.280> not<00:09:53.520> very<00:09:53.760> fast 00:09:54.190 --> 00:09:54.200 align:start position:0% taking a long time. It's not very fast 00:09:54.200 --> 00:09:55.830 align:start position:0% taking a long time. It's not very fast as<00:09:54.320> you<00:09:54.440> can<00:09:54.560> imagine. 00:09:55.830 --> 00:09:55.840 align:start position:0% as you can imagine. 00:09:55.840 --> 00:09:58.270 align:start position:0% as you can imagine. It<00:09:55.960> was<00:09:56.160> taking<00:09:56.880> a<00:09:56.960> long<00:09:57.240> time<00:09:57.560> and<00:09:57.680> the<00:09:57.800> output 00:09:58.270 --> 00:09:58.280 align:start position:0% It was taking a long time and the output 00:09:58.280 --> 00:09:59.790 align:start position:0% It was taking a long time and the output you<00:09:58.360> can<00:09:58.520> see<00:09:58.640> that<00:09:58.920> it<00:09:59.040> is<00:09:59.440> going<00:09:59.600> to<00:09:59.680> be 00:09:59.790 --> 00:09:59.800 align:start position:0% you can see that it is going to be 00:09:59.800 --> 00:10:02.430 align:start position:0% you can see that it is going to be giving<00:10:00.080> you<00:10:00.640> very<00:10:00.880> very<00:10:01.080> slow<00:10:01.320> here.<00:10:01.840> Which<00:10:02.000> is 00:10:02.430 --> 00:10:02.440 align:start position:0% giving you very very slow here. Which is 00:10:02.440 --> 00:10:04.110 align:start position:0% giving you very very slow here. Which is quite<00:10:02.720> expected<00:10:03.240> because<00:10:03.440> this<00:10:03.600> is<00:10:03.720> a<00:10:03.760> dense 00:10:04.110 --> 00:10:04.120 align:start position:0% quite expected because this is a dense 00:10:04.120 --> 00:10:06.550 align:start position:0% quite expected because this is a dense model.<00:10:04.720> And<00:10:04.880> I<00:10:05.000> now<00:10:05.120> have<00:10:05.320> told<00:10:05.600> it<00:10:05.720> to<00:10:06.080> analyze 00:10:06.550 --> 00:10:06.560 align:start position:0% model. And I now have told it to analyze 00:10:06.560 --> 00:10:08.590 align:start position:0% model. And I now have told it to analyze the<00:10:06.760> code<00:10:07.720> for<00:10:07.960> me<00:10:08.160> this<00:10:08.320> particular 00:10:08.590 --> 00:10:08.600 align:start position:0% the code for me this particular 00:10:08.600 --> 00:10:13.630 align:start position:0% the code for me this particular repository<00:10:09.600> and<00:10:09.880> that<00:10:10.200> took<00:10:10.400> me<00:10:10.920> more<00:10:11.320> than<00:10:12.200> 27 00:10:13.630 --> 00:10:13.640 align:start position:0% repository and that took me more than 27 00:10:13.640 --> 00:10:15.750 align:start position:0% repository and that took me more than 27 minutes<00:10:14.280> just<00:10:14.600> to<00:10:14.760> analyze<00:10:15.120> the<00:10:15.240> code.<00:10:15.440> I<00:10:15.520> mean 00:10:15.750 --> 00:10:15.760 align:start position:0% minutes just to analyze the code. I mean 00:10:15.760 --> 00:10:17.830 align:start position:0% minutes just to analyze the code. I mean it<00:10:15.920> went<00:10:16.240> so<00:10:16.520> deep<00:10:16.880> to<00:10:17.000> be<00:10:17.120> honest<00:10:17.400> like<00:10:17.560> going 00:10:17.830 --> 00:10:17.840 align:start position:0% it went so deep to be honest like going 00:10:17.840 --> 00:10:19.590 align:start position:0% it went so deep to be honest like going through<00:10:18.000> each<00:10:18.160> and<00:10:18.280> every<00:10:18.520> file<00:10:19.160> even<00:10:19.360> start 00:10:19.590 --> 00:10:19.600 align:start position:0% through each and every file even start 00:10:19.600 --> 00:10:20.830 align:start position:0% through each and every file even start to<00:10:19.720> figure<00:10:19.960> out<00:10:20.160> what<00:10:20.320> exactly<00:10:20.640> is<00:10:20.720> the 00:10:20.830 --> 00:10:20.840 align:start position:0% to figure out what exactly is the 00:10:20.840 --> 00:10:22.830 align:start position:0% to figure out what exactly is the problem<00:10:21.320> in<00:10:21.440> the<00:10:21.560> particular<00:10:21.880> code.<00:10:22.520> But 00:10:22.830 --> 00:10:22.840 align:start position:0% problem in the particular code. But 00:10:22.840 --> 00:10:25.430 align:start position:0% problem in the particular code. But actually<00:10:23.120> took<00:10:23.360> 27<00:10:24.000> minutes<00:10:24.680> to<00:10:24.880> give<00:10:25.120> me<00:10:25.240> the 00:10:25.430 --> 00:10:25.440 align:start position:0% actually took 27 minutes to give me the 00:10:25.440 --> 00:10:27.510 align:start position:0% actually took 27 minutes to give me the output<00:10:26.120> which<00:10:26.320> is<00:10:26.600> what<00:10:26.840> really<00:10:27.080> happened 00:10:27.510 --> 00:10:27.520 align:start position:0% output which is what really happened 00:10:27.520 --> 00:10:29.830 align:start position:0% output which is what really happened over<00:10:27.680> here<00:10:27.880> as<00:10:28.040> you<00:10:28.120> can<00:10:28.320> see.<00:10:29.120> So<00:10:29.280> long<00:10:29.520> story 00:10:29.830 --> 00:10:29.840 align:start position:0% over here as you can see. So long story 00:10:29.840 --> 00:10:32.150 align:start position:0% over here as you can see. So long story short,<00:10:30.280> you<00:10:30.480> can<00:10:30.960> run<00:10:31.440> a<00:10:31.840> two-bit 00:10:32.150 --> 00:10:32.160 align:start position:0% short, you can run a two-bit 00:10:32.160 --> 00:10:35.870 align:start position:0% short, you can run a two-bit quantization<00:10:33.080> Qwen<00:10:33.360> 3.8<00:10:34.520> 27<00:10:35.440> billion 00:10:35.870 --> 00:10:35.880 align:start position:0% quantization Qwen 3.8 27 billion 00:10:35.880 --> 00:10:39.790 align:start position:0% quantization Qwen 3.8 27 billion parameter<00:10:36.440> model<00:10:37.320> in<00:10:37.640> an<00:10:38.160> Apple<00:10:38.720> M4<00:10:39.280> Mac<00:10:39.600> mini 00:10:39.790 --> 00:10:39.800 align:start position:0% parameter model in an Apple M4 Mac mini 00:10:39.800 --> 00:10:41.310 align:start position:0% parameter model in an Apple M4 Mac mini machine<00:10:40.240> without<00:10:40.600> hitting<00:10:40.839> the<00:10:40.960> memory 00:10:41.310 --> 00:10:41.320 align:start position:0% machine without hitting the memory 00:10:41.320 --> 00:10:43.150 align:start position:0% machine without hitting the memory ceiling.<00:10:41.760> But<00:10:42.000> what<00:10:42.160> is<00:10:42.280> the<00:10:42.760> actual 00:10:43.150 --> 00:10:43.160 align:start position:0% ceiling. But what is the actual 00:10:43.160 --> 00:10:46.070 align:start position:0% ceiling. But what is the actual technique<00:10:43.880> which<00:10:44.160> makes<00:10:45.080> the<00:10:45.600> two-bit 00:10:46.070 --> 00:10:46.080 align:start position:0% technique which makes the two-bit 00:10:46.080 --> 00:10:49.230 align:start position:0% technique which makes the two-bit quantization<00:10:47.280> which<00:10:47.760> helps<00:10:48.120> you<00:10:48.360> run<00:10:48.800> in<00:10:48.960> a 00:10:49.230 --> 00:10:49.240 align:start position:0% quantization which helps you run in a 00:10:49.240 --> 00:10:51.550 align:start position:0% quantization which helps you run in a Mac<00:10:49.560> mini<00:10:49.800> M4<00:10:50.160> machine<00:10:50.600> is<00:10:50.920> actually<00:10:51.200> the<00:10:51.320> most 00:10:51.550 --> 00:10:51.560 align:start position:0% Mac mini M4 machine is actually the most 00:10:51.560 --> 00:10:53.110 align:start position:0% Mac mini M4 machine is actually the most important<00:10:51.960> part.<00:10:52.120> That's<00:10:52.240> the<00:10:52.280> reason<00:10:52.520> why<00:10:53.000> I 00:10:53.110 --> 00:10:53.120 align:start position:0% important part. That's the reason why I 00:10:53.120 --> 00:10:54.470 align:start position:0% important part. That's the reason why I actually<00:10:53.440> wanted<00:10:53.680> to<00:10:53.800> show<00:10:54.000> you<00:10:54.200> what's 00:10:54.470 --> 00:10:54.480 align:start position:0% actually wanted to show you what's 00:10:54.480 --> 00:10:56.510 align:start position:0% actually wanted to show you what's really<00:10:54.680> happening<00:10:55.040> over<00:10:55.200> here.<00:10:55.839> And<00:10:56.240> other 00:10:56.510 --> 00:10:56.520 align:start position:0% really happening over here. And other 00:10:56.520 --> 00:10:58.550 align:start position:0% really happening over here. And other thing<00:10:56.920> is<00:10:57.200> you<00:10:57.360> may<00:10:57.520> ask<00:10:58.040> what<00:10:58.120> happens<00:10:58.440> if<00:10:58.480> you 00:10:58.550 --> 00:10:58.560 align:start position:0% thing is you may ask what happens if you 00:10:58.560 --> 00:11:01.950 align:start position:0% thing is you may ask what happens if you try<00:10:58.760> to<00:10:59.000> run<00:10:59.280> the<00:10:59.400> same<00:10:59.800> exact<00:11:00.320> model<00:11:01.320> in<00:11:01.640> the 00:11:01.950 --> 00:11:01.960 align:start position:0% try to run the same exact model in the 00:11:01.960 --> 00:11:04.990 align:start position:0% try to run the same exact model in the Apple<00:11:02.520> M5<00:11:03.240> Max<00:11:03.600> machine,<00:11:04.040> right?<00:11:04.520> So<00:11:04.880> I'm 00:11:04.990 --> 00:11:05.000 align:start position:0% Apple M5 Max machine, right? So I'm 00:11:05.000 --> 00:11:06.870 align:start position:0% Apple M5 Max machine, right? So I'm going<00:11:05.120> to<00:11:05.200> be<00:11:05.320> loading<00:11:05.640> the<00:11:05.760> same<00:11:06.080> exact<00:11:06.520> model 00:11:06.870 --> 00:11:06.880 align:start position:0% going to be loading the same exact model 00:11:06.880 --> 00:11:09.630 align:start position:0% going to be loading the same exact model as<00:11:07.080> you<00:11:07.200> can<00:11:07.440> see<00:11:07.680> over<00:11:07.880> here<00:11:08.839> and<00:11:09.320> I'm<00:11:09.480> going 00:11:09.630 --> 00:11:09.640 align:start position:0% as you can see over here and I'm going 00:11:09.640 --> 00:11:13.150 align:start position:0% as you can see over here and I'm going to<00:11:09.720> be<00:11:10.120> doing<00:11:10.480> an<00:11:10.640> FP16<00:11:11.440> here<00:11:12.000> as<00:11:12.280> well.<00:11:12.880> So 00:11:13.150 --> 00:11:13.160 align:start position:0% to be doing an FP16 here as well. So 00:11:13.160 --> 00:11:16.190 align:start position:0% to be doing an FP16 here as well. So FP16<00:11:14.200> max<00:11:14.520> current<00:11:14.880> prediction<00:11:15.839> is<00:11:15.960> going<00:11:16.120> to 00:11:16.190 --> 00:11:16.200 align:start position:0% FP16 max current prediction is going to 00:11:16.200 --> 00:11:18.230 align:start position:0% FP16 max current prediction is going to be<00:11:16.320> one<00:11:16.680> as<00:11:16.880> well.<00:11:17.560> I'm<00:11:17.720> going<00:11:17.839> to<00:11:17.920> loading 00:11:18.230 --> 00:11:18.240 align:start position:0% be one as well. I'm going to loading 00:11:18.240 --> 00:11:19.990 align:start position:0% be one as well. I'm going to loading this<00:11:18.320> particular<00:11:18.600> model<00:11:19.240> and<00:11:19.480> let's<00:11:19.640> try<00:11:19.839> to 00:11:19.990 --> 00:11:20.000 align:start position:0% this particular model and let's try to 00:11:20.000 --> 00:11:21.670 align:start position:0% this particular model and let's try to ask<00:11:20.200> the<00:11:20.320> same<00:11:20.560> exact<00:11:20.880> question<00:11:21.240> over<00:11:21.440> here 00:11:21.670 --> 00:11:21.680 align:start position:0% ask the same exact question over here 00:11:21.680 --> 00:11:23.270 align:start position:0% ask the same exact question over here and<00:11:21.800> see<00:11:22.280> how<00:11:22.480> this<00:11:22.680> is<00:11:22.839> actually<00:11:23.080> going<00:11:23.200> to 00:11:23.270 --> 00:11:23.280 align:start position:0% and see how this is actually going to 00:11:23.280 --> 00:11:25.990 align:start position:0% and see how this is actually going to work<00:11:23.760> in<00:11:23.920> my<00:11:24.120> Apple<00:11:24.520> M5<00:11:25.000> Max<00:11:25.360> machine.<00:11:25.760> So<00:11:25.839> this 00:11:25.990 --> 00:11:26.000 align:start position:0% work in my Apple M5 Max machine. So this 00:11:26.000 --> 00:11:28.750 align:start position:0% work in my Apple M5 Max machine. So this is<00:11:26.120> a<00:11:26.560> 128<00:11:27.160> gig<00:11:27.400> machine.<00:11:28.080> So<00:11:28.240> you<00:11:28.400> might<00:11:28.640> be 00:11:28.750 --> 00:11:28.760 align:start position:0% is a 128 gig machine. So you might be 00:11:28.760 --> 00:11:31.030 align:start position:0% is a 128 gig machine. So you might be expecting<00:11:29.280> it<00:11:29.400> to<00:11:29.480> be<00:11:29.880> running<00:11:30.200> it<00:11:30.320> faster<00:11:30.880> and 00:11:31.030 --> 00:11:31.040 align:start position:0% expecting it to be running it faster and 00:11:31.040 --> 00:11:32.870 align:start position:0% expecting it to be running it faster and you'll<00:11:31.160> be<00:11:31.600> expecting<00:11:32.080> it<00:11:32.160> to<00:11:32.240> get<00:11:32.520> at<00:11:32.640> least<00:11:32.840> a 00:11:32.870 --> 00:11:32.880 align:start position:0% you'll be expecting it to get at least a 00:11:32.880 --> 00:11:34.590 align:start position:0% you'll be expecting it to get at least a 100<00:11:33.280> tokens<00:11:33.600> per<00:11:33.760> second<00:11:34.080> if<00:11:34.200> you're<00:11:34.320> thinking 00:11:34.590 --> 00:11:34.600 align:start position:0% 100 tokens per second if you're thinking 00:11:34.600 --> 00:11:36.150 align:start position:0% 100 tokens per second if you're thinking that<00:11:34.760> way.<00:11:35.400> But,<00:11:35.560> if<00:11:35.760> I'm<00:11:35.880> going<00:11:36.000> to<00:11:36.080> be 00:11:36.150 --> 00:11:36.160 align:start position:0% that way. But, if I'm going to be 00:11:36.160 --> 00:11:39.590 align:start position:0% that way. But, if I'm going to be running<00:11:36.760> the<00:11:36.880> same<00:11:37.160> exact<00:11:38.280> model<00:11:39.160> in<00:11:39.360> this 00:11:39.590 --> 00:11:39.600 align:start position:0% running the same exact model in this 00:11:39.600 --> 00:11:42.550 align:start position:0% running the same exact model in this machine,<00:11:40.640> you<00:11:40.760> can<00:11:41.000> see<00:11:41.640> that<00:11:42.280> it<00:11:42.440> is 00:11:42.550 --> 00:11:42.560 align:start position:0% machine, you can see that it is 00:11:42.560 --> 00:11:45.990 align:start position:0% machine, you can see that it is responding<00:11:43.440> with<00:11:43.680> 27<00:11:44.560> tokens<00:11:44.920> per<00:11:45.080> second.<00:11:45.880> It 00:11:45.990 --> 00:11:46.000 align:start position:0% responding with 27 tokens per second. It 00:11:46.000 --> 00:11:49.150 align:start position:0% responding with 27 tokens per second. It is<00:11:46.120> not<00:11:46.360> like<00:11:46.520> 127<00:11:47.400> tokens<00:11:47.760> per<00:11:47.880> second.<00:11:48.840> So, 00:11:49.150 --> 00:11:49.160 align:start position:0% is not like 127 tokens per second. So, 00:11:49.160 --> 00:11:51.830 align:start position:0% is not like 127 tokens per second. So, it<00:11:49.440> proves<00:11:49.760> the<00:11:49.880> point<00:11:50.280> that<00:11:51.000> it's<00:11:51.200> not<00:11:51.520> just 00:11:51.830 --> 00:11:51.840 align:start position:0% it proves the point that it's not just 00:11:51.840 --> 00:11:55.030 align:start position:0% it proves the point that it's not just about<00:11:52.680> the<00:11:53.200> two-bit<00:11:53.560> quantization<00:11:54.440> model 00:11:55.030 --> 00:11:55.040 align:start position:0% about the two-bit quantization model 00:11:55.040 --> 00:11:57.430 align:start position:0% about the two-bit quantization model which<00:11:55.200> can<00:11:55.400> run<00:11:55.880> way<00:11:56.080> more<00:11:56.320> faster<00:11:56.800> in<00:11:56.920> an 00:11:57.430 --> 00:11:57.440 align:start position:0% which can run way more faster in an 00:11:57.440 --> 00:11:59.350 align:start position:0% which can run way more faster in an Apple<00:11:57.760> M5<00:11:58.080> Max<00:11:58.320> machine<00:11:58.680> because<00:11:58.960> it's<00:11:59.160> even 00:11:59.350 --> 00:11:59.360 align:start position:0% Apple M5 Max machine because it's even 00:11:59.360 --> 00:12:02.950 align:start position:0% Apple M5 Max machine because it's even running<00:11:59.640> in<00:11:59.720> the<00:12:00.320> M4<00:12:01.400> mini<00:12:01.720> machine.<00:12:02.360> But,<00:12:02.720> the 00:12:02.950 --> 00:12:02.960 align:start position:0% running in the M4 mini machine. But, the 00:12:02.960 --> 00:12:04.790 align:start position:0% running in the M4 mini machine. But, the the<00:12:03.080> thing<00:12:03.280> is<00:12:03.840> even<00:12:04.080> though<00:12:04.280> it<00:12:04.400> is<00:12:04.480> a<00:12:04.520> two-bit 00:12:04.790 --> 00:12:04.800 align:start position:0% the thing is even though it is a two-bit 00:12:04.800 --> 00:12:07.470 align:start position:0% the thing is even though it is a two-bit quantization,<00:12:05.720> the<00:12:05.880> quality<00:12:06.680> of<00:12:06.880> the<00:12:07.080> output 00:12:07.470 --> 00:12:07.480 align:start position:0% quantization, the quality of the output 00:12:07.480 --> 00:12:10.270 align:start position:0% quantization, the quality of the output still<00:12:08.040> remains,<00:12:08.800> and<00:12:08.920> that's<00:12:09.120> the<00:12:09.200> reason<00:12:09.920> the 00:12:10.270 --> 00:12:10.280 align:start position:0% still remains, and that's the reason the 00:12:10.280 --> 00:12:13.190 align:start position:0% still remains, and that's the reason the the<00:12:10.400> technique<00:12:10.880> of<00:12:11.080> the<00:12:11.520> GSQ<00:12:12.160> RCU<00:12:12.680> sits<00:12:13.000> in. 00:12:13.190 --> 00:12:13.200 align:start position:0% the technique of the GSQ RCU sits in. 00:12:13.200 --> 00:12:15.950 align:start position:0% the technique of the GSQ RCU sits in. So,<00:12:13.360> it<00:12:13.600> it<00:12:13.720> safely<00:12:14.160> chooses<00:12:14.600> the<00:12:14.760> best<00:12:15.200> tensor 00:12:15.950 --> 00:12:15.960 align:start position:0% So, it it safely chooses the best tensor 00:12:15.960 --> 00:12:18.750 align:start position:0% So, it it safely chooses the best tensor which<00:12:16.120> can<00:12:16.320> be<00:12:16.520> optimized<00:12:17.240> to<00:12:17.360> do<00:12:17.600> a<00:12:17.720> better 00:12:18.750 --> 00:12:18.760 align:start position:0% which can be optimized to do a better 00:12:18.760 --> 00:12:21.790 align:start position:0% which can be optimized to do a better work<00:12:19.160> for<00:12:19.800> better<00:12:20.160> precision<00:12:21.360> instead<00:12:21.680> of 00:12:21.790 --> 00:12:21.800 align:start position:0% work for better precision instead of 00:12:21.800 --> 00:12:23.510 align:start position:0% work for better precision instead of focusing<00:12:22.320> on<00:12:22.600> the<00:12:22.760> low<00:12:23.000> precision 00:12:23.510 --> 00:12:23.520 align:start position:0% focusing on the low precision 00:12:23.520 --> 00:12:26.150 align:start position:0% focusing on the low precision quantizations<00:12:24.560> that<00:12:24.760> is<00:12:24.880> there 00:12:26.150 --> 00:12:26.160 align:start position:0% quantizations that is there 00:12:26.160 --> 00:12:27.550 align:start position:0% quantizations that is there on<00:12:26.320> the<00:12:26.440> tensors<00:12:26.920> there.<00:12:27.360> So,<00:12:27.380> [music]<00:12:27.400> that's 00:12:27.550 --> 00:12:27.560 align:start position:0% on the tensors there. So, [music] that's 00:12:27.560 --> 00:12:28.870 align:start position:0% on the tensors there. So, [music] that's it,<00:12:27.640> guys.<00:12:27.920> This<00:12:28.080> is<00:12:28.120> all<00:12:28.320> I<00:12:28.360> wanted<00:12:28.600> to<00:12:28.640> talk 00:12:28.870 --> 00:12:28.880 align:start position:0% it, guys. This is all I wanted to talk 00:12:28.880 --> 00:12:31.710 align:start position:0% it, guys. This is all I wanted to talk to<00:12:29.000> you<00:12:29.160> about<00:12:29.560> the<00:12:29.680> Quant<00:12:30.000> 3.8<00:12:30.680> 27<00:12:31.240> billion 00:12:31.710 --> 00:12:31.720 align:start position:0% to you about the Quant 3.8 27 billion 00:12:31.720 --> 00:12:33.150 align:start position:0% to you about the Quant 3.8 27 billion parameter<00:12:32.080> model<00:12:32.440> with<00:12:32.640> two-bit 00:12:33.150 --> 00:12:33.160 align:start position:0% parameter model with two-bit 00:12:33.160 --> 00:12:35.230 align:start position:0% parameter model with two-bit quantization.<00:12:34.120> Once<00:12:34.280> again,<00:12:34.760> thank<00:12:34.960> you<00:12:35.080> so 00:12:35.230 --> 00:12:35.240 align:start position:0% quantization. Once again, thank you so 00:12:35.240 --> 00:12:36.630 align:start position:0% quantization. Once again, thank you so much<00:12:35.360> for<00:12:35.480> watching<00:12:35.680> this<00:12:35.800> video.<00:12:36.400> Catch<00:12:36.600> you 00:12:36.630 --> 00:12:36.640 align:start position:0% much for watching this video. Catch you 00:12:36.640 --> 00:12:39.160 align:start position:0% much for watching this video. Catch you in<00:12:36.680> the<00:12:36.760> next<00:12:36.960> one.