WEBVTT
Kind: captions
Language: en

00:00:00.000 --> 00:00:04.150 align:start position:0%
 
Quen<00:00:00.320><c> 3.8</c><00:00:01.280><c> 27B.</c><00:00:02.680><c> A</c><00:00:02.760><c> serious</c><00:00:03.360><c> local</c><00:00:03.760><c> coding</c>

00:00:04.150 --> 00:00:04.160 align:start position:0%
Quen 3.8 27B. A serious local coding
 

00:00:04.160 --> 00:00:07.350 align:start position:0%
Quen 3.8 27B. A serious local coding
model.<00:00:04.840><c> But</c><00:00:05.000><c> what</c><00:00:05.160><c> will</c><00:00:05.280><c> it</c><00:00:05.400><c> run</c><00:00:05.640><c> on?</c><00:00:06.320><c> Dense</c><00:00:06.680><c> 27</c>

00:00:07.350 --> 00:00:07.360 align:start position:0%
model. But what will it run on? Dense 27
 

00:00:07.360 --> 00:00:10.390 align:start position:0%
model. But what will it run on? Dense 27
billion<00:00:07.800><c> parameters,</c><00:00:08.920><c> 4-bit</c><00:00:09.360><c> quantization,</c>

00:00:10.390 --> 00:00:10.400 align:start position:0%
billion parameters, 4-bit quantization,
 

00:00:10.400 --> 00:00:13.350 align:start position:0%
billion parameters, 4-bit quantization,
about<00:00:10.760><c> 18</c><00:00:11.160><c> GB.</c><00:00:12.160><c> But</c><00:00:12.320><c> loading</c><00:00:12.720><c> the</c><00:00:12.840><c> weights</c><00:00:13.200><c> is</c>

00:00:13.350 --> 00:00:13.360 align:start position:0%
about 18 GB. But loading the weights is
 

00:00:13.360 --> 00:00:17.350 align:start position:0%
about 18 GB. But loading the weights is
not<00:00:13.560><c> the</c><00:00:13.680><c> whole</c><00:00:13.920><c> story.</c><00:00:14.920><c> 16</c><00:00:15.400><c> gigs,</c><00:00:16.160><c> not</c><00:00:16.360><c> a</c><00:00:16.400><c> Q4.</c>

00:00:17.350 --> 00:00:17.360 align:start position:0%
not the whole story. 16 gigs, not a Q4.
 

00:00:17.360 --> 00:00:19.030 align:start position:0%
not the whole story. 16 gigs, not a Q4.
You<00:00:17.480><c> would</c><00:00:17.600><c> need</c><00:00:17.800><c> aggressive</c><00:00:18.280><c> quantization</c>

00:00:19.030 --> 00:00:19.040 align:start position:0%
You would need aggressive quantization
 

00:00:19.040 --> 00:00:22.550 align:start position:0%
You would need aggressive quantization
or<00:00:19.240><c> CPU</c><00:00:20.000><c> offload.</c><00:00:20.920><c> 24</c><00:00:21.480><c> gigs</c><00:00:21.880><c> is</c><00:00:22.040><c> where</c><00:00:22.200><c> it</c><00:00:22.320><c> gets</c>

00:00:22.550 --> 00:00:22.560 align:start position:0%
or CPU offload. 24 gigs is where it gets
 

00:00:22.560 --> 00:00:25.910 align:start position:0%
or CPU offload. 24 gigs is where it gets
interesting.<00:00:23.600><c> Q4</c><00:00:24.200><c> fits</c><00:00:24.520><c> a</c><00:00:24.600><c> single</c><00:00:25.040><c> 24</c><00:00:25.680><c> gig</c>

00:00:25.910 --> 00:00:25.920 align:start position:0%
interesting. Q4 fits a single 24 gig
 

00:00:25.920 --> 00:00:28.790 align:start position:0%
interesting. Q4 fits a single 24 gig
GPU.<00:00:27.040><c> A</c><00:00:27.120><c> 24</c><00:00:27.720><c> gig</c><00:00:27.960><c> Mac</c><00:00:28.165><c> [music]</c><00:00:28.240><c> squeezes</c><00:00:28.680><c> it</c>

00:00:28.790 --> 00:00:28.800 align:start position:0%
GPU. A 24 gig Mac [music] squeezes it
 

00:00:28.800 --> 00:00:30.870 align:start position:0%
GPU. A 24 gig Mac [music] squeezes it
in,<00:00:28.960><c> but</c><00:00:29.120><c> that</c><00:00:29.360><c> memory</c><00:00:29.760><c> is</c><00:00:30.040><c> shared</c><00:00:30.320><c> with</c><00:00:30.480><c> Mac</c>

00:00:30.870 --> 00:00:30.880 align:start position:0%
in, but that memory is shared with Mac
 

00:00:30.880 --> 00:00:34.390 align:start position:0%
in, but that memory is shared with Mac
OS.<00:00:31.680><c> It</c><00:00:31.880><c> fits.</c><00:00:32.680><c> Not</c><00:00:32.920><c> the</c><00:00:33.040><c> tier</c><00:00:33.400><c> I</c><00:00:33.480><c> would</c><00:00:33.640><c> buy.</c>

00:00:34.390 --> 00:00:34.400 align:start position:0%
OS. It fits. Not the tier I would buy.
 

00:00:34.400 --> 00:00:36.670 align:start position:0%
OS. It fits. Not the tier I would buy.
32<00:00:35.000><c> gigs</c><00:00:35.360><c> of</c><00:00:35.480><c> unified</c><00:00:36.000><c> memory.</c>

00:00:36.670 --> 00:00:36.680 align:start position:0%
32 gigs of unified memory.
 

00:00:36.680 --> 00:00:36.790 align:start position:0%
32 gigs of unified memory.
&gt;&gt; [music]

00:00:36.790 --> 00:00:36.800 align:start position:0%
&gt;&gt; [music]
 

00:00:36.800 --> 00:00:39.790 align:start position:0%
&gt;&gt; [music]
&gt;&gt; Now<00:00:37.080><c> it</c><00:00:37.200><c> makes</c><00:00:37.480><c> real</c><00:00:37.800><c> sense</c><00:00:38.160><c> on</c><00:00:38.280><c> a</c><00:00:38.360><c> Mac.</c><00:00:39.080><c> 48</c><00:00:39.680><c> to</c>

00:00:39.790 --> 00:00:39.800 align:start position:0%
&gt;&gt; Now it makes real sense on a Mac. 48 to
 

00:00:39.800 --> 00:00:42.830 align:start position:0%
&gt;&gt; Now it makes real sense on a Mac. 48 to
64,<00:00:40.880><c> real</c><00:00:41.240><c> breathing</c><00:00:41.680><c> room</c><00:00:41.920><c> for</c><00:00:42.080><c> context</c><00:00:42.680><c> and</c>

00:00:42.830 --> 00:00:42.840 align:start position:0%
64, real breathing room for context and
 

00:00:42.840 --> 00:00:46.350 align:start position:0%
64, real breathing room for context and
agents.<00:00:43.800><c> And</c><00:00:43.920><c> a</c><00:00:43.960><c> 32</c><00:00:44.560><c> gig</c><00:00:44.800><c> GPU</c><00:00:45.960><c> keeps</c><00:00:46.240><c> the</c>

00:00:46.350 --> 00:00:46.360 align:start position:0%
agents. And a 32 gig GPU keeps the
 

00:00:46.360 --> 00:00:49.270 align:start position:0%
agents. And a 32 gig GPU keeps the
entire<00:00:46.840><c> model</c><00:00:47.240><c> accelerated.</c><00:00:48.480><c> So</c><00:00:48.720><c> yes,</c><00:00:49.200><c> a</c>

00:00:49.270 --> 00:00:49.280 align:start position:0%
entire model accelerated. So yes, a
 

00:00:49.280 --> 00:00:52.150 align:start position:0%
entire model accelerated. So yes, a
serious<00:00:49.800><c> 27</c><00:00:50.480><c> billion</c><00:00:50.840><c> parameter</c><00:00:51.360><c> model</c><00:00:51.800><c> runs</c>

00:00:52.150 --> 00:00:52.160 align:start position:0%
serious 27 billion parameter model runs
 

00:00:52.160 --> 00:00:56.150 align:start position:0%
serious 27 billion parameter model runs
on<00:00:52.320><c> consumer</c><00:00:52.880><c> hardware.</c><00:00:53.880><c> On</c><00:00:54.000><c> a</c><00:00:54.080><c> Mac,</c><00:00:54.760><c> 32</c><00:00:55.320><c> gigs,</c>

00:00:56.150 --> 00:00:56.160 align:start position:0%
on consumer hardware. On a Mac, 32 gigs,
 

00:00:56.160 --> 00:01:00.350 align:start position:0%
on consumer hardware. On a Mac, 32 gigs,
64<00:00:56.880><c> to</c><00:00:57.040><c> use</c><00:00:57.320><c> it</c><00:00:57.440><c> like</c><00:00:57.640><c> an</c><00:00:57.800><c> agent.</c><00:00:58.720><c> Quen</c><00:00:59.000><c> 3.8</c><00:01:00.160><c> is</c>

00:01:00.350 --> 00:01:00.360 align:start position:0%
64 to use it like an agent. Quen 3.8 is
 

00:01:00.360 --> 00:01:02.670 align:start position:0%
64 to use it like an agent. Quen 3.8 is
only<00:01:00.680><c> the</c><00:01:00.760><c> beginning.</c><00:01:01.760><c> Comment</c><00:01:02.160><c> the</c><00:01:02.280><c> model</c>

00:01:02.670 --> 00:01:02.680 align:start position:0%
only the beginning. Comment the model
 

00:01:02.680 --> 00:01:06.160 align:start position:0%
only the beginning. Comment the model
you<00:01:02.800><c> want</c><00:01:03.000><c> me</c><00:01:03.120><c> to</c><00:01:03.240><c> test</c><00:01:03.560><c> next.</c>