0:00.400–0:07.200
So in this video, I want to continue the series I started on refurbished data center GPUs.
0:07.200–0:13.600
I've been working on this video for the past two months and I want to start with an important premise.
0:13.600–0:21.760
I know some people will feel compelled to use the comment section to vent and compare the hardware that we discuss here
0:21.760–0:26.400
with stuff that's two to five times more expensive.
0:26.400–0:31.200
First of all, this video comes out of a collaboration with Bargain Hardware.
0:31.200–0:34.160
This is not a sponsored video.
0:34.160–0:40.240
If you don't know them, Bargain Hardware are a major reseller of refurbished hardware.
0:40.240–0:42.960
I am not getting paid for this.
0:43.520–0:49.280
This collaboration simply gives me access to some hardware so I can learn an experiment,
0:49.280–0:52.480
which is the main objective of this channel.
0:52.480–0:59.280
Now, of course, I know that people that want to get into local AI are after alternatives to more
0:59.280–1:00.720
expensive hardware.
1:00.720–1:09.920
For reference, in August last year, one could get a 128GB Strixello machine for just over $2000.
1:10.640–1:14.800
Now, we are looking at twice as much for the same machine.
1:14.800–1:17.920
The landscape has totally changed.
1:17.920–1:26.080
This is also why I am experimenting with refurbished data center hardware to see what kind of performance
1:26.080–1:33.920
we can achieve, what the cost is, and most importantly, the trade-offs you need to be aware of if you decide
1:33.920–1:35.600
to go this way.
1:35.600–1:43.920
On top of all of this, Bargain Hardware is offering a 10% discount on the GPUs listed in the description of
1:43.920–1:44.800
this video.
1:44.800–1:48.720
Just use the code DONATO10 at checkout.
1:48.720–1:54.640
This is not an affiliate link and I get zero if you buy or don't buy.
1:54.640–2:00.960
We could have done a 5% split where you get 5% discount and I get 5%.
2:01.920–2:08.880
But I am trying to see what I can do to make some hardware more affordable for my viewers,
2:08.880–2:15.120
since it looks like that the memory and GPU situation that we have now will last for a while.
2:15.120–2:19.680
So I'd rather get the full 10% discount to you.
2:19.680–2:22.960
That's the only thing that's in my power.
2:22.960–2:28.720
Now, the first consideration is that when you want to build a multi-GPU system,
2:28.720–2:32.160
it's not just the GPUs that cost money.
2:32.160–2:39.040
The issue is that unless you already have a rig to place your cards in, you also have to build a
2:39.040–2:40.560
system to host them.
2:40.560–2:47.440
If you watched my Dual Radio 9700 video, you'll have an idea of what that looks like.
2:47.440–2:56.560
But that setup with 64GB of RAM would cost around $4,000 to $5,000 with current prices,
2:56.560–3:00.640
maybe even more, I really cannot keep up myself.
3:00.640–3:08.080
And if you want to scale up to four dual-slot GPUs, you need a motherboard that physically can
3:08.080–3:17.200
accommodate them, a GPU with enough PCIe lanes to avoid lane starvation, a high capacity power supply,
3:17.200–3:21.600
and a chassis with proper cooling, let alone system RAM.
3:21.600–3:27.440
If you try to build that base system from scratch using currently available hardware,
3:27.440–3:34.320
it can easily cost you around $4,000 before you even buy the GPUs.
3:34.320–3:41.120
That is where refurbished data center servers start making sense in 2026.
3:41.120–3:48.320
They offer great value by giving you a base system for a fraction of the cost.
3:48.320–3:54.080
Now, Bargain Hardware is a simple online builder where you can select all the components for the
3:54.080–3:54.800
system.
3:54.800–4:02.160
For a quad GPU configuration, one of the cheapest options is to get a super micro server.
4:02.160–4:04.720
I'll get this 12th generation one.
4:04.720–4:09.920
The motherboard in these servers typically hosts two Intel Xeon CPUs.
4:09.920–4:12.400
So we can now configure these.
4:12.400–4:15.360
You can see there is a base price, of course.
4:15.360–4:20.880
So the first thing we want to do is to select the CPUs.
4:20.880–4:28.400
And here I'm going for two of these Xeon 18 core CPUs that you can see here.
4:28.400–4:30.640
They are incredibly cheap.
4:30.640–4:33.280
Then, of course, the heat sinks are already included.
4:33.280–4:37.120
And for the RAM, as you can see here, there are a lot of options.
4:37.120–4:44.640
Actually, Bargain Hardware stocks right now, a lot of DDR4 at a very, very good price.
4:44.640–4:49.360
Here I'm getting four of these 16 GB sticks.
4:49.360–4:52.000
Now we want to select the storage.
4:52.000–4:54.880
And for the storage, I went with SATA.
4:54.880–5:03.440
And I selected two of these 480 GB SSDs, which gives me almost one terabyte, which is what you want
5:04.080–5:08.640
for LLMs, at least one terabyte, because you're going to be downloading a lot of weights.
5:08.640–5:13.520
And moving on, now we select the GPU accelerators.
5:13.520–5:20.480
Now for this build, I have selected four P100s, 16 GB each.
5:20.480–5:22.560
So we get 64 GB.
5:22.560–5:32.080
But in this video and in the benchmarks, I've also looked at the V100s and also the AMD MI25.
5:32.080–5:37.920
And again, each of these options ultimately gives you four GPUs with 64 GB of RAM.
5:37.920–5:44.400
But of course, the V100 is much more expensive because it's a more modern architecture.
5:44.400–5:49.200
But check the website because more GPUs are coming.
5:49.200–5:55.040
And probably by the time this video is uploaded, you will see some more options.
5:55.040–5:57.840
I know that there are some other GPUs coming.
5:57.840–6:05.360
RTX 3080, 3090, Quadro RTX 5000, Quadro M6000.
6:05.360–6:15.600
But just check the stock because they are updating it and they will also make some A100s and RTX 8000 available.
6:15.600–6:19.040
So there is quite a lot that's coming to their stock.
6:19.760–6:26.880
But for this build, I will keep the P100s in there and we can add to the cart.
6:26.880–6:34.160
Oh, actually, we forgot to select the Super Macro NVMe Enablement Kit.
6:34.160–6:38.320
We add to the cart and now we can go and check out.
6:38.320–6:42.960
And I just want to show you the different options and costs for delivery.
6:42.960–6:48.560
That's actually at least in the UK for domestic delivery.
6:48.560–6:55.680
When you look at this, it brings us to a total of just over 2000 pounds.
6:55.680–7:04.560
But this is not including the 10% discount on the GPUs that you can get with the code Donato 10.
7:06.480–7:12.000
Now, I thought it'd be interesting to actually go in person to their warehouse,
7:12.000–7:20.000
see how an order like this is put together and generally get a look at the process they use to test and refurbish hardware.
7:20.000–7:24.640
And a thank you to Jack Moyers from the Bargain Hardware team,
7:24.640–7:27.520
who was kind enough to give me a tour.
7:27.520–7:30.640
The first process at Bargain Hardware is inbound.
7:31.200–7:36.720
So pallets of servers, workstations, components, they arrive on trucks.
7:36.720–7:42.240
So we buy equipment from all kinds of places all around Europe and some more worldwide.
7:42.720–7:46.640
And then before the servers and workstations and any equipment gets processed,
7:46.640–7:49.600
it gets stored in our rack in here.
7:49.600–7:53.200
So the first part of the refurbishment process is cleaning.
7:53.200–7:55.200
So this is one of our cleaning rooms.
7:55.200–7:58.000
So in here, the first part of the process is the servers,
7:58.560–8:01.040
they get blown out with our air compressor machine.
8:01.920–8:06.400
You can see here, Michael's doing some reskinning on a server.
8:06.400–8:11.840
So certain generations of servers get reskinned just so we have a nicer finish.
8:11.840–8:14.720
So it just gets rid of all these scuffs and scrapes off the lids.
8:14.720–8:22.800
So sometimes we have orders which are just a batch of drives or it could be CPUs or GPUs.
8:22.800–8:26.080
So not necessarily every component comes out of a machine.
8:26.080–8:30.400
Sometimes we buy them in batch, but still every single component needs testing.
8:30.400–8:33.120
So then we come to component testing here.
8:38.320–8:42.800
So when a machine has been tested, the next thing that we do,
8:42.800–8:47.360
because we stock our servers and workstations as configured to order,
8:47.360–8:51.280
the next thing we need to do is take the components out and that's called disassembly.
8:51.280–8:52.960
So this is the next section.
8:52.960–8:56.800
So here we've got Harry who's working on a workstation.
8:56.800–9:03.280
So he'll be taking out the CPU, the RAM, any GPUs and other cards and components so that
9:04.000–9:07.360
when we stock it and we offer it as configured to order on our website,
9:07.360–9:12.000
it's basically a chassis with a PSU and other base components.
9:15.040–9:22.560
As we were walking around, my attention was immediately captured by a stock of GPUs that
9:22.560–9:27.840
they got in and that they were testing and getting ready to refurbish.
9:29.840–9:31.360
So then we come to picking.
9:31.360–9:35.600
So if someone orders on our website, they've configured a server.
9:36.320–9:38.240
We've stocked it as a chassis and components.
9:38.240–9:41.760
So then our picking team will they'll receive the order.
9:41.760–9:44.640
They'll have a list of all the components that need to be picked.
9:45.600–9:51.760
And then it's just the case of them assembling it all onto a trolley ready for the next part of the process.
9:51.760–9:54.400
It's actually very well organized.
9:54.400–9:54.800
Yeah.
9:54.800–9:57.760
Wherever I look, it's tidy.
9:57.760–9:58.160
Yeah.
9:58.160–9:59.920
Like look around me.
9:59.920–10:02.800
There is nothing left to have.
10:02.800–10:03.680
Yeah.
10:03.680–10:06.080
This is where we keep all the RAM and CPUs.
10:06.080–10:10.080
So there's a lot of value behind a closed door here.
10:10.080–10:11.440
So we just need to keep it safe.
10:12.640–10:18.880
Given the current situation with memory, it's refreshing to see how much DDR4 memory
10:18.880–10:23.680
they've been able to stock at a pretty competitive price.
10:24.960–10:30.720
So when a member of the picking team has picked an order, it goes onto a trolley
10:31.600–10:36.080
and then the trolley is moved through to assembly so that that particular order can be assembled.
10:37.200–10:38.800
So this is the assembly area.
10:38.800–10:44.640
So you can see a member of staff is assembling an order that's been configured online.
10:44.640–10:47.840
And then when the machine is assembled, it then needs to be tested.
10:50.240–10:55.440
It's clear that they put a lot of effort in testing the builds before sending them out.
10:55.440–11:01.600
And of course, they give people warranty, but also the care they put in packaging
11:01.600–11:05.680
to make sure that stuff gets shipped in the best possible way.
11:05.680–11:09.440
We've invested a lot in the cardboard boxes we use, in the foam inserts,
11:09.440–11:13.680
making sure that every server is catered for and that it's really secure inside.
11:15.520–11:15.760
Yeah.
11:15.760–11:21.760
So once the order has been packed, then we've got our outbound department here.
11:25.440–11:27.280
So we've got our outbound department here.
11:27.280–11:31.760
At the end of the warehouse tour, I spent some time with Toby Sheriff,
11:31.760–11:37.440
who's the lead engineer that actually put together the server build you see me configure
11:37.440–11:39.680
at the beginning of this video.
11:39.680–11:47.040
What we've got here is a Supermicro DGQ in CSE 118 chassis.
11:47.040–11:51.760
It's got space for four double wide full height cards.
11:51.760–11:56.880
It takes the scalable CPUs, Intel first and second generation.
11:56.880–12:02.320
So currently we've got a gold 6150 or two gold 6150s in there.
12:02.320–12:06.080
So I believe they're 18 core CPUs, but they're also threaded.
12:06.080–12:08.400
So obviously you've got twice as many of them than that.
12:08.400–12:12.560
We've also got some DDR4 registered DIMMs.
12:12.560–12:15.680
So it can take up to six per CPU.
12:15.680–12:21.200
Currently we've got 64 gigs in there in 16 gig DIMMs.
12:21.200–12:25.840
That's running at 2666 megahertz mega transfers.
12:26.880–12:28.560
And that's just a limitation of the CPU.
12:28.560–12:30.960
So there's not really anything you can do about that.
12:30.960–12:35.760
If you go to a second generation CPU, you'd get slightly faster at 2933.
12:35.760–12:40.000
So it's not fully utilising the speed, but it shouldn't really be a problem.
12:40.000–12:43.200
We've also got on board 10 gig NICs.
12:43.200–12:45.280
So that's integrated into the motherboard.
12:45.280–12:47.680
We've also got two spare PCI slots at the rear.
12:48.240–12:53.440
If you wanted to add a network card, maybe a PCIe storage drive or something.
12:54.000–12:58.320
Half of these three slots are CPU2 dependent.
12:58.320–13:04.560
So you could, if you felt so inclined, just put two cards in it initially with one CPU.
13:04.560–13:09.600
And expand in the future if that was, you know, that was what you wanted to do.
13:09.600–13:11.040
Keep the cost down initially.
13:11.040–13:16.160
Storage wise, at the front, you've got two NVMe ports built into a backplane.
13:16.160–13:18.640
So that would do U.2 drives.
13:18.640–13:22.160
So two and a quarter inch NVMe drives.
13:22.160–13:27.120
And then we've also got two SATA slots just behind that backplane internally.
13:29.040–13:34.720
So because these are data center graphics cards, they're not designed like a consumer card with a fan.
13:34.720–13:36.080
As you said, they're passively cooled.
13:36.800–13:42.400
So that has got to travel through the the the fin stack as it, you know, travels through the server.
13:42.960–13:45.600
So you've got all these fans across the front here.
13:46.480–13:47.120
I won't lie.
13:47.840–13:48.480
It is loud.
13:48.480–13:54.240
But it's got to be to be able to push enough air through to keep all these cards and your CPU
13:54.800–13:57.200
and even your power supplies or network cards at the back.
13:57.200–13:57.440
Cool.
13:58.000–14:05.360
So right now we've got four P100s in here, but obviously we can swap any card that would fit in there.
14:06.000–14:12.400
In the video, I'm going to show you the performance with the, I think I have it here,
14:12.400–14:19.680
with one of the MI-50s, so MI-25 from AMD.
14:21.440–14:25.120
And then maybe we are also going to try, is this the...
14:25.120–14:27.600
It's the V100, that one.
14:27.600–14:30.720
The V100, which is going to perform so much better.
14:31.200–14:33.840
And in fact, we'll probably do some benchmarks with that.
14:33.840–14:38.160
So Toby is now closing the chassis, so we can power this on.
14:38.160–14:42.240
And mostly I want to give you an idea of the noise levels.
14:42.240–14:45.680
When you first power it on, this is very noisy.
14:45.680–14:51.520
And then it becomes a little bit quieter, but obviously it is still quite noisy.
14:51.520–14:53.680
And we need to be aware of that.
14:53.680–15:10.800
So I wanted to also show you quickly the specs of these three GPUs side by side,
15:10.800–15:18.480
along with more modern GPUs like the R9700 AI Pro, that I'm sure you've already seen on my channel,
15:18.480–15:20.960
and also the RTX 5090.
15:20.960–15:26.640
Now, during the comparison, keep in mind the considerable price difference.
15:26.640–15:35.120
I have put it here, and obviously for the 5090 and the R9700, I've had to estimate some brackets,
15:35.120–15:37.520
because the price is very variable.
15:37.520–15:41.040
But for the other GPUs that you can get on bargain hardware,
15:41.040–15:45.600
I have put the price that you would get with the 10% discount.
15:45.600–15:50.480
One of the most important things to consider is memory bandwidth.
15:50.480–15:57.440
During inference, generating each token requires reading a lot of data, such as model weights,
15:57.440–15:59.760
back and forth from memory.
15:59.760–16:06.320
So the bandwidth here directly limits how many tokens per second you can generate and process.
16:06.320–16:11.840
So the V100 stands out at 900 gigabytes per second,
16:11.840–16:21.200
which is actually higher than the more modern R9700, which has a memory bandwidth of 640 gigabytes per second.
16:21.200–16:28.320
And the MI25 is significantly lower at 484 gigabytes per second.
16:28.320–16:30.800
And that shows in the benchmarks.
16:30.800–16:34.880
The other important factor is the data format support.
16:34.880–16:41.600
Modern GPU architectures support formats like BF16 and FP8 in a native way.
16:41.600–16:46.080
And these are the standard precisions used by current LLMs.
16:46.080–16:50.400
None of these three older cards support these formats natively.
16:50.400–16:58.320
They do support FP16, which is also a 16-bit format and uses the same amount of memory,
16:58.320–17:05.360
but FP16 cannot represent as wide a range of values as BF16.
17:05.360–17:11.040
When an inference framework converts BF16 model weights to FP16,
17:11.040–17:14.720
some values can overflow or lose accuracy.
17:14.720–17:19.920
And depending on the model, this can downgrade output quality.
17:19.920–17:27.760
One last difference to keep in mind is that the V100 is more expensive because it has Tensor Cores,
17:27.760–17:34.640
which are essentially hardware parts optimized for matrix operations used in machine learning.
17:34.640–17:39.760
And you'd find such support also on all the modern GPUs.
17:39.760–17:48.080
The P100 and MI25 do not have any equivalent, and this will be clearly reflected in the benchmarks.
17:48.080–17:53.840
So very quickly, I want to show you how to set up this server.
17:53.840–17:57.680
Right now, I've got the four V100s installed.
17:57.680–18:05.040
So this is the repository I am going to use with the V100 AI toolboxes.
18:05.040–18:12.240
As usual, with all of my repositories, you are going to need to create a toolbox.
18:12.240–18:14.640
And this is essentially a Docker container.
18:14.640–18:19.680
I'm not going to repeat myself. I explain about toolboxes, Podman and Docker containers
18:20.240–18:22.960
in a lot of the other videos.
18:22.960–18:27.920
You can choose between the CUDA backend and the Vulkan backend.
18:27.920–18:33.520
And CUDA is almost always going to work better for Nvidia cards.
18:33.520–18:37.680
So we're going to take that one and we're going to create the toolbox,
18:37.680–18:43.920
which is essentially going to connect to Docker Hub and pull this toolbox that I have pre-built
18:43.920–18:47.200
with Lama CPP. So let us do that.
18:48.480–18:53.600
And actually, I have already created this toolbox, so I can enter it.
18:53.600–18:56.000
Lama V100 CUDA.
18:57.200–19:05.360
Now, once you enter the toolbox, you can run Lama CLI, list devices.
19:05.360–19:14.480
And this essentially confirms that Lama via the CUDA backend can see the four V100s, each of them with 16
19:14.480–19:15.680
gigabytes of VRAM.
19:15.680–19:21.440
So now, if you want to run a model, the easiest way to do that is to download the model weights.
19:21.440–19:27.200
I've already downloaded some of these in GGUF format from Hugging Face.
19:27.200–19:35.440
And let's run, for example, one of the best models that we can have today for local agentic workflows,
19:35.440–19:39.360
which is QAN 3.6, 27 billion parameters.
19:39.920–19:44.720
And I think I have these in Q4 KXL quantization.
19:44.720–19:47.360
So I'm just going to say Lama server.
19:47.920–19:50.400
I'm going to pass the model.
19:50.400–19:56.400
Actually, I also have it in Q8 quantization, for a matter of fact.
19:56.960–20:03.760
And we are going to enable flash attention, and we can pass a context size.
20:04.960–20:05.360
All right.
20:05.360–20:08.240
So the server is up and running on port 8080.
20:08.240–20:12.160
And now I need to forward that port to my actual laptop.
20:12.160–20:15.040
So I can SSH into the box.
20:15.040–20:18.240
I've called it BH for bargain hardware.
20:18.240–20:24.880
And I'm going to say that I want to forward that port to 8081 on my host,
20:24.880–20:28.800
because 8080 is already used by something else.
20:28.800–20:29.520
And there we go.
20:29.520–20:31.840
This is Lama CPP web UI.
20:31.840–20:40.960
So I can type a prompt, such as write a CUDA kernel to multiply to
20:43.120–20:45.280
As you can see now, it's thinking about it.
20:45.280–20:49.440
When models spend a lot of time thinking,
20:49.440–20:54.000
and you can see it's going at around 31 tokens per second.
20:54.000–20:59.520
So for being a 27 billion parameter dense model, this is really good.
20:59.520–21:02.880
But in a bit, we'll take a look at the proper benchmarks
21:02.880–21:05.680
and what happens when the context size grows,
21:05.680–21:09.440
and obviously what happens with other models and quantizations.
21:09.440–21:14.560
But I just thought I'd show you how to get one of these up and running.
21:14.560–21:20.560
Now, Lama CPP is my recommendation, especially for these older GPUs,
21:20.560–21:26.000
not just the V100, but also the P100 and the MI25.
21:26.000–21:30.160
And I've got toolboxes and links to all of those.
21:30.160–21:36.640
And the reason for that is that Lama CPP has a much more uniform ecosystem,
21:36.640–21:39.120
and it works pretty much everywhere.
21:39.120–21:43.600
If your hardware is supported, you can run any model on it.
21:43.600–21:47.840
And it's got pretty much any model and any quantization.
21:47.840–21:51.440
And quantizations are going to be really important because, of course, here,
21:51.440–21:53.600
you don't have that much memory available.
21:53.600–21:57.040
I mean, in this case, you've got 64 gigabytes, which is quite a lot.
21:57.040–22:01.920
But again, even if you want to run the QEM model that we just saw 27 billion parameters
22:01.920–22:04.960
is a lot, then you are going to need a quantization.
22:04.960–22:10.000
However, I know that a lot of people really, really like VLLM.
22:10.000–22:15.920
So I've also created a VLLM toolbox that you can get up and running like this.
22:16.480–22:20.880
So here, I've already created the toolbox so I can simply enter it.
22:20.880–22:27.520
And you will see that, obviously, on this server, I have also the VLLM toolbox for the P100
22:27.520–22:31.360
and the one for the AMD MI25.
22:31.360–22:34.880
But obviously, in this case, we enter this particular one.
22:34.880–22:41.440
And what I do in my toolboxes for VLLM, I always give you a start VLLM script
22:41.440–22:43.840
with a list of models that I have tested.
22:44.560–22:46.720
So at least you have a starting point.
22:46.720–22:50.080
I use LAMA 3.1 just as a benchmark.
22:50.080–22:57.200
But then if you want to run some proper models here, you can see the QEM 3.6 family, the 27 billion
22:57.200–22:57.840
parameter.
22:57.840–23:01.680
Now, this is in GPTQ 4-bit quantization.
23:02.240–23:09.040
You cannot run, or at least I haven't been able to find a way on VLLM to run AWQ quants,
23:09.040–23:13.120
which are a little bit better activation aware.
23:13.120–23:22.480
But again, this is why I tell people that LAMA CPP is 90% of the times better for most people,
23:22.480–23:25.600
especially on older hardware.
23:25.600–23:27.440
But anyway, I've got four GPUs.
23:27.440–23:35.280
So Tensor Parallelism 4, I can set concurrent requests, size of the context GPU utilization.
23:35.280–23:37.440
I don't remember why I set it this low.
23:37.440–23:39.520
You should probably set it a little bit higher.
23:39.520–23:42.240
And then you can launch the server.
23:42.240–23:47.760
And this should show you exactly how we are running that particular model.
23:47.760–23:48.960
And it will take a little bit.
23:48.960–23:53.520
And then VLLM will come up and you'll be able to use the model.
23:53.520–23:56.160
So here we can see the model loading.
23:56.160–23:58.400
It's taking quite a bit of time.
23:58.400–24:02.960
And that's basically one of the caveats with PCI 3.0.
24:02.960–24:07.360
Especially model loading is going to be fairly slow.
24:07.360–24:13.200
And you can see here, it's also looking for different attention backhand that it can use.
24:13.200–24:20.400
And ultimately, I think it's falling back to PyTorch, SDPA attention, which is okay,
24:20.400–24:25.360
but it's not as good as some of the things you can get on modern hardware, of course.
24:25.360–24:33.280
But again, this will work and you'll be able to run some models, even with VLLM, if you use these
24:33.280–24:34.560
tool boxes.
24:34.560–24:35.600
So here we go.
24:35.600–24:38.000
This is now up and running.
24:38.000–24:42.240
And you can start using it, for example, with a coding agent.
24:42.240–24:47.840
But again, I would not recommend to use perhaps VLLM.
24:47.840–24:54.000
Just stick to Lama CPP and you're going to get probably the best performance here.
24:54.800–25:00.800
Let's now take a look at the benchmarks, which are arguably one of the most important factor in
25:00.800–25:04.800
deciding whether or not some of these cards might be good for you.
25:04.800–25:10.320
Here, I've got the individual repositories for all the cards that I tested.
25:10.320–25:14.960
I should have the MI25 here as well.
25:14.960–25:20.000
And I have put all of the benchmark results in the readme.
25:20.000–25:28.160
So you can probably scroll and find them for Lama CPP and VLLM, token generation and prompt processing.
25:28.160–25:33.760
So this is the V100 and obviously the P100.
25:33.760–25:40.560
But obviously for this video to make things a little bit easier to compare, I just put everything together.
25:40.560–25:46.720
You can see the comparison here of the three cards on different models.
25:46.720–25:54.400
Now, these are modern LLMs that I recommend you run on these cards, mostly for agentic workflows and coding.
25:54.400–26:04.240
The QAN 3.5 and 3.6 families are right now some of the best you can run with 32 to 64 gigabytes of RAM.
26:04.240–26:13.920
And you can see obviously that the V100, the blue one here, it's the best performer at prompt processing and token generation.
26:13.920–26:19.040
Even when the context goes to 32,000 tokens per second.
26:19.040–26:29.680
And that's because this is just a more modern architecture and it's got tensor cores, which essentially allow to do matrix multiplication much better.
26:29.680–26:33.040
And that's the core operation in LLMs.
26:33.040–26:51.440
Just to give you an idea for a model like 27 billion parameters, you get on the V100, 852 tokens per second in prompt processing with the Q4 quantization, which is a very good quantization.
26:51.440–26:54.080
And that's a very good prompt processing speed.
26:54.080–26:57.680
And you get around 34 tokens per second.
26:57.680–27:10.400
And even when you scale up the context, you see that you're still getting around 622 tokens per second on this 27 billion parameter model, which is a dense model.
27:10.400–27:13.920
So these are the hardest model to run.
27:13.920–27:16.800
And the performance is really good for these.
27:16.800–27:22.880
And the tokens per second that you get in token generation is around 28 tokens per second.
27:22.880–27:29.200
Again, this would be perfectly useful if you were using this model, let's say, in the PI coding agent.
27:29.200–27:37.200
Actually, check out the video I've done on coding agents and it's going to give you an idea of the type of performance you can get.
27:37.200–27:41.520
So you will also see that the MI25 is missing from some of these.
27:41.520–27:44.640
It's just because it's the first card that I tested.
27:44.640–27:48.080
And back then, I didn't include all the models.
27:48.080–27:57.440
But the ones that I included, you can see that on LLAMA CPP, it does perform close to the P100, but just below it.
27:57.440–27:59.200
So be aware of that.
27:59.200–28:03.520
The MI25 is the one that's going to give you the least performance.
28:03.520–28:07.520
So you can see that the other ones that you can see in the LLAMA CPP, but this is what you get on LLAMA CPP.
28:07.520–28:17.200
You can also run VLLM, but with some caveats, VLLM is much more sensible to the different GPU architectures.
28:17.200–28:26.560
You require specific kernels for specific architectures, and some of them are just not available for a lot of these cards.
28:26.560–28:32.400
So you are not really able to pick and choose like you do with LLAMA CPP and run anything you want.
28:32.400–28:44.560
But you can see that the MI25, when there are kernels available, actually performs better than the P100 on VLLM.
28:44.560–28:47.280
So that's something to keep in mind.
28:47.280–28:52.160
Again, I do not recommend using VLLM with these cards.
28:52.160–28:55.680
A lot of models you just cannot easily run.
28:55.680–29:04.800
But on the V100, at least, you can run some quantization of the QAN 3.6 27 billion parameter model.
29:05.200–29:08.760
This is the GPT-Q 4-bit quantization.
29:08.760–29:11.720
And you can see this throughput that you get.
29:11.720–29:15.880
And you can also run the QAN 3.5 9 billion parameter model.
29:15.880–29:19.560
But again, I do not recommend running any of these.
29:19.560–29:25.400
I did try these, and I have all the toolboxes if you want to experiment,
29:25.840–29:30.840
but probably stick to LLAMA CPP for these older GPUs.
29:31.760–29:33.280
So we've seen the benchmarks.
29:33.560–29:38.600
Now let's discuss the caveats and trade-offs you need to be aware of.
29:38.760–29:41.640
First, the GPU architectures.
29:41.640–29:52.120
The Pascal-based P100 and the Vega-based MI25 do not have hardware paths optimized for modern machine learning.
29:52.120–30:00.760
The V100 does have Tensor Cores, but it is still a legacy architecture compared to modern GPUs.
30:01.080–30:05.480
Because of this, although these cards will be able to run modern LLMs,
30:05.480–30:10.600
they just cannot match the throughput that a modern GPU gives you.
30:10.600–30:13.320
And that's reflected in the price you pay.
30:13.800–30:16.920
Second, these setups are noisy.
30:17.320–30:24.220
Because these passive cards require high airflow fans to stay cool, the system is loud.
30:24.440–30:27.500
You will not want this server under your desk.
30:27.880–30:32.580
This more likely belongs in a garage, a basement, or a dedicated room.
30:32.580–30:34.920
Third, the system bus.
30:34.920–30:39.260
These older servers use PCIe Gen 3 slots.
30:39.640–30:46.280
A PCIe Gen 3 slot has a theoretical bandwidth of around 16GB per second,
30:46.640–30:54.720
whereas a modern PCIe Gen 5 slot provides 64GB, four times the speed.
30:54.720–30:59.280
For single GPU setups, the difference is not that noticeable,
30:59.280–31:05.820
only adding a few seconds when you load the model weights from storage into the GPU memory.
31:06.140–31:15.040
But if you run multi-GPU workloads, the GPUs must constantly exchange information and synchronize their activity.
31:15.040–31:25.900
So, over a Gen 3 bus, this exchange of information will be slower and it will limit the overall throughput and token generation performance.
31:26.280–31:28.320
The same is true for the memory.
31:28.700–31:38.240
The service runs on DDR4-ACC memory, which usually operates at around 2400 to 3200 megatransfer per second.
31:38.240–31:47.620
In comparison, DDR5 starts at 4800 megatransfer and goes much higher, effectively doubling the bandwidth.
31:48.100–31:55.060
This does not affect you much if you choose models that fully fit inside the GPU memory.
31:55.220–32:02.060
But if you want to do some offloading to the CPU and to the system memory, this will be a bottleneck.
32:02.060–32:10.200
As I was editing this video, I realized that I was talking about PCIe 3.0 and DDR4.
32:10.200–32:21.960
And I chose those for the workstations that we built here because I wanted to show you the cheapest available options to get a viable setup.
32:21.960–32:33.860
But if you go on bargain hardware, for example, you can find servers and workstations with DDR5 and PCIe 4 and 5.
32:34.120–32:47.040
Just to give you an example, these Dell Precision 7960, well, of course, more expensive, but these will come with DDR5 memory.
32:47.040–32:53.280
And it will also come with PCIe 5.0.
32:53.280–32:59.760
So these are the main trade-offs you make, which allow you to keep the total cost at this price point.
33:00.100–33:02.640
Even with these caveats, the value is clear.
33:02.640–33:11.100
If you have a budget of 2,000 to 3,000 pounds and you need 64 gigabytes of VRAM to run certain models,
33:11.500–33:18.040
a refurbished enterprise server like the one I showed you is one of the real options available to you today.
33:18.320–33:28.080
You get a pre-built base system that can host four cards for a fraction of the cost of a DIY workstation.
33:28.080–33:38.080
You can even start with cheap cards like the MI25 and the P100 and then upgrade to faster GPUs when you are ready for that switch.
33:38.580–33:47.720
As I mentioned at the start of the video, bargain hardware is offering a 10% discount on all of the GPUs that they offer at the link in the description.
33:48.160–33:53.520
So if you want to use that discount, just type Donato 10 at checkout.
33:53.520–33:58.640
Again, this is not an affiliate link and I get zero commission.
33:58.980–34:07.520
My goal is just to see what I can do to make hardware a bit more affordable for anyone looking to get started with local inference.
34:08.460–34:18.400
So in the description of this video, you will also find all of the GitHub links to the toolboxes and configurations that I put together for the GPUs that I tested in this video.
34:18.400–34:26.720
So to wrap up, as usual, remember this channel is a hobby project and a significant amount of time goes into making these videos.
34:26.880–34:34.480
If you find the work useful and want to support my research and me maintaining all of the different toolboxes and containers,
34:35.000–34:37.840
the link to support the channel is in the description.
34:38.220–34:41.200
You can use Buy Me A Coffee to make a donation.
34:41.740–34:44.440
Thanks for watching and I'll see you in the next one.
34:48.400–34:49.000
Thank you.
0:00.400–0:07.200
（此句尚無繁中翻譯）
0:07.200–0:13.600
（此句尚無繁中翻譯）
0:13.600–0:21.760
（此句尚無繁中翻譯）
0:21.760–0:26.400
（此句尚無繁中翻譯）
0:26.400–0:31.200
（此句尚無繁中翻譯）
0:31.200–0:34.160
（此句尚無繁中翻譯）
0:34.160–0:40.240
（此句尚無繁中翻譯）
0:40.240–0:42.960
（此句尚無繁中翻譯）
0:43.520–0:49.280
（此句尚無繁中翻譯）
0:49.280–0:52.480
（此句尚無繁中翻譯）
0:52.480–0:59.280
（此句尚無繁中翻譯）
0:59.280–1:00.720
（此句尚無繁中翻譯）
1:00.720–1:09.920
（此句尚無繁中翻譯）
1:10.640–1:14.800
（此句尚無繁中翻譯）
1:14.800–1:17.920
（此句尚無繁中翻譯）
1:17.920–1:26.080
（此句尚無繁中翻譯）
1:26.080–1:33.920
（此句尚無繁中翻譯）
1:33.920–1:35.600
（此句尚無繁中翻譯）
1:35.600–1:43.920
（此句尚無繁中翻譯）
1:43.920–1:44.800
（此句尚無繁中翻譯）
1:44.800–1:48.720
（此句尚無繁中翻譯）
1:48.720–1:54.640
（此句尚無繁中翻譯）
1:54.640–2:00.960
（此句尚無繁中翻譯）
2:01.920–2:08.880
（此句尚無繁中翻譯）
2:08.880–2:15.120
（此句尚無繁中翻譯）
2:15.120–2:19.680
（此句尚無繁中翻譯）
2:19.680–2:22.960
（此句尚無繁中翻譯）
2:22.960–2:28.720
（此句尚無繁中翻譯）
2:28.720–2:32.160
（此句尚無繁中翻譯）
2:32.160–2:39.040
（此句尚無繁中翻譯）
2:39.040–2:40.560
（此句尚無繁中翻譯）
2:40.560–2:47.440
（此句尚無繁中翻譯）
2:47.440–2:56.560
（此句尚無繁中翻譯）
2:56.560–3:00.640
（此句尚無繁中翻譯）
3:00.640–3:08.080
（此句尚無繁中翻譯）
3:08.080–3:17.200
（此句尚無繁中翻譯）
3:17.200–3:21.600
（此句尚無繁中翻譯）
3:21.600–3:27.440
（此句尚無繁中翻譯）
3:27.440–3:34.320
（此句尚無繁中翻譯）
3:34.320–3:41.120
（此句尚無繁中翻譯）
3:41.120–3:48.320
（此句尚無繁中翻譯）
3:48.320–3:54.080
（此句尚無繁中翻譯）
3:54.080–3:54.800
（此句尚無繁中翻譯）
3:54.800–4:02.160
（此句尚無繁中翻譯）
4:02.160–4:04.720
（此句尚無繁中翻譯）
4:04.720–4:09.920
（此句尚無繁中翻譯）
4:09.920–4:12.400
（此句尚無繁中翻譯）
4:12.400–4:15.360
（此句尚無繁中翻譯）
4:15.360–4:20.880
（此句尚無繁中翻譯）
4:20.880–4:28.400
（此句尚無繁中翻譯）
4:28.400–4:30.640
（此句尚無繁中翻譯）
4:30.640–4:33.280
（此句尚無繁中翻譯）
4:33.280–4:37.120
（此句尚無繁中翻譯）
4:37.120–4:44.640
（此句尚無繁中翻譯）
4:44.640–4:49.360
（此句尚無繁中翻譯）
4:49.360–4:52.000
（此句尚無繁中翻譯）
4:52.000–4:54.880
（此句尚無繁中翻譯）
4:54.880–5:03.440
（此句尚無繁中翻譯）
5:04.080–5:08.640
（此句尚無繁中翻譯）
5:08.640–5:13.520
（此句尚無繁中翻譯）
5:13.520–5:20.480
（此句尚無繁中翻譯）
5:20.480–5:22.560
（此句尚無繁中翻譯）
5:22.560–5:32.080
（此句尚無繁中翻譯）
5:32.080–5:37.920
（此句尚無繁中翻譯）
5:37.920–5:44.400
（此句尚無繁中翻譯）
5:44.400–5:49.200
（此句尚無繁中翻譯）
5:49.200–5:55.040
（此句尚無繁中翻譯）
5:55.040–5:57.840
（此句尚無繁中翻譯）
5:57.840–6:05.360
（此句尚無繁中翻譯）
6:05.360–6:15.600
（此句尚無繁中翻譯）
6:15.600–6:19.040
（此句尚無繁中翻譯）
6:19.760–6:26.880
（此句尚無繁中翻譯）
6:26.880–6:34.160
（此句尚無繁中翻譯）
6:34.160–6:38.320
（此句尚無繁中翻譯）
6:38.320–6:42.960
（此句尚無繁中翻譯）
6:42.960–6:48.560
（此句尚無繁中翻譯）
6:48.560–6:55.680
（此句尚無繁中翻譯）
6:55.680–7:04.560
（此句尚無繁中翻譯）
7:06.480–7:12.000
（此句尚無繁中翻譯）
7:12.000–7:20.000
（此句尚無繁中翻譯）
7:20.000–7:24.640
（此句尚無繁中翻譯）
7:24.640–7:27.520
（此句尚無繁中翻譯）
7:27.520–7:30.640
（此句尚無繁中翻譯）
7:31.200–7:36.720
（此句尚無繁中翻譯）
7:36.720–7:42.240
（此句尚無繁中翻譯）
7:42.720–7:46.640
（此句尚無繁中翻譯）
7:46.640–7:49.600
（此句尚無繁中翻譯）
7:49.600–7:53.200
（此句尚無繁中翻譯）
7:53.200–7:55.200
（此句尚無繁中翻譯）
7:55.200–7:58.000
（此句尚無繁中翻譯）
7:58.560–8:01.040
（此句尚無繁中翻譯）
8:01.920–8:06.400
（此句尚無繁中翻譯）
8:06.400–8:11.840
（此句尚無繁中翻譯）
8:11.840–8:14.720
（此句尚無繁中翻譯）
8:14.720–8:22.800
（此句尚無繁中翻譯）
8:22.800–8:26.080
（此句尚無繁中翻譯）
8:26.080–8:30.400
（此句尚無繁中翻譯）
8:30.400–8:33.120
（此句尚無繁中翻譯）
8:38.320–8:42.800
（此句尚無繁中翻譯）
8:42.800–8:47.360
（此句尚無繁中翻譯）
8:47.360–8:51.280
（此句尚無繁中翻譯）
8:51.280–8:52.960
（此句尚無繁中翻譯）
8:52.960–8:56.800
（此句尚無繁中翻譯）
8:56.800–9:03.280
（此句尚無繁中翻譯）
9:04.000–9:07.360
（此句尚無繁中翻譯）
9:07.360–9:12.000
（此句尚無繁中翻譯）
9:15.040–9:22.560
（此句尚無繁中翻譯）
9:22.560–9:27.840
（此句尚無繁中翻譯）
9:29.840–9:31.360
（此句尚無繁中翻譯）
9:31.360–9:35.600
（此句尚無繁中翻譯）
9:36.320–9:38.240
（此句尚無繁中翻譯）
9:38.240–9:41.760
（此句尚無繁中翻譯）
9:41.760–9:44.640
（此句尚無繁中翻譯）
9:45.600–9:51.760
（此句尚無繁中翻譯）
9:51.760–9:54.400
（此句尚無繁中翻譯）
9:54.400–9:54.800
（此句尚無繁中翻譯）
9:54.800–9:57.760
（此句尚無繁中翻譯）
9:57.760–9:58.160
（此句尚無繁中翻譯）
9:58.160–9:59.920
（此句尚無繁中翻譯）
9:59.920–10:02.800
（此句尚無繁中翻譯）
10:02.800–10:03.680
（此句尚無繁中翻譯）
10:03.680–10:06.080
（此句尚無繁中翻譯）
10:06.080–10:10.080
（此句尚無繁中翻譯）
10:10.080–10:11.440
（此句尚無繁中翻譯）
10:12.640–10:18.880
（此句尚無繁中翻譯）
10:18.880–10:23.680
（此句尚無繁中翻譯）
10:24.960–10:30.720
（此句尚無繁中翻譯）
10:31.600–10:36.080
（此句尚無繁中翻譯）
10:37.200–10:38.800
（此句尚無繁中翻譯）
10:38.800–10:44.640
（此句尚無繁中翻譯）
10:44.640–10:47.840
（此句尚無繁中翻譯）
10:50.240–10:55.440
（此句尚無繁中翻譯）
10:55.440–11:01.600
（此句尚無繁中翻譯）
11:01.600–11:05.680
（此句尚無繁中翻譯）
11:05.680–11:09.440
（此句尚無繁中翻譯）
11:09.440–11:13.680
（此句尚無繁中翻譯）
11:15.520–11:15.760
（此句尚無繁中翻譯）
11:15.760–11:21.760
（此句尚無繁中翻譯）
11:25.440–11:27.280
（此句尚無繁中翻譯）
11:27.280–11:31.760
（此句尚無繁中翻譯）
11:31.760–11:37.440
（此句尚無繁中翻譯）
11:37.440–11:39.680
（此句尚無繁中翻譯）
11:39.680–11:47.040
（此句尚無繁中翻譯）
11:47.040–11:51.760
（此句尚無繁中翻譯）
11:51.760–11:56.880
（此句尚無繁中翻譯）
11:56.880–12:02.320
（此句尚無繁中翻譯）
12:02.320–12:06.080
（此句尚無繁中翻譯）
12:06.080–12:08.400
（此句尚無繁中翻譯）
12:08.400–12:12.560
（此句尚無繁中翻譯）
12:12.560–12:15.680
（此句尚無繁中翻譯）
12:15.680–12:21.200
（此句尚無繁中翻譯）
12:21.200–12:25.840
（此句尚無繁中翻譯）
12:26.880–12:28.560
（此句尚無繁中翻譯）
12:28.560–12:30.960
（此句尚無繁中翻譯）
12:30.960–12:35.760
（此句尚無繁中翻譯）
12:35.760–12:40.000
（此句尚無繁中翻譯）
12:40.000–12:43.200
（此句尚無繁中翻譯）
12:43.200–12:45.280
（此句尚無繁中翻譯）
12:45.280–12:47.680
（此句尚無繁中翻譯）
12:48.240–12:53.440
（此句尚無繁中翻譯）
12:54.000–12:58.320
（此句尚無繁中翻譯）
12:58.320–13:04.560
（此句尚無繁中翻譯）
13:04.560–13:09.600
（此句尚無繁中翻譯）
13:09.600–13:11.040
（此句尚無繁中翻譯）
13:11.040–13:16.160
（此句尚無繁中翻譯）
13:16.160–13:18.640
（此句尚無繁中翻譯）
13:18.640–13:22.160
（此句尚無繁中翻譯）
13:22.160–13:27.120
（此句尚無繁中翻譯）
13:29.040–13:34.720
（此句尚無繁中翻譯）
13:34.720–13:36.080
（此句尚無繁中翻譯）
13:36.800–13:42.400
（此句尚無繁中翻譯）
13:42.960–13:45.600
（此句尚無繁中翻譯）
13:46.480–13:47.120
（此句尚無繁中翻譯）
13:47.840–13:48.480
（此句尚無繁中翻譯）
13:48.480–13:54.240
（此句尚無繁中翻譯）
13:54.800–13:57.200
（此句尚無繁中翻譯）
13:57.200–13:57.440
（此句尚無繁中翻譯）
13:58.000–14:05.360
（此句尚無繁中翻譯）
14:06.000–14:12.400
（此句尚無繁中翻譯）
14:12.400–14:19.680
（此句尚無繁中翻譯）
14:21.440–14:25.120
（此句尚無繁中翻譯）
14:25.120–14:27.600
（此句尚無繁中翻譯）
14:27.600–14:30.720
（此句尚無繁中翻譯）
14:31.200–14:33.840
（此句尚無繁中翻譯）
14:33.840–14:38.160
（此句尚無繁中翻譯）
14:38.160–14:42.240
（此句尚無繁中翻譯）
14:42.240–14:45.680
（此句尚無繁中翻譯）
14:45.680–14:51.520
（此句尚無繁中翻譯）
14:51.520–14:53.680
（此句尚無繁中翻譯）
14:53.680–15:10.800
（此句尚無繁中翻譯）
15:10.800–15:18.480
（此句尚無繁中翻譯）
15:18.480–15:20.960
（此句尚無繁中翻譯）
15:20.960–15:26.640
（此句尚無繁中翻譯）
15:26.640–15:35.120
（此句尚無繁中翻譯）
15:35.120–15:37.520
（此句尚無繁中翻譯）
15:37.520–15:41.040
（此句尚無繁中翻譯）
15:41.040–15:45.600
（此句尚無繁中翻譯）
15:45.600–15:50.480
（此句尚無繁中翻譯）
15:50.480–15:57.440
（此句尚無繁中翻譯）
15:57.440–15:59.760
（此句尚無繁中翻譯）
15:59.760–16:06.320
（此句尚無繁中翻譯）
16:06.320–16:11.840
（此句尚無繁中翻譯）
16:11.840–16:21.200
（此句尚無繁中翻譯）
16:21.200–16:28.320
（此句尚無繁中翻譯）
16:28.320–16:30.800
（此句尚無繁中翻譯）
16:30.800–16:34.880
（此句尚無繁中翻譯）
16:34.880–16:41.600
（此句尚無繁中翻譯）
16:41.600–16:46.080
（此句尚無繁中翻譯）
16:46.080–16:50.400
（此句尚無繁中翻譯）
16:50.400–16:58.320
（此句尚無繁中翻譯）
16:58.320–17:05.360
（此句尚無繁中翻譯）
17:05.360–17:11.040
（此句尚無繁中翻譯）
17:11.040–17:14.720
（此句尚無繁中翻譯）
17:14.720–17:19.920
（此句尚無繁中翻譯）
17:19.920–17:27.760
（此句尚無繁中翻譯）
17:27.760–17:34.640
（此句尚無繁中翻譯）
17:34.640–17:39.760
（此句尚無繁中翻譯）
17:39.760–17:48.080
（此句尚無繁中翻譯）
17:48.080–17:53.840
（此句尚無繁中翻譯）
17:53.840–17:57.680
（此句尚無繁中翻譯）
17:57.680–18:05.040
（此句尚無繁中翻譯）
18:05.040–18:12.240
（此句尚無繁中翻譯）
18:12.240–18:14.640
（此句尚無繁中翻譯）
18:14.640–18:19.680
（此句尚無繁中翻譯）
18:20.240–18:22.960
（此句尚無繁中翻譯）
18:22.960–18:27.920
（此句尚無繁中翻譯）
18:27.920–18:33.520
（此句尚無繁中翻譯）
18:33.520–18:37.680
（此句尚無繁中翻譯）
18:37.680–18:43.920
（此句尚無繁中翻譯）
18:43.920–18:47.200
（此句尚無繁中翻譯）
18:48.480–18:53.600
（此句尚無繁中翻譯）
18:53.600–18:56.000
（此句尚無繁中翻譯）
18:57.200–19:05.360
（此句尚無繁中翻譯）
19:05.360–19:14.480
（此句尚無繁中翻譯）
19:14.480–19:15.680
（此句尚無繁中翻譯）
19:15.680–19:21.440
（此句尚無繁中翻譯）
19:21.440–19:27.200
（此句尚無繁中翻譯）
19:27.200–19:35.440
（此句尚無繁中翻譯）
19:35.440–19:39.360
（此句尚無繁中翻譯）
19:39.920–19:44.720
（此句尚無繁中翻譯）
19:44.720–19:47.360
（此句尚無繁中翻譯）
19:47.920–19:50.400
（此句尚無繁中翻譯）
19:50.400–19:56.400
（此句尚無繁中翻譯）
19:56.960–20:03.760
（此句尚無繁中翻譯）
20:04.960–20:05.360
（此句尚無繁中翻譯）
20:05.360–20:08.240
（此句尚無繁中翻譯）
20:08.240–20:12.160
（此句尚無繁中翻譯）
20:12.160–20:15.040
（此句尚無繁中翻譯）
20:15.040–20:18.240
（此句尚無繁中翻譯）
20:18.240–20:24.880
（此句尚無繁中翻譯）
20:24.880–20:28.800
（此句尚無繁中翻譯）
20:28.800–20:29.520
（此句尚無繁中翻譯）
20:29.520–20:31.840
（此句尚無繁中翻譯）
20:31.840–20:40.960
（此句尚無繁中翻譯）
20:43.120–20:45.280
（此句尚無繁中翻譯）
20:45.280–20:49.440
（此句尚無繁中翻譯）
20:49.440–20:54.000
（此句尚無繁中翻譯）
20:54.000–20:59.520
（此句尚無繁中翻譯）
20:59.520–21:02.880
（此句尚無繁中翻譯）
21:02.880–21:05.680
（此句尚無繁中翻譯）
21:05.680–21:09.440
（此句尚無繁中翻譯）
21:09.440–21:14.560
（此句尚無繁中翻譯）
21:14.560–21:20.560
（此句尚無繁中翻譯）
21:20.560–21:26.000
（此句尚無繁中翻譯）
21:26.000–21:30.160
（此句尚無繁中翻譯）
21:30.160–21:36.640
（此句尚無繁中翻譯）
21:36.640–21:39.120
（此句尚無繁中翻譯）
21:39.120–21:43.600
（此句尚無繁中翻譯）
21:43.600–21:47.840
（此句尚無繁中翻譯）
21:47.840–21:51.440
（此句尚無繁中翻譯）
21:51.440–21:53.600
（此句尚無繁中翻譯）
21:53.600–21:57.040
（此句尚無繁中翻譯）
21:57.040–22:01.920
（此句尚無繁中翻譯）
22:01.920–22:04.960
（此句尚無繁中翻譯）
22:04.960–22:10.000
（此句尚無繁中翻譯）
22:10.000–22:15.920
（此句尚無繁中翻譯）
22:16.480–22:20.880
（此句尚無繁中翻譯）
22:20.880–22:27.520
（此句尚無繁中翻譯）
22:27.520–22:31.360
（此句尚無繁中翻譯）
22:31.360–22:34.880
（此句尚無繁中翻譯）
22:34.880–22:41.440
（此句尚無繁中翻譯）
22:41.440–22:43.840
（此句尚無繁中翻譯）
22:44.560–22:46.720
（此句尚無繁中翻譯）
22:46.720–22:50.080
（此句尚無繁中翻譯）
22:50.080–22:57.200
（此句尚無繁中翻譯）
22:57.200–22:57.840
（此句尚無繁中翻譯）
22:57.840–23:01.680
（此句尚無繁中翻譯）
23:02.240–23:09.040
（此句尚無繁中翻譯）
23:09.040–23:13.120
（此句尚無繁中翻譯）
23:13.120–23:22.480
（此句尚無繁中翻譯）
23:22.480–23:25.600
（此句尚無繁中翻譯）
23:25.600–23:27.440
（此句尚無繁中翻譯）
23:27.440–23:35.280
（此句尚無繁中翻譯）
23:35.280–23:37.440
（此句尚無繁中翻譯）
23:37.440–23:39.520
（此句尚無繁中翻譯）
23:39.520–23:42.240
（此句尚無繁中翻譯）
23:42.240–23:47.760
（此句尚無繁中翻譯）
23:47.760–23:48.960
（此句尚無繁中翻譯）
23:48.960–23:53.520
（此句尚無繁中翻譯）
23:53.520–23:56.160
（此句尚無繁中翻譯）
23:56.160–23:58.400
（此句尚無繁中翻譯）
23:58.400–24:02.960
（此句尚無繁中翻譯）
24:02.960–24:07.360
（此句尚無繁中翻譯）
24:07.360–24:13.200
（此句尚無繁中翻譯）
24:13.200–24:20.400
（此句尚無繁中翻譯）
24:20.400–24:25.360
（此句尚無繁中翻譯）
24:25.360–24:33.280
（此句尚無繁中翻譯）
24:33.280–24:34.560
（此句尚無繁中翻譯）
24:34.560–24:35.600
（此句尚無繁中翻譯）
24:35.600–24:38.000
（此句尚無繁中翻譯）
24:38.000–24:42.240
（此句尚無繁中翻譯）
24:42.240–24:47.840
（此句尚無繁中翻譯）
24:47.840–24:54.000
（此句尚無繁中翻譯）
24:54.800–25:00.800
（此句尚無繁中翻譯）
25:00.800–25:04.800
（此句尚無繁中翻譯）
25:04.800–25:10.320
（此句尚無繁中翻譯）
25:10.320–25:14.960
（此句尚無繁中翻譯）
25:14.960–25:20.000
（此句尚無繁中翻譯）
25:20.000–25:28.160
（此句尚無繁中翻譯）
25:28.160–25:33.760
（此句尚無繁中翻譯）
25:33.760–25:40.560
（此句尚無繁中翻譯）
25:40.560–25:46.720
（此句尚無繁中翻譯）
25:46.720–25:54.400
（此句尚無繁中翻譯）
25:54.400–26:04.240
（此句尚無繁中翻譯）
26:04.240–26:13.920
（此句尚無繁中翻譯）
26:13.920–26:19.040
（此句尚無繁中翻譯）
26:19.040–26:29.680
（此句尚無繁中翻譯）
26:29.680–26:33.040
（此句尚無繁中翻譯）
26:33.040–26:51.440
（此句尚無繁中翻譯）
26:51.440–26:54.080
（此句尚無繁中翻譯）
26:54.080–26:57.680
（此句尚無繁中翻譯）
26:57.680–27:10.400
（此句尚無繁中翻譯）
27:10.400–27:13.920
（此句尚無繁中翻譯）
27:13.920–27:16.800
（此句尚無繁中翻譯）
27:16.800–27:22.880
（此句尚無繁中翻譯）
27:22.880–27:29.200
（此句尚無繁中翻譯）
27:29.200–27:37.200
（此句尚無繁中翻譯）
27:37.200–27:41.520
（此句尚無繁中翻譯）
27:41.520–27:44.640
（此句尚無繁中翻譯）
27:44.640–27:48.080
（此句尚無繁中翻譯）
27:48.080–27:57.440
（此句尚無繁中翻譯）
27:57.440–27:59.200
（此句尚無繁中翻譯）
27:59.200–28:03.520
（此句尚無繁中翻譯）
28:03.520–28:07.520
（此句尚無繁中翻譯）
28:07.520–28:17.200
（此句尚無繁中翻譯）
28:17.200–28:26.560
（此句尚無繁中翻譯）
28:26.560–28:32.400
（此句尚無繁中翻譯）
28:32.400–28:44.560
（此句尚無繁中翻譯）
28:44.560–28:47.280
（此句尚無繁中翻譯）
28:47.280–28:52.160
（此句尚無繁中翻譯）
28:52.160–28:55.680
（此句尚無繁中翻譯）
28:55.680–29:04.800
（此句尚無繁中翻譯）
29:05.200–29:08.760
（此句尚無繁中翻譯）
29:08.760–29:11.720
（此句尚無繁中翻譯）
29:11.720–29:15.880
（此句尚無繁中翻譯）
29:15.880–29:19.560
（此句尚無繁中翻譯）
29:19.560–29:25.400
（此句尚無繁中翻譯）
29:25.840–29:30.840
（此句尚無繁中翻譯）
29:31.760–29:33.280
（此句尚無繁中翻譯）
29:33.560–29:38.600
（此句尚無繁中翻譯）
29:38.760–29:41.640
（此句尚無繁中翻譯）
29:41.640–29:52.120
（此句尚無繁中翻譯）
29:52.120–30:00.760
（此句尚無繁中翻譯）
30:01.080–30:05.480
（此句尚無繁中翻譯）
30:05.480–30:10.600
（此句尚無繁中翻譯）
30:10.600–30:13.320
（此句尚無繁中翻譯）
30:13.800–30:16.920
（此句尚無繁中翻譯）
30:17.320–30:24.220
（此句尚無繁中翻譯）
30:24.440–30:27.500
（此句尚無繁中翻譯）
30:27.880–30:32.580
（此句尚無繁中翻譯）
30:32.580–30:34.920
（此句尚無繁中翻譯）
30:34.920–30:39.260
（此句尚無繁中翻譯）
30:39.640–30:46.280
（此句尚無繁中翻譯）
30:46.640–30:54.720
（此句尚無繁中翻譯）
30:54.720–30:59.280
（此句尚無繁中翻譯）
30:59.280–31:05.820
（此句尚無繁中翻譯）
31:06.140–31:15.040
（此句尚無繁中翻譯）
31:15.040–31:25.900
（此句尚無繁中翻譯）
31:26.280–31:28.320
（此句尚無繁中翻譯）
31:28.700–31:38.240
（此句尚無繁中翻譯）
31:38.240–31:47.620
（此句尚無繁中翻譯）
31:48.100–31:55.060
（此句尚無繁中翻譯）
31:55.220–32:02.060
（此句尚無繁中翻譯）
32:02.060–32:10.200
（此句尚無繁中翻譯）
32:10.200–32:21.960
（此句尚無繁中翻譯）
32:21.960–32:33.860
（此句尚無繁中翻譯）
32:34.120–32:47.040
（此句尚無繁中翻譯）
32:47.040–32:53.280
（此句尚無繁中翻譯）
32:53.280–32:59.760
（此句尚無繁中翻譯）
33:00.100–33:02.640
（此句尚無繁中翻譯）
33:02.640–33:11.100
（此句尚無繁中翻譯）
33:11.500–33:18.040
（此句尚無繁中翻譯）
33:18.320–33:28.080
（此句尚無繁中翻譯）
33:28.080–33:38.080
（此句尚無繁中翻譯）
33:38.580–33:47.720
（此句尚無繁中翻譯）
33:48.160–33:53.520
（此句尚無繁中翻譯）
33:53.520–33:58.640
（此句尚無繁中翻譯）
33:58.980–34:07.520
（此句尚無繁中翻譯）
34:08.460–34:18.400
（此句尚無繁中翻譯）
34:18.400–34:26.720
（此句尚無繁中翻譯）
34:26.880–34:34.480
（此句尚無繁中翻譯）
34:35.000–34:37.840
（此句尚無繁中翻譯）
34:38.220–34:41.200
（此句尚無繁中翻譯）
34:41.740–34:44.440
（此句尚無繁中翻譯）
34:48.400–34:49.000
（此句尚無繁中翻譯）
0:00.400–0:07.200
So in this video, I want to continue the series I started on refurbished data center GPUs.
（此句尚無繁中翻譯）
0:07.200–0:13.600
I've been working on this video for the past two months and I want to start with an important premise.
（此句尚無繁中翻譯）
0:13.600–0:21.760
I know some people will feel compelled to use the comment section to vent and compare the hardware that we discuss here
（此句尚無繁中翻譯）
0:21.760–0:26.400
with stuff that's two to five times more expensive.
（此句尚無繁中翻譯）
0:26.400–0:31.200
First of all, this video comes out of a collaboration with Bargain Hardware.
（此句尚無繁中翻譯）
0:31.200–0:34.160
This is not a sponsored video.
（此句尚無繁中翻譯）
0:34.160–0:40.240
If you don't know them, Bargain Hardware are a major reseller of refurbished hardware.
（此句尚無繁中翻譯）
0:40.240–0:42.960
I am not getting paid for this.
（此句尚無繁中翻譯）
0:43.520–0:49.280
This collaboration simply gives me access to some hardware so I can learn an experiment,
（此句尚無繁中翻譯）
0:49.280–0:52.480
which is the main objective of this channel.
（此句尚無繁中翻譯）
0:52.480–0:59.280
Now, of course, I know that people that want to get into local AI are after alternatives to more
（此句尚無繁中翻譯）
0:59.280–1:00.720
expensive hardware.
（此句尚無繁中翻譯）
1:00.720–1:09.920
For reference, in August last year, one could get a 128GB Strixello machine for just over $2000.
（此句尚無繁中翻譯）
1:10.640–1:14.800
Now, we are looking at twice as much for the same machine.
（此句尚無繁中翻譯）
1:14.800–1:17.920
The landscape has totally changed.
（此句尚無繁中翻譯）
1:17.920–1:26.080
This is also why I am experimenting with refurbished data center hardware to see what kind of performance
（此句尚無繁中翻譯）
1:26.080–1:33.920
we can achieve, what the cost is, and most importantly, the trade-offs you need to be aware of if you decide
（此句尚無繁中翻譯）
1:33.920–1:35.600
to go this way.
（此句尚無繁中翻譯）
1:35.600–1:43.920
On top of all of this, Bargain Hardware is offering a 10% discount on the GPUs listed in the description of
（此句尚無繁中翻譯）
1:43.920–1:44.800
this video.
（此句尚無繁中翻譯）
1:44.800–1:48.720
Just use the code DONATO10 at checkout.
（此句尚無繁中翻譯）
1:48.720–1:54.640
This is not an affiliate link and I get zero if you buy or don't buy.
（此句尚無繁中翻譯）
1:54.640–2:00.960
We could have done a 5% split where you get 5% discount and I get 5%.
（此句尚無繁中翻譯）
2:01.920–2:08.880
But I am trying to see what I can do to make some hardware more affordable for my viewers,
（此句尚無繁中翻譯）
2:08.880–2:15.120
since it looks like that the memory and GPU situation that we have now will last for a while.
（此句尚無繁中翻譯）
2:15.120–2:19.680
So I'd rather get the full 10% discount to you.
（此句尚無繁中翻譯）
2:19.680–2:22.960
That's the only thing that's in my power.
（此句尚無繁中翻譯）
2:22.960–2:28.720
Now, the first consideration is that when you want to build a multi-GPU system,
（此句尚無繁中翻譯）
2:28.720–2:32.160
it's not just the GPUs that cost money.
（此句尚無繁中翻譯）
2:32.160–2:39.040
The issue is that unless you already have a rig to place your cards in, you also have to build a
（此句尚無繁中翻譯）
2:39.040–2:40.560
system to host them.
（此句尚無繁中翻譯）
2:40.560–2:47.440
If you watched my Dual Radio 9700 video, you'll have an idea of what that looks like.
（此句尚無繁中翻譯）
2:47.440–2:56.560
But that setup with 64GB of RAM would cost around $4,000 to $5,000 with current prices,
（此句尚無繁中翻譯）
2:56.560–3:00.640
maybe even more, I really cannot keep up myself.
（此句尚無繁中翻譯）
3:00.640–3:08.080
And if you want to scale up to four dual-slot GPUs, you need a motherboard that physically can
（此句尚無繁中翻譯）
3:08.080–3:17.200
accommodate them, a GPU with enough PCIe lanes to avoid lane starvation, a high capacity power supply,
（此句尚無繁中翻譯）
3:17.200–3:21.600
and a chassis with proper cooling, let alone system RAM.
（此句尚無繁中翻譯）
3:21.600–3:27.440
If you try to build that base system from scratch using currently available hardware,
（此句尚無繁中翻譯）
3:27.440–3:34.320
it can easily cost you around $4,000 before you even buy the GPUs.
（此句尚無繁中翻譯）
3:34.320–3:41.120
That is where refurbished data center servers start making sense in 2026.
（此句尚無繁中翻譯）
3:41.120–3:48.320
They offer great value by giving you a base system for a fraction of the cost.
（此句尚無繁中翻譯）
3:48.320–3:54.080
Now, Bargain Hardware is a simple online builder where you can select all the components for the
（此句尚無繁中翻譯）
3:54.080–3:54.800
system.
（此句尚無繁中翻譯）
3:54.800–4:02.160
For a quad GPU configuration, one of the cheapest options is to get a super micro server.
（此句尚無繁中翻譯）
4:02.160–4:04.720
I'll get this 12th generation one.
（此句尚無繁中翻譯）
4:04.720–4:09.920
The motherboard in these servers typically hosts two Intel Xeon CPUs.
（此句尚無繁中翻譯）
4:09.920–4:12.400
So we can now configure these.
（此句尚無繁中翻譯）
4:12.400–4:15.360
You can see there is a base price, of course.
（此句尚無繁中翻譯）
4:15.360–4:20.880
So the first thing we want to do is to select the CPUs.
（此句尚無繁中翻譯）
4:20.880–4:28.400
And here I'm going for two of these Xeon 18 core CPUs that you can see here.
（此句尚無繁中翻譯）
4:28.400–4:30.640
They are incredibly cheap.
（此句尚無繁中翻譯）
4:30.640–4:33.280
Then, of course, the heat sinks are already included.
（此句尚無繁中翻譯）
4:33.280–4:37.120
And for the RAM, as you can see here, there are a lot of options.
（此句尚無繁中翻譯）
4:37.120–4:44.640
Actually, Bargain Hardware stocks right now, a lot of DDR4 at a very, very good price.
（此句尚無繁中翻譯）
4:44.640–4:49.360
Here I'm getting four of these 16 GB sticks.
（此句尚無繁中翻譯）
4:49.360–4:52.000
Now we want to select the storage.
（此句尚無繁中翻譯）
4:52.000–4:54.880
And for the storage, I went with SATA.
（此句尚無繁中翻譯）
4:54.880–5:03.440
And I selected two of these 480 GB SSDs, which gives me almost one terabyte, which is what you want
（此句尚無繁中翻譯）
5:04.080–5:08.640
for LLMs, at least one terabyte, because you're going to be downloading a lot of weights.
（此句尚無繁中翻譯）
5:08.640–5:13.520
And moving on, now we select the GPU accelerators.
（此句尚無繁中翻譯）
5:13.520–5:20.480
Now for this build, I have selected four P100s, 16 GB each.
（此句尚無繁中翻譯）
5:20.480–5:22.560
So we get 64 GB.
（此句尚無繁中翻譯）
5:22.560–5:32.080
But in this video and in the benchmarks, I've also looked at the V100s and also the AMD MI25.
（此句尚無繁中翻譯）
5:32.080–5:37.920
And again, each of these options ultimately gives you four GPUs with 64 GB of RAM.
（此句尚無繁中翻譯）
5:37.920–5:44.400
But of course, the V100 is much more expensive because it's a more modern architecture.
（此句尚無繁中翻譯）
5:44.400–5:49.200
But check the website because more GPUs are coming.
（此句尚無繁中翻譯）
5:49.200–5:55.040
And probably by the time this video is uploaded, you will see some more options.
（此句尚無繁中翻譯）
5:55.040–5:57.840
I know that there are some other GPUs coming.
（此句尚無繁中翻譯）
5:57.840–6:05.360
RTX 3080, 3090, Quadro RTX 5000, Quadro M6000.
（此句尚無繁中翻譯）
6:05.360–6:15.600
But just check the stock because they are updating it and they will also make some A100s and RTX 8000 available.
（此句尚無繁中翻譯）
6:15.600–6:19.040
So there is quite a lot that's coming to their stock.
（此句尚無繁中翻譯）
6:19.760–6:26.880
But for this build, I will keep the P100s in there and we can add to the cart.
（此句尚無繁中翻譯）
6:26.880–6:34.160
Oh, actually, we forgot to select the Super Macro NVMe Enablement Kit.
（此句尚無繁中翻譯）
6:34.160–6:38.320
We add to the cart and now we can go and check out.
（此句尚無繁中翻譯）
6:38.320–6:42.960
And I just want to show you the different options and costs for delivery.
（此句尚無繁中翻譯）
6:42.960–6:48.560
That's actually at least in the UK for domestic delivery.
（此句尚無繁中翻譯）
6:48.560–6:55.680
When you look at this, it brings us to a total of just over 2000 pounds.
（此句尚無繁中翻譯）
6:55.680–7:04.560
But this is not including the 10% discount on the GPUs that you can get with the code Donato 10.
（此句尚無繁中翻譯）
7:06.480–7:12.000
Now, I thought it'd be interesting to actually go in person to their warehouse,
（此句尚無繁中翻譯）
7:12.000–7:20.000
see how an order like this is put together and generally get a look at the process they use to test and refurbish hardware.
（此句尚無繁中翻譯）
7:20.000–7:24.640
And a thank you to Jack Moyers from the Bargain Hardware team,
（此句尚無繁中翻譯）
7:24.640–7:27.520
who was kind enough to give me a tour.
（此句尚無繁中翻譯）
7:27.520–7:30.640
The first process at Bargain Hardware is inbound.
（此句尚無繁中翻譯）
7:31.200–7:36.720
So pallets of servers, workstations, components, they arrive on trucks.
（此句尚無繁中翻譯）
7:36.720–7:42.240
So we buy equipment from all kinds of places all around Europe and some more worldwide.
（此句尚無繁中翻譯）
7:42.720–7:46.640
And then before the servers and workstations and any equipment gets processed,
（此句尚無繁中翻譯）
7:46.640–7:49.600
it gets stored in our rack in here.
（此句尚無繁中翻譯）
7:49.600–7:53.200
So the first part of the refurbishment process is cleaning.
（此句尚無繁中翻譯）
7:53.200–7:55.200
So this is one of our cleaning rooms.
（此句尚無繁中翻譯）
7:55.200–7:58.000
So in here, the first part of the process is the servers,
（此句尚無繁中翻譯）
7:58.560–8:01.040
they get blown out with our air compressor machine.
（此句尚無繁中翻譯）
8:01.920–8:06.400
You can see here, Michael's doing some reskinning on a server.
（此句尚無繁中翻譯）
8:06.400–8:11.840
So certain generations of servers get reskinned just so we have a nicer finish.
（此句尚無繁中翻譯）
8:11.840–8:14.720
So it just gets rid of all these scuffs and scrapes off the lids.
（此句尚無繁中翻譯）
8:14.720–8:22.800
So sometimes we have orders which are just a batch of drives or it could be CPUs or GPUs.
（此句尚無繁中翻譯）
8:22.800–8:26.080
So not necessarily every component comes out of a machine.
（此句尚無繁中翻譯）
8:26.080–8:30.400
Sometimes we buy them in batch, but still every single component needs testing.
（此句尚無繁中翻譯）
8:30.400–8:33.120
So then we come to component testing here.
（此句尚無繁中翻譯）
8:38.320–8:42.800
So when a machine has been tested, the next thing that we do,
（此句尚無繁中翻譯）
8:42.800–8:47.360
because we stock our servers and workstations as configured to order,
（此句尚無繁中翻譯）
8:47.360–8:51.280
the next thing we need to do is take the components out and that's called disassembly.
（此句尚無繁中翻譯）
8:51.280–8:52.960
So this is the next section.
（此句尚無繁中翻譯）
8:52.960–8:56.800
So here we've got Harry who's working on a workstation.
（此句尚無繁中翻譯）
8:56.800–9:03.280
So he'll be taking out the CPU, the RAM, any GPUs and other cards and components so that
（此句尚無繁中翻譯）
9:04.000–9:07.360
when we stock it and we offer it as configured to order on our website,
（此句尚無繁中翻譯）
9:07.360–9:12.000
it's basically a chassis with a PSU and other base components.
（此句尚無繁中翻譯）
9:15.040–9:22.560
As we were walking around, my attention was immediately captured by a stock of GPUs that
（此句尚無繁中翻譯）
9:22.560–9:27.840
they got in and that they were testing and getting ready to refurbish.
（此句尚無繁中翻譯）
9:29.840–9:31.360
So then we come to picking.
（此句尚無繁中翻譯）
9:31.360–9:35.600
So if someone orders on our website, they've configured a server.
（此句尚無繁中翻譯）
9:36.320–9:38.240
We've stocked it as a chassis and components.
（此句尚無繁中翻譯）
9:38.240–9:41.760
So then our picking team will they'll receive the order.
（此句尚無繁中翻譯）
9:41.760–9:44.640
They'll have a list of all the components that need to be picked.
（此句尚無繁中翻譯）
9:45.600–9:51.760
And then it's just the case of them assembling it all onto a trolley ready for the next part of the process.
（此句尚無繁中翻譯）
9:51.760–9:54.400
It's actually very well organized.
（此句尚無繁中翻譯）
9:54.400–9:54.800
Yeah.
（此句尚無繁中翻譯）
9:54.800–9:57.760
Wherever I look, it's tidy.
（此句尚無繁中翻譯）
9:57.760–9:58.160
Yeah.
（此句尚無繁中翻譯）
9:58.160–9:59.920
Like look around me.
（此句尚無繁中翻譯）
9:59.920–10:02.800
There is nothing left to have.
（此句尚無繁中翻譯）
10:02.800–10:03.680
Yeah.
（此句尚無繁中翻譯）
10:03.680–10:06.080
This is where we keep all the RAM and CPUs.
（此句尚無繁中翻譯）
10:06.080–10:10.080
So there's a lot of value behind a closed door here.
（此句尚無繁中翻譯）
10:10.080–10:11.440
So we just need to keep it safe.
（此句尚無繁中翻譯）
10:12.640–10:18.880
Given the current situation with memory, it's refreshing to see how much DDR4 memory
（此句尚無繁中翻譯）
10:18.880–10:23.680
they've been able to stock at a pretty competitive price.
（此句尚無繁中翻譯）
10:24.960–10:30.720
So when a member of the picking team has picked an order, it goes onto a trolley
（此句尚無繁中翻譯）
10:31.600–10:36.080
and then the trolley is moved through to assembly so that that particular order can be assembled.
（此句尚無繁中翻譯）
10:37.200–10:38.800
So this is the assembly area.
（此句尚無繁中翻譯）
10:38.800–10:44.640
So you can see a member of staff is assembling an order that's been configured online.
（此句尚無繁中翻譯）
10:44.640–10:47.840
And then when the machine is assembled, it then needs to be tested.
（此句尚無繁中翻譯）
10:50.240–10:55.440
It's clear that they put a lot of effort in testing the builds before sending them out.
（此句尚無繁中翻譯）
10:55.440–11:01.600
And of course, they give people warranty, but also the care they put in packaging
（此句尚無繁中翻譯）
11:01.600–11:05.680
to make sure that stuff gets shipped in the best possible way.
（此句尚無繁中翻譯）
11:05.680–11:09.440
We've invested a lot in the cardboard boxes we use, in the foam inserts,
（此句尚無繁中翻譯）
11:09.440–11:13.680
making sure that every server is catered for and that it's really secure inside.
（此句尚無繁中翻譯）
11:15.520–11:15.760
Yeah.
（此句尚無繁中翻譯）
11:15.760–11:21.760
So once the order has been packed, then we've got our outbound department here.
（此句尚無繁中翻譯）
11:25.440–11:27.280
So we've got our outbound department here.
（此句尚無繁中翻譯）
11:27.280–11:31.760
At the end of the warehouse tour, I spent some time with Toby Sheriff,
（此句尚無繁中翻譯）
11:31.760–11:37.440
who's the lead engineer that actually put together the server build you see me configure
（此句尚無繁中翻譯）
11:37.440–11:39.680
at the beginning of this video.
（此句尚無繁中翻譯）
11:39.680–11:47.040
What we've got here is a Supermicro DGQ in CSE 118 chassis.
（此句尚無繁中翻譯）
11:47.040–11:51.760
It's got space for four double wide full height cards.
（此句尚無繁中翻譯）
11:51.760–11:56.880
It takes the scalable CPUs, Intel first and second generation.
（此句尚無繁中翻譯）
11:56.880–12:02.320
So currently we've got a gold 6150 or two gold 6150s in there.
（此句尚無繁中翻譯）
12:02.320–12:06.080
So I believe they're 18 core CPUs, but they're also threaded.
（此句尚無繁中翻譯）
12:06.080–12:08.400
So obviously you've got twice as many of them than that.
（此句尚無繁中翻譯）
12:08.400–12:12.560
We've also got some DDR4 registered DIMMs.
（此句尚無繁中翻譯）
12:12.560–12:15.680
So it can take up to six per CPU.
（此句尚無繁中翻譯）
12:15.680–12:21.200
Currently we've got 64 gigs in there in 16 gig DIMMs.
（此句尚無繁中翻譯）
12:21.200–12:25.840
That's running at 2666 megahertz mega transfers.
（此句尚無繁中翻譯）
12:26.880–12:28.560
And that's just a limitation of the CPU.
（此句尚無繁中翻譯）
12:28.560–12:30.960
So there's not really anything you can do about that.
（此句尚無繁中翻譯）
12:30.960–12:35.760
If you go to a second generation CPU, you'd get slightly faster at 2933.
（此句尚無繁中翻譯）
12:35.760–12:40.000
So it's not fully utilising the speed, but it shouldn't really be a problem.
（此句尚無繁中翻譯）
12:40.000–12:43.200
We've also got on board 10 gig NICs.
（此句尚無繁中翻譯）
12:43.200–12:45.280
So that's integrated into the motherboard.
（此句尚無繁中翻譯）
12:45.280–12:47.680
We've also got two spare PCI slots at the rear.
（此句尚無繁中翻譯）
12:48.240–12:53.440
If you wanted to add a network card, maybe a PCIe storage drive or something.
（此句尚無繁中翻譯）
12:54.000–12:58.320
Half of these three slots are CPU2 dependent.
（此句尚無繁中翻譯）
12:58.320–13:04.560
So you could, if you felt so inclined, just put two cards in it initially with one CPU.
（此句尚無繁中翻譯）
13:04.560–13:09.600
And expand in the future if that was, you know, that was what you wanted to do.
（此句尚無繁中翻譯）
13:09.600–13:11.040
Keep the cost down initially.
（此句尚無繁中翻譯）
13:11.040–13:16.160
Storage wise, at the front, you've got two NVMe ports built into a backplane.
（此句尚無繁中翻譯）
13:16.160–13:18.640
So that would do U.2 drives.
（此句尚無繁中翻譯）
13:18.640–13:22.160
So two and a quarter inch NVMe drives.
（此句尚無繁中翻譯）
13:22.160–13:27.120
And then we've also got two SATA slots just behind that backplane internally.
（此句尚無繁中翻譯）
13:29.040–13:34.720
So because these are data center graphics cards, they're not designed like a consumer card with a fan.
（此句尚無繁中翻譯）
13:34.720–13:36.080
As you said, they're passively cooled.
（此句尚無繁中翻譯）
13:36.800–13:42.400
So that has got to travel through the the the fin stack as it, you know, travels through the server.
（此句尚無繁中翻譯）
13:42.960–13:45.600
So you've got all these fans across the front here.
（此句尚無繁中翻譯）
13:46.480–13:47.120
I won't lie.
（此句尚無繁中翻譯）
13:47.840–13:48.480
It is loud.
（此句尚無繁中翻譯）
13:48.480–13:54.240
But it's got to be to be able to push enough air through to keep all these cards and your CPU
（此句尚無繁中翻譯）
13:54.800–13:57.200
and even your power supplies or network cards at the back.
（此句尚無繁中翻譯）
13:57.200–13:57.440
Cool.
（此句尚無繁中翻譯）
13:58.000–14:05.360
So right now we've got four P100s in here, but obviously we can swap any card that would fit in there.
（此句尚無繁中翻譯）
14:06.000–14:12.400
In the video, I'm going to show you the performance with the, I think I have it here,
（此句尚無繁中翻譯）
14:12.400–14:19.680
with one of the MI-50s, so MI-25 from AMD.
（此句尚無繁中翻譯）
14:21.440–14:25.120
And then maybe we are also going to try, is this the...
（此句尚無繁中翻譯）
14:25.120–14:27.600
It's the V100, that one.
（此句尚無繁中翻譯）
14:27.600–14:30.720
The V100, which is going to perform so much better.
（此句尚無繁中翻譯）
14:31.200–14:33.840
And in fact, we'll probably do some benchmarks with that.
（此句尚無繁中翻譯）
14:33.840–14:38.160
So Toby is now closing the chassis, so we can power this on.
（此句尚無繁中翻譯）
14:38.160–14:42.240
And mostly I want to give you an idea of the noise levels.
（此句尚無繁中翻譯）
14:42.240–14:45.680
When you first power it on, this is very noisy.
（此句尚無繁中翻譯）
14:45.680–14:51.520
And then it becomes a little bit quieter, but obviously it is still quite noisy.
（此句尚無繁中翻譯）
14:51.520–14:53.680
And we need to be aware of that.
（此句尚無繁中翻譯）
14:53.680–15:10.800
So I wanted to also show you quickly the specs of these three GPUs side by side,
（此句尚無繁中翻譯）
15:10.800–15:18.480
along with more modern GPUs like the R9700 AI Pro, that I'm sure you've already seen on my channel,
（此句尚無繁中翻譯）
15:18.480–15:20.960
and also the RTX 5090.
（此句尚無繁中翻譯）
15:20.960–15:26.640
Now, during the comparison, keep in mind the considerable price difference.
（此句尚無繁中翻譯）
15:26.640–15:35.120
I have put it here, and obviously for the 5090 and the R9700, I've had to estimate some brackets,
（此句尚無繁中翻譯）
15:35.120–15:37.520
because the price is very variable.
（此句尚無繁中翻譯）
15:37.520–15:41.040
But for the other GPUs that you can get on bargain hardware,
（此句尚無繁中翻譯）
15:41.040–15:45.600
I have put the price that you would get with the 10% discount.
（此句尚無繁中翻譯）
15:45.600–15:50.480
One of the most important things to consider is memory bandwidth.
（此句尚無繁中翻譯）
15:50.480–15:57.440
During inference, generating each token requires reading a lot of data, such as model weights,
（此句尚無繁中翻譯）
15:57.440–15:59.760
back and forth from memory.
（此句尚無繁中翻譯）
15:59.760–16:06.320
So the bandwidth here directly limits how many tokens per second you can generate and process.
（此句尚無繁中翻譯）
16:06.320–16:11.840
So the V100 stands out at 900 gigabytes per second,
（此句尚無繁中翻譯）
16:11.840–16:21.200
which is actually higher than the more modern R9700, which has a memory bandwidth of 640 gigabytes per second.
（此句尚無繁中翻譯）
16:21.200–16:28.320
And the MI25 is significantly lower at 484 gigabytes per second.
（此句尚無繁中翻譯）
16:28.320–16:30.800
And that shows in the benchmarks.
（此句尚無繁中翻譯）
16:30.800–16:34.880
The other important factor is the data format support.
（此句尚無繁中翻譯）
16:34.880–16:41.600
Modern GPU architectures support formats like BF16 and FP8 in a native way.
（此句尚無繁中翻譯）
16:41.600–16:46.080
And these are the standard precisions used by current LLMs.
（此句尚無繁中翻譯）
16:46.080–16:50.400
None of these three older cards support these formats natively.
（此句尚無繁中翻譯）
16:50.400–16:58.320
They do support FP16, which is also a 16-bit format and uses the same amount of memory,
（此句尚無繁中翻譯）
16:58.320–17:05.360
but FP16 cannot represent as wide a range of values as BF16.
（此句尚無繁中翻譯）
17:05.360–17:11.040
When an inference framework converts BF16 model weights to FP16,
（此句尚無繁中翻譯）
17:11.040–17:14.720
some values can overflow or lose accuracy.
（此句尚無繁中翻譯）
17:14.720–17:19.920
And depending on the model, this can downgrade output quality.
（此句尚無繁中翻譯）
17:19.920–17:27.760
One last difference to keep in mind is that the V100 is more expensive because it has Tensor Cores,
（此句尚無繁中翻譯）
17:27.760–17:34.640
which are essentially hardware parts optimized for matrix operations used in machine learning.
（此句尚無繁中翻譯）
17:34.640–17:39.760
And you'd find such support also on all the modern GPUs.
（此句尚無繁中翻譯）
17:39.760–17:48.080
The P100 and MI25 do not have any equivalent, and this will be clearly reflected in the benchmarks.
（此句尚無繁中翻譯）
17:48.080–17:53.840
So very quickly, I want to show you how to set up this server.
（此句尚無繁中翻譯）
17:53.840–17:57.680
Right now, I've got the four V100s installed.
（此句尚無繁中翻譯）
17:57.680–18:05.040
So this is the repository I am going to use with the V100 AI toolboxes.
（此句尚無繁中翻譯）
18:05.040–18:12.240
As usual, with all of my repositories, you are going to need to create a toolbox.
（此句尚無繁中翻譯）
18:12.240–18:14.640
And this is essentially a Docker container.
（此句尚無繁中翻譯）
18:14.640–18:19.680
I'm not going to repeat myself. I explain about toolboxes, Podman and Docker containers
（此句尚無繁中翻譯）
18:20.240–18:22.960
in a lot of the other videos.
（此句尚無繁中翻譯）
18:22.960–18:27.920
You can choose between the CUDA backend and the Vulkan backend.
（此句尚無繁中翻譯）
18:27.920–18:33.520
And CUDA is almost always going to work better for Nvidia cards.
（此句尚無繁中翻譯）
18:33.520–18:37.680
So we're going to take that one and we're going to create the toolbox,
（此句尚無繁中翻譯）
18:37.680–18:43.920
which is essentially going to connect to Docker Hub and pull this toolbox that I have pre-built
（此句尚無繁中翻譯）
18:43.920–18:47.200
with Lama CPP. So let us do that.
（此句尚無繁中翻譯）
18:48.480–18:53.600
And actually, I have already created this toolbox, so I can enter it.
（此句尚無繁中翻譯）
18:53.600–18:56.000
Lama V100 CUDA.
（此句尚無繁中翻譯）
18:57.200–19:05.360
Now, once you enter the toolbox, you can run Lama CLI, list devices.
（此句尚無繁中翻譯）
19:05.360–19:14.480
And this essentially confirms that Lama via the CUDA backend can see the four V100s, each of them with 16
（此句尚無繁中翻譯）
19:14.480–19:15.680
gigabytes of VRAM.
（此句尚無繁中翻譯）
19:15.680–19:21.440
So now, if you want to run a model, the easiest way to do that is to download the model weights.
（此句尚無繁中翻譯）
19:21.440–19:27.200
I've already downloaded some of these in GGUF format from Hugging Face.
（此句尚無繁中翻譯）
19:27.200–19:35.440
And let's run, for example, one of the best models that we can have today for local agentic workflows,
（此句尚無繁中翻譯）
19:35.440–19:39.360
which is QAN 3.6, 27 billion parameters.
（此句尚無繁中翻譯）
19:39.920–19:44.720
And I think I have these in Q4 KXL quantization.
（此句尚無繁中翻譯）
19:44.720–19:47.360
So I'm just going to say Lama server.
（此句尚無繁中翻譯）
19:47.920–19:50.400
I'm going to pass the model.
（此句尚無繁中翻譯）
19:50.400–19:56.400
Actually, I also have it in Q8 quantization, for a matter of fact.
（此句尚無繁中翻譯）
19:56.960–20:03.760
And we are going to enable flash attention, and we can pass a context size.
（此句尚無繁中翻譯）
20:04.960–20:05.360
All right.
（此句尚無繁中翻譯）
20:05.360–20:08.240
So the server is up and running on port 8080.
（此句尚無繁中翻譯）
20:08.240–20:12.160
And now I need to forward that port to my actual laptop.
（此句尚無繁中翻譯）
20:12.160–20:15.040
So I can SSH into the box.
（此句尚無繁中翻譯）
20:15.040–20:18.240
I've called it BH for bargain hardware.
（此句尚無繁中翻譯）
20:18.240–20:24.880
And I'm going to say that I want to forward that port to 8081 on my host,
（此句尚無繁中翻譯）
20:24.880–20:28.800
because 8080 is already used by something else.
（此句尚無繁中翻譯）
20:28.800–20:29.520
And there we go.
（此句尚無繁中翻譯）
20:29.520–20:31.840
This is Lama CPP web UI.
（此句尚無繁中翻譯）
20:31.840–20:40.960
So I can type a prompt, such as write a CUDA kernel to multiply to
（此句尚無繁中翻譯）
20:43.120–20:45.280
As you can see now, it's thinking about it.
（此句尚無繁中翻譯）
20:45.280–20:49.440
When models spend a lot of time thinking,
（此句尚無繁中翻譯）
20:49.440–20:54.000
and you can see it's going at around 31 tokens per second.
（此句尚無繁中翻譯）
20:54.000–20:59.520
So for being a 27 billion parameter dense model, this is really good.
（此句尚無繁中翻譯）
20:59.520–21:02.880
But in a bit, we'll take a look at the proper benchmarks
（此句尚無繁中翻譯）
21:02.880–21:05.680
and what happens when the context size grows,
（此句尚無繁中翻譯）
21:05.680–21:09.440
and obviously what happens with other models and quantizations.
（此句尚無繁中翻譯）
21:09.440–21:14.560
But I just thought I'd show you how to get one of these up and running.
（此句尚無繁中翻譯）
21:14.560–21:20.560
Now, Lama CPP is my recommendation, especially for these older GPUs,
（此句尚無繁中翻譯）
21:20.560–21:26.000
not just the V100, but also the P100 and the MI25.
（此句尚無繁中翻譯）
21:26.000–21:30.160
And I've got toolboxes and links to all of those.
（此句尚無繁中翻譯）
21:30.160–21:36.640
And the reason for that is that Lama CPP has a much more uniform ecosystem,
（此句尚無繁中翻譯）
21:36.640–21:39.120
and it works pretty much everywhere.
（此句尚無繁中翻譯）
21:39.120–21:43.600
If your hardware is supported, you can run any model on it.
（此句尚無繁中翻譯）
21:43.600–21:47.840
And it's got pretty much any model and any quantization.
（此句尚無繁中翻譯）
21:47.840–21:51.440
And quantizations are going to be really important because, of course, here,
（此句尚無繁中翻譯）
21:51.440–21:53.600
you don't have that much memory available.
（此句尚無繁中翻譯）
21:53.600–21:57.040
I mean, in this case, you've got 64 gigabytes, which is quite a lot.
（此句尚無繁中翻譯）
21:57.040–22:01.920
But again, even if you want to run the QEM model that we just saw 27 billion parameters
（此句尚無繁中翻譯）
22:01.920–22:04.960
is a lot, then you are going to need a quantization.
（此句尚無繁中翻譯）
22:04.960–22:10.000
However, I know that a lot of people really, really like VLLM.
（此句尚無繁中翻譯）
22:10.000–22:15.920
So I've also created a VLLM toolbox that you can get up and running like this.
（此句尚無繁中翻譯）
22:16.480–22:20.880
So here, I've already created the toolbox so I can simply enter it.
（此句尚無繁中翻譯）
22:20.880–22:27.520
And you will see that, obviously, on this server, I have also the VLLM toolbox for the P100
（此句尚無繁中翻譯）
22:27.520–22:31.360
and the one for the AMD MI25.
（此句尚無繁中翻譯）
22:31.360–22:34.880
But obviously, in this case, we enter this particular one.
（此句尚無繁中翻譯）
22:34.880–22:41.440
And what I do in my toolboxes for VLLM, I always give you a start VLLM script
（此句尚無繁中翻譯）
22:41.440–22:43.840
with a list of models that I have tested.
（此句尚無繁中翻譯）
22:44.560–22:46.720
So at least you have a starting point.
（此句尚無繁中翻譯）
22:46.720–22:50.080
I use LAMA 3.1 just as a benchmark.
（此句尚無繁中翻譯）
22:50.080–22:57.200
But then if you want to run some proper models here, you can see the QEM 3.6 family, the 27 billion
（此句尚無繁中翻譯）
22:57.200–22:57.840
parameter.
（此句尚無繁中翻譯）
22:57.840–23:01.680
Now, this is in GPTQ 4-bit quantization.
（此句尚無繁中翻譯）
23:02.240–23:09.040
You cannot run, or at least I haven't been able to find a way on VLLM to run AWQ quants,
（此句尚無繁中翻譯）
23:09.040–23:13.120
which are a little bit better activation aware.
（此句尚無繁中翻譯）
23:13.120–23:22.480
But again, this is why I tell people that LAMA CPP is 90% of the times better for most people,
（此句尚無繁中翻譯）
23:22.480–23:25.600
especially on older hardware.
（此句尚無繁中翻譯）
23:25.600–23:27.440
But anyway, I've got four GPUs.
（此句尚無繁中翻譯）
23:27.440–23:35.280
So Tensor Parallelism 4, I can set concurrent requests, size of the context GPU utilization.
（此句尚無繁中翻譯）
23:35.280–23:37.440
I don't remember why I set it this low.
（此句尚無繁中翻譯）
23:37.440–23:39.520
You should probably set it a little bit higher.
（此句尚無繁中翻譯）
23:39.520–23:42.240
And then you can launch the server.
（此句尚無繁中翻譯）
23:42.240–23:47.760
And this should show you exactly how we are running that particular model.
（此句尚無繁中翻譯）
23:47.760–23:48.960
And it will take a little bit.
（此句尚無繁中翻譯）
23:48.960–23:53.520
And then VLLM will come up and you'll be able to use the model.
（此句尚無繁中翻譯）
23:53.520–23:56.160
So here we can see the model loading.
（此句尚無繁中翻譯）
23:56.160–23:58.400
It's taking quite a bit of time.
（此句尚無繁中翻譯）
23:58.400–24:02.960
And that's basically one of the caveats with PCI 3.0.
（此句尚無繁中翻譯）
24:02.960–24:07.360
Especially model loading is going to be fairly slow.
（此句尚無繁中翻譯）
24:07.360–24:13.200
And you can see here, it's also looking for different attention backhand that it can use.
（此句尚無繁中翻譯）
24:13.200–24:20.400
And ultimately, I think it's falling back to PyTorch, SDPA attention, which is okay,
（此句尚無繁中翻譯）
24:20.400–24:25.360
but it's not as good as some of the things you can get on modern hardware, of course.
（此句尚無繁中翻譯）
24:25.360–24:33.280
But again, this will work and you'll be able to run some models, even with VLLM, if you use these
（此句尚無繁中翻譯）
24:33.280–24:34.560
tool boxes.
（此句尚無繁中翻譯）
24:34.560–24:35.600
So here we go.
（此句尚無繁中翻譯）
24:35.600–24:38.000
This is now up and running.
（此句尚無繁中翻譯）
24:38.000–24:42.240
And you can start using it, for example, with a coding agent.
（此句尚無繁中翻譯）
24:42.240–24:47.840
But again, I would not recommend to use perhaps VLLM.
（此句尚無繁中翻譯）
24:47.840–24:54.000
Just stick to Lama CPP and you're going to get probably the best performance here.
（此句尚無繁中翻譯）
24:54.800–25:00.800
Let's now take a look at the benchmarks, which are arguably one of the most important factor in
（此句尚無繁中翻譯）
25:00.800–25:04.800
deciding whether or not some of these cards might be good for you.
（此句尚無繁中翻譯）
25:04.800–25:10.320
Here, I've got the individual repositories for all the cards that I tested.
（此句尚無繁中翻譯）
25:10.320–25:14.960
I should have the MI25 here as well.
（此句尚無繁中翻譯）
25:14.960–25:20.000
And I have put all of the benchmark results in the readme.
（此句尚無繁中翻譯）
25:20.000–25:28.160
So you can probably scroll and find them for Lama CPP and VLLM, token generation and prompt processing.
（此句尚無繁中翻譯）
25:28.160–25:33.760
So this is the V100 and obviously the P100.
（此句尚無繁中翻譯）
25:33.760–25:40.560
But obviously for this video to make things a little bit easier to compare, I just put everything together.
（此句尚無繁中翻譯）
25:40.560–25:46.720
You can see the comparison here of the three cards on different models.
（此句尚無繁中翻譯）
25:46.720–25:54.400
Now, these are modern LLMs that I recommend you run on these cards, mostly for agentic workflows and coding.
（此句尚無繁中翻譯）
25:54.400–26:04.240
The QAN 3.5 and 3.6 families are right now some of the best you can run with 32 to 64 gigabytes of RAM.
（此句尚無繁中翻譯）
26:04.240–26:13.920
And you can see obviously that the V100, the blue one here, it's the best performer at prompt processing and token generation.
（此句尚無繁中翻譯）
26:13.920–26:19.040
Even when the context goes to 32,000 tokens per second.
（此句尚無繁中翻譯）
26:19.040–26:29.680
And that's because this is just a more modern architecture and it's got tensor cores, which essentially allow to do matrix multiplication much better.
（此句尚無繁中翻譯）
26:29.680–26:33.040
And that's the core operation in LLMs.
（此句尚無繁中翻譯）
26:33.040–26:51.440
Just to give you an idea for a model like 27 billion parameters, you get on the V100, 852 tokens per second in prompt processing with the Q4 quantization, which is a very good quantization.
（此句尚無繁中翻譯）
26:51.440–26:54.080
And that's a very good prompt processing speed.
（此句尚無繁中翻譯）
26:54.080–26:57.680
And you get around 34 tokens per second.
（此句尚無繁中翻譯）
26:57.680–27:10.400
And even when you scale up the context, you see that you're still getting around 622 tokens per second on this 27 billion parameter model, which is a dense model.
（此句尚無繁中翻譯）
27:10.400–27:13.920
So these are the hardest model to run.
（此句尚無繁中翻譯）
27:13.920–27:16.800
And the performance is really good for these.
（此句尚無繁中翻譯）
27:16.800–27:22.880
And the tokens per second that you get in token generation is around 28 tokens per second.
（此句尚無繁中翻譯）
27:22.880–27:29.200
Again, this would be perfectly useful if you were using this model, let's say, in the PI coding agent.
（此句尚無繁中翻譯）
27:29.200–27:37.200
Actually, check out the video I've done on coding agents and it's going to give you an idea of the type of performance you can get.
（此句尚無繁中翻譯）
27:37.200–27:41.520
So you will also see that the MI25 is missing from some of these.
（此句尚無繁中翻譯）
27:41.520–27:44.640
It's just because it's the first card that I tested.
（此句尚無繁中翻譯）
27:44.640–27:48.080
And back then, I didn't include all the models.
（此句尚無繁中翻譯）
27:48.080–27:57.440
But the ones that I included, you can see that on LLAMA CPP, it does perform close to the P100, but just below it.
（此句尚無繁中翻譯）
27:57.440–27:59.200
So be aware of that.
（此句尚無繁中翻譯）
27:59.200–28:03.520
The MI25 is the one that's going to give you the least performance.
（此句尚無繁中翻譯）
28:03.520–28:07.520
So you can see that the other ones that you can see in the LLAMA CPP, but this is what you get on LLAMA CPP.
（此句尚無繁中翻譯）
28:07.520–28:17.200
You can also run VLLM, but with some caveats, VLLM is much more sensible to the different GPU architectures.
（此句尚無繁中翻譯）
28:17.200–28:26.560
You require specific kernels for specific architectures, and some of them are just not available for a lot of these cards.
（此句尚無繁中翻譯）
28:26.560–28:32.400
So you are not really able to pick and choose like you do with LLAMA CPP and run anything you want.
（此句尚無繁中翻譯）
28:32.400–28:44.560
But you can see that the MI25, when there are kernels available, actually performs better than the P100 on VLLM.
（此句尚無繁中翻譯）
28:44.560–28:47.280
So that's something to keep in mind.
（此句尚無繁中翻譯）
28:47.280–28:52.160
Again, I do not recommend using VLLM with these cards.
（此句尚無繁中翻譯）
28:52.160–28:55.680
A lot of models you just cannot easily run.
（此句尚無繁中翻譯）
28:55.680–29:04.800
But on the V100, at least, you can run some quantization of the QAN 3.6 27 billion parameter model.
（此句尚無繁中翻譯）
29:05.200–29:08.760
This is the GPT-Q 4-bit quantization.
（此句尚無繁中翻譯）
29:08.760–29:11.720
And you can see this throughput that you get.
（此句尚無繁中翻譯）
29:11.720–29:15.880
And you can also run the QAN 3.5 9 billion parameter model.
（此句尚無繁中翻譯）
29:15.880–29:19.560
But again, I do not recommend running any of these.
（此句尚無繁中翻譯）
29:19.560–29:25.400
I did try these, and I have all the toolboxes if you want to experiment,
（此句尚無繁中翻譯）
29:25.840–29:30.840
but probably stick to LLAMA CPP for these older GPUs.
（此句尚無繁中翻譯）
29:31.760–29:33.280
So we've seen the benchmarks.
（此句尚無繁中翻譯）
29:33.560–29:38.600
Now let's discuss the caveats and trade-offs you need to be aware of.
（此句尚無繁中翻譯）
29:38.760–29:41.640
First, the GPU architectures.
（此句尚無繁中翻譯）
29:41.640–29:52.120
The Pascal-based P100 and the Vega-based MI25 do not have hardware paths optimized for modern machine learning.
（此句尚無繁中翻譯）
29:52.120–30:00.760
The V100 does have Tensor Cores, but it is still a legacy architecture compared to modern GPUs.
（此句尚無繁中翻譯）
30:01.080–30:05.480
Because of this, although these cards will be able to run modern LLMs,
（此句尚無繁中翻譯）
30:05.480–30:10.600
they just cannot match the throughput that a modern GPU gives you.
（此句尚無繁中翻譯）
30:10.600–30:13.320
And that's reflected in the price you pay.
（此句尚無繁中翻譯）
30:13.800–30:16.920
Second, these setups are noisy.
（此句尚無繁中翻譯）
30:17.320–30:24.220
Because these passive cards require high airflow fans to stay cool, the system is loud.
（此句尚無繁中翻譯）
30:24.440–30:27.500
You will not want this server under your desk.
（此句尚無繁中翻譯）
30:27.880–30:32.580
This more likely belongs in a garage, a basement, or a dedicated room.
（此句尚無繁中翻譯）
30:32.580–30:34.920
Third, the system bus.
（此句尚無繁中翻譯）
30:34.920–30:39.260
These older servers use PCIe Gen 3 slots.
（此句尚無繁中翻譯）
30:39.640–30:46.280
A PCIe Gen 3 slot has a theoretical bandwidth of around 16GB per second,
（此句尚無繁中翻譯）
30:46.640–30:54.720
whereas a modern PCIe Gen 5 slot provides 64GB, four times the speed.
（此句尚無繁中翻譯）
30:54.720–30:59.280
For single GPU setups, the difference is not that noticeable,
（此句尚無繁中翻譯）
30:59.280–31:05.820
only adding a few seconds when you load the model weights from storage into the GPU memory.
（此句尚無繁中翻譯）
31:06.140–31:15.040
But if you run multi-GPU workloads, the GPUs must constantly exchange information and synchronize their activity.
（此句尚無繁中翻譯）
31:15.040–31:25.900
So, over a Gen 3 bus, this exchange of information will be slower and it will limit the overall throughput and token generation performance.
（此句尚無繁中翻譯）
31:26.280–31:28.320
The same is true for the memory.
（此句尚無繁中翻譯）
31:28.700–31:38.240
The service runs on DDR4-ACC memory, which usually operates at around 2400 to 3200 megatransfer per second.
（此句尚無繁中翻譯）
31:38.240–31:47.620
In comparison, DDR5 starts at 4800 megatransfer and goes much higher, effectively doubling the bandwidth.
（此句尚無繁中翻譯）
31:48.100–31:55.060
This does not affect you much if you choose models that fully fit inside the GPU memory.
（此句尚無繁中翻譯）
31:55.220–32:02.060
But if you want to do some offloading to the CPU and to the system memory, this will be a bottleneck.
（此句尚無繁中翻譯）
32:02.060–32:10.200
As I was editing this video, I realized that I was talking about PCIe 3.0 and DDR4.
（此句尚無繁中翻譯）
32:10.200–32:21.960
And I chose those for the workstations that we built here because I wanted to show you the cheapest available options to get a viable setup.
（此句尚無繁中翻譯）
32:21.960–32:33.860
But if you go on bargain hardware, for example, you can find servers and workstations with DDR5 and PCIe 4 and 5.
（此句尚無繁中翻譯）
32:34.120–32:47.040
Just to give you an example, these Dell Precision 7960, well, of course, more expensive, but these will come with DDR5 memory.
（此句尚無繁中翻譯）
32:47.040–32:53.280
And it will also come with PCIe 5.0.
（此句尚無繁中翻譯）
32:53.280–32:59.760
So these are the main trade-offs you make, which allow you to keep the total cost at this price point.
（此句尚無繁中翻譯）
33:00.100–33:02.640
Even with these caveats, the value is clear.
（此句尚無繁中翻譯）
33:02.640–33:11.100
If you have a budget of 2,000 to 3,000 pounds and you need 64 gigabytes of VRAM to run certain models,
（此句尚無繁中翻譯）
33:11.500–33:18.040
a refurbished enterprise server like the one I showed you is one of the real options available to you today.
（此句尚無繁中翻譯）
33:18.320–33:28.080
You get a pre-built base system that can host four cards for a fraction of the cost of a DIY workstation.
（此句尚無繁中翻譯）
33:28.080–33:38.080
You can even start with cheap cards like the MI25 and the P100 and then upgrade to faster GPUs when you are ready for that switch.
（此句尚無繁中翻譯）
33:38.580–33:47.720
As I mentioned at the start of the video, bargain hardware is offering a 10% discount on all of the GPUs that they offer at the link in the description.
（此句尚無繁中翻譯）
33:48.160–33:53.520
So if you want to use that discount, just type Donato 10 at checkout.
（此句尚無繁中翻譯）
33:53.520–33:58.640
Again, this is not an affiliate link and I get zero commission.
（此句尚無繁中翻譯）
33:58.980–34:07.520
My goal is just to see what I can do to make hardware a bit more affordable for anyone looking to get started with local inference.
（此句尚無繁中翻譯）
34:08.460–34:18.400
So in the description of this video, you will also find all of the GitHub links to the toolboxes and configurations that I put together for the GPUs that I tested in this video.
（此句尚無繁中翻譯）
34:18.400–34:26.720
So to wrap up, as usual, remember this channel is a hobby project and a significant amount of time goes into making these videos.
（此句尚無繁中翻譯）
34:26.880–34:34.480
If you find the work useful and want to support my research and me maintaining all of the different toolboxes and containers,
（此句尚無繁中翻譯）
34:35.000–34:37.840
the link to support the channel is in the description.
（此句尚無繁中翻譯）
34:38.220–34:41.200
You can use Buy Me A Coffee to make a donation.
（此句尚無繁中翻譯）
34:41.740–34:44.440
Thanks for watching and I'll see you in the next one.
（此句尚無繁中翻譯）
34:48.400–34:49.000
Thank you.
（此句尚無繁中翻譯）
影片筆記：Refurbished 64GB VRAM AI Server for Local AI: 4x NVIDIA V100/P100, AMD MI25
一句話總結
影片透過實測翻新資料中心伺服器（配備 4x P100/V100/MI25），驗證其在預算有限（約 2,000-3,000 英鎊）下運行本地大語言模型（LLM）的可行性，並比較 Lama CPP 與 VLLM 在舊硬體上的效能差異與限制。
核心重點
- 性價比優勢：自建多 GPU 系統（主機板、電源、散熱、機殼）基礎成本高昂（約 $4,000-$5,000），翻新資料中心伺服器（如 Supermicro DGQ）提供極高性價比，能以約 2,000 英鎊獲得 64GB VRAM 配置。
- 硬體規格與限制：
- 使用 PCIe Gen 3 與 DDR4 記憶體，頻寬較低，可能成為多卡同步或模型卸載至 CPU 時的瓶頸。
- 資料中心 GPU（P100/V100/MI25）多為被動散熱（Passively Cooled），依賴機箱高風量風扇，導致運轉噪音極大。
- GPU 效能比較：
- V100：具備 Tensor Cores 與高記憶體頻寬（900 GB/s），在推論效能上優於 P100 與 MI25。
- P100 & MI25：無 Tensor Cores 支援，頻寬較低（MI25 僅 484 GB/s），整體效能較差，但在 Lama CPP 上仍具可用性。
- 舊款 GPU 不原生支援 BF16/FP8，需轉換為 FP16，可能導致精度損失。
- 軟體生態建議：
- Lama CPP：推薦用於舊硬體，生態系統統一，相容性佳，支援多種量化格式。
- VLLM：對架構敏感，缺乏特定核心支援，載入模型慢，不建議在舊硬體上使用。
- 適用場景：適合預算在 2,000-3,000 英鎊，需要 64GB VRAM 運行小型模型（如 27B 參數量化版）的用戶，或作為進階升級前的過渡方案。
詳細大綱
1. 前言與合作聲明
- 本影片為「翻新資料中心 GPU」系列的延續，探討如何利用翻新硬體降低本地 AI 運算成本。
- 合作關係：與 Bargain Hardware 合作，但非贊助影片。創作者未獲直接報酬，僅為獲取硬體進行實驗。
- 折扣資訊：提供觀眾專屬折扣碼
DONATO10（非聯盟連結），可獲得 GPU 10% 折扣。 - 市場背景：
- 過去（去年 8 月）可花約 $2,000 購買 128GB Strixello 機器，現在價格翻倍。
- 記憶體與 GPU 短缺狀況可能持續，促使探索翻新硬體作為替代方案。
2. 自建系統的成本挑戰
- 多 GPU 系統不僅是 GPU 的成本，還包含主機系統建置成本。
- 參考之前的 Dual Radio 9700 影片，64GB RAM 配置成本約 $4,000-$5,000。
- 四張雙槽（Dual-slot）GPU 所需基礎設施：
- 支援物理安裝的主機板。
- 足夠 PCIe 頻寬以避免頻寬瓶頸（Lane starvation）。
- 高容量電源供應器。
- 具備適當散熱的機殼。
- 系統記憶體（System RAM）。
- 結論：從零開始自建基礎系統成本高昂，翻新資料中心伺服器提供極高性價比。
3. Bargain Hardware 線上組裝示範
- 目標配置：四卡 GPU 配置（Quad GPU Configuration）。
- 硬體選擇步驟：
- 主機板/伺服器：選擇 Supermicro 第 12 代伺服器。
- CPU：選擇兩顆 Intel Xeon 18 核心 CPU（價格低廉）。
- 散熱器：已包含在內。
- 記憶體：選擇四根 16GB DDR4 記憶體條（DDR4 庫存充足且價格優）。
- 儲存：選擇兩顆 480GB SATA SSD（總計近 1TB，適合 LLM 權重下載）。
- GPU 加速器：
- 本配置選擇四張 P100（每張 16GB，總計 64GB）。
- 影片中也測試了 V100 和 AMD MI25。
- 其他即將到貨選項：RTX 3080, 3090, Quadro RTX 5000, Quadro M6000, A100, RTX 8000。
- 額外配件：Super Micro NVMe Enablement Kit。
- 價格與折扣：
- 總價約 2000 英鎊（英國國內運費）。
- 可使用折扣碼
Donato 10獲得 GPU 10% 折扣。
4. 倉庫翻新流程導覽
- 入庫（Inbound）：
- 伺服器、工作站、組件通過貨車抵達。
- 從歐洲及全球各地採購設備。
- 暫存於機架中。
- 清潔（Cleaning）：
- 使用空氣壓縮機吹除灰塵。
- 部分世代伺服器進行重新貼皮（Reskinning）以去除刮痕。
- 即使是單獨購買的驅動器、CPU 或 GPU 也需測試。
- 組件測試與拆解（Component Testing & Disassembly）：
- 測試後的機器進行拆解。
- 取出 CPU、RAM、GPU 及其他卡片。
- 將機殼、PSU 及基礎組件作為庫存儲存，以便後續客製化組裝。
- 揀貨（Picking）：
- 收到線上訂單後，根據清單揀選組件。
- 將組件裝上推車。
- 倉庫整潔有序，DDR4 記憶體庫存豐富。
- 組裝（Assembly）：
- 將揀選的組件組裝成訂單配置的伺服器。
- 測試與包裝（Testing & Packaging）：
- 組裝後的機器進行測試。
- 提供保修服務。
- 投資於紙箱與泡沫襯墊，確保運輸安全。
- 出貨（Outbound）：
- 最後的出貨部門處理發運。
5. 最終硬體規格與噪音測試
- 硬體規格詳情（由 Lead Engineer Toby Sheriff 展示）：
- 機型：Supermicro DGQ in CSE 118 chassis。
- 擴充能力：支援四張雙寬全高（Double wide full height）卡片。
- CPU：支援可擴展 CPU（Intel 1st & 2nd Gen），目前安裝兩顆 Gold 6150（18 核心，支援超執行緒）。
- 記憶體：DDR4 Registered DIMMs，每 CPU 最多 6 根。目前安裝 64GB（4x 16GB DIMMs），運行於 2666 MHz（受 CPU 限制，第 2 代 CPU 可達 2933 MHz）。
- 網路：內建 10 Gig NIC（主機板整合）。
- PCIe 插槽：後方有兩個備用 PCIe 插槽（部分依賴 CPU2）。
- 儲存：
- 前方背板內建兩個 NVMe 端口（支援 U.2 / 2.5 英寸 NVMe 驅動器）。
- 背板後方內建兩個 SATA 插槽。
- 散熱與噪音：
- 資料中心 GPU 為被動散熱（Passively Cooled），無風扇。
- 依賴機箱前方大量風扇強制風扇通過散熱片堆疊。
- 噪音極大：開機時非常吵雜，後續稍安靜但仍顯著，需考慮環境噪音問題。
- 測試計畫：
- 目前安裝四張 P100。
- 影片將展示 AMD MI-50/MI-25 及 V100 的性能基準測試。
- 將與現代 GPU（如 R9700 AI Pro, RTX 5090）進行價格與性能對比。
6. 硬體規格關鍵差異比較
- 記憶體頻寬 (Memory Bandwidth)
- 推論時生成每個 token 需頻繁讀取模型權重，頻寬直接限制每秒生成的 token 數量。
- V100: 900 GB/s（表現最佳，高於現代 R9700）。
- R9700: 640 GB/s。
- MI25: 484 GB/s（顯著較低）。
- 資料格式支援 (Data Format Support)
- 現代 GPU 原生支援 BF16 和 FP8（當前 LLM 標準精度）。
- 舊款三張卡（V100, P100, MI25）均不原生支援 BF16/FP8。
- 支援 FP16（16-bit），但範圍不如 BF16廣。
- 風險：將 BF16 權重轉換為 FP16 時可能發生溢位或精度損失，導致輸出品質下降。
- 硬體加速單元
- V100: 具備 Tensor Cores（針對機器學習矩陣運算優化的硬體），效能較佳。
- P100 & MI25: 無此等效支援，基準測試中表現較差。
7. 伺服器環境設定與部署
- 基礎架構
- 使用 Docker container（稱為 toolbox）。
- 後端選擇：CUDA（Nvidia 卡幾乎總是最佳選擇）或 Vulkan。
- 工具來源：從 Docker Hub 拉取預先建構的 toolbox（內含 Lama CPP）。
- Lama CPP 部署流程
- 確認裝置：運行
Lama CLI, list devices確認 CUDA 後端識別到四張 V100（每張 16GB VRAM）。 - 模型下載：從 Hugging Face 下載 GGUF 格式權重。
- 執行伺服器：
- 使用
Lama server命令。 - 指定模型（例如 QAN 3.6，270 億參數，Q4 KXL 或 Q8 量化）。
- 啟用 Flash Attention。
- 設定上下文大小（Context size）。
- 網路轉發：透過 SSH 將伺服器埠（如 8080）轉發至主機埠（如 8081）。
- 介面：使用 Lama CPP web UI 進行提示輸入。
- 效能觀察：270 億參數稠密模型約 31 tokens/秒，表現良好。
- VLLM 部署流程
- 建立 VLLM toolbox。
- 使用提供的
start VLLM script啟動伺服器。 - 設定參數：Tensor Parallelism 4、並發請求數、上下文大小、GPU 利用率。
- 缺點：
- 載入模型時間長（受 PCIe 3.0 限制）。
- 尋找注意力後端（Attention backend），最終可能回退至 PyTorch SDPA attention，效能不如現代硬體。
- 不建議在舊硬體上使用 VLLM。
8. 軟體推薦與生態系統比較
- Lama CPP
- 推薦原因：生態系統更統一，幾乎適用於所有支援的硬體（V100, P100, MI25）。
- 優勢：支援幾乎所有模型與量化格式（Quantization）。
- 必要性：舊硬體記憶體有限（如 64GB），必須使用量化才能運行大參數模型（如 270 億參數）。
- VLLM
- 限制：對不同 GPU 架構非常敏感，需要特定核心（Kernels）。
- 問題：許多舊卡缺乏可用核心，無法像 Lama CPP 那樣自由選擇模型。
- 例外：在 MI25 上若有可用核心，效能可能高於 P100，但整體不建議使用。
9. 基準測試 (Benchmarks) 結果分析
- 測試對象：現代 LLM（QAN 3.5/3.6 系列，適合 32-64GB RAM 的代理工作流與編碼）。
- V100 表現
- 提示處理 (Prompt Processing)：最佳表現者。
- 270 億參數模型 (Q4 量化)：約 852 tokens/秒。
- 上下文擴展至 32,000 tokens：仍維持約 622 tokens/秒。
- Token 生成 (Token Generation)：
- 270 億參數稠密模型：約 28 tokens/秒。
- 對於 PI 編碼代理（PI coding agent）用途足夠。
- 原因：現代架構 + Tensor Cores 優化矩陣乘法。
- MI25 表現
- 在 Lama CPP 上表現接近但略低於 P100。
- 整體給出最低效能。
- 部分模型測試缺失（因早期測試未包含所有模型）。
- VLLM 在舊卡上的表現
- MI25 在有核心支援時，效能可能優於 P100。
- 但整體不建議使用，因許多模型無法輕鬆運行。
- V100 可運行 QAN 3.6 270 億參數模型的 GPT-Q 4-bit 量化版本。
10. 缺點與權衡 (Caveats and Trade-offs)
- 架構限制
- P100 (Pascal) 與 MI25 (Vega) 無現代機器學習硬體路徑。
- V100 雖有 Tensor Cores，但相較現代 GPU 仍為舊架構，吞吐量無法匹敵。
- 反映在價格上：便宜但效能有限。
- 噪音問題
- 被動散熱卡（Passive cards）需高風量風扇冷卻。
- 系統噪音大，不適合放在書桌下，建議放置於車庫、地下室或專用房間。
- 系統匯流排 (System Bus)
- 使用 PCIe Gen 3 插槽。
- 理論頻寬約 16GB/秒（現代 PCIe Gen 5 為 64GB/秒，四倍速度）。
- 單卡影響：載入模型權重時僅增加幾秒延遲。
- 多卡影響：GPU 間需頻繁交換資訊與同步活動，頻寬可能成為瓶頸。
11. 舊規格硬體對效能的影響與總結
- PCIe 3.0 限制：資訊交換較慢，限制整體吞吐量（throughput）與 token 生成效能。
- 記憶體頻寬比較：
- DDR4：運作於 2400 至 3200 megatransfer per second。
- DDR5：起跳 4800 megatransfer，頻寬幾乎翻倍。
- 瓶頸情境：若模型完全放入 GPU 記憶體，影響不大；但若需卸載（offloading）至 CPU 或系統記憶體，舊規格會成為瓶頸。
- 硬體選擇的取捨與升級選項：
- 低成本方案：選擇 PCIe 3.0 與 DDR4 是為了展示最便宜的可行組裝方案。
- 高階硬體選項：若尋找便宜硬體（bargain hardware），可找到配備 DDR5 與 PCIe 4/5 的伺服器與工作站。
- 具體範例：Dell Precision 7960（價格較高，但配備 DDR5 與 PCIe 5.0）。
- 目標受眾與解決方案價值：
- 預算範圍：2,000 至 3,000 英鎊。
- 需求：需要 64GB VRAM 來運行特定模型。
- **推薦
尚未產生學習筆記
請在 Telegram 指令最後加上「學習」，例如：videonote 網址 英文 雙語 學習