1
00:00:00,000 --> 00:00:12,000
開放了千萬3.8RGB的模型權重。很多人看到RGB之後的第一個問題就是,這東西,我的Mac到底能不能跑呢?

2
00:00:12,000 --> 00:00:21,000
有人看到量化以後只需要十幾GB的內存,覺得自己的24GB Mac應該也沒有什麼問題。

3
00:00:21,000 --> 00:00:24,060
结果模型下载完以后

4
00:00:24,060 --> 00:00:25,780
要么加载失败

5
00:00:25,780 --> 00:00:27,880
要么刚开始生成

6
00:00:27,880 --> 00:00:30,380
系统内存就直接爆掉

7
00:00:30,380 --> 00:00:34,880
有人花了半天时间把模型跑起来

8
00:00:34,880 --> 00:00:38,600
最后发现生成速度只有每秒八九个Token

9
00:00:38,600 --> 00:00:41,700
打开MTP之后不仅没有加速

10
00:00:41,700 --> 00:00:42,980
反而更慢了

11
00:00:42,980 --> 00:00:45,520
所以今天这支影片

12
00:00:45,520 --> 00:00:47,660
我会从零开始

13
00:00:47,660 --> 00:00:50,840
带你在Mac上完整部署一次

14
00:00:50,840 --> 00:00:52,680
千万3.8 RGB

15
00:00:52,680 --> 00:00:56,100
我们会先搞清楚RGP到底代表什么

16
00:00:56,100 --> 00:00:58,960
它和Cloud GPT这些B原模型

17
00:00:58,960 --> 00:01:00,620
大概处于什么水平

18
00:01:00,620 --> 00:01:03,880
然后算清楚模型运行时

19
00:01:03,880 --> 00:01:05,880
到底会占用多少内存

20
00:01:05,880 --> 00:01:11,120
接下来我会带你选择正确的MLX量化版本

21
00:01:11,120 --> 00:01:13,560
使用Hugging Face的并行下载工具

22
00:01:13,560 --> 00:01:16,440
把模型下载到本地

23
00:01:17,660 --> 00:01:21,500
先跑出一个没有任何加速的基础速度

24
00:01:21,500 --> 00:01:24,140
最后再安装DFlesh 2

25
00:01:24,140 --> 00:01:26,280
在下载模型之前

26
00:01:26,280 --> 00:01:28,460
我们首先得搞清楚一件事

27
00:01:28,460 --> 00:01:33,620
前往3.827B里面的这个27B到底是什么意思

28
00:01:33,620 --> 00:01:37,580
这里的B是Billing

29
00:01:37,580 --> 00:01:39,000
也就是实义

30
00:01:39,000 --> 00:01:42,840
27B代表这个语言的模型部分

31
00:01:42,840 --> 00:01:46,140
大约拥有271个参数

32
00:01:46,140 --> 00:01:53,780
你可以把参数理解成有模型训练完成以后保留下来的271个数字

33
00:01:53,780 --> 00:01:58,600
模型每生成一个token都要读取这些数字进行计算

34
00:01:58,600 --> 00:02:02,640
然后判断下一个token最有可能是什么

35
00:02:02,640 --> 00:02:04,720
如果想象的更具体一点

36
00:02:04,720 --> 00:02:10,900
你可以把它理解成一台拥有271个旋钮的巨大机器

37
00:02:10,900 --> 00:02:15,780
训练模型的过程就是不断调整这271个旋钮

38
00:02:15,780 --> 00:02:18,740
而我们现在进行的本地推理

39
00:02:18,740 --> 00:02:22,160
就是把这些调整好的旋钮加载到内存里

40
00:02:22,160 --> 00:02:24,400
然后用它们来生成答案

41
00:02:24,400 --> 00:02:27,240
这里还有一个非常重要的概念

42
00:02:27,240 --> 00:02:32,100
也就是说千问3.827B是一个Distance模型

43
00:02:32,100 --> 00:02:33,820
也就是稠密模型

44
00:02:33,820 --> 00:02:36,720
而不是MROE混合专家模型

45
00:02:36,720 --> 00:02:41,840
有些模型名称可能写着总参数35B

46
00:02:41,840 --> 00:02:44,160
但是后面还有一个A3B

47
00:02:44,160 --> 00:02:45,980
所谓的A3B呢

48
00:02:45,980 --> 00:02:47,380
这个A3B的意思就是

49
00:02:47,380 --> 00:02:49,980
它虽然一共有350亿参数

50
00:02:49,980 --> 00:02:51,780
但每次生成Token的时候

51
00:02:51,780 --> 00:02:54,600
大约只有30亿参数会被激活

52
00:02:54,600 --> 00:02:57,620
前往3.827B不一样

53
00:02:57,620 --> 00:02:59,700
它没有A几B的后缀

54
00:02:59,700 --> 00:03:02,980
它的语言模型拥有270亿参数

55
00:03:02,980 --> 00:03:05,020
每生成一个Token

56
00:03:05,020 --> 00:03:07,800
那么就代表着

57
00:03:07,800 --> 00:03:10,740
基本要经过这270亿参数的计算

58
00:03:10,740 --> 00:03:13,680
所以它的总参数大约是27B

59
00:03:13,680 --> 00:03:16,120
激活参数呢

60
00:03:16,120 --> 00:03:18,160
同样也是27B

61
00:03:18,160 --> 00:03:23,320
这也是为什么同样叫做30B左右的模型

62
00:03:23,320 --> 00:03:25,180
千万3.8 27B

63
00:03:25,180 --> 00:03:28,800
可能比一些35B A3B的MOE模型大很多

64
00:03:28,800 --> 00:03:29,400
当然

65
00:03:29,400 --> 00:03:31,860
稠密模型也不是只有缺点

66
00:03:31,860 --> 00:03:36,140
它的优势是所有能力都集中在同一套参数里

67
00:03:36,140 --> 00:03:37,940
通常在输出稳定性

68
00:03:37,940 --> 00:03:41,060
复杂任务和知识调用上会更加均衡

69
00:03:41,060 --> 00:03:44,860
如果可以把本地运行的模型简单承评成三个类别

70
00:03:44,860 --> 00:03:47,660
3B到8B通常属于轻量模型

71
00:03:47,660 --> 00:03:49,220
这类模型速度很快

72
00:03:49,220 --> 00:03:51,580
但复杂任务比较容易走偏

73
00:03:51,580 --> 00:03:57,680
14B到30B可以看成本地大模型目前最实用的高质量区间

74
00:03:57,680 --> 00:04:02,380
能力已经足以处理比较复杂的编程文章生成和工具调用

75
00:04:02,380 --> 00:04:03,720
以及agent任务

76
00:04:03,720 --> 00:04:10,860
70B以上的稠密模型能力通常会更加稳定

77
00:04:10,860 --> 00:04:15,080
但需要的内存和内存带宽也会明显上升

78
00:04:15,080 --> 00:04:17,440
对于大部分个人用户来说

79
00:04:17,440 --> 00:04:19,880
部署成本其实是比较高的

80
00:04:19,880 --> 00:04:26,880
所以千万3.8RGB基本属于个人电脑还能够比较现实的部署

81
00:04:26,880 --> 00:04:30,900
同时能力有足够强的高端本地模型

82
00:04:30,900 --> 00:04:34,000
那它跟市面上的B1模型相比怎么样呢

83
00:04:34,000 --> 00:04:40,020
首先我们不能直接拿RGB去和Cloud和GPT的比较参数量

84
00:04:40,020 --> 00:04:45,080
因为大部分B原模型不会公开完整参数量和激活参数

85
00:04:45,080 --> 00:04:47,720
即使参数量一样

86
00:04:47,720 --> 00:04:51,800
训练数据、模型架构、后训练和推理预算不同

87
00:04:51,800 --> 00:04:54,380
最终能力也可能差很多

88
00:04:54,380 --> 00:04:56,240
比较稳妥的说法是

89
00:04:56,240 --> 00:05:00,680
千万3.8RGB已经进入了可以和B原旗舰模型

90
00:05:00,680 --> 00:05:02,920
在部分任务上正面竞争的范围

91
00:05:02,920 --> 00:05:06,920
但它还不能全面替代B原旗舰

92
00:05:06,920 --> 00:05:11,260
以签问官方测试为例

93
00:05:11,260 --> 00:05:13,000
可以看到

94
00:05:13,000 --> 00:05:17,640
在Batch Pro里签问3.8

95
00:05:17,640 --> 00:05:19,080
RTB是61.7

96
00:05:19,080 --> 00:05:21,440
OPLA 4.6 MAX是53.4

97
00:05:21,440 --> 00:05:25,620
Terminal Bench则是73对78.2

98
00:05:25,620 --> 00:05:28,840
HLE是30.8对40

99
00:05:28,840 --> 00:05:32,240
那么这组数据就可以说明

100
00:05:32,240 --> 00:05:35,080
它在某些编程和工具调用任务上

101
00:05:35,080 --> 00:05:37,180
可以接近甚至超过B元

102
00:05:37,180 --> 00:05:39,120
但是复杂开放式推理

103
00:05:39,120 --> 00:05:41,060
知识广度和长容稳定性

104
00:05:41,060 --> 00:05:44,040
B元旗舰通常还是更强一些

105
00:05:44,040 --> 00:05:48,320
而且这是签问官方公布的测试

106
00:05:48,320 --> 00:05:52,180
它不是不同模型可能使用了不同的harness

107
00:05:52,180 --> 00:05:57,040
所以这也包含官方自己的测试集

108
00:05:57,040 --> 00:06:00,320
可以作为一部分定位参考

109
00:06:00,320 --> 00:06:03,080
但不能因为某一个分数更高

110
00:06:03,080 --> 00:06:05,320
就是说千万已经全面超越了Cloud

111
00:06:05,320 --> 00:06:07,600
对我来说最准确的定位是

112
00:06:07,600 --> 00:06:10,680
它不是B原旗舰的完整替代品

113
00:06:10,680 --> 00:06:15,680
但已经是一款可以真正承担生产任务的高端本地模型

114
00:06:15,680 --> 00:06:18,980
接下来我们来算一下它到底需要多少内存

115
00:06:18,980 --> 00:06:23,680
很多人看到27B会直接认为它需要27G的内存

116
00:06:23,680 --> 00:06:28,160
这个说法只在非常特定的条件下才大致成立

117
00:06:29,440 --> 00:06:32,840
如果使用BF16一个参数呢

118
00:06:32,840 --> 00:06:34,160
需要两个字节

119
00:06:34,160 --> 00:06:36,900
270一个参数乘以两个字节的话

120
00:06:36,900 --> 00:06:41,480
光语言模型原始权重就要大约54G

121
00:06:41,480 --> 00:06:43,640
这还没有算计算系统

122
00:06:43,640 --> 00:06:46,620
缓存和退理过程中的临时内存

123
00:06:46,620 --> 00:06:48,280
所以64G Mac

124
00:06:48,280 --> 00:06:50,540
虽然理论上可以放下BF16权重

125
00:06:50,540 --> 00:06:53,100
但运行语量其实非常有限

126
00:06:53,100 --> 00:06:55,900
量化的作用呢

127
00:06:55,900 --> 00:06:59,060
就是使用更少的位数保存这些参数

128
00:06:59,060 --> 00:07:02,340
8bit理论权重大约27G

129
00:07:02,340 --> 00:07:06,120
4bit理论上大约13.5G

130
00:07:06,120 --> 00:07:10,080
实际MLX版本还包含量化参数

131
00:07:10,080 --> 00:07:11,400
词表和配置

132
00:07:11,400 --> 00:07:14,140
8bit纯文字版约28.6G

133
00:07:14,140 --> 00:07:16,420
带视觉能力约29.5G

134
00:07:16,420 --> 00:07:18,820
4bit版本大约16G

135
00:07:18,820 --> 00:07:22,120
那么从这些地方的这些数字

136
00:07:22,120 --> 00:07:24,120
只代表模型权重

137
00:07:24,120 --> 00:07:27,280
并不是模型启动后的总内存占用

138
00:07:27,280 --> 00:07:29,140
每次运行时

139
00:07:29,140 --> 00:07:31,200
内存里至少还要放进五类东西

140
00:07:31,200 --> 00:07:33,000
第一类也就是最大的一类

141
00:07:33,000 --> 00:07:35,040
就是模型群众

142
00:07:35,040 --> 00:07:39,440
运行8bit版本光这一向就已经接近29G

143
00:07:39,440 --> 00:07:42,700
而且启动后基本都要进入Mac统一内存

144
00:07:42,700 --> 00:07:47,140
第二类是KVKISH和线性注意力技术

145
00:07:47,140 --> 00:07:50,160
模型需要保存已经处理过的上下文

146
00:07:50,160 --> 00:07:56,700
否则每生成一个Token都需要重新计算前文

147
00:07:56,700 --> 00:08:00,000
KVKISH和上下文长度直接下文

148
00:08:01,400 --> 00:08:07,760
前文原生支持约26万Token,但支持不代表你的Mac适合直接开满。

149
00:08:07,760 --> 00:08:09,780
粗略计算

150
00:08:09,780 --> 00:08:11,220
32K上下文

151
00:08:11,220 --> 00:08:12,620
仅标准注意力

152
00:08:12,620 --> 00:08:14,260
64K约4G

153
00:08:14,260 --> 00:08:16,640
128K约8G

154
00:08:16,640 --> 00:08:19,120
256K就要16G

155
00:08:19,120 --> 00:08:22,760
此外还没包含线性注意力状态

156
00:08:22,760 --> 00:08:25,340
如果平常只是聊天

157
00:08:25,340 --> 00:08:26,920
或者处理普通代码任务

158
00:08:26,920 --> 00:08:29,160
先从16K和32K开始

159
00:08:29,160 --> 00:08:31,080
通常就已经足够了

160
00:08:31,080 --> 00:08:34,060
第三类是MLX和META运行时

161
00:08:34,060 --> 00:08:35,540
使用的计算空间

162
00:08:35,540 --> 00:08:38,860
包括临时激活值和计算缓冲区

163
00:08:38,860 --> 00:08:41,640
那么它不是固定数字

164
00:08:41,640 --> 00:08:44,020
会随输入程度和并发变化

165
00:08:44,020 --> 00:08:47,860
尤其是在第一次读取大段Prompt

166
00:08:47,860 --> 00:08:49,460
也就是Prefill阶段

167
00:08:49,460 --> 00:08:52,020
内存占用可能突然上升

168
00:08:52,020 --> 00:08:56,000
第四类是视觉编码器

169
00:08:56,000 --> 00:08:59,140
那么只做文字

170
00:08:59,140 --> 00:09:00,880
写代码或运行Agent

171
00:09:00,880 --> 00:09:02,300
可以选择TextOnly

172
00:09:02,300 --> 00:09:04,260
需要读取图片或视频

173
00:09:04,260 --> 00:09:06,260
才选择完整多模态版本

174
00:09:06,260 --> 00:09:09,500
第五类是DFLASH2草稿模型

175
00:09:09,500 --> 00:09:11,020
它不是普通开关

176
00:09:11,020 --> 00:09:13,220
还需要一个配套的草稿模型

177
00:09:13,220 --> 00:09:17,020
本身它大约21个参数

178
00:09:17,020 --> 00:09:21,480
并额外占用缓存和缓冲区

179
00:09:21,480 --> 00:09:26,380
所以最终的内存杖不是27B等于27G

180
00:09:26,380 --> 00:09:28,220
而是模型权重加Cache

181
00:09:28,220 --> 00:09:31,020
加运行缓冲加系统占用

182
00:09:31,020 --> 00:09:33,840
再加DFLASH2草稿模型

183
00:09:33,840 --> 00:09:37,880
如果想运行8bit版本

184
00:09:37,880 --> 00:09:41,820
那么48G统一内存可以作为比较现实的起点

185
00:09:41,820 --> 00:09:44,220
例如4G会更加舒服

186
00:09:44,220 --> 00:09:47,300
32G不是绝对加载不了

187
00:09:47,300 --> 00:09:48,780
但余量非常小

188
00:09:48,780 --> 00:09:51,220
如果Mac只有24G或32G的话

189
00:09:51,220 --> 00:09:56,980
我建议更加直接使用4bit版本是比较好的

190
00:09:56,980 --> 00:09:59,960
模型文件大约16G

191
00:09:59,960 --> 00:10:05,000
给系统Cache和D-Flash 2留下的空间会宽松很多

192
00:10:05,000 --> 00:10:07,600
Mac运行本地模型常见的聊线

193
00:10:07,600 --> 00:10:10,500
GGUF加Lama CPP一节MLX

194
00:10:10,500 --> 00:10:13,320
这次选择MLX

195
00:10:13,320 --> 00:10:18,240
因为它针对苹果电脑有比较特殊的一些优化

196
00:10:18,240 --> 00:10:21,880
而且后面的D-Flash 2工具可以直接运行MLX权重

197
00:10:21,880 --> 00:10:28,860
我这次选择千万3.8RGB的MLX 8bit版本

198
00:10:28,860 --> 00:10:30,860
如果内存比较小

199
00:10:30,860 --> 00:10:34,120
把8bit换成对应的4bit仓库即可

200
00:10:34,120 --> 00:10:37,680
其他流程基本是一样的

201
00:10:37,680 --> 00:10:41,360
模型可以直接去HarkinFace下载

202
00:10:41,360 --> 00:10:42,960
那么下载的时候

203
00:10:42,960 --> 00:10:44,620
不建议打开浏览器

204
00:10:44,620 --> 00:10:46,520
一个文件一个文件点击

205
00:10:46,520 --> 00:10:49,940
大模型通常拆成多个权重封片

206
00:10:49,940 --> 00:10:52,500
浏览器下载非常容易中断

207
00:10:52,500 --> 00:10:54,500
也可能漏掉配置文件

208
00:10:54,500 --> 00:10:58,180
那么这里更推荐用HarkinFace官方的

209
00:10:58,180 --> 00:10:59,660
HF命令航工具

210
00:10:59,660 --> 00:11:02,580
使用新版的HUB

211
00:11:02,580 --> 00:11:04,920
使用XNET后端

212
00:11:04,920 --> 00:11:06,800
它可以支持并行下载

213
00:11:06,800 --> 00:11:09,700
就造成你的Transfer

214
00:11:09,700 --> 00:11:11,780
已经逐步被淘汰了

215
00:11:11,780 --> 00:11:12,840
那么我们这里

216
00:11:12,840 --> 00:11:15,680
首先安装最新版Hagenface CLI

217
00:11:15,680 --> 00:11:17,440
接下来

218
00:11:17,440 --> 00:11:19,340
开启高性能模式

219
00:11:19,340 --> 00:11:22,560
然后把模型下载到指定目录

220
00:11:22,560 --> 00:11:25,960
那么高性能模式会提高并发

221
00:11:25,960 --> 00:11:28,200
尽可能利用网络和CPU资源

222
00:11:28,200 --> 00:11:29,280
模型比较大

223
00:11:29,280 --> 00:11:32,560
下载前记得磁盘至少还有几十T的空余空间

224
00:11:32,560 --> 00:11:34,000
下载完成之后

225
00:11:34,000 --> 00:11:36,580
我们先别着急安装D-Flash 2

226
00:11:36,580 --> 00:11:40,600
先跑出没有加速的基础版本

227
00:11:40,600 --> 00:11:43,160
建议给本地模型单独创建

228
00:11:43,160 --> 00:11:44,660
Python虚拟环境

229
00:11:44,660 --> 00:11:48,880
避免不同的项目依赖会互相污染

230
00:11:48,880 --> 00:11:50,480
创建环境

231
00:11:50,480 --> 00:11:51,880
激活环境

232
00:11:51,880 --> 00:11:54,800
以及安装MXM和MX

233
00:11:54,800 --> 00:11:56,720
第一次测试呢

234
00:11:56,720 --> 00:11:58,420
不要开超长的上下纹

235
00:11:58,420 --> 00:12:00,560
不要开Dflash 2

236
00:12:00,560 --> 00:12:02,220
同时呢

237
00:12:02,220 --> 00:12:04,960
也不要修改一大堆采样参数

238
00:12:04,960 --> 00:12:07,300
只指定模型路径

239
00:12:07,300 --> 00:12:08,340
固定Prompt

240
00:12:08,340 --> 00:12:09,960
生成几百个Token

241
00:12:09,960 --> 00:12:12,120
运行最基础的生成命令

242
00:12:12,120 --> 00:12:14,200
运行完成之后呢

243
00:12:14,200 --> 00:12:17,180
我们可以把模型加载时间

244
00:12:17,180 --> 00:12:18,900
Prompt处理速度

245
00:12:18,900 --> 00:12:21,660
守Token等待时间和生成Token速度

246
00:12:21,660 --> 00:12:23,860
这四个数字记录下来

247
00:12:23,860 --> 00:12:26,040
如果没有基础数据

248
00:12:26,040 --> 00:12:29,800
后面看到每每20几个token也不知道到底快了多少

249
00:12:29,800 --> 00:12:32,020
基础9变27是3倍

250
00:12:32,020 --> 00:12:37,380
那么基础20变22可能就不值得增加内存和复杂度

251
00:12:37,380 --> 00:12:42,600
确认基础模型稳定之后呢

252
00:12:42,600 --> 00:12:45,480
我们就可以安装D-Flash 2

253
00:12:45,480 --> 00:12:49,960
D-Flash 2本质上是一种推测解码技术

254
00:12:49,960 --> 00:12:51,200
正常情况下

255
00:12:51,200 --> 00:12:54,160
千万3.827B每生成一个token的话

256
00:12:54,160 --> 00:12:57,580
都要让完整270亿参数计算一次

257
00:12:57,580 --> 00:13:00,280
那么生成10个token

258
00:13:00,280 --> 00:13:03,280
就要连续做10次完整的推理

259
00:13:03,280 --> 00:13:05,400
Dflash 2

260
00:13:05,400 --> 00:13:08,760
让一个大约20亿参数的草稿模型

261
00:13:08,760 --> 00:13:12,300
一次性并行预测接下来的一组token

262
00:13:12,300 --> 00:13:18,080
然后再让27B主模型一次检查这一整组答案

263
00:13:18,080 --> 00:13:20,920
如果草稿模型猜的比较准

264
00:13:20,920 --> 00:13:24,200
主模型一次计算就可以接受多个Token

265
00:13:24,200 --> 00:13:27,260
原本连续走五六四的流程

266
00:13:27,260 --> 00:13:29,620
现在可能只需要一次验证

267
00:13:29,620 --> 00:13:32,460
草稿模型只负责猜

268
00:13:32,460 --> 00:13:35,960
最终决定权依然在27B主模型手上

269
00:13:35,960 --> 00:13:37,740
贪心解码下最终输出

270
00:13:37,740 --> 00:13:39,860
可以与基础模型一致

271
00:13:39,860 --> 00:13:42,720
随机采样则保留主模型原本的分布

272
00:13:42,720 --> 00:13:44,340
在Mac上

273
00:13:44,340 --> 00:13:47,320
比较直接的实现是MX的D Spark

274
00:13:47,320 --> 00:13:51,240
那么这个虽然名字写着Dspark

275
00:13:51,240 --> 00:13:54,280
但它同时支持Dspark,Dflash和Dflash2

276
00:13:54,280 --> 00:14:02,040
在用的时候对于千万3.8,27B,Ultra模式会直接使用Dflash2

277
00:14:02,040 --> 00:14:07,200
安装命令也非常简单

278
00:14:11,880 --> 00:14:13,360
屏幕上的安装命令

279
00:14:13,360 --> 00:14:17,060
安装完成以后不要凭感觉判断有没有变快

280
00:14:17,060 --> 00:14:19,520
直接运行项目自带的Benchmark

281
00:14:19,520 --> 00:14:21,720
使用三轮测试

282
00:14:21,720 --> 00:14:27,140
第一次运行会自动下载匹配的DFlesh2草稿模型

283
00:14:27,140 --> 00:14:30,340
并根据这台Mac的实际成本测试

284
00:14:30,340 --> 00:14:32,580
合适的推理长度

285
00:14:32,580 --> 00:14:35,780
M1 M3 M4 M5的最佳参数呢

286
00:14:35,780 --> 00:14:37,440
它是不一定一样的

287
00:14:37,440 --> 00:14:40,080
如果你想直接生成内容的话

288
00:14:40,080 --> 00:14:42,320
建议用这个命令

289
00:14:42,320 --> 00:14:44,780
可以使用MXD Spark Generate

290
00:14:44,780 --> 00:14:50,840
默认AUTO模式会为千万3.8RCB匹配DFLASH2

291
00:14:50,840 --> 00:14:54,300
再用完全相同的Prompt测试

292
00:14:54,300 --> 00:14:55,660
对比基础速度

293
00:14:55,660 --> 00:14:59,100
DFLASH2速度接受长度和内存占用

294
00:14:59,100 --> 00:15:00,980
那么比下来之后

295
00:15:00,980 --> 00:15:03,360
项目在特定的M4 Pro测试中

296
00:15:03,360 --> 00:15:06,160
8bit的基础约8.4Token每秒

297
00:15:06,160 --> 00:15:09,160
DFLASH2平均约30.5Token每秒

298
00:15:09,160 --> 00:15:13,040
不同任务约2.8到4倍

299
00:15:13,040 --> 00:15:17,220
但这不是你的Mac一定能达到的速度

300
00:15:17,220 --> 00:15:17,980
因为呢

301
00:15:17,980 --> 00:15:20,620
DFLASH2不是打开以后

302
00:15:20,620 --> 00:15:22,000
它就一定变快

303
00:15:22,000 --> 00:15:23,980
如果草稿接受率低的话

304
00:15:23,980 --> 00:15:26,980
主模型会不断拒绝和重新生成

305
00:15:26,980 --> 00:15:30,080
草稿计算反而成为额外的负担

306
00:15:30,080 --> 00:15:33,440
对于只有几十个Token的短回答

307
00:15:33,440 --> 00:15:36,240
模型刚加载完成和校准

308
00:15:36,240 --> 00:15:37,440
任务就结束了

309
00:15:37,440 --> 00:15:39,980
实际体感提升可能不是很明显

310
00:15:39,980 --> 00:15:43,040
D-Flash 2还会增加内存占用

311
00:15:43,040 --> 00:15:44,460
所以呢

312
00:15:44,460 --> 00:15:45,936
如果8bit主

313
00:15:45,936 --> 00:15:52,936
所以,如果8bit主模型以及頻繁swap,那麼再加載草稿模型,最終速度反而更慢。

314
00:15:52,936 --> 00:15:58,936
所以要用自己最常執行的任務去測試一下。寫代碼就測真實代碼修改。

315
00:15:58,936 --> 00:16:02,376
生成口播稿就连续输出一两千的token

316
00:16:02,376 --> 00:16:06,236
运行agent就测试完整工具调用

317
00:16:06,236 --> 00:16:09,936
最后一步是把模型启动成API

318
00:16:09,936 --> 00:16:14,076
MLX的Dspark可以兼容OpenAI

319
00:16:14,076 --> 00:16:18,036
那么和Astorbit格式的本地服务

320
00:16:18,036 --> 00:16:20,716
启动命令是MLXDsparkServe

321
00:16:22,796 --> 00:16:27,656
默认是开在本级8080端口去提供接口的

322
00:16:27,656 --> 00:16:32,296
客户端BaseURL设置成127.0.18080

323
00:16:32,296 --> 00:16:35,576
设置好之后

324
00:16:35,576 --> 00:16:37,276
只在自己电脑使用

325
00:16:37,276 --> 00:16:40,016
可以不设置真正的API Key

326
00:16:40,016 --> 00:16:41,836
需要局域网访问的时候

327
00:16:41,836 --> 00:16:43,396
要调整监听地址

328
00:16:43,396 --> 00:16:44,976
检查防火墙

329
00:16:44,976 --> 00:16:49,256
但不要把无件权接口直接暴露到公网

330
00:16:49,256 --> 00:16:52,756
到这里我们完成了整套部署流程

331
00:16:52,756 --> 00:16:53,476
第一

332
00:16:53,476 --> 00:16:56,576
R7B代表约270亿参数

333
00:16:56,576 --> 00:16:59,596
而且它是全量激活的稠密模型

334
00:16:59,596 --> 00:17:00,196
第二

335
00:17:00,196 --> 00:17:03,456
RGB不等于只需RGB内存

336
00:17:03,456 --> 00:17:05,556
把比特权重接近R9G运行

337
00:17:05,556 --> 00:17:09,416
是要加CASE缓冲区系统和草稿模型

338
00:17:09,416 --> 00:17:12,136
那么第三

339
00:17:12,136 --> 00:17:15,016
它属于本地模型高端档

340
00:17:15,016 --> 00:17:18,116
部分编程和Agent任务接近B原起见

341
00:17:18,116 --> 00:17:19,996
但不是全面的替代

342
00:17:19,996 --> 00:17:21,076
第四

343
00:17:21,076 --> 00:17:22,596
不要一上来

344
00:17:22,596 --> 00:17:24,516
就打开所有加速箱

345
00:17:24,516 --> 00:17:26,976
先用MLX跑出基础速度

346
00:17:26,976 --> 00:17:29,436
再用相同Prompt做A-B测试

347
00:17:29,436 --> 00:17:32,596
决定是否长期使用D-Flash 2

348
00:17:32,596 --> 00:17:35,716
如果真实任务从每秒9个Token

349
00:17:35,716 --> 00:17:37,776
能够提高到2-30个的话

350
00:17:37,776 --> 00:17:39,356
而且内存稳定

351
00:17:39,356 --> 00:17:41,856
那么它就值得长期去打开

352
00:17:41,856 --> 00:17:45,996
如果频繁Swap或者主要都是短问答的话

353
00:17:45,996 --> 00:17:48,736
4bit基础模型可能更舒服一些

354
00:17:48,736 --> 00:17:53,036
本地部署大模型的价值不是省API费用

355
00:17:53,036 --> 00:17:55,936
当它变成Mac上的本地服务

356
00:17:55,936 --> 00:17:58,696
它就可以接入代码Langraph

357
00:17:58,696 --> 00:17:59,936
自动剪辑系统

358
00:17:59,936 --> 00:18:00,776
知识库

359
00:18:00,776 --> 00:18:03,916
以及agent工作流

360
00:18:03,916 --> 00:18:07,556
真正好的本地布置方案

361
00:18:07,556 --> 00:18:08,876
不是参数最大

362
00:18:08,876 --> 00:18:10,636
也不是测试数字最高

363
00:18:10,636 --> 00:18:12,116
而是在自己的设备上

364
00:18:12,116 --> 00:18:13,076
质量足够好

365
00:18:13,076 --> 00:18:14,316
速度能够接受

366
00:18:14,316 --> 00:18:16,236
内存保持稳定

367
00:18:16,236 --> 00:18:19,156
并且真的进入日常工作流

368
00:18:19,156 --> 00:18:20,756
才能长期去使用

369
00:18:20,756 --> 00:18:24,196
影片领誘到的模型地址安装命令和测试命令

370
00:18:24,196 --> 00:18:26,416
我都会整理在下方

371
00:18:26,416 --> 00:18:33,436
如果希望下一期测试4bit 6bit和8bit的品质差距的话

372
00:18:33,436 --> 00:18:36,856
以及DFlex 2速度也可以留言给我

373
00:18:36,856 --> 00:18:39,056
喜欢这支影片的话

374
00:18:39,056 --> 00:18:42,776
记得帮我按赞订阅加分享哦

375
00:18:42,776 --> 00:18:44,956
这是我持续创作的最大动力

376
00:18:44,956 --> 00:18:45,916
我们下次再见