{ "text": "開放了千萬3.8RGB的模型權重。很多人看到RGB之後的第一個問題就是,這東西,我的Mac到底能不能跑呢?\n有人看到量化以後只需要十幾GB的內存,覺得自己的24GB Mac應該也沒有什麼問題。\n结果模型下载完以后\n要么加载失败\n要么刚开始生成\n系统内存就直接爆掉\n有人花了半天时间把模型跑起来\n最后发现生成速度只有每秒八九个Token\n打开MTP之后不仅没有加速\n反而更慢了\n所以今天这支影片\n我会从零开始\n带你在Mac上完整部署一次\n千万3.8 RGB\n我们会先搞清楚RGP到底代表什么\n它和Cloud GPT这些B原模型\n大概处于什么水平\n然后算清楚模型运行时\n到底会占用多少内存\n接下来我会带你选择正确的MLX量化版本\n使用Hugging Face的并行下载工具\n把模型下载到本地\n先跑出一个没有任何加速的基础速度\n最后再安装DFlesh 2\n在下载模型之前\n我们首先得搞清楚一件事\n前往3.827B里面的这个27B到底是什么意思\n这里的B是Billing\n也就是实义\n27B代表这个语言的模型部分\n大约拥有271个参数\n你可以把参数理解成有模型训练完成以后保留下来的271个数字\n模型每生成一个token都要读取这些数字进行计算\n然后判断下一个token最有可能是什么\n如果想象的更具体一点\n你可以把它理解成一台拥有271个旋钮的巨大机器\n训练模型的过程就是不断调整这271个旋钮\n而我们现在进行的本地推理\n就是把这些调整好的旋钮加载到内存里\n然后用它们来生成答案\n这里还有一个非常重要的概念\n也就是说千问3.827B是一个Distance模型\n也就是稠密模型\n而不是MROE混合专家模型\n有些模型名称可能写着总参数35B\n但是后面还有一个A3B\n所谓的A3B呢\n这个A3B的意思就是\n它虽然一共有350亿参数\n但每次生成Token的时候\n大约只有30亿参数会被激活\n前往3.827B不一样\n它没有A几B的后缀\n它的语言模型拥有270亿参数\n每生成一个Token\n那么就代表着\n基本要经过这270亿参数的计算\n所以它的总参数大约是27B\n激活参数呢\n同样也是27B\n这也是为什么同样叫做30B左右的模型\n千万3.8 27B\n可能比一些35B A3B的MOE模型大很多\n当然\n稠密模型也不是只有缺点\n它的优势是所有能力都集中在同一套参数里\n通常在输出稳定性\n复杂任务和知识调用上会更加均衡\n如果可以把本地运行的模型简单承评成三个类别\n3B到8B通常属于轻量模型\n这类模型速度很快\n但复杂任务比较容易走偏\n14B到30B可以看成本地大模型目前最实用的高质量区间\n能力已经足以处理比较复杂的编程文章生成和工具调用\n以及agent任务\n70B以上的稠密模型能力通常会更加稳定\n但需要的内存和内存带宽也会明显上升\n对于大部分个人用户来说\n部署成本其实是比较高的\n所以千万3.8RGB基本属于个人电脑还能够比较现实的部署\n同时能力有足够强的高端本地模型\n那它跟市面上的B1模型相比怎么样呢\n首先我们不能直接拿RGB去和Cloud和GPT的比较参数量\n因为大部分B原模型不会公开完整参数量和激活参数\n即使参数量一样\n训练数据、模型架构、后训练和推理预算不同\n最终能力也可能差很多\n比较稳妥的说法是\n千万3.8RGB已经进入了可以和B原旗舰模型\n在部分任务上正面竞争的范围\n但它还不能全面替代B原旗舰\n以签问官方测试为例\n可以看到\n在Batch Pro里签问3.8\nRTB是61.7\nOPLA 4.6 MAX是53.4\nTerminal Bench则是73对78.2\nHLE是30.8对40\n那么这组数据就可以说明\n它在某些编程和工具调用任务上\n可以接近甚至超过B元\n但是复杂开放式推理\n知识广度和长容稳定性\nB元旗舰通常还是更强一些\n而且这是签问官方公布的测试\n它不是不同模型可能使用了不同的harness\n所以这也包含官方自己的测试集\n可以作为一部分定位参考\n但不能因为某一个分数更高\n就是说千万已经全面超越了Cloud\n对我来说最准确的定位是\n它不是B原旗舰的完整替代品\n但已经是一款可以真正承担生产任务的高端本地模型\n接下来我们来算一下它到底需要多少内存\n很多人看到27B会直接认为它需要27G的内存\n这个说法只在非常特定的条件下才大致成立\n如果使用BF16一个参数呢\n需要两个字节\n270一个参数乘以两个字节的话\n光语言模型原始权重就要大约54G\n这还没有算计算系统\n缓存和退理过程中的临时内存\n所以64G Mac\n虽然理论上可以放下BF16权重\n但运行语量其实非常有限\n量化的作用呢\n就是使用更少的位数保存这些参数\n8bit理论权重大约27G\n4bit理论上大约13.5G\n实际MLX版本还包含量化参数\n词表和配置\n8bit纯文字版约28.6G\n带视觉能力约29.5G\n4bit版本大约16G\n那么从这些地方的这些数字\n只代表模型权重\n并不是模型启动后的总内存占用\n每次运行时\n内存里至少还要放进五类东西\n第一类也就是最大的一类\n就是模型群众\n运行8bit版本光这一向就已经接近29G\n而且启动后基本都要进入Mac统一内存\n第二类是KVKISH和线性注意力技术\n模型需要保存已经处理过的上下文\n否则每生成一个Token都需要重新计算前文\nKVKISH和上下文长度直接下文\n前文原生支持约26万Token,但支持不代表你的Mac适合直接开满。\n粗略计算\n32K上下文\n仅标准注意力\n64K约4G\n128K约8G\n256K就要16G\n此外还没包含线性注意力状态\n如果平常只是聊天\n或者处理普通代码任务\n先从16K和32K开始\n通常就已经足够了\n第三类是MLX和META运行时\n使用的计算空间\n包括临时激活值和计算缓冲区\n那么它不是固定数字\n会随输入程度和并发变化\n尤其是在第一次读取大段Prompt\n也就是Prefill阶段\n内存占用可能突然上升\n第四类是视觉编码器\n那么只做文字\n写代码或运行Agent\n可以选择TextOnly\n需要读取图片或视频\n才选择完整多模态版本\n第五类是DFLASH2草稿模型\n它不是普通开关\n还需要一个配套的草稿模型\n本身它大约21个参数\n并额外占用缓存和缓冲区\n所以最终的内存杖不是27B等于27G\n而是模型权重加Cache\n加运行缓冲加系统占用\n再加DFLASH2草稿模型\n如果想运行8bit版本\n那么48G统一内存可以作为比较现实的起点\n例如4G会更加舒服\n32G不是绝对加载不了\n但余量非常小\n如果Mac只有24G或32G的话\n我建议更加直接使用4bit版本是比较好的\n模型文件大约16G\n给系统Cache和D-Flash 2留下的空间会宽松很多\nMac运行本地模型常见的聊线\nGGUF加Lama CPP一节MLX\n这次选择MLX\n因为它针对苹果电脑有比较特殊的一些优化\n而且后面的D-Flash 2工具可以直接运行MLX权重\n我这次选择千万3.8RGB的MLX 8bit版本\n如果内存比较小\n把8bit换成对应的4bit仓库即可\n其他流程基本是一样的\n模型可以直接去HarkinFace下载\n那么下载的时候\n不建议打开浏览器\n一个文件一个文件点击\n大模型通常拆成多个权重封片\n浏览器下载非常容易中断\n也可能漏掉配置文件\n那么这里更推荐用HarkinFace官方的\nHF命令航工具\n使用新版的HUB\n使用XNET后端\n它可以支持并行下载\n就造成你的Transfer\n已经逐步被淘汰了\n那么我们这里\n首先安装最新版Hagenface CLI\n接下来\n开启高性能模式\n然后把模型下载到指定目录\n那么高性能模式会提高并发\n尽可能利用网络和CPU资源\n模型比较大\n下载前记得磁盘至少还有几十T的空余空间\n下载完成之后\n我们先别着急安装D-Flash 2\n先跑出没有加速的基础版本\n建议给本地模型单独创建\nPython虚拟环境\n避免不同的项目依赖会互相污染\n创建环境\n激活环境\n以及安装MXM和MX\n第一次测试呢\n不要开超长的上下纹\n不要开Dflash 2\n同时呢\n也不要修改一大堆采样参数\n只指定模型路径\n固定Prompt\n生成几百个Token\n运行最基础的生成命令\n运行完成之后呢\n我们可以把模型加载时间\nPrompt处理速度\n守Token等待时间和生成Token速度\n这四个数字记录下来\n如果没有基础数据\n后面看到每每20几个token也不知道到底快了多少\n基础9变27是3倍\n那么基础20变22可能就不值得增加内存和复杂度\n确认基础模型稳定之后呢\n我们就可以安装D-Flash 2\nD-Flash 2本质上是一种推测解码技术\n正常情况下\n千万3.827B每生成一个token的话\n都要让完整270亿参数计算一次\n那么生成10个token\n就要连续做10次完整的推理\nDflash 2\n让一个大约20亿参数的草稿模型\n一次性并行预测接下来的一组token\n然后再让27B主模型一次检查这一整组答案\n如果草稿模型猜的比较准\n主模型一次计算就可以接受多个Token\n原本连续走五六四的流程\n现在可能只需要一次验证\n草稿模型只负责猜\n最终决定权依然在27B主模型手上\n贪心解码下最终输出\n可以与基础模型一致\n随机采样则保留主模型原本的分布\n在Mac上\n比较直接的实现是MX的D Spark\n那么这个虽然名字写着Dspark\n但它同时支持Dspark,Dflash和Dflash2\n在用的时候对于千万3.8,27B,Ultra模式会直接使用Dflash2\n安装命令也非常简单\n屏幕上的安装命令\n安装完成以后不要凭感觉判断有没有变快\n直接运行项目自带的Benchmark\n使用三轮测试\n第一次运行会自动下载匹配的DFlesh2草稿模型\n并根据这台Mac的实际成本测试\n合适的推理长度\nM1 M3 M4 M5的最佳参数呢\n它是不一定一样的\n如果你想直接生成内容的话\n建议用这个命令\n可以使用MXD Spark Generate\n默认AUTO模式会为千万3.8RCB匹配DFLASH2\n再用完全相同的Prompt测试\n对比基础速度\nDFLASH2速度接受长度和内存占用\n那么比下来之后\n项目在特定的M4 Pro测试中\n8bit的基础约8.4Token每秒\nDFLASH2平均约30.5Token每秒\n不同任务约2.8到4倍\n但这不是你的Mac一定能达到的速度\n因为呢\nDFLASH2不是打开以后\n它就一定变快\n如果草稿接受率低的话\n主模型会不断拒绝和重新生成\n草稿计算反而成为额外的负担\n对于只有几十个Token的短回答\n模型刚加载完成和校准\n任务就结束了\n实际体感提升可能不是很明显\nD-Flash 2还会增加内存占用\n所以呢\n如果8bit主\n所以,如果8bit主模型以及頻繁swap,那麼再加載草稿模型,最終速度反而更慢。\n所以要用自己最常執行的任務去測試一下。寫代碼就測真實代碼修改。\n生成口播稿就连续输出一两千的token\n运行agent就测试完整工具调用\n最后一步是把模型启动成API\nMLX的Dspark可以兼容OpenAI\n那么和Astorbit格式的本地服务\n启动命令是MLXDsparkServe\n默认是开在本级8080端口去提供接口的\n客户端BaseURL设置成127.0.18080\n设置好之后\n只在自己电脑使用\n可以不设置真正的API Key\n需要局域网访问的时候\n要调整监听地址\n检查防火墙\n但不要把无件权接口直接暴露到公网\n到这里我们完成了整套部署流程\n第一\nR7B代表约270亿参数\n而且它是全量激活的稠密模型\n第二\nRGB不等于只需RGB内存\n把比特权重接近R9G运行\n是要加CASE缓冲区系统和草稿模型\n那么第三\n它属于本地模型高端档\n部分编程和Agent任务接近B原起见\n但不是全面的替代\n第四\n不要一上来\n就打开所有加速箱\n先用MLX跑出基础速度\n再用相同Prompt做A-B测试\n决定是否长期使用D-Flash 2\n如果真实任务从每秒9个Token\n能够提高到2-30个的话\n而且内存稳定\n那么它就值得长期去打开\n如果频繁Swap或者主要都是短问答的话\n4bit基础模型可能更舒服一些\n本地部署大模型的价值不是省API费用\n当它变成Mac上的本地服务\n它就可以接入代码Langraph\n自动剪辑系统\n知识库\n以及agent工作流\n真正好的本地布置方案\n不是参数最大\n也不是测试数字最高\n而是在自己的设备上\n质量足够好\n速度能够接受\n内存保持稳定\n并且真的进入日常工作流\n才能长期去使用\n影片领誘到的模型地址安装命令和测试命令\n我都会整理在下方\n如果希望下一期测试4bit 6bit和8bit的品质差距的话\n以及DFlex 2速度也可以留言给我\n喜欢这支影片的话\n记得帮我按赞订阅加分享哦\n这是我持续创作的最大动力\n我们下次再见", "language": "zh", "duration": 1125.935625, "segments": [ { "start": 0.0, "end": 12.0, "text": "開放了千萬3.8RGB的模型權重。很多人看到RGB之後的第一個問題就是,這東西,我的Mac到底能不能跑呢?", "chunk": 0, "language": "zh", "avg_logprob": -0.23163795471191406, "no_speech_prob": 1.0186812331169914e-11, "compression_ratio": 1.059090909090909, "source": "whisper" }, { "start": 12.0, "end": 21.0, "text": "有人看到量化以後只需要十幾GB的內存,覺得自己的24GB Mac應該也沒有什麼問題。", "chunk": 0, "language": "zh", "avg_logprob": -0.23163795471191406, "no_speech_prob": 1.0186812331169914e-11, "compression_ratio": 1.059090909090909, "source": "whisper" }, { "start": 21.0, "end": 24.06, "text": "结果模型下载完以后", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 24.06, "end": 25.78, "text": "要么加载失败", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 25.78, "end": 27.88, "text": "要么刚开始生成", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 27.88, "end": 30.38, "text": "系统内存就直接爆掉", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 30.38, "end": 34.88, "text": "有人花了半天时间把模型跑起来", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 34.88, "end": 38.6, "text": "最后发现生成速度只有每秒八九个Token", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 38.6, "end": 41.7, "text": "打开MTP之后不仅没有加速", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 41.7, "end": 42.98, "text": "反而更慢了", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 42.98, "end": 45.52, "text": "所以今天这支影片", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 45.52, "end": 47.66, "text": "我会从零开始", "chunk": 0, "language": "zh", "avg_logprob": -0.13290001465393617, "no_speech_prob": 1.0647512385664193e-11, "compression_ratio": 1.1652542372881356, "source": "whisper" }, { "start": 47.66, "end": 50.84, "text": "带你在Mac上完整部署一次", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 50.84, "end": 52.68, "text": "千万3.8 RGB", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 52.68, "end": 56.1, "text": "我们会先搞清楚RGP到底代表什么", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 56.1, "end": 58.96, "text": "它和Cloud GPT这些B原模型", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 58.96, "end": 60.62, "text": "大概处于什么水平", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 60.62, "end": 63.88, "text": "然后算清楚模型运行时", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 63.88, "end": 65.88, "text": "到底会占用多少内存", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 65.88, "end": 71.12, "text": "接下来我会带你选择正确的MLX量化版本", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 71.12, "end": 73.56, "text": "使用Hugging Face的并行下载工具", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 73.56, "end": 76.44, "text": "把模型下载到本地", "chunk": 0, "language": "zh", "avg_logprob": -0.1355231190455779, "no_speech_prob": 9.142876560008784e-12, "compression_ratio": 1.1214285714285714, "source": "whisper" }, { "start": 77.66, "end": 81.5, "text": "先跑出一个没有任何加速的基础速度", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 81.5, "end": 84.14, "text": "最后再安装DFlesh 2", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 84.14, "end": 86.28, "text": "在下载模型之前", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 86.28, "end": 88.46, "text": "我们首先得搞清楚一件事", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 88.46, "end": 93.62, "text": "前往3.827B里面的这个27B到底是什么意思", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 93.62, "end": 97.58, "text": "这里的B是Billing", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 97.58, "end": 99.0, "text": "也就是实义", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 99.0, "end": 102.84, "text": "27B代表这个语言的模型部分", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 102.84, "end": 106.14, "text": "大约拥有271个参数", "chunk": 0, "language": "zh", "avg_logprob": -0.19504516465323313, "no_speech_prob": 1.090696803707214e-11, "compression_ratio": 1.1061224489795918, "source": "whisper" }, { "start": 106.14, "end": 113.78, "text": "你可以把参数理解成有模型训练完成以后保留下来的271个数字", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 113.78, "end": 118.6, "text": "模型每生成一个token都要读取这些数字进行计算", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 118.6, "end": 122.64, "text": "然后判断下一个token最有可能是什么", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 122.64, "end": 124.72, "text": "如果想象的更具体一点", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 124.72, "end": 130.9, "text": "你可以把它理解成一台拥有271个旋钮的巨大机器", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 130.9, "end": 135.78, "text": "训练模型的过程就是不断调整这271个旋钮", "chunk": 0, "language": "zh", "avg_logprob": -0.06313332579189673, "no_speech_prob": 8.079360097612565e-12, "compression_ratio": 1.3373015873015872, "source": "whisper" }, { "start": 135.78, "end": 138.74, "text": "而我们现在进行的本地推理", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 138.74, "end": 142.16, "text": "就是把这些调整好的旋钮加载到内存里", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 142.16, "end": 144.4, "text": "然后用它们来生成答案", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 144.4, "end": 147.24, "text": "这里还有一个非常重要的概念", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 147.24, "end": 152.1, "text": "也就是说千问3.827B是一个Distance模型", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 152.1, "end": 153.82, "text": "也就是稠密模型", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 153.82, "end": 156.72, "text": "而不是MROE混合专家模型", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 156.72, "end": 161.84, "text": "有些模型名称可能写着总参数35B", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 161.84, "end": 164.16, "text": "但是后面还有一个A3B", "chunk": 0, "language": "zh", "avg_logprob": -0.11350999559674944, "no_speech_prob": 1.0213775138157022e-11, "compression_ratio": 1.173913043478261, "source": "whisper" }, { "start": 164.16, "end": 165.98, "text": "所谓的A3B呢", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 165.98, "end": 167.38, "text": "这个A3B的意思就是", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 167.38, "end": 169.98, "text": "它虽然一共有350亿参数", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 169.98, "end": 171.78, "text": "但每次生成Token的时候", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 171.78, "end": 174.6, "text": "大约只有30亿参数会被激活", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 174.6, "end": 177.62, "text": "前往3.827B不一样", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 177.62, "end": 179.7, "text": "它没有A几B的后缀", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 179.7, "end": 182.98, "text": "它的语言模型拥有270亿参数", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 182.98, "end": 185.02, "text": "每生成一个Token", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 185.02, "end": 187.8, "text": "那么就代表着", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 187.8, "end": 190.74, "text": "基本要经过这270亿参数的计算", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 190.74, "end": 193.68, "text": "所以它的总参数大约是27B", "chunk": 0, "language": "zh", "avg_logprob": -0.08554289842906751, "no_speech_prob": 8.697074310726194e-12, "compression_ratio": 1.2568093385214008, "source": "whisper" }, { "start": 193.68, "end": 196.12, "text": "激活参数呢", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 196.12, "end": 198.16, "text": "同样也是27B", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 198.16, "end": 203.32, "text": "这也是为什么同样叫做30B左右的模型", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 203.32, "end": 205.18, "text": "千万3.8 27B", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 205.18, "end": 208.8, "text": "可能比一些35B A3B的MOE模型大很多", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 208.8, "end": 209.4, "text": "当然", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 209.4, "end": 211.86, "text": "稠密模型也不是只有缺点", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 211.86, "end": 216.14, "text": "它的优势是所有能力都集中在同一套参数里", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 216.14, "end": 217.94, "text": "通常在输出稳定性", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 217.94, "end": 221.06, "text": "复杂任务和知识调用上会更加均衡", "chunk": 0, "language": "zh", "avg_logprob": -0.09012216596461053, "no_speech_prob": 1.0095654347230809e-11, "compression_ratio": 1.1725490196078432, "source": "whisper" }, { "start": 221.06, "end": 224.86, "text": "如果可以把本地运行的模型简单承评成三个类别", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 224.86, "end": 227.66, "text": "3B到8B通常属于轻量模型", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 227.66, "end": 229.22, "text": "这类模型速度很快", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 229.22, "end": 231.58, "text": "但复杂任务比较容易走偏", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 231.58, "end": 237.68, "text": "14B到30B可以看成本地大模型目前最实用的高质量区间", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 237.68, "end": 242.38, "text": "能力已经足以处理比较复杂的编程文章生成和工具调用", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 242.38, "end": 243.72, "text": "以及agent任务", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 243.72, "end": 250.86, "text": "70B以上的稠密模型能力通常会更加稳定", "chunk": 0, "language": "zh", "avg_logprob": -0.11881552275663577, "no_speech_prob": 1.121300274853354e-11, "compression_ratio": 1.25, "source": "whisper" }, { "start": 250.86, "end": 255.08, "text": "但需要的内存和内存带宽也会明显上升", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 255.08, "end": 257.44, "text": "对于大部分个人用户来说", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 257.44, "end": 259.88, "text": "部署成本其实是比较高的", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 259.88, "end": 266.88, "text": "所以千万3.8RGB基本属于个人电脑还能够比较现实的部署", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 266.88, "end": 270.9, "text": "同时能力有足够强的高端本地模型", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 270.9, "end": 274.0, "text": "那它跟市面上的B1模型相比怎么样呢", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 274.0, "end": 280.02, "text": "首先我们不能直接拿RGB去和Cloud和GPT的比较参数量", "chunk": 0, "language": "zh", "avg_logprob": -0.12798802835957987, "no_speech_prob": 1.113035011379715e-11, "compression_ratio": 1.193103448275862, "source": "whisper" }, { "start": 280.02, "end": 285.08, "text": "因为大部分B原模型不会公开完整参数量和激活参数", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 285.08, "end": 287.72, "text": "即使参数量一样", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 287.72, "end": 291.8, "text": "训练数据、模型架构、后训练和推理预算不同", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 291.8, "end": 294.38, "text": "最终能力也可能差很多", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 294.38, "end": 296.24, "text": "比较稳妥的说法是", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 296.24, "end": 300.68, "text": "千万3.8RGB已经进入了可以和B原旗舰模型", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 300.68, "end": 302.92, "text": "在部分任务上正面竞争的范围", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 302.92, "end": 306.92, "text": "但它还不能全面替代B原旗舰", "chunk": 0, "language": "zh", "avg_logprob": -0.10511265825105194, "no_speech_prob": 1.0455998046554615e-11, "compression_ratio": 1.2595419847328244, "source": "whisper" }, { "start": 306.92, "end": 311.26, "text": "以签问官方测试为例", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 311.26, "end": 313.0, "text": "可以看到", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 313.0, "end": 317.64, "text": "在Batch Pro里签问3.8", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 317.64, "end": 319.08, "text": "RTB是61.7", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 319.08, "end": 321.44, "text": "OPLA 4.6 MAX是53.4", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 321.44, "end": 325.62, "text": "Terminal Bench则是73对78.2", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 325.62, "end": 328.84, "text": "HLE是30.8对40", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 328.84, "end": 332.24, "text": "那么这组数据就可以说明", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 332.24, "end": 335.08, "text": "它在某些编程和工具调用任务上", "chunk": 0, "language": "zh", "avg_logprob": -0.17018574207752674, "no_speech_prob": 9.299311320987158e-12, "compression_ratio": 0.985981308411215, "source": "whisper" }, { "start": 335.08, "end": 337.18, "text": "可以接近甚至超过B元", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 337.18, "end": 339.12, "text": "但是复杂开放式推理", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 339.12, "end": 341.06, "text": "知识广度和长容稳定性", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 341.06, "end": 344.04, "text": "B元旗舰通常还是更强一些", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 344.04, "end": 348.32, "text": "而且这是签问官方公布的测试", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 348.32, "end": 352.18, "text": "它不是不同模型可能使用了不同的harness", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 352.18, "end": 357.04, "text": "所以这也包含官方自己的测试集", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 357.04, "end": 360.32, "text": "可以作为一部分定位参考", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 360.32, "end": 363.08, "text": "但不能因为某一个分数更高", "chunk": 0, "language": "zh", "avg_logprob": -0.126101606474148, "no_speech_prob": 1.1087587445390845e-11, "compression_ratio": 1.202247191011236, "source": "whisper" }, { "start": 363.08, "end": 365.32, "text": "就是说千万已经全面超越了Cloud", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 365.32, "end": 367.6, "text": "对我来说最准确的定位是", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 367.6, "end": 370.68, "text": "它不是B原旗舰的完整替代品", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 370.68, "end": 375.68, "text": "但已经是一款可以真正承担生产任务的高端本地模型", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 375.68, "end": 378.98, "text": "接下来我们来算一下它到底需要多少内存", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 378.98, "end": 383.68, "text": "很多人看到27B会直接认为它需要27G的内存", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 383.68, "end": 388.16, "text": "这个说法只在非常特定的条件下才大致成立", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 389.44, "end": 392.84, "text": "如果使用BF16一个参数呢", "chunk": 0, "language": "zh", "avg_logprob": -0.11010003425705601, "no_speech_prob": 1.1479082441534505e-11, "compression_ratio": 1.2168284789644013, "source": "whisper" }, { "start": 392.84, "end": 394.16, "text": "需要两个字节", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 394.16, "end": 396.9, "text": "270一个参数乘以两个字节的话", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 396.9, "end": 401.48, "text": "光语言模型原始权重就要大约54G", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 401.48, "end": 403.64, "text": "这还没有算计算系统", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 403.64, "end": 406.62, "text": "缓存和退理过程中的临时内存", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 406.62, "end": 408.28, "text": "所以64G Mac", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 408.28, "end": 410.54, "text": "虽然理论上可以放下BF16权重", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 410.54, "end": 413.1, "text": "但运行语量其实非常有限", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 413.1, "end": 415.9, "text": "量化的作用呢", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 415.9, "end": 419.06, "text": "就是使用更少的位数保存这些参数", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 419.06, "end": 422.34, "text": "8bit理论权重大约27G", "chunk": 0, "language": "zh", "avg_logprob": -0.10623672940083687, "no_speech_prob": 1.0663069385796753e-11, "compression_ratio": 1.1914893617021276, "source": "whisper" }, { "start": 422.34, "end": 426.12, "text": "4bit理论上大约13.5G", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 426.12, "end": 430.08, "text": "实际MLX版本还包含量化参数", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 430.08, "end": 431.4, "text": "词表和配置", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 431.4, "end": 434.14, "text": "8bit纯文字版约28.6G", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 434.14, "end": 436.42, "text": "带视觉能力约29.5G", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 436.42, "end": 438.82, "text": "4bit版本大约16G", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 438.82, "end": 442.12, "text": "那么从这些地方的这些数字", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 442.12, "end": 444.12, "text": "只代表模型权重", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 444.12, "end": 447.28, "text": "并不是模型启动后的总内存占用", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 447.28, "end": 449.14, "text": "每次运行时", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 449.14, "end": 451.2, "text": "内存里至少还要放进五类东西", "chunk": 0, "language": "zh", "avg_logprob": -0.09094584069284452, "no_speech_prob": 9.820813433314424e-12, "compression_ratio": 1.1439688715953307, "source": "whisper" }, { "start": 451.2, "end": 453.0, "text": "第一类也就是最大的一类", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 453.0, "end": 455.04, "text": "就是模型群众", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 455.04, "end": 459.44, "text": "运行8bit版本光这一向就已经接近29G", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 459.44, "end": 462.7, "text": "而且启动后基本都要进入Mac统一内存", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 462.7, "end": 467.14, "text": "第二类是KVKISH和线性注意力技术", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 467.14, "end": 470.16, "text": "模型需要保存已经处理过的上下文", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 470.16, "end": 476.7, "text": "否则每生成一个Token都需要重新计算前文", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 476.7, "end": 480.0, "text": "KVKISH和上下文长度直接下文", "chunk": 0, "language": "zh", "avg_logprob": -0.1759010728939559, "no_speech_prob": 1.2706707214205082e-11, "compression_ratio": 1.2067669172932332, "source": "whisper" }, { "start": 481.4, "end": 487.76, "text": "前文原生支持约26万Token,但支持不代表你的Mac适合直接开满。", "chunk": 1, "language": "zh", "avg_logprob": -0.19848033360072545, "no_speech_prob": 9.994729870121954e-12, "compression_ratio": 1.1036036036036037, "source": "whisper" }, { "start": 487.76, "end": 489.78, "text": "粗略计算", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 489.78, "end": 491.22, "text": "32K上下文", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 491.22, "end": 492.62, "text": "仅标准注意力", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 492.62, "end": 494.26, "text": "64K约4G", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 494.26, "end": 496.64, "text": "128K约8G", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 496.64, "end": 499.12, "text": "256K就要16G", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 499.12, "end": 502.76, "text": "此外还没包含线性注意力状态", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 502.76, "end": 505.34, "text": "如果平常只是聊天", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 505.34, "end": 506.92, "text": "或者处理普通代码任务", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 506.92, "end": 509.16, "text": "先从16K和32K开始", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 509.16, "end": 511.08, "text": "通常就已经足够了", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 511.08, "end": 514.06, "text": "第三类是MLX和META运行时", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 514.06, "end": 515.54, "text": "使用的计算空间", "chunk": 1, "language": "zh", "avg_logprob": -0.12527999877929688, "no_speech_prob": 9.11952111048997e-12, "compression_ratio": 1.0871369294605808, "source": "whisper" }, { "start": 515.54, "end": 518.86, "text": "包括临时激活值和计算缓冲区", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 518.86, "end": 521.64, "text": "那么它不是固定数字", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 521.64, "end": 524.02, "text": "会随输入程度和并发变化", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 524.02, "end": 527.86, "text": "尤其是在第一次读取大段Prompt", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 527.86, "end": 529.46, "text": "也就是Prefill阶段", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 529.46, "end": 532.02, "text": "内存占用可能突然上升", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 532.02, "end": 536.0, "text": "第四类是视觉编码器", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 536.0, "end": 539.14, "text": "那么只做文字", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 539.14, "end": 540.88, "text": "写代码或运行Agent", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 540.88, "end": 542.3, "text": "可以选择TextOnly", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 542.3, "end": 544.26, "text": "需要读取图片或视频", "chunk": 1, "language": "zh", "avg_logprob": -0.12596155891955738, "no_speech_prob": 1.0079851883726398e-11, "compression_ratio": 1.1254612546125462, "source": "whisper" }, { "start": 544.26, "end": 546.26, "text": "才选择完整多模态版本", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 546.26, "end": 549.5, "text": "第五类是DFLASH2草稿模型", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 549.5, "end": 551.02, "text": "它不是普通开关", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 551.02, "end": 553.22, "text": "还需要一个配套的草稿模型", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 553.22, "end": 557.02, "text": "本身它大约21个参数", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 557.02, "end": 561.48, "text": "并额外占用缓存和缓冲区", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 561.48, "end": 566.38, "text": "所以最终的内存杖不是27B等于27G", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 566.38, "end": 568.22, "text": "而是模型权重加Cache", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 568.22, "end": 571.02, "text": "加运行缓冲加系统占用", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 571.02, "end": 573.84, "text": "再加DFLASH2草稿模型", "chunk": 1, "language": "zh", "avg_logprob": -0.10984228778359116, "no_speech_prob": 1.2561422388368548e-11, "compression_ratio": 1.214574898785425, "source": "whisper" }, { "start": 573.84, "end": 577.88, "text": "如果想运行8bit版本", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 577.88, "end": 581.82, "text": "那么48G统一内存可以作为比较现实的起点", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 581.82, "end": 584.22, "text": "例如4G会更加舒服", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 584.22, "end": 587.3, "text": "32G不是绝对加载不了", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 587.3, "end": 588.78, "text": "但余量非常小", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 588.78, "end": 591.22, "text": "如果Mac只有24G或32G的话", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 591.22, "end": 596.98, "text": "我建议更加直接使用4bit版本是比较好的", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 596.98, "end": 599.96, "text": "模型文件大约16G", "chunk": 1, "language": "zh", "avg_logprob": -0.10592486479572047, "no_speech_prob": 8.896260464408279e-12, "compression_ratio": 1.1061946902654867, "source": "whisper" }, { "start": 599.96, "end": 605.0, "text": "给系统Cache和D-Flash 2留下的空间会宽松很多", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 605.0, "end": 607.6, "text": "Mac运行本地模型常见的聊线", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 607.6, "end": 610.5, "text": "GGUF加Lama CPP一节MLX", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 610.5, "end": 613.32, "text": "这次选择MLX", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 613.32, "end": 618.24, "text": "因为它针对苹果电脑有比较特殊的一些优化", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 618.24, "end": 621.88, "text": "而且后面的D-Flash 2工具可以直接运行MLX权重", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 621.88, "end": 628.86, "text": "我这次选择千万3.8RGB的MLX 8bit版本", "chunk": 1, "language": "zh", "avg_logprob": -0.1602724738742994, "no_speech_prob": 1.0403664041369609e-11, "compression_ratio": 1.0703703703703704, "source": "whisper" }, { "start": 628.86, "end": 630.86, "text": "如果内存比较小", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 630.86, "end": 634.12, "text": "把8bit换成对应的4bit仓库即可", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 634.12, "end": 637.68, "text": "其他流程基本是一样的", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 637.68, "end": 641.36, "text": "模型可以直接去HarkinFace下载", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 641.36, "end": 642.96, "text": "那么下载的时候", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 642.96, "end": 644.62, "text": "不建议打开浏览器", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 644.62, "end": 646.52, "text": "一个文件一个文件点击", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 646.52, "end": 649.94, "text": "大模型通常拆成多个权重封片", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 649.94, "end": 652.5, "text": "浏览器下载非常容易中断", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 652.5, "end": 654.5, "text": "也可能漏掉配置文件", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 654.5, "end": 658.18, "text": "那么这里更推荐用HarkinFace官方的", "chunk": 1, "language": "zh", "avg_logprob": -0.13267044118932775, "no_speech_prob": 1.0164106535592854e-11, "compression_ratio": 1.2472727272727273, "source": "whisper" }, { "start": 658.18, "end": 659.66, "text": "HF命令航工具", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 659.66, "end": 662.58, "text": "使用新版的HUB", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 662.58, "end": 664.92, "text": "使用XNET后端", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 664.92, "end": 666.8, "text": "它可以支持并行下载", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 666.8, "end": 669.7, "text": "就造成你的Transfer", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 669.7, "end": 671.78, "text": "已经逐步被淘汰了", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 671.78, "end": 672.84, "text": "那么我们这里", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 672.84, "end": 675.68, "text": "首先安装最新版Hagenface CLI", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 675.68, "end": 677.44, "text": "接下来", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 677.44, "end": 679.34, "text": "开启高性能模式", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 679.34, "end": 682.56, "text": "然后把模型下载到指定目录", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 682.56, "end": 685.96, "text": "那么高性能模式会提高并发", "chunk": 1, "language": "zh", "avg_logprob": -0.1552984176143523, "no_speech_prob": 1.5371773298689106e-11, "compression_ratio": 1.1204819277108433, "source": "whisper" }, { "start": 685.96, "end": 688.2, "text": "尽可能利用网络和CPU资源", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 688.2, "end": 689.28, "text": "模型比较大", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 689.28, "end": 692.56, "text": "下载前记得磁盘至少还有几十T的空余空间", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 692.56, "end": 694.0, "text": "下载完成之后", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 694.0, "end": 696.58, "text": "我们先别着急安装D-Flash 2", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 696.58, "end": 700.6, "text": "先跑出没有加速的基础版本", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 700.6, "end": 703.16, "text": "建议给本地模型单独创建", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 703.16, "end": 704.66, "text": "Python虚拟环境", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 704.66, "end": 708.88, "text": "避免不同的项目依赖会互相污染", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 708.88, "end": 710.48, "text": "创建环境", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 710.48, "end": 711.88, "text": "激活环境", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 711.88, "end": 714.8, "text": "以及安装MXM和MX", "chunk": 1, "language": "zh", "avg_logprob": -0.09388711338951475, "no_speech_prob": 1.16956670043189e-11, "compression_ratio": 1.1354166666666667, "source": "whisper" }, { "start": 714.8, "end": 716.72, "text": "第一次测试呢", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 716.72, "end": 718.42, "text": "不要开超长的上下纹", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 718.42, "end": 720.56, "text": "不要开Dflash 2", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 720.56, "end": 722.22, "text": "同时呢", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 722.22, "end": 724.96, "text": "也不要修改一大堆采样参数", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 724.96, "end": 727.3, "text": "只指定模型路径", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 727.3, "end": 728.34, "text": "固定Prompt", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 728.34, "end": 729.96, "text": "生成几百个Token", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 729.96, "end": 732.12, "text": "运行最基础的生成命令", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 732.12, "end": 734.2, "text": "运行完成之后呢", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 734.2, "end": 737.18, "text": "我们可以把模型加载时间", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 737.18, "end": 738.9, "text": "Prompt处理速度", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 738.9, "end": 741.66, "text": "守Token等待时间和生成Token速度", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 741.66, "end": 743.86, "text": "这四个数字记录下来", "chunk": 1, "language": "zh", "avg_logprob": -0.13190226523291987, "no_speech_prob": 9.908660697499627e-12, "compression_ratio": 1.2185185185185186, "source": "whisper" }, { "start": 743.86, "end": 746.04, "text": "如果没有基础数据", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 746.04, "end": 749.8, "text": "后面看到每每20几个token也不知道到底快了多少", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 749.8, "end": 752.02, "text": "基础9变27是3倍", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 752.02, "end": 757.38, "text": "那么基础20变22可能就不值得增加内存和复杂度", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 757.38, "end": 762.6, "text": "确认基础模型稳定之后呢", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 762.6, "end": 765.48, "text": "我们就可以安装D-Flash 2", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 765.48, "end": 769.96, "text": "D-Flash 2本质上是一种推测解码技术", "chunk": 1, "language": "zh", "avg_logprob": -0.11997632743898502, "no_speech_prob": 9.513278186046303e-12, "compression_ratio": 1.1097560975609757, "source": "whisper" }, { "start": 769.96, "end": 771.2, "text": "正常情况下", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 771.2, "end": 774.16, "text": "千万3.827B每生成一个token的话", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 774.16, "end": 777.58, "text": "都要让完整270亿参数计算一次", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 777.58, "end": 780.28, "text": "那么生成10个token", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 780.28, "end": 783.28, "text": "就要连续做10次完整的推理", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 783.28, "end": 785.4, "text": "Dflash 2", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 785.4, "end": 788.76, "text": "让一个大约20亿参数的草稿模型", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 788.76, "end": 792.3, "text": "一次性并行预测接下来的一组token", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 792.3, "end": 798.08, "text": "然后再让27B主模型一次检查这一整组答案", "chunk": 1, "language": "zh", "avg_logprob": -0.11271844693084261, "no_speech_prob": 8.400256185092658e-12, "compression_ratio": 1.1983805668016194, "source": "whisper" }, { "start": 798.08, "end": 800.92, "text": "如果草稿模型猜的比较准", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 800.92, "end": 804.2, "text": "主模型一次计算就可以接受多个Token", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 804.2, "end": 807.26, "text": "原本连续走五六四的流程", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 807.26, "end": 809.62, "text": "现在可能只需要一次验证", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 809.62, "end": 812.46, "text": "草稿模型只负责猜", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 812.46, "end": 815.96, "text": "最终决定权依然在27B主模型手上", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 815.96, "end": 817.74, "text": "贪心解码下最终输出", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 817.74, "end": 819.86, "text": "可以与基础模型一致", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 819.86, "end": 822.72, "text": "随机采样则保留主模型原本的分布", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 822.72, "end": 824.34, "text": "在Mac上", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 824.34, "end": 827.32, "text": "比较直接的实现是MX的D Spark", "chunk": 1, "language": "zh", "avg_logprob": -0.10677342270359849, "no_speech_prob": 1.0741073828979264e-11, "compression_ratio": 1.2535211267605635, "source": "whisper" }, { "start": 827.32, "end": 831.24, "text": "那么这个虽然名字写着Dspark", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 831.24, "end": 834.28, "text": "但它同时支持Dspark,Dflash和Dflash2", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 834.28, "end": 842.04, "text": "在用的时候对于千万3.8,27B,Ultra模式会直接使用Dflash2", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 842.04, "end": 847.2, "text": "安装命令也非常简单", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 851.88, "end": 853.36, "text": "屏幕上的安装命令", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 853.36, "end": 857.06, "text": "安装完成以后不要凭感觉判断有没有变快", "chunk": 1, "language": "zh", "avg_logprob": -0.19419280043593398, "no_speech_prob": 9.472695197687564e-12, "compression_ratio": 1.0964912280701755, "source": "whisper" }, { "start": 857.06, "end": 859.52, "text": "直接运行项目自带的Benchmark", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 859.52, "end": 861.72, "text": "使用三轮测试", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 861.72, "end": 867.14, "text": "第一次运行会自动下载匹配的DFlesh2草稿模型", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 867.14, "end": 870.34, "text": "并根据这台Mac的实际成本测试", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 870.34, "end": 872.58, "text": "合适的推理长度", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 872.58, "end": 875.78, "text": "M1 M3 M4 M5的最佳参数呢", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 875.78, "end": 877.44, "text": "它是不一定一样的", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 877.44, "end": 880.08, "text": "如果你想直接生成内容的话", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 880.08, "end": 882.32, "text": "建议用这个命令", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 882.32, "end": 884.78, "text": "可以使用MXD Spark Generate", "chunk": 1, "language": "zh", "avg_logprob": -0.15595877587378443, "no_speech_prob": 1.0164874150730974e-11, "compression_ratio": 1.1223021582733812, "source": "whisper" }, { "start": 884.78, "end": 890.84, "text": "默认AUTO模式会为千万3.8RCB匹配DFLASH2", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 890.84, "end": 894.3, "text": "再用完全相同的Prompt测试", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 894.3, "end": 895.66, "text": "对比基础速度", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 895.66, "end": 899.1, "text": "DFLASH2速度接受长度和内存占用", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 899.1, "end": 900.98, "text": "那么比下来之后", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 900.98, "end": 903.36, "text": "项目在特定的M4 Pro测试中", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 903.36, "end": 906.16, "text": "8bit的基础约8.4Token每秒", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 906.16, "end": 909.16, "text": "DFLASH2平均约30.5Token每秒", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 909.16, "end": 913.04, "text": "不同任务约2.8到4倍", "chunk": 1, "language": "zh", "avg_logprob": -0.13123943723481277, "no_speech_prob": 1.0129710438511186e-11, "compression_ratio": 1.1075697211155378, "source": "whisper" }, { "start": 913.04, "end": 917.22, "text": "但这不是你的Mac一定能达到的速度", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 917.22, "end": 917.98, "text": "因为呢", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 917.98, "end": 920.62, "text": "DFLASH2不是打开以后", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 920.62, "end": 922.0, "text": "它就一定变快", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 922.0, "end": 923.98, "text": "如果草稿接受率低的话", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 923.98, "end": 926.98, "text": "主模型会不断拒绝和重新生成", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 926.98, "end": 930.08, "text": "草稿计算反而成为额外的负担", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 930.08, "end": 933.44, "text": "对于只有几十个Token的短回答", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 933.44, "end": 936.24, "text": "模型刚加载完成和校准", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 936.24, "end": 937.44, "text": "任务就结束了", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 937.44, "end": 939.98, "text": "实际体感提升可能不是很明显", "chunk": 1, "language": "zh", "avg_logprob": -0.11758167213863796, "no_speech_prob": 1.016427220168481e-11, "compression_ratio": 1.1743772241992882, "source": "whisper" }, { "start": 939.98, "end": 943.04, "text": "D-Flash 2还会增加内存占用", "chunk": 1, "language": "zh", "avg_logprob": -0.2704872025383843, "no_speech_prob": 8.407554166756093e-12, "compression_ratio": 0.8333333333333334, "source": "whisper" }, { "start": 943.04, "end": 944.46, "text": "所以呢", "chunk": 1, "language": "zh", "avg_logprob": -0.2704872025383843, "no_speech_prob": 8.407554166756093e-12, "compression_ratio": 0.8333333333333334, "source": "whisper" }, { "start": 944.46, "end": 945.936, "text": "如果8bit主", "chunk": 1, "language": "zh", "avg_logprob": -0.2704872025383843, "no_speech_prob": 8.407554166756093e-12, "compression_ratio": 0.8333333333333334, "source": "whisper" }, { "start": 945.936, "end": 952.936, "text": "所以,如果8bit主模型以及頻繁swap,那麼再加載草稿模型,最終速度反而更慢。", "chunk": 2, "language": "zh", "avg_logprob": -0.16911011847896854, "no_speech_prob": 8.654795630169687e-12, "compression_ratio": 1.173202614379085, "source": "whisper" }, { "start": 952.936, "end": 958.936, "text": "所以要用自己最常執行的任務去測試一下。寫代碼就測真實代碼修改。", "chunk": 2, "language": "zh", "avg_logprob": -0.16911011847896854, "no_speech_prob": 8.654795630169687e-12, "compression_ratio": 1.173202614379085, "source": "whisper" }, { "start": 958.936, "end": 962.376, "text": "生成口播稿就连续输出一两千的token", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 962.376, "end": 966.236, "text": "运行agent就测试完整工具调用", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 966.236, "end": 969.936, "text": "最后一步是把模型启动成API", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 969.936, "end": 974.076, "text": "MLX的Dspark可以兼容OpenAI", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 974.076, "end": 978.036, "text": "那么和Astorbit格式的本地服务", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 978.036, "end": 980.716, "text": "启动命令是MLXDsparkServe", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 982.796, "end": 987.656, "text": "默认是开在本级8080端口去提供接口的", "chunk": 2, "language": "zh", "avg_logprob": -0.1894981563091278, "no_speech_prob": 9.814134747931913e-12, "compression_ratio": 1.0595238095238095, "source": "whisper" }, { "start": 987.656, "end": 992.296, "text": "客户端BaseURL设置成127.0.18080", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 992.296, "end": 995.576, "text": "设置好之后", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 995.576, "end": 997.276, "text": "只在自己电脑使用", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 997.276, "end": 1000.016, "text": "可以不设置真正的API Key", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1000.016, "end": 1001.836, "text": "需要局域网访问的时候", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1001.836, "end": 1003.396, "text": "要调整监听地址", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1003.396, "end": 1004.976, "text": "检查防火墙", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1004.976, "end": 1009.256, "text": "但不要把无件权接口直接暴露到公网", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1009.256, "end": 1012.756, "text": "到这里我们完成了整套部署流程", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1012.756, "end": 1013.476, "text": "第一", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1013.476, "end": 1016.576, "text": "R7B代表约270亿参数", "chunk": 2, "language": "zh", "avg_logprob": -0.15905124157458753, "no_speech_prob": 1.1606281040410504e-11, "compression_ratio": 1.0774907749077491, "source": "whisper" }, { "start": 1016.576, "end": 1019.596, "text": "而且它是全量激活的稠密模型", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1019.596, "end": 1020.196, "text": "第二", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1020.196, "end": 1023.456, "text": "RGB不等于只需RGB内存", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1023.456, "end": 1025.556, "text": "把比特权重接近R9G运行", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1025.556, "end": 1029.416, "text": "是要加CASE缓冲区系统和草稿模型", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1029.416, "end": 1032.136, "text": "那么第三", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1032.136, "end": 1035.016, "text": "它属于本地模型高端档", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1035.016, "end": 1038.116, "text": "部分编程和Agent任务接近B原起见", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1038.116, "end": 1039.996, "text": "但不是全面的替代", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1039.996, "end": 1041.076, "text": "第四", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1041.076, "end": 1042.596, "text": "不要一上来", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1042.596, "end": 1044.516, "text": "就打开所有加速箱", "chunk": 2, "language": "zh", "avg_logprob": -0.14509812763759067, "no_speech_prob": 1.1283685791285691e-11, "compression_ratio": 1.146153846153846, "source": "whisper" }, { "start": 1044.516, "end": 1046.976, "text": "先用MLX跑出基础速度", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1046.976, "end": 1049.436, "text": "再用相同Prompt做A-B测试", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1049.436, "end": 1052.596, "text": "决定是否长期使用D-Flash 2", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1052.596, "end": 1055.716, "text": "如果真实任务从每秒9个Token", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1055.716, "end": 1057.776, "text": "能够提高到2-30个的话", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1057.776, "end": 1059.356, "text": "而且内存稳定", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1059.356, "end": 1061.856, "text": "那么它就值得长期去打开", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1061.856, "end": 1065.996, "text": "如果频繁Swap或者主要都是短问答的话", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1065.996, "end": 1068.736, "text": "4bit基础模型可能更舒服一些", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1068.736, "end": 1073.036, "text": "本地部署大模型的价值不是省API费用", "chunk": 2, "language": "zh", "avg_logprob": -0.15267231320374763, "no_speech_prob": 1.2173688272720806e-11, "compression_ratio": 1.1006493506493507, "source": "whisper" }, { "start": 1073.036, "end": 1075.936, "text": "当它变成Mac上的本地服务", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1075.936, "end": 1078.696, "text": "它就可以接入代码Langraph", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1078.696, "end": 1079.936, "text": "自动剪辑系统", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1079.936, "end": 1080.776, "text": "知识库", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1080.776, "end": 1083.916, "text": "以及agent工作流", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1083.916, "end": 1087.556, "text": "真正好的本地布置方案", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1087.556, "end": 1088.876, "text": "不是参数最大", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1088.876, "end": 1090.636, "text": "也不是测试数字最高", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1090.636, "end": 1092.116, "text": "而是在自己的设备上", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1092.116, "end": 1093.076, "text": "质量足够好", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1093.076, "end": 1094.316, "text": "速度能够接受", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1094.316, "end": 1096.236, "text": "内存保持稳定", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1096.236, "end": 1099.156, "text": "并且真的进入日常工作流", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1099.156, "end": 1100.756, "text": "才能长期去使用", "chunk": 2, "language": "zh", "avg_logprob": -0.1148639284331223, "no_speech_prob": 1.0730094764099807e-11, "compression_ratio": 1.2037735849056603, "source": "whisper" }, { "start": 1100.756, "end": 1104.196, "text": "影片领誘到的模型地址安装命令和测试命令", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1104.196, "end": 1106.416, "text": "我都会整理在下方", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1106.416, "end": 1113.436, "text": "如果希望下一期测试4bit 6bit和8bit的品质差距的话", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1113.436, "end": 1116.856, "text": "以及DFlex 2速度也可以留言给我", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1116.856, "end": 1119.056, "text": "喜欢这支影片的话", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1119.056, "end": 1122.776, "text": "记得帮我按赞订阅加分享哦", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1122.776, "end": 1124.956, "text": "这是我持续创作的最大动力", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" }, { "start": 1124.956, "end": 1125.916, "text": "我们下次再见", "chunk": 2, "language": "zh", "avg_logprob": -0.13539819394127797, "no_speech_prob": 1.1650425416065424e-11, "compression_ratio": 1.1544401544401544, "source": "whisper" } ], "transcript_selection": { "schema": "videonote.transcript_selection.v1", "status": "SELECTED", "requested_language": "zh", "duration": 1125.935625, "reference_path": null, "reference_kind": null, "reference_language": null, "selected_source": "whisper_segmented", "decision_reason": "source_unavailable_or_untrusted_whisper_passed", "whisper_rejected": false, "source_profile": { "segment_count": 0, "duration_coverage": 0.0, "usable_full": false, "usable_partial": false, "hallucination_region_count": 0, "hallucination_regions": [], "language": { "requested": "zh", "matched": false } }, "whisper_profile": { "segment_count": 376, "multilingual": false, "first_start": 0.0, "final_end": 1125.916, "duration_coverage": 0.999983, "maximum_gap_seconds": 4.68, "inverted_count": 0, "language": { "requested": "zh", "matched": true, "latin": 857, "han": 3653, "kana": 0, "hangul": 0 }, "hallucination_region_count": 0, "hallucination_regions": [], "usable_full": true, "usable_partial": true }, "comparison": { "window_seconds": 30.0, "compared_window_count": 0, "median_score": 0.0, "divergent_window_count": 0, "divergent_ratio": 0.0, "divergent_windows": [] }, "timestamp_alignment": { "applied": false, "estimated_offset_seconds": 0.0, "matched_segment_count": 0, "match_scores": [] }, "caption_timing_repair": null, "filled_source_profile": null, "whisper_duration_guard": { "media_duration": 1125.936, "rejected_segment_count": 0, "rejected_segments": [] } } }