{ "text": "OK 今天来讲一下千问 3.8 27B\n这个模型我最近发现了一个新框架\n这个框架叫 NInfer\n在特定的长上下文再加\n上 MTP 的场景里面啊官方和\n社区的测试速度甚至能达到两倍\n那我自己这张 4090 到底能快多少呢\n看到最后你就知道了\n上上个星期我不是\n刚部署好千问 3.8 27B 嘛那时候用 的呢是 llama.cpp\n也是为了优化卯足了劲儿\n十几个小时的矩阵测试\n就是为了得到最好的那一个参数\n啊然后终于搞满意了固定下来了\n200K 上下文\n七八十 tokens 每秒\n那 llama.cpp 在这个阶段呢\n对于我自己的硬件而言基本上是被优化到顶了\n但是呢就在前两天我发现全白测了\n就是在研究 VLLM 或者 SGLang 这些框架的时候呢\n我发现了一个叫做 NInfer 的东西\n看大家的评价是非常的\n疯狂什么直接暴涨两倍的速度四舍九入那将像\n四舍九入\n四舍五入那就相当于多给了我一张 4090 啊\n嗯这还不赶紧部署起来试试\n于是呢我就让 GPT-5.6 Sol 写 了一个详细的方案\n一整套啊包括部署包括测试的方案\n然后把整个 plan 丢给我服务器上的 DeepSeek V4 Flash 去干\nV4 Flash 一个人是吭哧吭哧干了\n五个小时包括配置包括\n对 NInfer 的各种参数进行矩阵测试\n之后再和 llama.cpp 进行矩阵对 比\n我就出了个门\n然后回来他就直接给我\n交付了一份能用的最佳配置和整个对比\n结果的报告\n这感觉太爽了就\n要知道就在四五个月前\n这种事情都是不可能的那时候我是逮着 GPT 对线\n手搓配置 vLLM\n整整搓了好几天才跑起来\n来啊那给不了解的小伙伴先解释一下所谓的 NInfer 其\n其实就是类似于 llama.cpp 或者 vLLM\n这种框架但它和那些都不一样啊它\n不是那些的派生而是一个从零写的推理引擎\n它特殊的地方就在于其\n他的框架是要需要考虑\n不同的模型不同的硬件\n怎么兼容而 NInfer 特别的地方就在 于\n它把模型和硬件这一条路径给固定死了\n原版主要盯的就是 RTX 5090 这个 显卡\n模型呢主要就是千问 3.8 这个家族\n后面有大神把它移植到了 3090 上面后 来呢又有人\n根据 3090 做了 4090 的优化那 我这次用的就是这个 4090 fork\n所以基本上就是把\n千问 3.8 27B 这一个模型和 RTX 4090 这一张显卡\n这两个固定下来往死里优化\n它不需要像 llama.cpp 那样去\n考虑\n他拿到一个 GGUF 之后要去看\n一下它的架构是什么再解析\n然后再根据设备去选\n一个后端再去运行这一\n整套啊 llama.cpp 是为了兼容很 多模型和设备嘛\n所以不可避免的会有很多妥协还有一些小 bug\n但 NInfer 就是这样一个极端优化极 端固定的概念就\n一旦这一条路跑通了就\n优化到极致之后就连参数它都基本上不变了\n那我比喻一下啊我又要来了就是一个比喻\nllama.cpp 它就像一辆很优秀的\n丰田 S U V 你把它开去哪里\n都可以啊全世界都有零件你去哪儿都能维修\n雪地沙地山路上都可以啊速度呢也还行\n那 vLLM 呢可以说它像一个\n地铁系统啊它在乎的不是我一个乘客能跑多快\n它在乎的是我每个小时\n的总吞吐量可以把多少人给并行的运起来\n它最经典的技术呢就是 PagedAttention\n把 KV 缓存像虚拟内存一样分页管理啊\n减少显存碎片\n再配合一个\ncontinuous batching 就是\n连续批处理\n非常适合高并发的服务\n它现在呢也有\nprefix caching 啊就是可以 把以前算过的前缀直接复用\n然后呢就是 SGLang 这个是特别\n强调复杂工作流里面的\n缓存复用比如说他会去\n考虑哪些货物可以共享\n路线啊哪些可以缓存哪\n些可以有共同的 prefix 怎么复用这 些状态\n所以呢它可能就像一个智能物流中心\n它很有代表性的技\n术叫做 RadixAttention 粗 略理解一下它\n就是把大量的请求的 prefix 组成一 棵树\n然后呢重复的 prefix 它就不重复计 算了\n所以它对于 agent 的 loop 来 讲有很\n多东西都是重复的嘛非常有效\n那那最后就是我这回尝试的 NInfer\n它呢就是一辆 F1 赛车\n你赛道就是这一个赛道发动机就这一台发动机\n轮胎我都是这一组轮胎\n变速箱悬挂什么的都不变就\n就连天气啊我都给它固定住了\n所有的都是已知条件然后就往死里去刷圈\n这样极端优化的\n结果呢就是我这样\n一张 4090 24GB 的卡 NInfer 可以\n把整个 256K 的模型原生上下文啊全都 塞进去\n而且生成速度可以达到平均一百 tokens 每秒\n关于速度啊这些测试我后面会再展开讲一下\n啊其实千问 3.8 27B 这个模型它本 身也比较特殊啊它\n它不是每一层都有巨大\n的 KV 缓存它是一个混合架构\n所以才能在 24GB 级别的卡上\n把 256K 的上下文给开满就放在以前想 都不敢想\n但这个缓存呢并不是就\n说原始精度而是量化后的 Q4 呃但又不是 普通的 K4V4\n而是 RK4V4-E8\n那这个参数具体是什么\n意思呢我就不展开讲了\n讲了你也听不懂啊主要是因为呢我也看不懂\n啊反正就是一种比普通\n压缩精度要更好一点的\n旋转跳跃的压缩方式它前面那个二代表的是 rotate 嘛\n啊\n所以可以在精度损失很\n小的前提下可以比 Q8 或者 FP16 的 KV 缓存能更省显存\n就它主要的目的其实不\n是提速啊它反而是比 Q8 要慢一点点\n它主要是为了用尽可能\n小的质量代价把 KV 压的更小\n好把完整的 256K 上下文全都塞进 24GB 里面\n所以总结来说啊\n别的框架是要模型和\n硬件向框架妥协以获取兼容\n但 NInfer 呢是把自己改成了 千问 3.8 的那个形状\n特别是 4090 这个 fork 只为 千问 3.8 27B 一个人服务\n这是什么样的硬件榨汁儿行为啊\n所以如果你有对应的九\n零系列的硬件真的可以去尝试一下\n呃我都我想象了一下这个框架用来跑 4090 48GB 哦\n那那应该是一种天下无双的体验\n果如果有优化好的话啊\n他我不知道他有没有单独为了 48GB 版 本优化\n哎如果有大佬试过的话可以来评论区炫耀一下\n让我长长见识\n那 NInfer 整个部署好之后呢我\n首先让它去做了一个小\n任务去修改我的 A I 本地控制台\n让他把自己的 profile 给加进去\n我是做了这样一个网站\n部署在我自己的服务器\n上这样我可以非常简单\n的去切换各种服务各种\n模型或者就监控大模型的流量\n那给我的第一个感受啊\n就是它首 token 变慢了就 llama.cpp 里面\n大概是五秒钟左右就会\n出首 token 它要十二三秒但这\n里得说清楚啊我这里拿\n的是实际 Agent 跑的时候第一\n轮冷请求看到的 TTFT 也就是首 token 延迟\n如果我们真的去做那个\nbenchmark 去测试的话呢其实纯 prefill 的吞吐没有慢那么两倍啊\n大概就是落后个两三成左右\n那主要是这种冷启动的\n情况下一次性灌一大堆\n上下文进去的场景它是要慢一点\n后续呢就是缓存命中越来越高之后会越用越低\n一开始大概十几秒后面就是能达到几秒\n然后呢就是速度啊用下来之后\n我发现一些小任务能达到 130 tokens/s 左右\n就我按照以前的体感啊给他派一个任务\n然后我自己切去干别的事情了\n结果没过一会儿他跟我\n说干好了我就有点不习惯\n就跟我以前的那个体感不太一样\n然后呢长链开发现在是\n稳定在一百多一点 tokens 啊这\n也是一个非常不错的成绩\n呃因为在上下文稍微长\n一点之后呢 llama.cpp 你是即便 开\nMTP 它的速度大概其实也就五六十个 tokens 每秒了\n好一点的情况也只有七十\n所以基本上就是一个一点五倍的速度提升\n然后在这个速度之外还\n有一点非常牛逼的就是\n呢在 NInfer 框架里面把上下文开到 顶\n也就是 256K 上下文全都塞进\n去的情况下还剩 1.3GB 左右的显存也 就是说啊\n我们可以把视觉模型也同时给开起来\nllama.cpp 呢在我个人的这台机器\n上对最多我是开到 208K 左右的上下文\n然后再开一个视觉基本\n上就是你看到那根红线\n顶着跑了啊跑着跑着就就不太稳定啊\n就是命悬一线那种感觉\n就每次发一个请求我菊花都紧紧的\n生怕它直接爆了\n啊但开 NInfer 之后呢我就直接全上 下文\n然后视觉模型同时开起来\n呃所以总结下来孩子吃\n了 NInfer 粉之后呢显存占用更\n少了上下文更长了速度更快了这有三个优点\n唯一的缺点就是冷 prefill 的速度 要降低百分之二三十\n再然后就是关于功耗啊这个非常的重要啊\n因为我这台机器 decode 的时候\n整机墙上功耗最高是会到五百三四十瓦\n但是 prefill 的阶段呢就没有那么 高\n特别是当缓存命中的时\n候有时候大概是两三百瓦这样\n然啊当然这是我说这套\nNInfer 再加上 4090 的这个配 置的实测\n并不是说所有的或者你\n的机器也是这样你自己也可以实测一下\n你可以看我这个墙上功耗的实测就是用 llama.cpp 那几天\ntoken 总量这几天用的都是差不多的都 是一个亿左右\nllama.cpp 还要再少一点啊但是呢 一天下来墙上功耗 llama.cpp 是\n六千瓦到七千瓦之间就\n奔着七我感觉它有时候都要破七了\n但换成 NInfer 之后呢我这两天\n大概就是四点五到五千瓦时之间\n所以这个就光电费的区\n别按照我现在的电价来\n粗算一下年化下来是能省一千块钱\n就换了这么个框架而已\n啊当然目前这个样本还比较少没有办法确定说\n但就我体感而言它的那个发热的量还\n还有风扇呼啸的时间都是要更少的\n最后啊总结一下我的体\n验最开始我用的是 llama.cpp 啊 部署完之后用了一阵子\n跟在线 API 比起来就还是\n有一点点慢就有一点点\n比三点六要好很多了但是就还是\n有那么一点点便秘的感觉\n但换成 NInfer 之后这个一百 token 每\n秒的速度啊这就相当可用了\n呀呃因为之\n前的 DeepSeek V4 Flash 呃我\n用 Open Code Go 的那个 plan\n它给到的速度也就是一百到一百五之间\n当然你要跟最新的\nDeepSeek V4.1 Flash 这种复写式生成的速度那肯定是没法比的\n啊但反正啊总之这个优\n化下来我个人是非常非\n常满意的我现在是直接\n把整个生产都切换到了 NInfer 上面\n而且用起来非常稳定就\n没有触发任何问题也没有觉得质量有下降\n反而啊至少在我现在\n这套配置里面 NInfer 是比 llama.cpp 更省心的\n因为它很多容量或者设\n置在启动的时候就已经\n固定和校验好了超过预算就直接起不来\n而不是跑一半突然给你来个 OOM\n这个之前我在 llama.cpp 里面经 历过就是在长 session 里面会\n会出现一些显存和 slot 管理上的问题\n因为 llama 如果你默认不去设置 KV slot 你\n你不把它固定在一的话它现在默认是四\n然后用的是 unified KV\n这个功能概念上很好我很喜欢\n但是那个我目前的配置\n好像有点硬件啊可能有点水土不服\n啊反正就是上下文越长越不稳定最后会崩掉\n那么目前啊本地部署对我个人来说\n我觉得还差最后一个问题就是并发\n目前呢我个人是不希望\n损失上下文去搞并发的啊\n因为就 4090 单张卡而言我现\n在把上下文开到 256K 个人是\n非常舒爽的一个状态特\n别是在 agent 搞长链开发的时\n候呢有的人会觉得说一\n百二十八 k 上下文够用但\n我个人啊就是觉得有点\n不够我甚至希望再长一点\n然后在保持这个上下文\n的前提下有办法啊把并发给做起来那是更好\n个啊希望有什么框架可\n以这个打破一下物理定律\n可以让 4090 二十四 G 搞一个啊\n四个左右吧我也不贪心四个长上下文并发啊\n上下文在 256K 左右生成速度都维持在 一百左右\n的哼宛如做梦\n啊但也不是不可能的我\n不是痴人说梦你看现在软件的优化是真的\n翻天覆地前几个月同样\n的硬件更弱的模型就\n千问 3.6 那个版本嘛我\n是开了 MTP 都只有\n40–50 tokens/s 那才过了这 么几个月\n更强的三点八模型更快\n的速度啊原生四五十 MTP 直接破百\n搞得我上个星期跑了十\n几个小时的矩阵测试啊好不容易定下来的参数\n到今天已经没卵用了\n了啊但没关系啊毕竟我\n自己就是非常喜欢折腾这种事情嘛啊多\n多折腾一下也有更多的话题可以跟你们聊\n有空欢迎再来听我唠嗑下期再见", "language": "zh", "duration": 679.0, "segments": [ { "id": 0, "start": 0.166, "end": 2.335, "text": "OK 今天来讲一下千问 3.8 27B", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 1, "start": 2.335, "end": 4.771, "text": "这个模型我最近发现了一个新框架", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 2, "start": 4.938, "end": 6.139, "text": "这个框架叫 NInfer", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 3, "start": 6.206, "end": 7.907, "text": "在特定的长上下文再加", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 4, "start": 7.907, "end": 9.743, "text": "上 MTP 的场景里面啊官方和", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 5, "start": 9.743, "end": 12.479, "text": "社区的测试速度甚至能达到两倍", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 6, "start": 12.712, "end": 15.081, "text": "那我自己这张 4090 到底能快多少呢", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 7, "start": 15.281, "end": 16.249, "text": "看到最后你就知道了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 8, "start": 16.416, "end": 17.117, "text": "上上个星期我不是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 9, "start": 17.117, "end": 21.154, "text": "刚部署好千问 3.8 27B 嘛那时候用 的呢是 llama.cpp", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 10, "start": 21.287, "end": 22.889, "text": "也是为了优化卯足了劲儿", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 11, "start": 22.922, "end": 24.624, "text": "十几个小时的矩阵测试", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 12, "start": 24.724, "end": 27.26, "text": "就是为了得到最好的那一个参数", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 13, "start": 27.36, "end": 29.396, "text": "啊然后终于搞满意了固定下来了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 14, "start": 29.429, "end": 30.33, "text": "200K 上下文", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 15, "start": 30.497, "end": 31.664, "text": "七八十 tokens 每秒", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 16, "start": 31.765, "end": 33.4, "text": "那 llama.cpp 在这个阶段呢", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 17, "start": 33.4, "end": 36.202, "text": "对于我自己的硬件而言基本上是被优化到顶了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 18, "start": 36.469, "end": 38.905, "text": "但是呢就在前两天我发现全白测了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 19, "start": 39.005, "end": 41.808, "text": "就是在研究 VLLM 或者 SGLang 这些框架的时候呢", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 20, "start": 41.808, "end": 43.51, "text": "我发现了一个叫做 NInfer 的东西", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 21, "start": 43.576, "end": 44.944, "text": "看大家的评价是非常的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 22, "start": 44.944, "end": 48.581, "text": "疯狂什么直接暴涨两倍的速度四舍九入那将像", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 23, "start": 48.581, "end": 49.349, "text": "四舍九入", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 24, "start": 49.682, "end": 52.585, "text": "四舍五入那就相当于多给了我一张 4090 啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 25, "start": 52.819, "end": 54.454, "text": "嗯这还不赶紧部署起来试试", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 26, "start": 54.721, "end": 57.557, "text": "于是呢我就让 GPT-5.6 Sol 写 了一个详细的方案", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 27, "start": 57.59, "end": 59.859, "text": "一整套啊包括部署包括测试的方案", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 28, "start": 60.126, "end": 63.129, "text": "然后把整个 plan 丢给我服务器上的 DeepSeek V4 Flash 去干", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 29, "start": 63.329, "end": 65.498, "text": "V4 Flash 一个人是吭哧吭哧干了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 30, "start": 65.498, "end": 67.267, "text": "五个小时包括配置包括", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 31, "start": 67.267, "end": 69.769, "text": "对 NInfer 的各种参数进行矩阵测试", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 32, "start": 69.836, "end": 71.538, "text": "之后再和 llama.cpp 进行矩阵对 比", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 33, "start": 71.604, "end": 72.305, "text": "我就出了个门", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 34, "start": 72.405, "end": 73.873, "text": "然后回来他就直接给我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 35, "start": 73.873, "end": 76.509, "text": "交付了一份能用的最佳配置和整个对比", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 36, "start": 76.509, "end": 77.177, "text": "结果的报告", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 37, "start": 77.31, "end": 78.711, "text": "这感觉太爽了就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 38, "start": 78.711, "end": 80.113, "text": "要知道就在四五个月前", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 39, "start": 80.113, "end": 83.85, "text": "这种事情都是不可能的那时候我是逮着 GPT 对线", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 40, "start": 83.85, "end": 85.485, "text": "手搓配置 vLLM", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 41, "start": 85.518, "end": 87.22, "text": "整整搓了好几天才跑起来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 42, "start": 87.353, "end": 90.523, "text": "来啊那给不了解的小伙伴先解释一下所谓的 NInfer 其", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 43, "start": 90.523, "end": 92.926, "text": "其实就是类似于 llama.cpp 或者 vLLM", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 44, "start": 92.926, "end": 95.595, "text": "这种框架但它和那些都不一样啊它", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 45, "start": 95.595, "end": 98.731, "text": "不是那些的派生而是一个从零写的推理引擎", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 46, "start": 98.865, "end": 100.3, "text": "它特殊的地方就在于其", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 47, "start": 100.3, "end": 102.001, "text": "他的框架是要需要考虑", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 48, "start": 102.001, "end": 103.736, "text": "不同的模型不同的硬件", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 49, "start": 103.736, "end": 106.339, "text": "怎么兼容而 NInfer 特别的地方就在 于", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 50, "start": 106.439, "end": 109.976, "text": "它把模型和硬件这一条路径给固定死了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 51, "start": 110.176, "end": 113.146, "text": "原版主要盯的就是 RTX 5090 这个 显卡", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 52, "start": 113.313, "end": 115.682, "text": "模型呢主要就是千问 3.8 这个家族", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 53, "start": 115.815, "end": 118.818, "text": "后面有大神把它移植到了 3090 上面后 来呢又有人", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 54, "start": 118.852, "end": 123.523, "text": "根据 3090 做了 4090 的优化那 我这次用的就是这个 4090 fork", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 55, "start": 123.59, "end": 124.591, "text": "所以基本上就是把", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 56, "start": 124.624, "end": 129.162, "text": "千问 3.8 27B 这一个模型和 RTX 4090 这一张显卡", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 57, "start": 129.195, "end": 131.231, "text": "这两个固定下来往死里优化", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 58, "start": 131.331, "end": 132.999, "text": "它不需要像 llama.cpp 那样去", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 59, "start": 132.999, "end": 133.266, "text": "考虑", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 60, "start": 133.333, "end": 135.101, "text": "他拿到一个 GGUF 之后要去看", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 61, "start": 135.101, "end": 137.003, "text": "一下它的架构是什么再解析", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 62, "start": 137.003, "end": 138.705, "text": "然后再根据设备去选", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 63, "start": 138.705, "end": 140.607, "text": "一个后端再去运行这一", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 64, "start": 140.607, "end": 144.01, "text": "整套啊 llama.cpp 是为了兼容很 多模型和设备嘛", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 65, "start": 144.31, "end": 147.213, "text": "所以不可避免的会有很多妥协还有一些小 bug", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 66, "start": 147.413, "end": 151.117, "text": "但 NInfer 就是这样一个极端优化极 端固定的概念就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 67, "start": 151.117, "end": 152.685, "text": "一旦这一条路跑通了就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 68, "start": 152.685, "end": 155.788, "text": "优化到极致之后就连参数它都基本上不变了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 69, "start": 155.955, "end": 158.858, "text": "那我比喻一下啊我又要来了就是一个比喻", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 70, "start": 159.025, "end": 161.127, "text": "llama.cpp 它就像一辆很优秀的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 71, "start": 161.127, "end": 163.196, "text": "丰田 S U V 你把它开去哪里", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 72, "start": 163.196, "end": 166.599, "text": "都可以啊全世界都有零件你去哪儿都能维修", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 73, "start": 166.866, "end": 170.336, "text": "雪地沙地山路上都可以啊速度呢也还行", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 74, "start": 170.67, "end": 172.405, "text": "那 vLLM 呢可以说它像一个", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 75, "start": 172.405, "end": 176.042, "text": "地铁系统啊它在乎的不是我一个乘客能跑多快", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 76, "start": 176.209, "end": 177.977, "text": "它在乎的是我每个小时", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 77, "start": 177.977, "end": 181.281, "text": "的总吞吐量可以把多少人给并行的运起来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 78, "start": 181.447, "end": 183.383, "text": "它最经典的技术呢就是 PagedAttention", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 79, "start": 183.65, "end": 186.419, "text": "把 KV 缓存像虚拟内存一样分页管理啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 80, "start": 186.419, "end": 187.387, "text": "减少显存碎片", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 81, "start": 187.52, "end": 188.221, "text": "再配合一个", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 82, "start": 188.354, "end": 189.589, "text": "continuous batching 就是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 83, "start": 189.589, "end": 190.523, "text": "连续批处理", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 84, "start": 190.557, "end": 192.125, "text": "非常适合高并发的服务", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 85, "start": 192.492, "end": 193.193, "text": "它现在呢也有", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 86, "start": 193.193, "end": 196.596, "text": "prefix caching 啊就是可以 把以前算过的前缀直接复用", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 87, "start": 196.896, "end": 198.798, "text": "然后呢就是 SGLang 这个是特别", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 88, "start": 198.798, "end": 200.333, "text": "强调复杂工作流里面的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 89, "start": 200.333, "end": 202.068, "text": "缓存复用比如说他会去", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 90, "start": 202.068, "end": 203.836, "text": "考虑哪些货物可以共享", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 91, "start": 203.836, "end": 205.672, "text": "路线啊哪些可以缓存哪", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 92, "start": 205.672, "end": 208.508, "text": "些可以有共同的 prefix 怎么复用这 些状态", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 93, "start": 208.708, "end": 211.544, "text": "所以呢它可能就像一个智能物流中心", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 94, "start": 211.744, "end": 212.679, "text": "它很有代表性的技", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 95, "start": 212.679, "end": 215.281, "text": "术叫做 RadixAttention 粗 略理解一下它", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 96, "start": 215.281, "end": 218.284, "text": "就是把大量的请求的 prefix 组成一 棵树", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 97, "start": 218.551, "end": 220.987, "text": "然后呢重复的 prefix 它就不重复计 算了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 98, "start": 221.354, "end": 223.289, "text": "所以它对于 agent 的 loop 来 讲有很", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 99, "start": 223.289, "end": 225.158, "text": "多东西都是重复的嘛非常有效", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 100, "start": 225.391, "end": 227.493, "text": "那那最后就是我这回尝试的 NInfer", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 101, "start": 227.627, "end": 229.262, "text": "它呢就是一辆 F1 赛车", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 102, "start": 229.395, "end": 232.532, "text": "你赛道就是这一个赛道发动机就这一台发动机", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 103, "start": 232.699, "end": 234.334, "text": "轮胎我都是这一组轮胎", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 104, "start": 234.434, "end": 236.135, "text": "变速箱悬挂什么的都不变就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 105, "start": 236.135, "end": 237.971, "text": "就连天气啊我都给它固定住了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 106, "start": 238.037, "end": 241.04, "text": "所有的都是已知条件然后就往死里去刷圈", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 107, "start": 241.407, "end": 242.442, "text": "这样极端优化的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 108, "start": 242.442, "end": 243.576, "text": "结果呢就是我这样", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 109, "start": 243.576, "end": 246.145, "text": "一张 4090 24GB 的卡 NInfer 可以", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 110, "start": 246.145, "end": 249.916, "text": "把整个 256K 的模型原生上下文啊全都 塞进去", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 111, "start": 250.049, "end": 252.585, "text": "而且生成速度可以达到平均一百 tokens 每秒", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 112, "start": 252.719, "end": 254.988, "text": "关于速度啊这些测试我后面会再展开讲一下", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 113, "start": 255.388, "end": 258.591, "text": "啊其实千问 3.8 27B 这个模型它本 身也比较特殊啊它", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 114, "start": 258.591, "end": 260.093, "text": "它不是每一层都有巨大", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 115, "start": 260.093, "end": 262.028, "text": "的 KV 缓存它是一个混合架构", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 116, "start": 262.128, "end": 263.863, "text": "所以才能在 24GB 级别的卡上", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 117, "start": 263.896, "end": 267.533, "text": "把 256K 的上下文给开满就放在以前想 都不敢想", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 118, "start": 267.867, "end": 269.302, "text": "但这个缓存呢并不是就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 119, "start": 269.302, "end": 274.34, "text": "说原始精度而是量化后的 Q4 呃但又不是 普通的 K4V4", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 120, "start": 274.34, "end": 277.01, "text": "而是 RK4V4-E8", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 121, "start": 277.176, "end": 278.378, "text": "那这个参数具体是什么", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 122, "start": 278.378, "end": 279.579, "text": "意思呢我就不展开讲了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 123, "start": 279.579, "end": 282.515, "text": "讲了你也听不懂啊主要是因为呢我也看不懂", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 124, "start": 282.682, "end": 284.05, "text": "啊反正就是一种比普通", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 125, "start": 284.05, "end": 285.785, "text": "压缩精度要更好一点的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 126, "start": 285.818, "end": 289.455, "text": "旋转跳跃的压缩方式它前面那个二代表的是 rotate 嘛", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 127, "start": 290.023, "end": 290.056, "text": "啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 128, "start": 290.256, "end": 291.557, "text": "所以可以在精度损失很", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 129, "start": 291.557, "end": 296.062, "text": "小的前提下可以比 Q8 或者 FP16 的 KV 缓存能更省显存", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 130, "start": 296.396, "end": 297.53, "text": "就它主要的目的其实不", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 131, "start": 297.53, "end": 300.366, "text": "是提速啊它反而是比 Q8 要慢一点点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 132, "start": 300.466, "end": 301.834, "text": "它主要是为了用尽可能", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 133, "start": 301.834, "end": 304.103, "text": "小的质量代价把 KV 压的更小", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 134, "start": 304.37, "end": 307.84, "text": "好把完整的 256K 上下文全都塞进 24GB 里面", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 135, "start": 308.074, "end": 309.142, "text": "所以总结来说啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 136, "start": 309.275, "end": 311.077, "text": "别的框架是要模型和", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 137, "start": 311.11, "end": 313.88, "text": "硬件向框架妥协以获取兼容", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 138, "start": 314.047, "end": 317.283, "text": "但 NInfer 呢是把自己改成了 千问 3.8 的那个形状", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 139, "start": 317.317, "end": 321.688, "text": "特别是 4090 这个 fork 只为 千问 3.8 27B 一个人服务", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 140, "start": 321.854, "end": 324.123, "text": "这是什么样的硬件榨汁儿行为啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 141, "start": 324.324, "end": 325.491, "text": "所以如果你有对应的九", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 142, "start": 325.491, "end": 327.126, "text": "零系列的硬件真的可以去尝试一下", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 143, "start": 327.36, "end": 331.564, "text": "呃我都我想象了一下这个框架用来跑 4090 48GB 哦", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 144, "start": 331.731, "end": 333.9, "text": "那那应该是一种天下无双的体验", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 145, "start": 333.966, "end": 335.368, "text": "果如果有优化好的话啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 146, "start": 335.368, "end": 337.97, "text": "他我不知道他有没有单独为了 48GB 版 本优化", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 147, "start": 338.004, "end": 340.74, "text": "哎如果有大佬试过的话可以来评论区炫耀一下", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 148, "start": 341.007, "end": 342.075, "text": "让我长长见识", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 149, "start": 342.108, "end": 343.643, "text": "那 NInfer 整个部署好之后呢我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 150, "start": 343.643, "end": 344.811, "text": "首先让它去做了一个小", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 151, "start": 344.811, "end": 347.113, "text": "任务去修改我的 A I 本地控制台", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 152, "start": 347.28, "end": 349.148, "text": "让他把自己的 profile 给加进去", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 153, "start": 349.315, "end": 350.683, "text": "我是做了这样一个网站", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 154, "start": 350.683, "end": 351.784, "text": "部署在我自己的服务器", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 155, "start": 351.784, "end": 353.086, "text": "上这样我可以非常简单", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 156, "start": 353.086, "end": 354.687, "text": "的去切换各种服务各种", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 157, "start": 354.687, "end": 356.723, "text": "模型或者就监控大模型的流量", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 158, "start": 357.156, "end": 358.458, "text": "那给我的第一个感受啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 159, "start": 358.458, "end": 361.694, "text": "就是它首 token 变慢了就 llama.cpp 里面", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 160, "start": 361.694, "end": 363.463, "text": "大概是五秒钟左右就会", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 161, "start": 363.463, "end": 366.432, "text": "出首 token 它要十二三秒但这", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 162, "start": 366.432, "end": 367.533, "text": "里得说清楚啊我这里拿", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 163, "start": 367.533, "end": 369.535, "text": "的是实际 Agent 跑的时候第一", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 164, "start": 369.535, "end": 373.005, "text": "轮冷请求看到的 TTFT 也就是首 token 延迟", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 165, "start": 373.306, "end": 374.607, "text": "如果我们真的去做那个", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 166, "start": 374.607, "end": 379.112, "text": "benchmark 去测试的话呢其实纯 prefill 的吞吐没有慢那么两倍啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 167, "start": 379.145, "end": 380.947, "text": "大概就是落后个两三成左右", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 168, "start": 381.214, "end": 382.749, "text": "那主要是这种冷启动的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 169, "start": 382.749, "end": 384.417, "text": "情况下一次性灌一大堆", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 170, "start": 384.417, "end": 386.386, "text": "上下文进去的场景它是要慢一点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 171, "start": 386.452, "end": 389.522, "text": "后续呢就是缓存命中越来越高之后会越用越低", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 172, "start": 389.722, "end": 392.091, "text": "一开始大概十几秒后面就是能达到几秒", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 173, "start": 392.225, "end": 394.293, "text": "然后呢就是速度啊用下来之后", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 174, "start": 394.293, "end": 397.864, "text": "我发现一些小任务能达到 130 tokens/s 左右", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 175, "start": 398.03, "end": 400.066, "text": "就我按照以前的体感啊给他派一个任务", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 176, "start": 400.066, "end": 401.567, "text": "然后我自己切去干别的事情了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 177, "start": 401.701, "end": 402.702, "text": "结果没过一会儿他跟我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 178, "start": 402.702, "end": 404.47, "text": "说干好了我就有点不习惯", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 179, "start": 404.57, "end": 406.372, "text": "就跟我以前的那个体感不太一样", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 180, "start": 406.539, "end": 407.974, "text": "然后呢长链开发现在是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 181, "start": 407.974, "end": 409.976, "text": "稳定在一百多一点 tokens 啊这", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 182, "start": 409.976, "end": 411.31, "text": "也是一个非常不错的成绩", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 183, "start": 411.577, "end": 412.912, "text": "呃因为在上下文稍微长", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 184, "start": 412.912, "end": 415.448, "text": "一点之后呢 llama.cpp 你是即便 开", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 185, "start": 415.481, "end": 419.752, "text": "MTP 它的速度大概其实也就五六十个 tokens 每秒了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 186, "start": 419.786, "end": 421.32, "text": "好一点的情况也只有七十", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 187, "start": 421.587, "end": 424.19, "text": "所以基本上就是一个一点五倍的速度提升", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 188, "start": 424.357, "end": 425.491, "text": "然后在这个速度之外还", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 189, "start": 425.491, "end": 426.759, "text": "有一点非常牛逼的就是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 190, "start": 426.759, "end": 429.595, "text": "呢在 NInfer 框架里面把上下文开到 顶", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 191, "start": 429.929, "end": 431.864, "text": "也就是 256K 上下文全都塞进", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 192, "start": 431.864, "end": 435.802, "text": "去的情况下还剩 1.3GB 左右的显存也 就是说啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 193, "start": 435.802, "end": 437.97, "text": "我们可以把视觉模型也同时给开起来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 194, "start": 438.204, "end": 440.306, "text": "llama.cpp 呢在我个人的这台机器", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 195, "start": 440.306, "end": 443.409, "text": "上对最多我是开到 208K 左右的上下文", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 196, "start": 443.409, "end": 445.178, "text": "然后再开一个视觉基本", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 197, "start": 445.178, "end": 446.512, "text": "上就是你看到那根红线", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 198, "start": 446.512, "end": 449.215, "text": "顶着跑了啊跑着跑着就就不太稳定啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 199, "start": 449.282, "end": 450.85, "text": "就是命悬一线那种感觉", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 200, "start": 450.95, "end": 453.152, "text": "就每次发一个请求我菊花都紧紧的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 201, "start": 453.386, "end": 454.487, "text": "生怕它直接爆了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 202, "start": 454.554, "end": 456.856, "text": "啊但开 NInfer 之后呢我就直接全上 下文", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 203, "start": 456.956, "end": 458.424, "text": "然后视觉模型同时开起来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 204, "start": 458.624, "end": 460.126, "text": "呃所以总结下来孩子吃", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 205, "start": 460.126, "end": 461.894, "text": "了 NInfer 粉之后呢显存占用更", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 206, "start": 461.894, "end": 464.797, "text": "少了上下文更长了速度更快了这有三个优点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 207, "start": 464.964, "end": 468.501, "text": "唯一的缺点就是冷 prefill 的速度 要降低百分之二三十", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 208, "start": 468.901, "end": 471.771, "text": "再然后就是关于功耗啊这个非常的重要啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 209, "start": 471.771, "end": 473.639, "text": "因为我这台机器 decode 的时候", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 210, "start": 473.94, "end": 477.243, "text": "整机墙上功耗最高是会到五百三四十瓦", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 211, "start": 477.543, "end": 479.245, "text": "但是 prefill 的阶段呢就没有那么 高", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 212, "start": 479.478, "end": 480.847, "text": "特别是当缓存命中的时", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 213, "start": 480.847, "end": 482.648, "text": "候有时候大概是两三百瓦这样", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 214, "start": 482.915, "end": 484.35, "text": "然啊当然这是我说这套", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 215, "start": 484.35, "end": 487.186, "text": "NInfer 再加上 4090 的这个配 置的实测", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 216, "start": 487.353, "end": 488.654, "text": "并不是说所有的或者你", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 217, "start": 488.654, "end": 490.59, "text": "的机器也是这样你自己也可以实测一下", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 218, "start": 490.79, "end": 494.26, "text": "你可以看我这个墙上功耗的实测就是用 llama.cpp 那几天", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 219, "start": 494.36, "end": 497.263, "text": "token 总量这几天用的都是差不多的都 是一个亿左右", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 220, "start": 497.363, "end": 501.968, "text": "llama.cpp 还要再少一点啊但是呢 一天下来墙上功耗 llama.cpp 是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 221, "start": 502.201, "end": 503.636, "text": "六千瓦到七千瓦之间就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 222, "start": 503.636, "end": 506.405, "text": "奔着七我感觉它有时候都要破七了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 223, "start": 506.706, "end": 508.474, "text": "但换成 NInfer 之后呢我这两天", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 224, "start": 508.474, "end": 510.977, "text": "大概就是四点五到五千瓦时之间", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 225, "start": 511.11, "end": 512.712, "text": "所以这个就光电费的区", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 226, "start": 512.712, "end": 514.046, "text": "别按照我现在的电价来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 227, "start": 514.046, "end": 516.983, "text": "粗算一下年化下来是能省一千块钱", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 228, "start": 517.149, "end": 518.684, "text": "就换了这么个框架而已", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 229, "start": 519.051, "end": 521.654, "text": "啊当然目前这个样本还比较少没有办法确定说", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 230, "start": 521.754, "end": 524.323, "text": "但就我体感而言它的那个发热的量还", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 231, "start": 524.357, "end": 526.459, "text": "还有风扇呼啸的时间都是要更少的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 232, "start": 526.726, "end": 528.728, "text": "最后啊总结一下我的体", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 233, "start": 528.728, "end": 532.198, "text": "验最开始我用的是 llama.cpp 啊 部署完之后用了一阵子", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 234, "start": 532.265, "end": 534.533, "text": "跟在线 API 比起来就还是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 235, "start": 534.7, "end": 536.002, "text": "有一点点慢就有一点点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 236, "start": 536.168, "end": 538.604, "text": "比三点六要好很多了但是就还是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 237, "start": 538.771, "end": 540.106, "text": "有那么一点点便秘的感觉", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 238, "start": 540.239, "end": 542.308, "text": "但换成 NInfer 之后这个一百 token 每", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 239, "start": 542.308, "end": 544.41, "text": "秒的速度啊这就相当可用了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 240, "start": 544.61, "end": 544.977, "text": "呀呃因为之", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 241, "start": 544.977, "end": 546.812, "text": "前的 DeepSeek V4 Flash 呃我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 242, "start": 546.812, "end": 548.447, "text": "用 Open Code Go 的那个 plan", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 243, "start": 548.748, "end": 551.017, "text": "它给到的速度也就是一百到一百五之间", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 244, "start": 551.284, "end": 552.485, "text": "当然你要跟最新的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 245, "start": 552.485, "end": 557.39, "text": "DeepSeek V4.1 Flash 这种复写式生成的速度那肯定是没法比的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 246, "start": 557.657, "end": 559.325, "text": "啊但反正啊总之这个优", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 247, "start": 559.325, "end": 560.76, "text": "化下来我个人是非常非", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 248, "start": 560.76, "end": 562.061, "text": "常满意的我现在是直接", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 249, "start": 562.061, "end": 563.996, "text": "把整个生产都切换到了 NInfer 上面", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 250, "start": 564.363, "end": 565.398, "text": "而且用起来非常稳定就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 251, "start": 565.398, "end": 568.1, "text": "没有触发任何问题也没有觉得质量有下降", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 252, "start": 568.301, "end": 569.902, "text": "反而啊至少在我现在", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 253, "start": 569.902, "end": 572.738, "text": "这套配置里面 NInfer 是比 llama.cpp 更省心的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 254, "start": 573.105, "end": 574.473, "text": "因为它很多容量或者设", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 255, "start": 574.473, "end": 575.942, "text": "置在启动的时候就已经", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 256, "start": 575.975, "end": 578.744, "text": "固定和校验好了超过预算就直接起不来", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 257, "start": 578.811, "end": 580.746, "text": "而不是跑一半突然给你来个 OOM", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 258, "start": 581.08, "end": 584.483, "text": "这个之前我在 llama.cpp 里面经 历过就是在长 session 里面会", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 259, "start": 584.483, "end": 586.752, "text": "会出现一些显存和 slot 管理上的问题", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 260, "start": 586.852, "end": 589.822, "text": "因为 llama 如果你默认不去设置 KV slot 你", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 261, "start": 589.822, "end": 592.525, "text": "你不把它固定在一的话它现在默认是四", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 262, "start": 592.525, "end": 594.226, "text": "然后用的是 unified KV", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 263, "start": 594.293, "end": 596.562, "text": "这个功能概念上很好我很喜欢", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 264, "start": 596.696, "end": 598.064, "text": "但是那个我目前的配置", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 265, "start": 598.064, "end": 600.633, "text": "好像有点硬件啊可能有点水土不服", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 266, "start": 600.9, "end": 603.269, "text": "啊反正就是上下文越长越不稳定最后会崩掉", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 267, "start": 603.569, "end": 606.372, "text": "那么目前啊本地部署对我个人来说", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 268, "start": 606.372, "end": 608.808, "text": "我觉得还差最后一个问题就是并发", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 269, "start": 609.008, "end": 610.543, "text": "目前呢我个人是不希望", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 270, "start": 610.543, "end": 612.378, "text": "损失上下文去搞并发的啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 271, "start": 612.378, "end": 614.447, "text": "因为就 4090 单张卡而言我现", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 272, "start": 614.447, "end": 617.016, "text": "在把上下文开到 256K 个人是", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 273, "start": 617.016, "end": 619.118, "text": "非常舒爽的一个状态特", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 274, "start": 619.118, "end": 620.72, "text": "别是在 agent 搞长链开发的时", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 275, "start": 620.72, "end": 622.722, "text": "候呢有的人会觉得说一", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 276, "start": 622.722, "end": 624.49, "text": "百二十八 k 上下文够用但", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 277, "start": 624.49, "end": 625.925, "text": "我个人啊就是觉得有点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 278, "start": 625.925, "end": 627.293, "text": "不够我甚至希望再长一点", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 279, "start": 627.727, "end": 629.095, "text": "然后在保持这个上下文", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 280, "start": 629.095, "end": 632.231, "text": "的前提下有办法啊把并发给做起来那是更好", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 281, "start": 632.565, "end": 633.933, "text": "个啊希望有什么框架可", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 282, "start": 633.933, "end": 635.468, "text": "以这个打破一下物理定律", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 283, "start": 635.735, "end": 638.304, "text": "可以让 4090 二十四 G 搞一个啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 284, "start": 638.304, "end": 641.707, "text": "四个左右吧我也不贪心四个长上下文并发啊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 285, "start": 641.774, "end": 645.011, "text": "上下文在 256K 左右生成速度都维持在 一百左右", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 286, "start": 645.344, "end": 646.178, "text": "的哼宛如做梦", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 287, "start": 646.479, "end": 647.947, "text": "啊但也不是不可能的我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 288, "start": 648.014, "end": 651.217, "text": "不是痴人说梦你看现在软件的优化是真的", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 289, "start": 651.217, "end": 653.119, "text": "翻天覆地前几个月同样", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 290, "start": 653.119, "end": 654.553, "text": "的硬件更弱的模型就", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 291, "start": 654.553, "end": 656.322, "text": "千问 3.6 那个版本嘛我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 292, "start": 656.322, "end": 658.157, "text": "是开了 MTP 都只有", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 293, "start": 658.157, "end": 660.693, "text": "40–50 tokens/s 那才过了这 么几个月", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 294, "start": 660.96, "end": 662.728, "text": "更强的三点八模型更快", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 295, "start": 662.728, "end": 665.297, "text": "的速度啊原生四五十 MTP 直接破百", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 296, "start": 665.431, "end": 666.799, "text": "搞得我上个星期跑了十", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 297, "start": 666.799, "end": 669.568, "text": "几个小时的矩阵测试啊好不容易定下来的参数", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 298, "start": 669.635, "end": 670.87, "text": "到今天已经没卵用了", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 299, "start": 671.07, "end": 672.071, "text": "了啊但没关系啊毕竟我", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 300, "start": 672.071, "end": 674.206, "text": "自己就是非常喜欢折腾这种事情嘛啊多", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 301, "start": 674.273, "end": 676.375, "text": "多折腾一下也有更多的话题可以跟你们聊", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 302, "start": 676.542, "end": 678.244, "text": "有空欢迎再来听我唠嗑下期再见", "source": "video_source_subtitle", "authority": "video_source_subtitle" } ], "transcript_selection": { "schema": "videonote.transcript_selection.v1", "status": "SELECTED", "requested_language": "zh", "duration": 679.0, "reference_path": "/Users/sagawa/AI_WORK/video_notes/url/20260918_171303__NInfer_千问3.8_27b_1.5倍速_256K上下文_单张4090_24GB/reference_subtitle.vtt", "reference_kind": "manual", "reference_language": "zh", "selected_source": "video_source_subtitle", "source_caption_kind": "manual", "source_caption_processing": "verbatim_direct", "youtube_overlap_clamps": [], "general_high_confidence_repairs": [], "decision_reason": "complete_source_subtitle_fast_path_no_whisper", "whisper_rejected": false, "source_profile": { "segment_count": 303, "duration_coverage": 0.954468 } } }