{"text": "千萬3.827B就是開源模型的Deep Seek時刻。它性能好、體積小、個人也能部署,實現很不錯的效果。最近幾天我研究了各種本地部署的方案,過程都有發在知識星球NewType裡面。那本期視頻我來做一個總結。咱們從一個基礎問題開始。在本地跑開源大模型需要什麼樣的配置?重點看兩個指標。第一,容量。也就是先看統一內存或者GPU顯存的容量。模型運行的時候,需要把權重裝進GPU顯存或者統一內存。如果裝不下,要麼無法運行,要麼就只能把部分的權重放進系統內存,通過CPU或者PCIe讀取。那這麼一來,速度就會大幅下降。用大白話來說就是,內存或者顯存容量決定了你能跑多大尺寸、多高精度的模型。舉個例子,一個27B的模型如果使用BF16的精度,那麼光是權重就需要大約50GB。量化到8bit之後,可以降到30GB左右。而量化到4bit之後,只需要17-20GB。除了權重之外,系統本身推理框架上下紋,它們都需要占有內存。所以4bit的模型,你光有17GB還不夠。大概得要20GB才能勉強能跑。第二,帶框。把模型裝進內存之後,第二個關鍵指標就是內存帶框。容量決定了你能不能跑,而帶框決定模型生成有多快。注意,內存容量大,並不意味著帶框就高。比如英偉達的DGX Spark,它有128GB,比RTX Pro 6000的就是6GB還大。看起來很大很划算對吧?但是它的內存帶框只有273GB,遠小於6000的1792。這麼低的帶框意味著,DGX Spark雖然能裝下很大的模型,但是跑大型Dance模型的時候,生成速度不會太快。它更適合MOE模型,因為MOE的總參數雖然很大,但每次生成Token的時候,它只激活其中的一小部分。你看,我們星球裡就有一個小夥伴,買了兩台,用來跑Deep Seek V4 Flash,速度有30多,夠用了。容量和帶框是理解本地推理最重要的兩個指標,但它們不是全部。你實際的性能還會受到GPU的數量力、模型架構和推理框架的影響。但是對於普通人來說,先抓住這兩個核心指標完全沒有問題。OK,理解了原理之後,那選擇就容易多了。咱們有三個選擇。第一,Mac。蘋果芯片用的是統一記憶,也就是CPU和GPU共享一個記憶池。因此,買Mac的話,可以用相對較低的價格獲得更高的記憶。但是,它的不足之處在於,帶框沒有那麼高,只有5090和6000的一半甚至更低。所以,即使Mac擁有同樣甚至更多的記憶,在運行Dance模型的時候,速度不會太快。而這個是需要咱們取捨的地方。第二,RTX5090。5090擁有3.4G顯存和1792GB每秒的記憶體帶框,非常適合千萬3.827B這樣的模型。所以,如果你的目標只是本地運行千萬3.827B的話,那一張5090通常會比Mac Studio更加合適。如果需要多Agent并發的話,可以上兩張5090。注意啊,5090沒有NV-Link。你非要跨卡通信的話,要走PCIE,效率不會像一塊原生的64G顯卡那麼高。所以,插兩張卡的目的,是讓每張卡都運行一個完整的模型。第一張卡用來處理一部分的Agent。第二張卡處理另一部分的Agent。它特別適合短或者中上下文,高并發多Agent任務的情況。5090雖然夠快,但它容量只有32G。你跑27B的模型沒有問題。如果要運行70B的模型超長上下文,或者同時常駐多個模型,那這樣的顯存就不夠了。這是它的不足之處。第三,RTX PRO 6000。這是最完整的本地AI方案。這張卡擁有96G顯存和1792GB每秒的帶寬。同時具備大容量和高帶寬。意味著它特別適合長上下文多模型常駐和生產及Agent服務。而且RTX PRO 6000還支持ECC和專業驅動,更適合長期運行。但是,它有一個很大的不足。太貴了。你去收下價格就知道了。剛才我介紹的三種方案,價格跨度非常大。從幾萬到十幾二十萬都有。所以,沒有一種硬件在所有維度都最好。你只能做取捨。大家目前用什麼樣的配置跑本地模型,效果如何?歡迎在評論區留言。OK,以上就是本期內容。想了解AI想奪回個體主權,想找到自從盜合的人就來我們Newtype社群。那咱們下期見。請問一下,請問一下。請問一下,請問一下。請問一下,請問一下。請問一下,請問一下。請問一下,請問一下。請問一下,請問一下。請問一下,請問一下。", "segments": [{"id": 0, "seek": 0, "start": 0.0, "end": 9.6, "text": "千萬3.827B就是開源模型的Deep Seek時刻。它性能好、體積小、個人也能部署,實現很不錯的效果。", "tokens": [50365, 20787, 14099, 18, 13, 23, 10076, 33, 5620, 8949, 47402, 41908, 39823, 1546, 11089, 595, 1100, 916, 6611, 45500, 1543, 11284, 21686, 8225, 2131, 1231, 23987, 13740, 235, 7322, 1231, 40104, 6404, 8225, 13470, 16469, 110, 11, 10376, 9581, 4563, 30967, 1546, 43076, 9319, 1543, 50845], "temperature": 0, "avg_logprob": -0.18840591943086082, "compression_ratio": 1.165625, "no_speech_prob": 9.936990466585804e-12}, {"id": 1, "seek": 0, "start": 9.6, "end": 18.6, "text": "最近幾天我研究了各種本地部署的方案,過程都有發在知識星球NewType裡面。那本期視頻我來做一個總結。", "tokens": [50845, 46777, 23575, 6135, 1654, 23230, 242, 44704, 2289, 17516, 16516, 8802, 10928, 13470, 16469, 110, 1546, 9249, 28899, 11, 8816, 29649, 48121, 14637, 3581, 6498, 43143, 20682, 28533, 18278, 22464, 494, 32399, 1543, 4184, 8802, 16786, 27333, 29940, 1654, 3763, 10907, 8990, 26575, 17144, 1543, 51295], "temperature": 0, "avg_logprob": -0.18840591943086082, "compression_ratio": 1.165625, "no_speech_prob": 9.936990466585804e-12}, {"id": 2, "seek": 0, "start": 18.6, "end": 24.6, "text": "咱們從一個基礎問題開始。在本地跑開源大模型需要什麼樣的配置?", "tokens": [51295, 8975, 109, 4623, 21068, 8990, 26008, 14971, 236, 17197, 21017, 1543, 3581, 8802, 10928, 32585, 8949, 47402, 3582, 41908, 39823, 35748, 7598, 6180, 1546, 38846, 34719, 30, 51595], "temperature": 0, "avg_logprob": -0.18840591943086082, "compression_ratio": 1.165625, "no_speech_prob": 9.936990466585804e-12}, {"id": 3, "seek": 0, "start": 24.6, "end": 26.8, "text": "重點看兩個指標。", "tokens": [51595, 12624, 8216, 4200, 34623, 25922, 46299, 1543, 51705], "temperature": 0, "avg_logprob": -0.18840591943086082, "compression_ratio": 1.165625, "no_speech_prob": 9.936990466585804e-12}, {"id": 4, "seek": 2680, "start": 26.8, "end": 38.4, "text": "第一,容量。也就是先看統一內存或者GPU顯存的容量。模型運行的時候,需要把權重裝進GPU顯存或者統一內存。", "tokens": [50365, 18049, 11, 25750, 26748, 1543, 6404, 5620, 10108, 4200, 33725, 2257, 28472, 39781, 31148, 38, 8115, 46605, 39781, 1546, 25750, 26748, 1543, 41908, 39823, 32772, 8082, 20643, 11, 35748, 16075, 40212, 12624, 44453, 18214, 38, 8115, 46605, 39781, 31148, 33725, 2257, 28472, 39781, 1543, 50945], "temperature": 0, "avg_logprob": -0.10713730766659692, "compression_ratio": 1.2974137931034482, "no_speech_prob": 1.1220590429017463e-11}, {"id": 5, "seek": 2680, "start": 38.4, "end": 51.0, "text": "如果裝不下,要麼無法運行,要麼就只能把部分的權重放進系統內存,通過CPU或者PCIe讀取。那這麼一來,速度就會大幅下降。", "tokens": [50945, 13119, 44453, 1960, 4438, 11, 4275, 4323, 16976, 11148, 32772, 8082, 11, 4275, 4323, 3111, 14003, 8225, 16075, 32174, 1546, 40212, 12624, 12744, 18214, 25368, 33725, 28472, 39781, 11, 19550, 8816, 34, 8115, 31148, 12986, 40, 68, 7422, 222, 29436, 1543, 4184, 21269, 2257, 3763, 11, 31217, 13127, 47778, 3582, 3509, 227, 4438, 47421, 1543, 51575], "temperature": 0, "avg_logprob": -0.10713730766659692, "compression_ratio": 1.2974137931034482, "no_speech_prob": 1.1220590429017463e-11}, {"id": 6, "seek": 5100, "start": 51.0, "end": 58.6, "text": "用大白話來說就是,內存或者顯存容量決定了你能跑多大尺寸、多高精度的模型。", "tokens": [50365, 9254, 3582, 13558, 11103, 36979, 5620, 11, 28472, 39781, 31148, 46605, 39781, 25750, 26748, 33540, 12088, 2289, 2166, 8225, 32585, 6392, 3582, 1530, 118, 4510, 116, 1231, 6392, 12979, 34910, 13127, 1546, 41908, 39823, 1543, 50745], "temperature": 0, "avg_logprob": -0.11743161388646776, "compression_ratio": 1.183673469387755, "no_speech_prob": 2.0302424671991837e-11}, {"id": 7, "seek": 5100, "start": 58.6, "end": 66.6, "text": "舉個例子,一個27B的模型如果使用BF16的精度,那麼光是權重就需要大約50GB。", "tokens": [50745, 42828, 3338, 17797, 7626, 11, 8990, 10076, 33, 1546, 41908, 39823, 13119, 22982, 9254, 33, 37, 6866, 1546, 34910, 13127, 11, 20504, 20690, 1541, 40212, 12624, 3111, 35748, 3582, 30889, 2803, 8769, 1543, 51145], "temperature": 0, "avg_logprob": -0.11743161388646776, "compression_ratio": 1.183673469387755, "no_speech_prob": 2.0302424671991837e-11}, {"id": 8, "seek": 5100, "start": 66.6, "end": 70.6, "text": "量化到8bit之後,可以降到30GB左右。", "tokens": [51145, 26748, 23756, 4511, 23, 5260, 26362, 11, 6723, 47421, 4511, 3446, 8769, 29457, 1543, 51345], "temperature": 0, "avg_logprob": -0.11743161388646776, "compression_ratio": 1.183673469387755, "no_speech_prob": 2.0302424671991837e-11}, {"id": 9, "seek": 5100, "start": 70.6, "end": 74.6, "text": "而量化到4bit之後,只需要17-20GB。", "tokens": [51345, 11070, 26748, 23756, 4511, 19, 5260, 26362, 11, 14003, 35748, 7773, 12, 2009, 8769, 1543, 51545], "temperature": 0, "avg_logprob": -0.11743161388646776, "compression_ratio": 1.183673469387755, "no_speech_prob": 2.0302424671991837e-11}, {"id": 10, "seek": 7460, "start": 74.6, "end": 80.6, "text": "除了權重之外,系統本身推理框架上下紋,它們都需要占有內存。", "tokens": [50365, 32999, 2289, 40212, 12624, 9574, 12022, 11, 25368, 33725, 8802, 19847, 33597, 13876, 21416, 228, 7360, 114, 5708, 4438, 7732, 233, 11, 11284, 4623, 7182, 35748, 5322, 254, 2412, 28472, 39781, 1543, 50665], "temperature": 0, "avg_logprob": -0.12449188232421875, "compression_ratio": 1.2298850574712643, "no_speech_prob": 2.1877468586728455e-11}, {"id": 11, "seek": 7460, "start": 80.6, "end": 87.6, "text": "所以4bit的模型,你光有17GB還不夠。大概得要20GB才能勉強能跑。", "tokens": [50665, 7239, 19, 5260, 1546, 41908, 39823, 11, 2166, 20690, 2412, 7773, 8769, 7824, 1960, 31649, 1543, 32044, 5916, 4275, 2009, 8769, 18888, 8225, 7978, 231, 22752, 8225, 32585, 1543, 51015], "temperature": 0, "avg_logprob": -0.12449188232421875, "compression_ratio": 1.2298850574712643, "no_speech_prob": 2.1877468586728455e-11}, {"id": 12, "seek": 7460, "start": 87.6, "end": 89.19999999999999, "text": "第二,帶框。", "tokens": [51015, 19693, 11, 32379, 21416, 228, 1543, 51095], "temperature": 0, "avg_logprob": -0.12449188232421875, "compression_ratio": 1.2298850574712643, "no_speech_prob": 2.1877468586728455e-11}, {"id": 13, "seek": 7460, "start": 89.19999999999999, "end": 94.19999999999999, "text": "把模型裝進內存之後,第二個關鍵指標就是內存帶框。", "tokens": [51095, 16075, 41908, 39823, 44453, 18214, 28472, 39781, 26362, 11, 49085, 14899, 41668, 113, 25922, 46299, 5620, 28472, 39781, 32379, 21416, 228, 1543, 51345], "temperature": 0, "avg_logprob": -0.12449188232421875, "compression_ratio": 1.2298850574712643, "no_speech_prob": 2.1877468586728455e-11}, {"id": 14, "seek": 7460, "start": 94.19999999999999, "end": 99.39999999999999, "text": "容量決定了你能不能跑,而帶框決定模型生成有多快。", "tokens": [51345, 25750, 26748, 33540, 12088, 2289, 2166, 8225, 28590, 32585, 11, 11070, 32379, 21416, 228, 33540, 12088, 41908, 39823, 8244, 11336, 2412, 6392, 10251, 1543, 51605], "temperature": 0, "avg_logprob": -0.12449188232421875, "compression_ratio": 1.2298850574712643, "no_speech_prob": 2.1877468586728455e-11}, {"id": 15, "seek": 9940, "start": 99.4, "end": 104.4, "text": "注意,內存容量大,並不意味著帶框就高。", "tokens": [50365, 40794, 11, 28472, 39781, 25750, 26748, 3582, 11, 35934, 1960, 9042, 17268, 19382, 32379, 21416, 228, 3111, 12979, 1543, 50615], "temperature": 0, "avg_logprob": -0.1305128896338308, "compression_ratio": 1.1299212598425197, "no_speech_prob": 2.2944387709222802e-11}, {"id": 16, "seek": 9940, "start": 104.4, "end": 111.0, "text": "比如英偉達的DGX Spark,它有128GB,比RTX Pro 6000的就是6GB還大。", "tokens": [50615, 36757, 27869, 7437, 231, 33086, 1546, 35, 38, 55, 23424, 11, 11284, 2412, 4762, 23, 8769, 11, 11706, 49, 51, 55, 1705, 41789, 1546, 5620, 21, 8769, 7824, 3582, 1543, 50945], "temperature": 0, "avg_logprob": -0.1305128896338308, "compression_ratio": 1.1299212598425197, "no_speech_prob": 2.2944387709222802e-11}, {"id": 17, "seek": 9940, "start": 111.0, "end": 113.4, "text": "看起來很大很划算對吧?", "tokens": [50945, 4200, 21670, 4563, 3582, 4563, 2437, 240, 19497, 47905, 30, 51065], "temperature": 0, "avg_logprob": -0.1305128896338308, "compression_ratio": 1.1299212598425197, "no_speech_prob": 2.2944387709222802e-11}, {"id": 18, "seek": 9940, "start": 113.4, "end": 120.0, "text": "但是它的內存帶框只有273GB,遠小於6000的1792。", "tokens": [51065, 11189, 45224, 28472, 39781, 32379, 21416, 228, 35244, 10076, 18, 8769, 11, 34489, 7322, 19488, 21, 1360, 1546, 7773, 21821, 1543, 51395], "temperature": 0, "avg_logprob": -0.1305128896338308, "compression_ratio": 1.1299212598425197, "no_speech_prob": 2.2944387709222802e-11}, {"id": 19, "seek": 9940, "start": 120.0, "end": 125.0, "text": "這麼低的帶框意味著,DGX Spark雖然能裝下很大的模型,", "tokens": [51395, 21269, 41377, 1546, 32379, 21416, 228, 9042, 17268, 19382, 11, 35, 38, 55, 23424, 6306, 244, 5823, 8225, 44453, 4438, 4563, 39156, 41908, 39823, 11, 51645], "temperature": 0, "avg_logprob": -0.1305128896338308, "compression_ratio": 1.1299212598425197, "no_speech_prob": 2.2944387709222802e-11}, {"id": 20, "seek": 12500, "start": 125.0, "end": 129.6, "text": "但是跑大型Dance模型的時候,生成速度不會太快。", "tokens": [50365, 11189, 32585, 3582, 39823, 35, 719, 41908, 39823, 20643, 11, 8244, 11336, 31217, 13127, 21121, 9455, 10251, 1543, 50595], "temperature": 0, "avg_logprob": -0.13111501382597796, "compression_ratio": 1.1826625386996905, "no_speech_prob": 1.6165039792848113e-11}, {"id": 21, "seek": 12500, "start": 129.6, "end": 134.6, "text": "它更適合MOE模型,因為MOE的總參數雖然很大,", "tokens": [50595, 11284, 19002, 3330, 102, 14245, 18976, 36, 41908, 39823, 11, 11471, 18976, 36, 1546, 26575, 47375, 30622, 6306, 244, 5823, 4563, 3582, 11, 50845], "temperature": 0, "avg_logprob": -0.13111501382597796, "compression_ratio": 1.1826625386996905, "no_speech_prob": 1.6165039792848113e-11}, {"id": 22, "seek": 12500, "start": 134.6, "end": 139.0, "text": "但每次生成Token的時候,它只激活其中的一小部分。", "tokens": [50845, 8395, 23664, 9487, 8244, 11336, 51, 8406, 20643, 11, 11284, 14003, 26373, 222, 25956, 9572, 5975, 1546, 2257, 7322, 32174, 1543, 51065], "temperature": 0, "avg_logprob": -0.13111501382597796, "compression_ratio": 1.1826625386996905, "no_speech_prob": 1.6165039792848113e-11}, {"id": 23, "seek": 12500, "start": 139.0, "end": 146.6, "text": "你看,我們星球裡就有一個小夥伴,買了兩台,用來跑Deep Seek V4 Flash,速度有30多,夠用了。", "tokens": [51065, 16529, 11, 5884, 20682, 28533, 11066, 3111, 2412, 8990, 7322, 1787, 98, 7384, 112, 11, 31001, 2289, 16313, 15433, 11, 9254, 3763, 32585, 11089, 595, 1100, 916, 691, 19, 20232, 11, 31217, 13127, 2412, 3446, 6392, 11, 31649, 9254, 2289, 1543, 51445], "temperature": 0, "avg_logprob": -0.13111501382597796, "compression_ratio": 1.1826625386996905, "no_speech_prob": 1.6165039792848113e-11}, {"id": 24, "seek": 12500, "start": 146.6, "end": 152.6, "text": "容量和帶框是理解本地推理最重要的兩個指標,但它們不是全部。", "tokens": [51445, 25750, 26748, 12565, 32379, 21416, 228, 1541, 13876, 17278, 8802, 10928, 33597, 13876, 8661, 24928, 1546, 34623, 25922, 46299, 11, 8395, 11284, 4623, 7296, 38714, 1543, 51745], "temperature": 0, "avg_logprob": -0.13111501382597796, "compression_ratio": 1.1826625386996905, "no_speech_prob": 1.6165039792848113e-11}, {"id": 25, "seek": 15260, "start": 152.6, "end": 158.2, "text": "你實際的性能還會受到GPU的數量力、模型架構和推理框架的影響。", "tokens": [50365, 2166, 10376, 34837, 1546, 21686, 8225, 7824, 6236, 23151, 4511, 38, 8115, 1546, 30622, 26748, 13486, 1231, 41908, 39823, 7360, 114, 43362, 12565, 33597, 13876, 21416, 228, 7360, 114, 1546, 16820, 13665, 123, 1543, 50645], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 26, "seek": 15260, "start": 158.2, "end": 162.79999999999998, "text": "但是對於普通人來說,先抓住這兩個核心指標完全沒有問題。", "tokens": [50645, 11189, 2855, 19488, 29993, 19550, 4035, 36979, 11, 10108, 43069, 21632, 2664, 34623, 8987, 116, 7945, 25922, 46299, 37100, 6963, 17197, 1543, 50875], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 27, "seek": 15260, "start": 162.79999999999998, "end": 167.0, "text": "OK,理解了原理之後,那選擇就容易多了。", "tokens": [50875, 9443, 11, 13876, 17278, 2289, 19683, 13876, 26362, 11, 4184, 21625, 13167, 229, 3111, 49212, 6392, 2289, 1543, 51085], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 28, "seek": 15260, "start": 167.0, "end": 169.0, "text": "咱們有三個選擇。", "tokens": [51085, 8975, 109, 4623, 2412, 10960, 3338, 21625, 13167, 229, 1543, 51185], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 29, "seek": 15260, "start": 169.0, "end": 170.4, "text": "第一,Mac。", "tokens": [51185, 18049, 11, 34355, 1543, 51255], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 30, "seek": 15260, "start": 170.4, "end": 176.79999999999998, "text": "蘋果芯片用的是統一記憶,也就是CPU和GPU共享一個記憶池。", "tokens": [51255, 31882, 233, 9319, 13778, 107, 16668, 9254, 24620, 33725, 2257, 16958, 29713, 114, 11, 6404, 5620, 34, 8115, 12565, 38, 8115, 15408, 27654, 8990, 16958, 29713, 114, 12800, 254, 1543, 51575], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 31, "seek": 15260, "start": 176.79999999999998, "end": 181.79999999999998, "text": "因此,買Mac的話,可以用相對較低的價格獲得更高的記憶。", "tokens": [51575, 8698, 17947, 11, 31001, 34355, 21358, 11, 6723, 9254, 15106, 2855, 23915, 41377, 1546, 45441, 30921, 29057, 110, 5916, 19002, 12979, 1546, 16958, 29713, 114, 1543, 51825], "temperature": 0, "avg_logprob": -0.12321382761001587, "compression_ratio": 1.2272727272727273, "no_speech_prob": 1.5795051030997875e-11}, {"id": 32, "seek": 18180, "start": 181.8, "end": 189.4, "text": "但是,它的不足之處在於,帶框沒有那麼高,只有5090和6000的一半甚至更低。", "tokens": [50365, 11189, 11, 45224, 1960, 37236, 9574, 39289, 3581, 19488, 11, 32379, 21416, 228, 6963, 20504, 12979, 11, 35244, 2803, 7771, 12565, 21, 1360, 1546, 2257, 30018, 17556, 20844, 19002, 41377, 1543, 50745], "temperature": 0, "avg_logprob": -0.14337227309959522, "compression_ratio": 1.17, "no_speech_prob": 2.2172656072294572e-11}, {"id": 33, "seek": 18180, "start": 189.4, "end": 196.60000000000002, "text": "所以,即使Mac擁有同樣甚至更多的記憶,在運行Dance模型的時候,速度不會太快。", "tokens": [50745, 7239, 11, 39127, 22982, 34355, 13167, 223, 2412, 13089, 6180, 17556, 20844, 19002, 6392, 1546, 16958, 29713, 114, 11, 3581, 32772, 8082, 35, 719, 41908, 39823, 20643, 11, 31217, 13127, 21121, 9455, 10251, 1543, 51105], "temperature": 0, "avg_logprob": -0.14337227309959522, "compression_ratio": 1.17, "no_speech_prob": 2.2172656072294572e-11}, {"id": 34, "seek": 18180, "start": 196.60000000000002, "end": 199.4, "text": "而這個是需要咱們取捨的地方。", "tokens": [51105, 11070, 41427, 35748, 8975, 109, 4623, 29436, 162, 18304, 1546, 30146, 1543, 51245], "temperature": 0, "avg_logprob": -0.14337227309959522, "compression_ratio": 1.17, "no_speech_prob": 2.2172656072294572e-11}, {"id": 35, "seek": 18180, "start": 199.4, "end": 202.0, "text": "第二,RTX5090。", "tokens": [51245, 19693, 11, 49, 51, 55, 2803, 7771, 1543, 51375], "temperature": 0, "avg_logprob": -0.14337227309959522, "compression_ratio": 1.17, "no_speech_prob": 2.2172656072294572e-11}, {"id": 36, "seek": 18180, "start": 202.0, "end": 210.4, "text": "5090擁有3.4G顯存和1792GB每秒的記憶體帶框,非常適合千萬3.827B這樣的模型。", "tokens": [51375, 2803, 7771, 13167, 223, 2412, 18, 13, 19, 38, 46605, 39781, 12565, 7773, 21821, 8769, 23664, 20956, 1546, 16958, 29713, 114, 23987, 32379, 21416, 228, 11, 14392, 3330, 102, 14245, 20787, 14099, 18, 13, 23, 10076, 33, 39425, 41908, 39823, 1543, 51795], "temperature": 0, "avg_logprob": -0.14337227309959522, "compression_ratio": 1.17, "no_speech_prob": 2.2172656072294572e-11}, {"id": 37, "seek": 21040, "start": 210.4, "end": 220.0, "text": "所以,如果你的目標只是本地運行千萬3.827B的話,那一張5090通常會比Mac Studio更加合適。", "tokens": [50365, 7239, 11, 13119, 18961, 11386, 46299, 36859, 8802, 10928, 32772, 8082, 20787, 14099, 18, 13, 23, 10076, 33, 21358, 11, 4184, 2257, 18889, 2803, 7771, 19550, 11279, 6236, 11706, 34355, 13500, 19002, 9990, 14245, 3330, 102, 1543, 50845], "temperature": 0, "avg_logprob": -0.14184354146321615, "compression_ratio": 0.9881656804733728, "no_speech_prob": 2.5244879103825113e-11}, {"id": 38, "seek": 21040, "start": 220.0, "end": 224.8, "text": "如果需要多Agent并發的話,可以上兩張5090。", "tokens": [50845, 13119, 35748, 6392, 32, 6930, 3509, 114, 14637, 21358, 11, 6723, 5708, 16313, 18889, 2803, 7771, 1543, 51085], "temperature": 0, "avg_logprob": -0.14184354146321615, "compression_ratio": 0.9881656804733728, "no_speech_prob": 2.5244879103825113e-11}, {"id": 39, "seek": 22480, "start": 224.8, "end": 227.4, "text": "注意啊,5090沒有NV-Link。", "tokens": [50365, 40794, 4905, 11, 2803, 7771, 6963, 45, 53, 12, 26822, 1543, 50495], "temperature": 0, "avg_logprob": -0.1126116576947664, "compression_ratio": 1.0154639175257731, "no_speech_prob": 2.2171002880821966e-11}, {"id": 40, "seek": 22480, "start": 227.4, "end": 234.4, "text": "你非要跨卡通信的話,要走PCIE,效率不會像一塊原生的64G顯卡那麼高。", "tokens": [50495, 2166, 12107, 4275, 6563, 101, 32681, 19550, 17665, 21358, 11, 4275, 9575, 12986, 6550, 11, 43076, 44866, 21121, 12760, 2257, 26268, 19683, 8244, 1546, 19395, 38, 46605, 32681, 20504, 12979, 1543, 50845], "temperature": 0, "avg_logprob": -0.1126116576947664, "compression_ratio": 1.0154639175257731, "no_speech_prob": 2.2171002880821966e-11}, {"id": 41, "seek": 22480, "start": 234.4, "end": 239.4, "text": "所以,插兩張卡的目的,是讓每張卡都運行一個完整的模型。", "tokens": [50845, 7239, 11, 11673, 240, 16313, 18889, 32681, 1546, 11386, 1546, 11, 1541, 21195, 23664, 18889, 32681, 7182, 32772, 8082, 8990, 14128, 27662, 1546, 41908, 39823, 1543, 51095], "temperature": 0, "avg_logprob": -0.1126116576947664, "compression_ratio": 1.0154639175257731, "no_speech_prob": 2.2171002880821966e-11}, {"id": 42, "seek": 23940, "start": 239.4, "end": 242.0, "text": "第一張卡用來處理一部分的Agent。", "tokens": [50365, 18049, 18889, 32681, 9254, 3763, 39289, 13876, 2257, 32174, 1546, 32, 6930, 1543, 50495], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 43, "seek": 23940, "start": 242.0, "end": 244.0, "text": "第二張卡處理另一部分的Agent。", "tokens": [50495, 19693, 18889, 32681, 39289, 13876, 22762, 2257, 32174, 1546, 32, 6930, 1543, 50595], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 44, "seek": 23940, "start": 244.0, "end": 250.0, "text": "它特別適合短或者中上下文,高并發多Agent任務的情況。", "tokens": [50595, 11284, 38446, 3330, 102, 14245, 5881, 255, 31148, 5975, 5708, 4438, 17174, 11, 12979, 3509, 114, 14637, 6392, 32, 6930, 26443, 39517, 1546, 39391, 1543, 50895], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 45, "seek": 23940, "start": 250.0, "end": 254.0, "text": "5090雖然夠快,但它容量只有32G。", "tokens": [50895, 2803, 7771, 6306, 244, 5823, 31649, 10251, 11, 8395, 11284, 25750, 26748, 35244, 11440, 38, 1543, 51095], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 46, "seek": 23940, "start": 254.0, "end": 256.0, "text": "你跑27B的模型沒有問題。", "tokens": [51095, 2166, 32585, 10076, 33, 1546, 41908, 39823, 6963, 17197, 1543, 51195], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 47, "seek": 23940, "start": 256.0, "end": 261.0, "text": "如果要運行70B的模型超長上下文,或者同時常駐多個模型,", "tokens": [51195, 13119, 4275, 32772, 8082, 5867, 33, 1546, 41908, 39823, 19869, 15353, 5708, 4438, 17174, 11, 31148, 13089, 6611, 11279, 165, 39122, 6392, 3338, 41908, 39823, 11, 51445], "temperature": 0, "avg_logprob": -0.12332593983617322, "compression_ratio": 1.2448979591836735, "no_speech_prob": 1.990836662191242e-11}, {"id": 48, "seek": 26100, "start": 261.0, "end": 263.6, "text": "那這樣的顯存就不夠了。", "tokens": [50365, 4184, 39425, 46605, 39781, 3111, 1960, 31649, 2289, 1543, 50495], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 49, "seek": 26100, "start": 263.6, "end": 265.6, "text": "這是它的不足之處。", "tokens": [50495, 16600, 45224, 1960, 37236, 9574, 39289, 1543, 50595], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 50, "seek": 26100, "start": 265.6, "end": 268.4, "text": "第三,RTX PRO 6000。", "tokens": [50595, 35878, 11, 49, 51, 55, 15008, 41789, 1543, 50735], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 51, "seek": 26100, "start": 268.4, "end": 271.6, "text": "這是最完整的本地AI方案。", "tokens": [50735, 16600, 8661, 14128, 27662, 1546, 8802, 10928, 48698, 9249, 28899, 1543, 50895], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 52, "seek": 26100, "start": 271.6, "end": 276.9, "text": "這張卡擁有96G顯存和1792GB每秒的帶寬。", "tokens": [50895, 2664, 18889, 32681, 13167, 223, 2412, 22962, 38, 46605, 39781, 12565, 7773, 21821, 8769, 23664, 20956, 1546, 32379, 4510, 105, 1543, 51160], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 53, "seek": 26100, "start": 276.9, "end": 279.6, "text": "同時具備大容量和高帶寬。", "tokens": [51160, 13089, 6611, 39806, 20854, 3582, 25750, 26748, 12565, 12979, 32379, 4510, 105, 1543, 51295], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 54, "seek": 26100, "start": 279.6, "end": 285.1, "text": "意味著它特別適合長上下文多模型常駐和生產及Agent服務。", "tokens": [51295, 9042, 17268, 19382, 11284, 38446, 3330, 102, 14245, 15353, 5708, 4438, 17174, 6392, 41908, 39823, 11279, 165, 39122, 12565, 8244, 33299, 25703, 32, 6930, 27408, 39517, 1543, 51570], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 55, "seek": 26100, "start": 285.1, "end": 290.6, "text": "而且RTX PRO 6000還支持ECC和專業驅動,更適合長期運行。", "tokens": [51570, 22942, 49, 51, 55, 15008, 41789, 7824, 35488, 8140, 34, 12565, 46072, 27119, 24023, 227, 12572, 11, 19002, 3330, 102, 14245, 15353, 16786, 32772, 8082, 1543, 51845], "temperature": 0, "avg_logprob": -0.13407287323217598, "compression_ratio": 1.1979522184300342, "no_speech_prob": 1.512857721042149e-11}, {"id": 56, "seek": 29100, "start": 291.0, "end": 294.0, "text": "但是,它有一個很大的不足。", "tokens": [50365, 11189, 11, 11284, 2412, 8990, 4563, 39156, 1960, 37236, 1543, 50515], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 57, "seek": 29100, "start": 294.0, "end": 295.6, "text": "太貴了。", "tokens": [50515, 9455, 11561, 112, 2289, 1543, 50595], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 58, "seek": 29100, "start": 295.6, "end": 297.6, "text": "你去收下價格就知道了。", "tokens": [50595, 2166, 6734, 18681, 4438, 45441, 30921, 3111, 7758, 2289, 1543, 50695], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 59, "seek": 29100, "start": 297.6, "end": 301.4, "text": "剛才我介紹的三種方案,價格跨度非常大。", "tokens": [50695, 16940, 18888, 1654, 30312, 38193, 1546, 10960, 16516, 9249, 28899, 11, 45441, 30921, 6563, 101, 13127, 14392, 3582, 1543, 50885], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 60, "seek": 29100, "start": 301.4, "end": 303.8, "text": "從幾萬到十幾二十萬都有。", "tokens": [50885, 21068, 23575, 14099, 4511, 20145, 23575, 11217, 20145, 14099, 48121, 1543, 51005], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 61, "seek": 29100, "start": 303.8, "end": 307.6, "text": "所以,沒有一種硬件在所有維度都最好。", "tokens": [51005, 7239, 11, 6963, 2257, 16516, 38114, 105, 20485, 3581, 39300, 9261, 255, 13127, 7182, 8661, 2131, 1543, 51195], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 62, "seek": 29100, "start": 307.6, "end": 309.2, "text": "你只能做取捨。", "tokens": [51195, 2166, 14003, 8225, 10907, 29436, 162, 18304, 1543, 51275], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 63, "seek": 29100, "start": 309.2, "end": 312.8, "text": "大家目前用什麼樣的配置跑本地模型,效果如何?", "tokens": [51275, 6868, 39004, 9254, 7598, 6180, 1546, 38846, 34719, 32585, 8802, 10928, 41908, 39823, 11, 43076, 9319, 43526, 30, 51455], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 64, "seek": 29100, "start": 312.8, "end": 314.8, "text": "歡迎在評論區留言。", "tokens": [51455, 21315, 3581, 39115, 29340, 44526, 43681, 1543, 51555], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 65, "seek": 29100, "start": 314.8, "end": 317.0, "text": "OK,以上就是本期內容。", "tokens": [51555, 9443, 11, 29497, 5620, 8802, 16786, 28472, 25750, 1543, 51665], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 66, "seek": 29100, "start": 317.0, "end": 319.2, "text": "想了解AI想奪回個體主權,", "tokens": [51665, 7093, 2289, 17278, 48698, 7093, 1881, 103, 8350, 3338, 23987, 13557, 40212, 11, 51775], "temperature": 0, "avg_logprob": -0.14848749527078592, "compression_ratio": 1.267515923566879, "no_speech_prob": 1.5728545202375876e-11}, {"id": 67, "seek": 31920, "start": 319.2, "end": 322.2, "text": "想找到自從盜合的人就來我們Newtype社群。", "tokens": [50365, 7093, 25085, 4511, 9722, 21068, 5419, 250, 14245, 29979, 3111, 3763, 5884, 18278, 20467, 27658, 7248, 97, 1543, 50515], "temperature": 0, "avg_logprob": -0.2885630130767822, "compression_ratio": 0.8735632183908046, "no_speech_prob": 1.8348000202506043e-11}, {"id": 68, "seek": 31920, "start": 322.2, "end": 323.8, "text": "那咱們下期見。", "tokens": [50515, 4184, 8975, 109, 4623, 4438, 16786, 10470, 1543, 50595], "temperature": 0, "avg_logprob": -0.2885630130767822, "compression_ratio": 0.8735632183908046, "no_speech_prob": 1.8348000202506043e-11}, {"id": 69, "seek": 32380, "start": 323.8, "end": 325.8, "text": "請問一下,請問一下。", "tokens": [50365, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50465], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 70, "seek": 32380, "start": 325.8, "end": 326.8, "text": "請問一下,請問一下。", "tokens": [50465, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50515], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 71, "seek": 32380, "start": 326.8, "end": 328.8, "text": "請問一下,請問一下。", "tokens": [50515, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50615], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 72, "seek": 32380, "start": 328.8, "end": 330.8, "text": "請問一下,請問一下。", "tokens": [50615, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50715], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 73, "seek": 32380, "start": 330.8, "end": 332.8, "text": "請問一下,請問一下。", "tokens": [50715, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50815], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 74, "seek": 32380, "start": 332.8, "end": 334.8, "text": "請問一下,請問一下。", "tokens": [50815, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 50915], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}, {"id": 75, "seek": 32380, "start": 334.8, "end": 336.8, "text": "請問一下,請問一下。", "tokens": [50915, 16302, 11361, 8861, 11, 16302, 11361, 8861, 1543, 51015], "temperature": 0, "avg_logprob": -0.5087868902418349, "compression_ratio": 6.125, "no_speech_prob": 4.355095450936375e-10}], "language": "zh"}