{
  "schema": "videonote.subtitle_normalize.v1",
  "input_cue_count": 177,
  "kept_cue_count": 177,
  "dropped_cue_count": 0,
  "duplicate_merged_count": 0,
  "overlap_repaired_count": 0,
  "rolling_prefix_removed_count": 0,
  "timing_uncertain_count": 0,
  "coverage_ratio": 0.999292,
  "gaps": [],
  "issues": [],
  "cues": [
    {
      "id": 0,
      "start": 0.08,
      "end": 6.72,
      "text": "Quen's 27 billion parameter model reads images and text. And the lab behind most local model",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 1,
      "start": 6.72,
      "end": 13.76,
      "text": "compression just shrank it into a single 11.8 GB file. A model that size normally needs more memory",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 2,
      "start": 13.76,
      "end": 19.68,
      "text": "than any gaming computer has. This one is smaller than a video game install. Needless to say,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 3,
      "start": 19.68,
      "end": 25.12,
      "text": "this is a seriously impressive demonstration of model compression. But the question must be asked,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 4,
      "start": 25.12,
      "end": 31.12,
      "text": "can a 12 GBTE graphics card actually run Quen's 27 billion parameter model? To find out,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 5,
      "start": 31.12,
      "end": 35.44,
      "text": "we have to look at the repository where we see that the download page doesn't actually offer",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 6,
      "start": 35.44,
      "end": 40.8,
      "text": "a single file, but a menu of four different sizes shipped two ways for a total of eight",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 7,
      "start": 40.8,
      "end": 47.28,
      "text": "model files. That 11.8 GB model from our title is the largest of the plane options, which the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 8,
      "start": 47.28,
      "end": 53.28,
      "text": "lab recommends as its task lossless build. But sitting right beside those models is a ninth file,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 9,
      "start": 53.28,
      "end": 60.32,
      "text": "a 0.93 GBTE vision projector called MM Proj. None of the eight main checkpoints actually contain",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 10,
      "start": 60.32,
      "end": 64.4,
      "text": "this vision projector, which means you have to get it separately if you want to use the model's",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 11,
      "start": 64.4,
      "end": 72.56,
      "text": "multimodal capabilities. On paper, the 11.8 GB of the GSQ RCO build and the 12 GB of your graphics",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 12,
      "start": 72.56,
      "end": 77.52,
      "text": "card sit right next to each other. But those two numbers are actually measured in entirely",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 13,
      "start": 77.52,
      "end": 83.36,
      "text": "different units. When a card is sold as having 12 GB of memory, the operating system actually",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 14,
      "start": 83.36,
      "end": 89.12,
      "text": "counts that space in binary gigabytes, which is what your hardware actually has to work with. On",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 15,
      "start": 89.12,
      "end": 97.12,
      "text": "the other side of the ledger, that recommended GSQ RCO build is listed on HuggingFace at 11.77 GB.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 16,
      "start": 97.12,
      "end": 104.32,
      "text": "But when you inspect the actual weights of that GSQ RCO build, they only take up 10.96 GB of space",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 17,
      "start": 104.32,
      "end": 112.0,
      "text": "because the download page is using decimal GB. If you subtract 10.96 from your 12 GB of capacity,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 18,
      "start": 112.0,
      "end": 118.96,
      "text": "you are left with 1.04 GB of free space, that means the actual headroom on your card is larger",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 19,
      "start": 118.96,
      "end": 123.76,
      "text": "than the 200 megabytes the headline number suggested. But that extra headroom is about",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 20,
      "start": 123.76,
      "end": 128.72,
      "text": "to vanish because the model's ability to read images is stored in a completely separate file",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 21,
      "start": 128.72,
      "end": 138.32,
      "text": "called mm praj, which weighs 931 million bytes or 0.87 GB. And in a release designed entirely around",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 22,
      "start": 138.32,
      "end": 144.32,
      "text": "model compression, this vision projector is the only file that wasn't compressed at all. Instead,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 23,
      "start": 144.32,
      "end": 149.84,
      "text": "the lab left it at its original 16-bit format. And because one copy serves every single build",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 24,
      "start": 149.84,
      "end": 154.56,
      "text": "on the menu, this memory penalty is exactly the same whether you pick the largest budget",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 25,
      "start": 154.56,
      "end": 161.04,
      "text": "or the smallest. In llama.cpp, you load this projector using a separate command line flag,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 26,
      "start": 161.04,
      "end": 168.48,
      "text": "which makes it easy to forget that it's even there. But once it loads, 0.8 87 GB of the 1.04 we",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 27,
      "start": 168.48,
      "end": 174.24,
      "text": "just found is instantly gone. This means the part of the model the title sells, the system that lets",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 28,
      "start": 174.24,
      "end": 179.28,
      "text": "it read images, is the one part the compression never touched, and it eats almost all of the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 29,
      "start": 179.28,
      "end": 184.4,
      "text": "headroom your card has left. The quantization paper behind this build splits the field of",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 30,
      "start": 184.4,
      "end": 191.68,
      "text": "model compression into simple scalar methods like GPTQ and AWQ that everyone deploys and vector or",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 31,
      "start": 191.68,
      "end": 198.8,
      "text": "trellis methods like Q-tip and AQLM that score better. In the paper's own words, scalar methods",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 32,
      "start": 198.8,
      "end": 204.0,
      "text": "plateau in accuracy at 3 to four bits, while the more accurate vector family is notoriously",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 33,
      "start": 204.0,
      "end": 210.64,
      "text": "hard to implement and scale. So GSQ closes most of that accuracy gap while staying on a scalar grid,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 34,
      "start": 210.64,
      "end": 214.88,
      "text": "ensuring the compressed model remains fully compatible with the existing scalar inference",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 35,
      "start": 214.88,
      "end": 221.28,
      "text": "kernels we already run on our cards. It turns out the same lab published GPTQ, which is one of the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 36,
      "start": 221.28,
      "end": 227.52,
      "text": "standard methods GSQ measures itself against. On top of that, GSQ can be applied directly to an",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 37,
      "start": 227.52,
      "end": 233.2,
      "text": "existing GGUF release you already have, improving its accuracy and exporting it right back into the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 38,
      "start": 233.2,
      "end": 240.48,
      "text": "exact same format. This file runs in your everyday apps like Llama.cpp, Olama, and LM Studio without",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 39,
      "start": 240.48,
      "end": 245.84,
      "text": "modifying any of them. And that's not luck. It happens because the lab deliberately chose the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 40,
      "start": 245.84,
      "end": 251.2,
      "text": "family of methods that scores worse in order to stay inside a format the existing software already",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 41,
      "start": 251.2,
      "end": 255.92,
      "text": "reads. Compressing a single tensor doesn't build a whole file because you still have to",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 42,
      "start": 255.92,
      "end": 261.6,
      "text": "decide how much compression to apply to each part of the model to fit your hardware. To solve that,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 43,
      "start": 261.6,
      "end": 267.28,
      "text": "Michael Hellig and Dan Allistar published a second paper titled Model Compression with Exact Budget",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 44,
      "start": 267.28,
      "end": 272.4,
      "text": "Constraints, framing the problem as assigning one of several options to each group under a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 45,
      "start": 272.4,
      "end": 278.24,
      "text": "total cost budget. While penalty-based approaches only enforce a budget approximately and require",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 46,
      "start": 278.24,
      "end": 284.8,
      "text": "extensive hyperparameter tuning, this method meets it exactly without any constraintspecific tuning.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 47,
      "start": 284.8,
      "end": 290.16,
      "text": "Relaxing the budget constraint describes a smooth curved surface called a remmanion manifold in",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 48,
      "start": 290.16,
      "end": 295.76,
      "text": "lojit space allowing a gradient method to walk along it without ever leaving it. The procedure",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 49,
      "start": 295.76,
      "end": 301.44,
      "text": "takes three steps. You quantize every tensor into every candidate format, search for the assignment",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 50,
      "start": 301.44,
      "end": 306.64,
      "text": "that meets your budget and stitch those winners into one ordinary file. And that means you can",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 51,
      "start": 306.64,
      "end": 311.36,
      "text": "spend your memory budget down to the last bite, which is exactly the problem you face when trying",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 52,
      "start": 311.36,
      "end": 317.92,
      "text": "to squeeze a 27 billion parameter model onto a 12 GB card. You don't even have to download the model",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 53,
      "start": 317.92,
      "end": 323.12,
      "text": "to see what that search algorithm decided because the lab published the entire decision list as a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 54,
      "start": 323.12,
      "end": 330.24,
      "text": "plain text file. Inside the recommended build, the search algorithm ended up slicing 851 individual",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 55,
      "start": 330.24,
      "end": 337.04,
      "text": "tensors across 11 different storage formats. It left 96 tensors at full 16-bit precision",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 56,
      "start": 337.04,
      "end": 342.24,
      "text": "while dropping the rest into a staircase of smaller quantization formats. For instance,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 57,
      "start": 342.24,
      "end": 348.88,
      "text": "the search assigned 144 tensors to the IQ3S format. And the measured average lands right",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 58,
      "start": 348.88,
      "end": 355.2,
      "text": "on the lab's 3.5 target, but non-uniform is not just a marketing word, and the search isn't afraid",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 59,
      "start": 355.2,
      "end": 360.16,
      "text": "to starve a single component if it means saving the rest of the network. In the published dump,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 60,
      "start": 360.16,
      "end": 367.12,
      "text": "block 13's feed forward gate is squeezed down to a starved format called IQ1M, running on just 1 and",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 61,
      "start": 367.12,
      "end": 373.28,
      "text": "3/4 bits. That single tensor isn't surrounded by full precision. It sits inside a block whose other",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 62,
      "start": 373.28,
      "end": 379.2,
      "text": "matrices were all given 2, 3, and 4bit formats. It's simply the cheapest thing in a neighborhood",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 63,
      "start": 379.2,
      "end": 387.04,
      "text": "of ordinary compromises. More than a tenth of the 11.77 GB file is just vocabulary. Specifically,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 64,
      "start": 387.04,
      "end": 393.28,
      "text": "two tensors that do nothing but turn tokens into numbers and back account for 1.12 GB of",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 65,
      "start": 393.28,
      "end": 398.72,
      "text": "the file size together, split between the output head and the embedding table. Against that,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 66,
      "start": 398.72,
      "end": 404.96,
      "text": "every uncompressed tensor in the file adds up to 58 megabytes, under half a percent. Inside a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 67,
      "start": 404.96,
      "end": 410.16,
      "text": "single attention block, the search gave the query projection the cheapest format in its block, while",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 68,
      "start": 410.16,
      "end": 415.52,
      "text": "the key and value neighbors beside it kept more expensive ones. If you watch that same starved",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 69,
      "start": 415.52,
      "end": 420.72,
      "text": "gate across the four size budgets, it starts at just one bit, goes up to four bits at the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 70,
      "start": 420.72,
      "end": 426.16,
      "text": "intermediate three-bit budget and drops back under two bits at the largest. This happens because a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 71,
      "start": 426.16,
      "end": 432.0,
      "text": "global budget optimizer isn't ranking tensors by importance. It's trading bytes and it'll happily",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 72,
      "start": 432.0,
      "end": 436.56,
      "text": "starve a tensor at a larger budget than it fed it at a smaller one because the only thing it's",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 73,
      "start": 436.56,
      "end": 440.96,
      "text": "protecting is the total. Every performance score for these compressed builds comes from the lab",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 74,
      "start": 440.96,
      "end": 445.92,
      "text": "that performed the compression, published on its own model card, and nobody outside that lab has",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 75,
      "start": 445.92,
      "end": 452.48,
      "text": "reproduced these comparisons. According to their results, the recommended 11.8 GB file matches the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 76,
      "start": 452.48,
      "end": 461.04,
      "text": "uncompressed 53.8 GB original exactly on AIM25 and Live Codebench, representing a 4.6 time size",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 77,
      "start": 461.04,
      "end": 466.96,
      "text": "reduction. On GPQA Diamond, this recommended build trails the original by only half a point,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 78,
      "start": 466.96,
      "end": 471.76,
      "text": "and the scores remain high even as you step down to the threebit build, which is a gigabyte and",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 79,
      "start": 471.76,
      "end": 479.2,
      "text": "a half smaller. At 10.09 GB, this 3bit version still hits a perfect 100 on that main reasoning",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 80,
      "start": 479.2,
      "end": 484.64,
      "text": "benchmark, while trailing the recommended build by only half a point on GPQA Diamond. And near",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 81,
      "start": 484.64,
      "end": 491.76,
      "text": "the bottom of the size scale, their 2.75 bit build actually recovers 101.8% 8% of the original",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 82,
      "start": 491.76,
      "end": 498.8,
      "text": "zeroshot average, scoring above the uncompressed baseline. At a matched 8.4 GB, their build beats",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 83,
      "start": 498.8,
      "end": 504.96,
      "text": "Unslaught's dynamic quantizations by 10 points on Aim25. Because the gap between the recommended",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 84,
      "start": 504.96,
      "end": 510.08,
      "text": "build and the one a gigabyte and a half smaller is only half a point on one benchmark, picking a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 85,
      "start": 510.08,
      "end": 516.4,
      "text": "build is not really a quality decision. The base model's configuration lists 64 transformer layers",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 86,
      "start": 516.4,
      "end": 522.24,
      "text": "in a repeating pattern of three linear attention layers followed by one full attention layer. That",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 87,
      "start": 522.24,
      "end": 529.04,
      "text": "leaves 48 gated delta layers and 16 full attention layers that the files own tensor names confirm. A",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 88,
      "start": 529.04,
      "end": 534.64,
      "text": "llama.cpp bug report describes the running model in those exact terms, which matters because the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 89,
      "start": 534.64,
      "end": 539.84,
      "text": "way these layers process information changes how memory behaves. While a full attention",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 90,
      "start": 539.84,
      "end": 544.88,
      "text": "layer has to keep every token of the conversation available, so its memory grows with every word,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 91,
      "start": 544.88,
      "end": 550.4,
      "text": "a recurrent layer keeps one fixed size summary that is the same size after 10 words as after",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 92,
      "start": 550.4,
      "end": 557.28,
      "text": "10,000. That fixed size summary is the reason 48 of its 64 layers never accumulate conversation",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 93,
      "start": 557.28,
      "end": 563.04,
      "text": "history. And that is the reason a 12 GB graphics card is in this conversation at all. We can work",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 94,
      "start": 563.04,
      "end": 568.56,
      "text": "this out directly from the base configuration. The model has 16 attention layers and each layer",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 95,
      "start": 568.56,
      "end": 576.32,
      "text": "holds two key value tensors containing four heads of 256 numbers stored at two bytes a piece. That",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 96,
      "start": 576.32,
      "end": 582.8,
      "text": "multiplies out to 64 kilob for every single token in the conversation. Compressing that cache brings",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 97,
      "start": 582.8,
      "end": 589.04,
      "text": "the price down. Drop it to 8 bits and you spend 34 kilob a token. Squeeze it to four bits and",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 98,
      "start": 589.04,
      "end": 596.4,
      "text": "you only spend 18. Meanwhile, those 48 recurrent layers cost a flat 150 megbytes at any context",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 99,
      "start": 596.4,
      "end": 603.6,
      "text": "length. So when Alibaba advertises a native 262,000 token context, that window has a very",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 100,
      "start": 603.6,
      "end": 612.08,
      "text": "real price. At full precision, the cache alone needs 16 GB. That drops to 8 1/2 GB at 8 bit and",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 101,
      "start": 612.08,
      "end": 618.08,
      "text": "4 1/2 GB at 4bit. We can check that arithmetic against a machine known to work. A developer",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 102,
      "start": 618.08,
      "end": 624.48,
      "text": "running this model at full context on a 16 GB card has to use a 4-bit cache, which the math puts at",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 103,
      "start": 624.48,
      "end": 631.92,
      "text": "4 1/2 GB. If all 64 layers were attention, that cache would require 18 GB and it wouldn't fit on",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 104,
      "start": 631.92,
      "end": 637.6,
      "text": "their card at all. So, what does that actually leave for our 12 GB card? But what happens at",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 105,
      "start": 637.6,
      "end": 642.56,
      "text": "the long context this model advertises on the software your card would actually be running,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 106,
      "start": 642.56,
      "end": 649.04,
      "text": "like llama.cpp. In late August, a developer filed a bug report showing that at long context,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 107,
      "start": 649.04,
      "end": 654.56,
      "text": "the model occasionally generates an end of text marker as its very first token with absolutely no",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 108,
      "start": 654.56,
      "end": 660.24,
      "text": "error. The failure is stubborn, reproducing on the graphics card, on the CPU with the same build,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 109,
      "start": 660.24,
      "end": 665.28,
      "text": "and on an independent fork of the codebase. But the reporter's differential test points to an",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 110,
      "start": 665.28,
      "end": 671.84,
      "text": "engine defect rather than the weights themselves since the exact same prompt passes cleanly on VLM.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 111,
      "start": 671.84,
      "end": 679.28,
      "text": "When testing context capacity, around 113,000 tokens passes cleanly, but around 130,000 fails",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 112,
      "start": 679.28,
      "end": 684.24,
      "text": "with that instant end of text marker, while a slightly deeper prompt works perfectly fine.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 113,
      "start": 684.24,
      "end": 689.44,
      "text": "Again, keep in mind this was measured on another lab's quantizations rather than this build. And",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 114,
      "start": 689.44,
      "end": 695.36,
      "text": "a sibling model with only 30 recurrent layers survives much deeper. The reporter eventually",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 115,
      "start": 695.36,
      "end": 700.24,
      "text": "withdrew his own proposed threshold because of these random holes of failure, advising that",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 116,
      "start": 700.24,
      "end": 705.6,
      "text": "there is no safe limit to quote, \"and you simply have to validate the exact depths you plan to",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 117,
      "start": 705.6,
      "end": 712.0,
      "text": "serve.\" Olama pulls this exact build straight from hugging face with a single command. And although a",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 118,
      "start": 712.0,
      "end": 717.04,
      "text": "maintainer demonstrated it working, actual usage breaks on the chat template rather than your",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 119,
      "start": 717.04,
      "end": 723.04,
      "text": "graphics card. The built-in template refuses any system message that isn't first. So if you send",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 120,
      "start": 723.04,
      "end": 729.28,
      "text": "one mid conversation, you get an empty string, a finished status, and absolutely no error,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 121,
      "start": 729.28,
      "end": 734.64,
      "text": "fixing it takes a threeline patch in a model file. And since this exact fault was filed twice",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 122,
      "start": 734.64,
      "end": 741.52,
      "text": "in August against the plain weights, it isn't this build's doing. Switching to llama.cpp's multimodal",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 123,
      "start": 741.52,
      "end": 746.8,
      "text": "tool is even worse because failing to reserve memory while encoding an image crashes the process",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 124,
      "start": 746.8,
      "end": 752.4,
      "text": "with a segmentation fault instead of reporting an error. Its crash log shows the tool dying on",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 125,
      "start": 752.4,
      "end": 759.84,
      "text": "a failed attempt to allocate just 247 megabytes while reading a single picture. LM Studio avoids",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 126,
      "start": 759.84,
      "end": 765.36,
      "text": "crashing entirely, but its offload slider will quietly push excess weights into ordinary system",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 127,
      "start": 765.36,
      "end": 771.12,
      "text": "memory rather than refuse to load them. The lab's entire LM Studio guidance is a single",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 128,
      "start": 771.12,
      "end": 776.4,
      "text": "sentence telling you to search the repository, and nobody has published a test of this build on a 12",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 129,
      "start": 776.4,
      "end": 782.32,
      "text": "GB card. None of the three applications tells you when the model won't fit, which is why the only",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 130,
      "start": 782.32,
      "end": 788.32,
      "text": "way to know if it runs is to count your bites before you click download. In early September,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 131,
      "start": 788.32,
      "end": 794.64,
      "text": "a developer running a 16 GB Blackwell card with a specific toolkit version found that Llama.cpp's",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 132,
      "start": 794.64,
      "end": 800.72,
      "text": "own back-end test failed matrix multiplication for three formats, including the exact format this",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 133,
      "start": 800.72,
      "end": 806.72,
      "text": "build is named after. The reported errors reached up to 1.1, which is large enough to mean you just",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 134,
      "start": 806.72,
      "end": 812.32,
      "text": "get random garbage output, but the neighboring formats in that same run passed, making the error",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 135,
      "start": 812.32,
      "end": 818.88,
      "text": "highly specific. Then a second user reported the same garbage output on another Blackwell card,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 136,
      "start": 818.88,
      "end": 825.04,
      "text": "fixing it locally with a small change to the bite handling code. But a third user on the same chip",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 137,
      "start": 825.04,
      "end": 829.68,
      "text": "generation couldn't reproduce it, arguing the fault lies in the build chain rather than the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 138,
      "start": 829.68,
      "end": 835.6,
      "text": "hardware since there's no chip specific path for these formats. Anyway, while this debate is still",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 139,
      "start": 835.6,
      "end": 841.12,
      "text": "unresolved, choosing a file assembled from 11 formats means you're depending on 11 distinct",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 140,
      "start": 841.12,
      "end": 847.28,
      "text": "code paths. And those least used paths are exactly where the arguments are still happening. The",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 141,
      "start": 847.28,
      "end": 857.12,
      "text": "recommended model file itself is 11.77 GB, but on your card, those weights consume 10.96 GB of VRAM.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 142,
      "start": 857.12,
      "end": 863.6,
      "text": "Adding the separate vision projector eats another 0.87 GB of memory, which brings the combined total",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 143,
      "start": 863.6,
      "end": 873.68,
      "text": "to 11.83 GB and leaves you with a tiny pocket of just 174 MGB. Subtracting the flat 150 megabytes",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 144,
      "start": 873.68,
      "end": 880.24,
      "text": "demanded by the recurrent state layers leaves 24 megabytes of free headroom. At 64 kilobytes",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 145,
      "start": 880.24,
      "end": 888.0,
      "text": "per token, that remaining space can hold at most about 386 tokens and fewer in practice. Of course,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 146,
      "start": 888.0,
      "end": 893.04,
      "text": "that is a theoretical ceiling that completely ignores whatever graphics memory your desktop",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 147,
      "start": 893.04,
      "end": 897.84,
      "text": "operating system is already using, not to mention the working memory that the inference software",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 148,
      "start": 897.84,
      "end": 904.24,
      "text": "itself reserves before it allocates a single token of cash. If you try to step up to the largest",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 149,
      "start": 904.24,
      "end": 912.08,
      "text": "12.12 GB build, adding the projector overshoots your card's memory by 308 MGB, meaning it cannot",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 150,
      "start": 912.08,
      "end": 918.32,
      "text": "hold its own eyes at zero context. But if you go text only and work out the math without that",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 151,
      "start": 918.32,
      "end": 925.28,
      "text": "vision projector, the recommended build leaves 912 megabytes of free space, which translates to about",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 152,
      "start": 925.28,
      "end": 934.32,
      "text": "14 12,000 tokens of context at full precision or up to 27,000 tokens at 8 bit. Your 12 GB graphics",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 153,
      "start": 934.32,
      "end": 940.64,
      "text": "card is fully capable of running this 27 billion parameter model, but it simply cannot handle the",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 154,
      "start": 940.64,
      "end": 946.8,
      "text": "specific 11.8 8 GB build that the laboratory recommends once that build can see. The file",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 155,
      "start": 946.8,
      "end": 954.64,
      "text": "you actually want is the 3bit build which comes in at 10.09 GB on the download page or about 9.40",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 156,
      "start": 954.64,
      "end": 959.84,
      "text": "in your card's own units. Once you load those weights together with the vision projector,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 157,
      "start": 959.84,
      "end": 967.2,
      "text": "you're left with 1,624 megabytes of headroom. That leaves a ceiling of about 26,000 tokens of active",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 158,
      "start": 967.2,
      "end": 973.04,
      "text": "conversation with images enabled, which jumps to 40,000 tokens if you're running text only. If you",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 159,
      "start": 973.04,
      "end": 978.8,
      "text": "optimize your cache or step down to the even smaller 2.75 bit build, those numbers stretch",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 160,
      "start": 978.8,
      "end": 984.0,
      "text": "even further. Keep in mind, these are absolute mathematical ceilings that ignore the runtime's",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 161,
      "start": 984.0,
      "end": 989.04,
      "text": "own working buffers, so your real capacity is lower. According to the laboratory's own",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 162,
      "start": 989.04,
      "end": 995.04,
      "text": "evaluation, that downgrade costs half a point of GPQA diamond and about a point of live codebench",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 163,
      "start": 995.04,
      "end": 1001.6,
      "text": "with the perfect score of 100 on AIM25 completely unchanged. And that tiny reduction in accuracy",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 164,
      "start": 1001.6,
      "end": 1007.6,
      "text": "buys you roughly 26,000 tokens of conversation. Two separate teams have filed the same request",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 165,
      "start": 1007.6,
      "end": 1013.28,
      "text": "on the lab's repository, but with no reply so far, outside developers can only recover the final map",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 166,
      "start": 1013.28,
      "end": 1018.96,
      "text": "of which format went to which tensor without any way to actually run the search themselves. They've",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 167,
      "start": 1018.96,
      "end": 1023.44,
      "text": "asked for the missing pieces because several critical steps like the candidate generation",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 168,
      "start": 1023.44,
      "end": 1029.04,
      "text": "are completely absent from the public code. Specifically, they're missing the per tensor cost",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 169,
      "start": 1029.04,
      "end": 1033.92,
      "text": "model, the settings for two of the budgets, and the export code that assembles the mixed format",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 170,
      "start": 1033.92,
      "end": 1039.28,
      "text": "file. One of those teams wants the code because their server fleet is limited by memory bandwidth",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 171,
      "start": 1039.28,
      "end": 1044.72,
      "text": "rather than capacity, making a smaller file a direct speed win, and this mixed format is their",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 172,
      "start": 1044.72,
      "end": 1051.36,
      "text": "only real deployment path. The budget paper names mixed precision quantization, non-uniform pruning,",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 173,
      "start": 1051.36,
      "end": 1056.4,
      "text": "and choosing which experts to keep as the same problem of assigning one of K options to each of",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 174,
      "start": 1056.4,
      "end": 1062.32,
      "text": "N groups, which means this is a method for fitting almost anything into a fixed memory footprint.",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 175,
      "start": 1062.32,
      "end": 1067.12,
      "text": "So, while the lab's release gives us a finished answer to one specific model's budget, the one",
      "source": "source_caption",
      "timing_uncertain": false
    },
    {
      "id": 176,
      "start": 1067.12,
      "end": 1072.32,
      "text": "thing they haven't shipped yet is the method for anyone else's. Anyways, that's all, folks.",
      "source": "source_caption",
      "timing_uncertain": false
    }
  ]
}