{ "text": "Can the open AI whisper model dostreaming ASR? My name is Bai. I'm a\nchine learning engineer and a PhD inatural language processing. And today Iwill answer this question. So if you\nhaven't seen the whisper model, it is aspeechtoext model. It is trained onabout 100 languages on 680,000 hours of\ndata. The model architecture is aencoder decoder transformer model and itcomes in five different sizes and it is\ngrowing in popularity recently becauseit is robust to noise and accents. Sothe question of the day is can this\nmodel do streaming speech recognition.So first of all what do we mean bystreaming ASR? Well, automated speech\nrecognition can come in two forms batchand streaming. Batch means that themodel needs to take a bunch of speech\nand produce a bunch of text. Butstreaming ASR is when the model needs toproduce an output as the speaker isaying things with a delay of not more\nthan a few seconds. So for example, ifyou're listening to a live broadcast,you don't want to first record the\nntire broadcast and then produce thesubtitles, you want it to appear at mosta few seconds after it is said. And\ngenerally you'll expect streaming ASR tobe a few percentage points loweraccuracy than batch ASR because the\nmodel does not have access to all thecontext only the point up until thecurrent word. The question of how to do\nstreaming ASR came up when I wasbuilding the voice writer. This is an AIwriting assistant that works in twosteps. In the first step, you just speak\nyour thoughts without worrying too muchabout the grammar. And in the secondstep, the AI corrects the grammar for\nyou. I've been finding it super helpfulfor writing all kinds of things,including emails, blog posts, and Slack\nmessages. You can try it out for free inthe link here. And I'll also post a linkin the description of the video. Now,\nback to the video. First of all, why isthis even difficult? Why is it not so\neasy to use the Whisper model forstreaming ASR? The issue is Whisper is\ntrained to process input audio of 30seconds long. If the audio is less than30 seconds, you can pad it to be longer.\nSo, this is not too much of a problem.But what if your audio is longer than 30seconds? There is no way to feed into\nWhisper anything longer than 30 seconds.So, the first thing you might think ofis what if you just split it up into\nchunks of 30 seconds at a time andprocess each chunk one at a time. Well,the first problem is you might split it\nin the middle of the word. Like if yousplit it while in the middle of theword, this word will probably not berecognized correctly. The second problem\nis the latency will be really high. Soimagine if you're processing audio,you're waiting for audio to fill up\nthese 30 seconds and only then will youprocess this entire chunk and then thelatency um for for the first word will\nbe up to 30 seconds. Fortunately, thereis an open source report called WhisperStreaming that does this for you and it\nurns the Whisper model into a streamingASR system. There are instructions onhow to set this up. So, let's try it\nout. So, I run this command to start amodel running um it's using a whispersmall model and then I go to this tab\nand run another command and I just startspeaking. So you can pretty quickly seethat it's um picking it up and it's\npretty quick and responsive and it evengives timestamps. So um it's pretty cooldemo. Um now let's talk about how this\nworks. So we start with this audio fileand what we do is we feed chunks ofincreasing size into whisper. The size\nof these chunks is configured by theminimum chunk size parameter which bydefault is 1 second. So in eachiteration, we increase the size of this\nbuffer by 1 second and feed all of itinto whisper. And this process continuesuntil we hit a end of sentence marker\nlike a period or a question mark. Thenwe move the buffer forward and start theprocess again. So each piece of audio is\nprocessed by whisper multiple times asmany times as it takes until we hit thend of the sentence. The reasons for\nthis is whisper is trained on sentences.So, it gives the best result when thestart of the audio aligns with the start\nof a sentence. And that's why the bufferonly moves forward when the previousentence is complete and we're beginning\na new sentence. This way, Whisper neverneeds to start transcribing from themiddle of a sentence, which would give\nsuboptimal results. In manyapplications, it is useful to have someresults available as soon as possible,\neven if they're not completely accurate.In the voice writer, I am showing theincomplete results in gray whereas the\nconfirmed results are in black. Thedifference is the gray part of thesentence is unconfirmed. So it maychange as the model gets new\ninformation. But the black part is theconfirmed results and they arepermanent. How this works is using analgorithm called local agreement with n\nequals 2. This means that for a token tobe confirmed, it needs to be generatedin two consecutive audio buffers. Let's\ngive an example. Let's say in the firstep, the worst per model outputs thethree tokens if you like and nothing is\nconfirmed at this step. In the secondstep, the model produces more tokens,but only the first two tokens agree with\ne previous step. So those two areconfirmed. In the third and the fourthsteps, more tokens are generated. But at\neach step, it does not confirm anytokens until it is at least generated intwo consecutive chunks. So anything in\nthe gray part is possible to change andgiven new information. For example, theword view might change to video once the\nmodel hears the rest of the sentence.But anything in the black part ispermanent. So even if the model wants to\nchange it to something else after moreiterations it cannot be changed anymore.One last thing that this model does when\nit generates a new sentence is it feedsthe previous sentence into the model asprompt tokens. So this is something that\nyou can do in the whisper format is youcan give it a bunch of prompt tokensbefore you start generating and this\ntends to improve the accuracy a littlebit because more context is always good.So in summary, the algorithm can be\nsummarized in three basic ideas. Thefirst idea is we feed longer and longerconsecutive audio buffers into whisper\nand then we emit tokens as soon asthey're confirmed by two iterations. Andfinally, we scroll forward the audio\nbuffer whenever a sentence is completed.So, it's a pretty simple algorithm thatyou can apply to any speech to text\nmodel that does not support streamingand basically turn it into a streamingmodel. If you like, you can check out\nall the details in this paper, whichI'll link in the description. One of thelimitations comes from the fact that\nwhisper was not really designed to be astreaming model. And because of this, itassumes that each audio buffer has to\nstart at the beginning of a sentence.Therefore, if you have a sentence thatis quite long, then the beginning of the\nsentence has to be fed into the modeland processed many times. This isinefficient and not necessary if themodel was trained from the beginning to\ndo streaming ASR. Now, if we look at anarchitecture that was designedspecifically to do streaming speechrecognition, it looks a little bit\ndifferent. Here is a model that wasproposed in 2021. At each step, itpredicts a token and it has access to afixed amount of past context and future\ncontext. During training, this rule isenforced by having an intention maskthat is mostly zeros, meaning that the\nmodel cannot use information that isoutside of this fixed window. And duringprediction, the model predicts a token\ngiven a limited amount of fixed context.To make the next prediction, the entireceptive field moves forward by one\nchunk, but the size of the receptivefield is fixed. This way, the beginningof the sentence will not be processed\nmultiple times. However, this is notpossible with the whisper model becauseit requires modifying the architecture\nand retraining the model and we do nothave access to whisper's training data.That's it for this video and I hope you\nenjoyed my explanation of how Whispercan be turned into a streaming model. Ifyou enjoyed this video, please leave a\ncomment, hit the subscribe button, andring the bell icon to stay notified whenI release future videos. Goodbye.", "language": "en", "duration": 521.0, "segments": [ { "id": 0, "start": 0.24, "end": 5.92, "text": "Can the open AI whisper model dostreaming ASR? My name is Bai. I'm a", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 1, "start": 5.92, "end": 12.0, "text": "chine learning engineer and a PhD inatural language processing. And today Iwill answer this question. So if you", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 2, "start": 12.0, "end": 19.84, "text": "haven't seen the whisper model, it is aspeechtoext model. It is trained onabout 100 languages on 680,000 hours of", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 3, "start": 19.84, "end": 27.279, "text": "data. The model architecture is aencoder decoder transformer model and itcomes in five different sizes and it is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 4, "start": 27.279, "end": 33.52, "text": "growing in popularity recently becauseit is robust to noise and accents. Sothe question of the day is can this", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 5, "start": 33.52, "end": 40.32, "text": "model do streaming speech recognition.So first of all what do we mean bystreaming ASR? Well, automated speech", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 6, "start": 40.32, "end": 46.32, "text": "recognition can come in two forms batchand streaming. Batch means that themodel needs to take a bunch of speech", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 7, "start": 46.32, "end": 57.68, "text": "and produce a bunch of text. Butstreaming ASR is when the model needs toproduce an output as the speaker isaying things with a delay of not more", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 8, "start": 57.68, "end": 64.96, "text": "than a few seconds. So for example, ifyou're listening to a live broadcast,you don't want to first record the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 9, "start": 64.96, "end": 72.4, "text": "ntire broadcast and then produce thesubtitles, you want it to appear at mosta few seconds after it is said. And", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 10, "start": 72.4, "end": 79.84, "text": "generally you'll expect streaming ASR tobe a few percentage points loweraccuracy than batch ASR because the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 11, "start": 79.84, "end": 87.04, "text": "model does not have access to all thecontext only the point up until thecurrent word. The question of how to do", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 12, "start": 87.04, "end": 95.84, "text": "streaming ASR came up when I wasbuilding the voice writer. This is an AIwriting assistant that works in twosteps. In the first step, you just speak", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 13, "start": 95.84, "end": 101.68, "text": "your thoughts without worrying too muchabout the grammar. And in the secondstep, the AI corrects the grammar for", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 14, "start": 101.68, "end": 107.92, "text": "you. I've been finding it super helpfulfor writing all kinds of things,including emails, blog posts, and Slack", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 15, "start": 107.92, "end": 114.56, "text": "messages. You can try it out for free inthe link here. And I'll also post a linkin the description of the video. Now,", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 16, "start": 114.56, "end": 119.6, "text": "back to the video. First of all, why isthis even difficult? Why is it not so", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 17, "start": 119.6, "end": 124.719, "text": "easy to use the Whisper model forstreaming ASR? The issue is Whisper is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 18, "start": 124.719, "end": 132.64, "text": "trained to process input audio of 30seconds long. If the audio is less than30 seconds, you can pad it to be longer.", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 19, "start": 132.64, "end": 140.08, "text": "So, this is not too much of a problem.But what if your audio is longer than 30seconds? There is no way to feed into", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 20, "start": 140.08, "end": 146.56, "text": "Whisper anything longer than 30 seconds.So, the first thing you might think ofis what if you just split it up into", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 21, "start": 146.56, "end": 153.68, "text": "chunks of 30 seconds at a time andprocess each chunk one at a time. Well,the first problem is you might split it", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 22, "start": 153.68, "end": 162.16, "text": "in the middle of the word. Like if yousplit it while in the middle of theword, this word will probably not berecognized correctly. The second problem", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 23, "start": 162.16, "end": 169.12, "text": "is the latency will be really high. Soimagine if you're processing audio,you're waiting for audio to fill up", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 24, "start": 169.12, "end": 177.519, "text": "these 30 seconds and only then will youprocess this entire chunk and then thelatency um for for the first word will", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 25, "start": 177.519, "end": 185.12, "text": "be up to 30 seconds. Fortunately, thereis an open source report called WhisperStreaming that does this for you and it", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 26, "start": 185.12, "end": 193.519, "text": "urns the Whisper model into a streamingASR system. There are instructions onhow to set this up. So, let's try it", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 27, "start": 193.519, "end": 201.12, "text": "out. So, I run this command to start amodel running um it's using a whispersmall model and then I go to this tab", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 28, "start": 201.12, "end": 210.0, "text": "and run another command and I just startspeaking. So you can pretty quickly seethat it's um picking it up and it's", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 29, "start": 210.0, "end": 217.519, "text": "pretty quick and responsive and it evengives timestamps. So um it's pretty cooldemo. Um now let's talk about how this", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 30, "start": 217.519, "end": 226.319, "text": "works. So we start with this audio fileand what we do is we feed chunks ofincreasing size into whisper. The size", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 31, "start": 226.319, "end": 234.879, "text": "of these chunks is configured by theminimum chunk size parameter which bydefault is 1 second. So in eachiteration, we increase the size of this", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 32, "start": 234.879, "end": 242.56, "text": "buffer by 1 second and feed all of itinto whisper. And this process continuesuntil we hit a end of sentence marker", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 33, "start": 242.56, "end": 249.68, "text": "like a period or a question mark. Thenwe move the buffer forward and start theprocess again. So each piece of audio is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 34, "start": 249.68, "end": 256.079, "text": "processed by whisper multiple times asmany times as it takes until we hit thend of the sentence. The reasons for", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 35, "start": 256.079, "end": 262.96, "text": "this is whisper is trained on sentences.So, it gives the best result when thestart of the audio aligns with the start", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 36, "start": 262.96, "end": 269.68, "text": "of a sentence. And that's why the bufferonly moves forward when the previousentence is complete and we're beginning", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 37, "start": 269.68, "end": 275.52, "text": "a new sentence. This way, Whisper neverneeds to start transcribing from themiddle of a sentence, which would give", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 38, "start": 275.52, "end": 282.32, "text": "suboptimal results. In manyapplications, it is useful to have someresults available as soon as possible,", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 39, "start": 282.32, "end": 288.96, "text": "even if they're not completely accurate.In the voice writer, I am showing theincomplete results in gray whereas the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 40, "start": 288.96, "end": 297.12, "text": "confirmed results are in black. Thedifference is the gray part of thesentence is unconfirmed. So it maychange as the model gets new", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 41, "start": 297.12, "end": 307.52, "text": "information. But the black part is theconfirmed results and they arepermanent. How this works is using analgorithm called local agreement with n", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 42, "start": 307.52, "end": 314.72, "text": "equals 2. This means that for a token tobe confirmed, it needs to be generatedin two consecutive audio buffers. Let's", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 43, "start": 314.72, "end": 322.32, "text": "give an example. Let's say in the firstep, the worst per model outputs thethree tokens if you like and nothing is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 44, "start": 322.32, "end": 328.96, "text": "confirmed at this step. In the secondstep, the model produces more tokens,but only the first two tokens agree with", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 45, "start": 328.96, "end": 336.0, "text": "e previous step. So those two areconfirmed. In the third and the fourthsteps, more tokens are generated. But at", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 46, "start": 336.0, "end": 344.32, "text": "each step, it does not confirm anytokens until it is at least generated intwo consecutive chunks. So anything in", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 47, "start": 344.32, "end": 353.28, "text": "the gray part is possible to change andgiven new information. For example, theword view might change to video once the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 48, "start": 353.28, "end": 360.16, "text": "model hears the rest of the sentence.But anything in the black part ispermanent. So even if the model wants to", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 49, "start": 360.16, "end": 367.12, "text": "change it to something else after moreiterations it cannot be changed anymore.One last thing that this model does when", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 50, "start": 367.12, "end": 375.039, "text": "it generates a new sentence is it feedsthe previous sentence into the model asprompt tokens. So this is something that", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 51, "start": 375.039, "end": 382.0, "text": "you can do in the whisper format is youcan give it a bunch of prompt tokensbefore you start generating and this", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 52, "start": 382.0, "end": 388.72, "text": "tends to improve the accuracy a littlebit because more context is always good.So in summary, the algorithm can be", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 53, "start": 388.72, "end": 397.36, "text": "summarized in three basic ideas. Thefirst idea is we feed longer and longerconsecutive audio buffers into whisper", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 54, "start": 397.36, "end": 405.28, "text": "and then we emit tokens as soon asthey're confirmed by two iterations. Andfinally, we scroll forward the audio", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 55, "start": 405.28, "end": 412.4, "text": "buffer whenever a sentence is completed.So, it's a pretty simple algorithm thatyou can apply to any speech to text", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 56, "start": 412.4, "end": 418.72, "text": "model that does not support streamingand basically turn it into a streamingmodel. If you like, you can check out", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 57, "start": 418.72, "end": 425.039, "text": "all the details in this paper, whichI'll link in the description. One of thelimitations comes from the fact that", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 58, "start": 425.039, "end": 432.479, "text": "whisper was not really designed to be astreaming model. And because of this, itassumes that each audio buffer has to", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 59, "start": 432.479, "end": 438.4, "text": "start at the beginning of a sentence.Therefore, if you have a sentence thatis quite long, then the beginning of the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 60, "start": 438.4, "end": 447.28, "text": "sentence has to be fed into the modeland processed many times. This isinefficient and not necessary if themodel was trained from the beginning to", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 61, "start": 447.28, "end": 456.08, "text": "do streaming ASR. Now, if we look at anarchitecture that was designedspecifically to do streaming speechrecognition, it looks a little bit", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 62, "start": 456.08, "end": 466.319, "text": "different. Here is a model that wasproposed in 2021. At each step, itpredicts a token and it has access to afixed amount of past context and future", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 63, "start": 466.319, "end": 473.199, "text": "context. During training, this rule isenforced by having an intention maskthat is mostly zeros, meaning that the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 64, "start": 473.199, "end": 480.08, "text": "model cannot use information that isoutside of this fixed window. And duringprediction, the model predicts a token", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 65, "start": 480.08, "end": 487.36, "text": "given a limited amount of fixed context.To make the next prediction, the entireceptive field moves forward by one", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 66, "start": 487.36, "end": 493.759, "text": "chunk, but the size of the receptivefield is fixed. This way, the beginningof the sentence will not be processed", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 67, "start": 493.759, "end": 500.24, "text": "multiple times. However, this is notpossible with the whisper model becauseit requires modifying the architecture", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 68, "start": 500.24, "end": 506.56, "text": "and retraining the model and we do nothave access to whisper's training data.That's it for this video and I hope you", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 69, "start": 506.56, "end": 513.279, "text": "enjoyed my explanation of how Whispercan be turned into a streaming model. Ifyou enjoyed this video, please leave a", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 70, "start": 513.279, "end": 521.0, "text": "comment, hit the subscribe button, andring the bell icon to stay notified whenI release future videos. Goodbye.", "source": "video_source_subtitle", "authority": "video_source_subtitle" } ], "transcript_selection": { "schema": "videonote.transcript_selection.v1", "status": "SELECTED", "requested_language": "en", "duration": 521.0, "reference_path": "/Users/sagawa/AI_WORK/video_notes/url/20260910_160201__Can_Whisper_be_used_for_real-time_streaming_ASR/reference_subtitle.vtt", "reference_kind": "automatic_original", "reference_language": "en-orig", "selected_source": "video_source_subtitle", "source_caption_kind": "automatic_original", "source_caption_processing": "rolling_dedup_and_sentence_merge", "youtube_overlap_clamps": [], "general_high_confidence_repairs": [], "decision_reason": "complete_source_subtitle_fast_path_no_whisper", "whisper_rejected": false, "source_profile": { "segment_count": 71, "duration_coverage": 0.999539 } } }