Can the open AI whisper model dostreaming ASR? My name is Bai. I'm a
chine learning engineer and a PhD inatural language processing. And today Iwill answer this question. So if you
haven't seen the whisper model, it is aspeechtoext model. It is trained onabout 100 languages on 680,000 hours of
data. The model architecture is aencoder decoder transformer model and itcomes in five different sizes and it is
growing in popularity recently becauseit is robust to noise and accents. Sothe question of the day is can this
model do streaming speech recognition.So first of all what do we mean bystreaming ASR? Well, automated speech
recognition can come in two forms batchand streaming. Batch means that themodel needs to take a bunch of speech
and produce a bunch of text. Butstreaming ASR is when the model needs toproduce an output as the speaker isaying things with a delay of not more
than a few seconds. So for example, ifyou're listening to a live broadcast,you don't want to first record the
ntire broadcast and then produce thesubtitles, you want it to appear at mosta few seconds after it is said. And
generally you'll expect streaming ASR tobe a few percentage points loweraccuracy than batch ASR because the
model does not have access to all thecontext only the point up until thecurrent word. The question of how to do
streaming ASR came up when I wasbuilding the voice writer. This is an AIwriting assistant that works in twosteps. In the first step, you just speak
your thoughts without worrying too muchabout the grammar. And in the secondstep, the AI corrects the grammar for
you. I've been finding it super helpfulfor writing all kinds of things,including emails, blog posts, and Slack
messages. You can try it out for free inthe link here. And I'll also post a linkin the description of the video. Now,
back to the video. First of all, why isthis even difficult? Why is it not so
easy to use the Whisper model forstreaming ASR? The issue is Whisper is
trained to process input audio of 30seconds long. If the audio is less than30 seconds, you can pad it to be longer.
So, this is not too much of a problem.But what if your audio is longer than 30seconds? There is no way to feed into
Whisper anything longer than 30 seconds.So, the first thing you might think ofis what if you just split it up into
chunks of 30 seconds at a time andprocess each chunk one at a time. Well,the first problem is you might split it
in the middle of the word. Like if yousplit it while in the middle of theword, this word will probably not berecognized correctly. The second problem
is the latency will be really high. Soimagine if you're processing audio,you're waiting for audio to fill up
these 30 seconds and only then will youprocess this entire chunk and then thelatency um for for the first word will
be up to 30 seconds. Fortunately, thereis an open source report called WhisperStreaming that does this for you and it
urns the Whisper model into a streamingASR system. There are instructions onhow to set this up. So, let's try it
out. So, I run this command to start amodel running um it's using a whispersmall model and then I go to this tab
and run another command and I just startspeaking. So you can pretty quickly seethat it's um picking it up and it's
pretty quick and responsive and it evengives timestamps. So um it's pretty cooldemo. Um now let's talk about how this
works. So we start with this audio fileand what we do is we feed chunks ofincreasing size into whisper. The size
of these chunks is configured by theminimum chunk size parameter which bydefault is 1 second. So in eachiteration, we increase the size of this
buffer by 1 second and feed all of itinto whisper. And this process continuesuntil we hit a end of sentence marker
like a period or a question mark. Thenwe move the buffer forward and start theprocess again. So each piece of audio is
processed by whisper multiple times asmany times as it takes until we hit thend of the sentence. The reasons for
this is whisper is trained on sentences.So, it gives the best result when thestart of the audio aligns with the start
of a sentence. And that's why the bufferonly moves forward when the previousentence is complete and we're beginning
a new sentence. This way, Whisper neverneeds to start transcribing from themiddle of a sentence, which would give
suboptimal results. In manyapplications, it is useful to have someresults available as soon as possible,
even if they're not completely accurate.In the voice writer, I am showing theincomplete results in gray whereas the
confirmed results are in black. Thedifference is the gray part of thesentence is unconfirmed. So it maychange as the model gets new
information. But the black part is theconfirmed results and they arepermanent. How this works is using analgorithm called local agreement with n
equals 2. This means that for a token tobe confirmed, it needs to be generatedin two consecutive audio buffers. Let's
give an example. Let's say in the firstep, the worst per model outputs thethree tokens if you like and nothing is
confirmed at this step. In the secondstep, the model produces more tokens,but only the first two tokens agree with
e previous step. So those two areconfirmed. In the third and the fourthsteps, more tokens are generated. But at
each step, it does not confirm anytokens until it is at least generated intwo consecutive chunks. So anything in
the gray part is possible to change andgiven new information. For example, theword view might change to video once the
model hears the rest of the sentence.But anything in the black part ispermanent. So even if the model wants to
change it to something else after moreiterations it cannot be changed anymore.One last thing that this model does when
it generates a new sentence is it feedsthe previous sentence into the model asprompt tokens. So this is something that
you can do in the whisper format is youcan give it a bunch of prompt tokensbefore you start generating and this
tends to improve the accuracy a littlebit because more context is always good.So in summary, the algorithm can be
summarized in three basic ideas. Thefirst idea is we feed longer and longerconsecutive audio buffers into whisper
and then we emit tokens as soon asthey're confirmed by two iterations. Andfinally, we scroll forward the audio
buffer whenever a sentence is completed.So, it's a pretty simple algorithm thatyou can apply to any speech to text
model that does not support streamingand basically turn it into a streamingmodel. If you like, you can check out
all the details in this paper, whichI'll link in the description. One of thelimitations comes from the fact that
whisper was not really designed to be astreaming model. And because of this, itassumes that each audio buffer has to
start at the beginning of a sentence.Therefore, if you have a sentence thatis quite long, then the beginning of the
sentence has to be fed into the modeland processed many times. This isinefficient and not necessary if themodel was trained from the beginning to
do streaming ASR. Now, if we look at anarchitecture that was designedspecifically to do streaming speechrecognition, it looks a little bit
different. Here is a model that wasproposed in 2021. At each step, itpredicts a token and it has access to afixed amount of past context and future
context. During training, this rule isenforced by having an intention maskthat is mostly zeros, meaning that the
model cannot use information that isoutside of this fixed window. And duringprediction, the model predicts a token
given a limited amount of fixed context.To make the next prediction, the entireceptive field moves forward by one
chunk, but the size of the receptivefield is fixed. This way, the beginningof the sentence will not be processed
multiple times. However, this is notpossible with the whisper model becauseit requires modifying the architecture
and retraining the model and we do nothave access to whisper's training data.That's it for this video and I hope you
enjoyed my explanation of how Whispercan be turned into a streaming model. Ifyou enjoyed this video, please leave a
comment, hit the subscribe button, andring the bell icon to stay notified whenI release future videos. Goodbye.