hey everyone welcome to nning IO I'm JD
and today we're going to be looking at
real time in browser speech recognition
using web GPU and Transformer JS to
actually use speech in the browser so
with that let's go ahead and get
started all right the first thing that
we're going to do everyone is we're
going to go to this hugging face space
and basically what we're going to do is
look at this demo for the whisper web
GPO uh it's real time in browser
recognition it's using the whisper base
which is a 73 million parameter speech
recognition mod model and using onx and
Transformer JS so you can see that the
audio audio visualizer is already
picking up my voice from the microphone
and uh able to do some kind of uh visual
what we're going to do is actually load
this model and then we should be able to
see real time uh transcription but what
we want to show
is also that if you go and look at your
network uh right now I have I'm on this
wasm tab and we want to pay attention to
both Fetch and wasm so we're going to go
ahead and look at the model and you can
see it pulled it in really quickly uh
could be uh because it's cached but uh
this was incredibly fast and it's
already doing the the transcription in
real time you can see that it's trying
add
punctuation and uh the token count down
here if we go over to the wasm tab this
is all that was loaded it's going
through a worker to establish this and
the size was 4 megabytes so again this
is crazy fast for doing real time uh
transcription in the browser super
helpful if you were trying to do uh like
chat based systems where you actually
wanted to have voice in with your chat
as without having to go out to open AI
or some other
um closed Source or even open source
model that uh requires a backend call
right again we didn't fetch anything
from our back end this was just loaded
through the browser so now what we're
going to do is we're going to take this
and we're going to actually look at the
repo and and dig into the code so if you
come over here to Transformers Js
the thing you want to note is you want
to be on uh the
V3 Branch this is the the new Branch
it's still experimental but it's going
to be the next version of Transformer JS
a lot of great examples we've been
trying to go through some of them it
feels like every day they're coming out
with a new demo a new model that they're
able to implement um definitely really
cool stuff so if we look at this example
so we go into Transformer JS examples
and Whisper then we're able to actually
D dive into this code so what we're
going to do is we're just going to pull
this down get this running on our local
machine and then we'll start digging
into the code so I'm going to use cursor
I'm already I already have it pulled
down but you can do uh get clone and
then come over
here and just do
this uh we are so I'm in the root so
what we need to do is go to
examples and then we're going to do web
GPU and go to
whisper and we'll do an npm
install and then we'll go ahead and do
an npm run
Dev and what we're going to do is we
have our Local Host here so we can get
this up and running we'll watch really
quickly if we can get
this uh loaded again super quick you saw
how quickly the the load time was for
that again some of it sometimes it's
cashed and the first one's going to be
slower but still really
impressive so if we look at some of the
core files in here we have our app which
is the main main file and what we're
going to be looking at is how we're
actually pulling in this worker so this
worker. JS is where we're actually going
to do our Logic for Transformer Js we're
pulling in the model we're actually
sending messages back and forth where
you can see right here that as messages
are coming being received we're actually
going to be looking for the type of
event that is being pass and then send
that uh data as well
so you can see most of this is for like
loading there's uh some update which is
where you're updating the text
we're actually going to look and see
where this is uh is working so if we
continue down and we are looking at the
recorder this is all the information
about starting and stopping the
recording and then what's Happening Here
is we are actually looking at how this
information is being sent with this code
so right
here it's saying as the recording is
coming in we're going to take a blob and
chunk it and so then we're going to make
a file reader and read those and then
send that on when the file is being
read get that bit information that's
decoded slice it and then send those
slices over to our worker so we're
posting the message to generate our
transcription on the Fly then this is
just our uh our HTML that we're going to
be pulling
in so what we'll do is we'll jump over
to the worker and then we'll take a look
at how this is actually functioning real
quick everyone if you haven't already
please remember to like And subscribe it
helps more than you know if you have any
questions please leave them in the
comments and with that let's get back to
it all right so now that we're in our
worker file we're going to start
looking at how this actually functions
first thing is we're loading in
Transformers we're going to pull in our
Auto
tokenizer our Auto processor and then uh
whisper and text stream as well as full
so if we look at what we're first doing
we have this class which is our Pipeline
and then we're actually using our
instance where we're getting our model
so we're saying this is our model ID we
have our tokenizer our processor we're
actually getting the model based on our
ID at this point uh we're telling it
what our dtype is and then specifically
we're using device web
GPU and once all of these have processed
uh then our git instance has been
established if we scroll down to the
bottom we can kind of look at like some
of the loader functions so you can see
here there's the load this happens like
where we're actually posting back
messages to actually see see the
progress bar things like that we're
establishing some events based on our
message and then once the the browser uh
or once the model is actually loaded
we're going to be looking for this
generate function this function is what
really is going to be taking the audio
and taking the language and then
actually processing it so the first
thing we'll be doing is saying is if if
processing return
if it is not processing then go ahead
and set that processing to True
maintaining State we're actually going
to send a message back to say that we're
trans that the thread is starting so
we're saying start as our status and
then we're actually pulling in our
instance so again where we set our
tokenizer and our processor and model
we're actually pulling that from our um
instance and we're starting to actually
process so we have our call back
function which is going to give us the
out
output of our post message of update and
information about our TPS our tokens and
our actual output and then we we have
our streamer and we this is where we're
actually
processing our information so what we're
saying is our input we're taking our AO
audio and we're uh putting it through
our processor we then this is our
transcription our transcription BAS
based on our model and our generate
we're saying we're passing our same
inputs our language and we're actually
streaming the events or streaming the
output so now we have our tokenizer this
gives us the information that's
displayed in the bottom right of how
many tokens we're actually using and
then we're posting back that chunk of
text as it is complete and so remember
if we are looking at what these post
messages are we're coming back here and
that is the information that is being on
on message received so if we look down
to
update we're seeing that we're setting
our TPS here as well as uh our message
so our on message right here this is
where we're being sending the
information back and then that's how
we're actually passing information down
here so this is where our
TPS is and then uh
the this the rest of the progress bar in
the audio I'm looking for
the
actual uh information around the text oh
it's right here so on ready we're
passing the information of the text
that's being
updated and that's how this works is we
basically are taking this information
from after after it passes passes
through the model and then right here so
being generated by the model and then
posting that information back
so it's processing the auto chunk it is
audio chunk and it's uh then converting
that and doing a transcription on this
generation then it's streaming it's
doing the tokenizer and posting all that
information back all right everyone
that's it for us today what we covered
was using real time speech recognition
in the browser with web GPU and
Transformers JS we're able to dig
through some of the repo and actually
get their demo up and running and so
with that happy nerding